Tech fries Ethernet current sensors ?

[Brough]

We have over 120 WISP switches powering radios on urban roof tops, so we have experienced the occasional grounding problem that fries the current sensors in one or more Ethernet interfaces. But recently (January 16, 2017), we had a strange event while adding to a site that has been in service without incident for over two years.

One of our techs was at the building to add a second WISP switch and a new radio. Both the original switch (a WS-12-250A) and the new switch (a WS-6-MINI) are powered by the same 120vAC UPS and the new radio is mounted on an existing non-penetrating roof mount that was already grounded to our equipment which is grounded to the building electrical ground via an outlet in a roof-level mechanical room. Also, January 16th in Boston was clear and cold; no storms; no precipitation.

There are several things in the logs I don't understand. See the log excerpts below.

My biggest question - what could have happened to drive all three power supplies to -0.7v and fry all the current sensors at once? I could understand a short circuit driving one of the power supply voltages to zero and frying one set of current sensors.

But also, how can there be events for ports 25 & 26 of a 12-port switch???


Log excerpt from existing WS-12-250A Jan 2017:
Jan 16 09:36:16 UI: Configuration changed by 10.130.0.192
Jan 16 09:36:16 UI: Port 14 Name: changed from 'sfp' to 'to-PoE-SW2'
Jan 16 09:36:16 UI: Port 14 Link: changed from '1G' to 'Auto'
Jan 16 09:36:16 UI: VLAN 2 PortSettings: changed from 'EUEEEEEEEETEEE' to 'EUEEEEEEEETEET'
Jan 16 09:36:16 UI: VLAN 3 PortSettings: changed from 'EEUEEEEEEETEEE' to 'EEUEEEEEEETEET'
Jan 16 09:36:16 UI: VLAN 4 PortSettings: changed from 'EEEUEEEETEETEE' to 'EEEUEEEETEETET'
Jan 16 09:36:16 UI: VLAN 5 PortSettings: changed from 'EEEEUEEETEETEE' to 'EEEEUEEETEETET'
Jan 16 09:36:16 UI: VLAN 6 PortSettings: changed from 'EEEEEUEETEETEE' to 'EEEEEUEETEETET'
Jan 16 09:36:16 UI: VLAN 7 PortSettings: changed from 'EEEEEEUETEETEE' to 'EEEEEEUETEETET'
Jan 16 09:36:16 UI: VLAN 8 PortSettings: changed from 'EEEEEEEUEEETEE' to 'EEEEEEEUEEETET'
Jan 16 09:36:16 UI: VLAN 9 PortSettings: changed from 'EEEEEEEEETTTUE' to 'EEEEEEEEETTTUT'
Jan 16 09:36:16 UI: VLAN 10 PortSettings: changed from 'EEEETEEEETTTEE' to 'EEEETEEEETTTET'
Jan 16 09:45:19 UI: Configuration changed by 10.130.0.192
Jan 16 09:45:20 UI: VLANs: added 'to-699Boylston'
Jan 16 11:25:56 kernel: link state changed to 'down' on port 25
Jan 16 11:26:07 kernel: link state changed to 'up' on port 26
Jan 16 11:26:07 switch[1358]: 48V is out of range (-0.7V)
Jan 16 11:26:07 switch[1358]: 24V is out of range (-0.7V)
Jan 16 11:26:07 switch[1358]: 3V is out of range (0.0V)
Jan 16 11:26:07 switch[1358]: Port 1: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 2: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 3: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 4: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 5: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 6: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 7: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 8: PoE is enabled and port has link, power should not be 0
Jan 16 11:26:07 switch[1358]: Port 9: PoE is enabled and port has link, power should not be 0
Jan 16 11:37:16 kernel: link state changed to 'up' on port 25
Jan 16 11:37:17 kernel: link state changed to 'down' on port 25
Jan 16 11:52:20 kernel: link state changed to 'up' on port 25
Jan 16 12:21:53 kernel: link state changed to 'down' on port 26
Jan 16 12:22:11 kernel: link state changed to 'up' on port 26
Jan 16 12:22:34 kernel: link state changed to 'down' on port 26
Jan 16 12:22:36 kernel: link state changed to 'up' on port 26
Jan 16 12:25:03 UI: Configuration changed by 10.130.0.182
Jan 16 12:25:03 UI: VLAN 11 PortSettings: changed from 'EEEEEEEEEEEEET' to 'EEEEEEEEEEETET'


Log excerpt from new WS-6-MINI Jan 2017:
Jan 1 00:00:09 netonix: 1.4.5rc1 on WS-6-MINI
Jan 1 00:00:13 system: Setting MAC address from flash configuration: EC:13:B2:91:57:00
Jan 1 00:00:16 netonix: adding lan (eth0) to firewall zone lan
Dec 31 19:00:30 netonix: removing lan (eth0) from firewall zone lan
Dec 31 19:00:34 netonix: adding lan (eth0) to firewall zone lan
Dec 31 19:00:38 netonix: adding lan (eth0) to firewall zone lan
Dec 31 19:00:39 main: packet_rx_filter_change NOT IMPLEMENTED
Dec 31 19:00:39 system: starting ntpclient
Jan 16 12:23:14 dropbear[1323]: Running in background
Jan 16 12:23:16 switch[1348]: Detected warm boot
Jan 16 12:36:10 Port: link state changed to 'down' on port 3
Jan 16 12:36:12 Port: link state changed to 'up' (1G) on port 3
Jan 16 12:37:09 Port: link state changed to 'down' on port 3
Jan 16 12:37:12 Port: link state changed to 'up' (1G) on port 3
Jan 16 12:38:35 Port: link state changed to 'down' on port 3
Jan 16 12:38:37 Port: link state changed to 'up' (1G) on port 3

[sirhc]

Does this building have multiple electrical service panels like in many large buildings?

If different service panels, the one the equipment is plugged into and the one the roof top mount in tied into ground then the ground potential for each service panel is different and all excess current on one service if it has a worse ground potential may try to get to ground through the Ethernet cables.

Other than that we would need to see the units to see what is damaged.

We can repair any unit even if out of warranty.

We do not try and profit from repairs even those out of warranty we simply try to cover our costs which is a nice service that we repair our stuff even if out of warranty.

We also apply the latest board MODs to any switch in for RMA service.

THe Ground Current MOD is something people with a little soldering skill can do themselves as it is detailed in this post: viewtopic.php?f=17&t=2584

[Julian]

Port numbers were mistranslated by the kernel in older versions of firmware.

Also, current sensors are easy fixes, let me at 'em.

Your log, though, isn't current sensors, it's the A/D converter or the analog multiplexing circuit, since all analog samples are reporting as 0v. (Voltages, power consumption, etc).. grounding still does that, just depends on what power is applied when the current sensors magic smoke escapes.

Long story slightly less long, I'm going to need that switch in for a checkup.

[Brough]

Thanks.

The site is a 12-story office building, but all relevant equipment is in one wall mount rack in the roof-top mechanical room, connected to one specific grounded outlet.

It may be a while before we get to replacing this switch as it's carrying traffic just fine and there are other maintenance items ahead of it. But I will send it back eventually and I'll be very interested in what your find.