Lost port(s) after load shedding

[jermudgeon]

WS-12-250-DC

Firmware 1.4.0
Revision E

Narrative of event:
multiple instances of input under-voltage, load shedding, 24v bus under voltage, resulting in multiple Netonix port failures and two damaged Nanobeams

Incident logged ports
7 (24v) UBNT Nanobeam 19 — failed to come up at all with NB, Nanobeam gig speed fried. Works @100 with another device though.
9 (24v) UBNT Nanobeam 19 — came back up at 100 Mb, Nanobeam gig speed fried. Works @1000 with another device.
10 (--) LAG to M*tik — came back up, operating at gigabit
11 (24v) Unifi (indoor) — came back up, fully operating (gigabit tested, running @100)
12 (48v) LAG to M*tik [Mikrotik has secondary AC/DC power] — came back up, gig speed fried, Mikrotik OK (according to loopback port speed/giga test)

Unaffected ports
1 (48v) Mimosa B11, no data connection
2 (48v) Mimosa B5
3 (48v) Mimosa B5
4 n/c
5 connected to Telrad Compact for failover, no data, diagnostics clean
6 (--) LAG to M*tik
8 (--) LAG to M*tik

Port, power priority
11 = 1
10 = 2
6 = 3 <-- no PoE, didn't go down
7 = 4 <-- affected port
9 = 5 <-- affected port
4 = 6
5 = 7
2 = 8
3 = 9
8 = 10
12 = 11 <- affected port
1 = 12

Notes:
Inexplicably, port 12 went down first, with a priority of 11. This is the PoE out to the Mikrotik router. It did not itself reboot, as it has a secondary 24v power supply.
Inexplicably, ports 7 and 9 (priority 4 and 5) go down first, and not 11 and 10 (priority 1 and 2).

Code: Select all


Jul 1 10:23:19 Port: link state changed to 'down' on port 12
Jul 1 10:23:19 Port: link state changed to 'down' on port 7
Jul 1 10:23:19 STP: set port 7 to discarding
Jul 1 10:23:19 Port: link state changed to 'down' on port 9
Jul 1 10:23:19 STP: set port 10 to discarding
Jul 1 10:23:19 STP: set port 9 to discarding
Jul 1 10:23:19 LACP: LACP changed state to Not active on port 12 (key 10)
Jul 1 10:23:21 STP: set port 10 to learning
Jul 1 10:23:21 STP: set port 10 to forwarding
Jul 1 10:23:35 Port: link state changed to 'up' (100M-F) on port 12
Jul 1 10:23:35 STP: set port 12 to discarding
Jul 1 10:23:35 LACP: starting negotiation with partner 4C-5E-0C-03-0E-44
Jul 1 10:23:37 STP: set port 12 to learning
Jul 1 10:23:37 STP: set port 12 to forwarding
Jul 1 10:23:37 LACP: LACP changed state to Active on port 12 (key 10)
Jul 1 10:24:38 Port: link state changed to 'up' (100M-F) on port 9
Jul 1 10:24:38 STP: set port 9 to discarding
Jul 1 10:24:38 Port: link state changed to 'up' (100M-F) on port 7
Jul 1 10:24:38 STP: set port 7 to discarding
Jul 1 10:24:40 STP: set port 7 to learning
Jul 1 10:24:40 STP: set port 7 to forwarding
Jul 1 10:24:40 STP: set port 9 to learning
Jul 1 10:24:40 STP: set port 9 to forwarding
Jul 1 10:24:55 Port: link state changed to 'down' on port 9
Jul 1 10:24:55 STP: set port 9 to discarding
Jul 1 10:24:55 Port: link state changed to 'down' on port 7
Jul 1 10:24:55 STP: set port 7 to discarding
Jul 1 10:25:00 Port: link state changed to 'down' on port 11
Jul 1 10:25:00 STP: set port 11 to discarding
Jul 1 10:25:01 switch[901]: 24V is out of range (7.2V)
Jul 1 10:25:02 Port: link state changed to 'up' (100M-F) on port 11
Jul 1 10:25:02 STP: set port 11 to discarding
Jul 1 10:25:02 switch[2450]: unexpected link change on port 11 100M-100M-F
Jul 1 10:25:04 STP: set port 11 to learning
Jul 1 10:25:04 STP: set port 11 to forwarding
Jul 1 10:25:05 Port: link state changed to 'up' (100M-F) on port 9
Jul 1 10:25:05 STP: set port 9 to discarding
Jul 1 10:25:07 STP: set port 9 to learning
Jul 1 10:25:07 STP: set port 9 to forwarding
Jul 1 10:25:20 Port: link state changed to 'down' on port 11
Jul 1 10:25:20 STP: set port 11 to discarding
Jul 1 10:25:22 Port: link state changed to 'down' on port 9
Jul 1 10:25:22 STP: set port 9 to discarding
Jul 1 10:25:22 Port: link state changed to 'up' (100M-F) on port 11
Jul 1 10:25:23 STP: set port 11 to discarding
Jul 1 10:25:25 STP: set port 11 to learning
Jul 1 10:25:25 STP: set port 11 to forwarding
Jul 1 10:25:39 Port: link state changed to 'up' (100M-F) on port 9
Jul 1 10:25:39 STP: set port 9 to discarding
Jul 1 10:25:40 Port: link state changed to 'down' on port 11
Jul 1 10:25:40 STP: set port 11 to discarding
Jul 1 10:25:41 Port: link state changed to 'up' (100M-F) on port 11
Jul 1 10:25:41 STP: set port 11 to discarding
Jul 1 10:25:43 STP: set port 11 to learning
Jul 1 10:25:43 STP: set port 11 to forwarding
Jul 1 10:26:14 STP: set port 9 to learning
Jul 1 10:26:29 STP: set port 9 to forwarding

On reboot:

Dec 31 15:00:37 switch[915]: Port 2 cable check results: Ok, Ok, Ok, Ok
Dec 31 15:00:37 switch[915]: Port 2 passed PoE Smart cable check, turning on power
Dec 31 15:00:37 switch[915]: Port 3 cable check results: Ok, Ok, Ok, Ok
Dec 31 15:00:37 switch[915]: Port 3 passed PoE Smart cable check, turning on power
Dec 31 15:00:37 switch[915]: Port 4 cable check results: Open, Open, Open, Short <- moved port 7 Nanobeam to here
Dec 31 15:00:37 switch[915]: Port 4 passed PoE Smart cable check, turning on power
Dec 31 15:00:37 switch[915]: Port 9 cable check results: Open, Open, Open, Short
Dec 31 15:00:37 switch[915]: Port 9 passed PoE Smart cable check, turning on power
Dec 31 15:00:37 switch[915]: Port 11 cable check results: Ok, Ok, Short, Short
Dec 31 15:00:37 switch[915]: Port 11 passed PoE Smart cable check, turning on power
Dec 31 15:00:37 switch[915]: Port 12 cable check results: Ok, Ok, Ok, Open <- this only runs @ 100 now
Dec 31 15:00:37 switch[915]: Port 12 passed PoE Smart cable check, turning on power



Diagnostics with no cable attached, and PoE off, with usual port voltage indicated.

Control (no voltage)
5
open
open
open
open
good


24v
7
open
open
short
short
100!


24v
9
open
open
open
short
100!


--
10
open
open
open
open
good


24v
11
open
open
open
open
good


48v
12
open
open
open
open
100!

[sirhc]

Diagnostics with no cable attached:
Load
Port
1
2
3
4
Status

OK this makes no sense?

You say diagnostic with no cable but if that is to test the port to make sure your Ethernet transformer is not damaged why would you say/indicate a "Load"? If you have no cable in the port and your testing the port then POE should be turned OFF?

Control (no voltage)
5
open
open
open
open
good

OK so I get this one, port 5 has no cable and POE is OFF so you should get open on all 4 pairs

24v
7
open
open
short
short
100!

As I said this makes no sense as you said at the top you were testing ports with no cable but OK lets move past this.
Look any UBNT 10/100 airMAX device where data is carried on pairs 1 and 2 will report open on pairs 1 and 2 when not powered and pairs 3 and 4 will report shot as UBNT shorts them to carry power but there is no way you can have a 100M link with pair 1 and 2 open???

When you apply power to a 10/100 airMAX radio pair 1 and 2 should report OK?

24v
9
open
open
open
short
100!

Once again if power is ON for an airMAX device you should see OK on pair 1 and 2 when powered up and short on part 3 and 4.

The fact that you see open on pair 3 means you have a bad crimp end or a cable with one of the wires in pair 3 broken. Pair three is the Blue and Blue White wires.

Once again you are saying you have 100M link but not possible with pair 1 and 2 open?
--

10
open
open
open
open
good

OK, nothing plugged into the port I guess and all open?

24v
11
open
open
open
open
good

Not sure why you have 24V applied and no cable but all open OK, as it should be with nothing plugged in.

48v
12
open
open
open
open
100!

Once again why 48V applied if nothing is plugged in which is indicated by all opens yet you show a 100M link?

I am confused, maybe go back and edit your post?

[sirhc]

But you need to upgrade to v1.4.2 please.

Our policy is to not look into bug reports unless your running the latest code as that version is closed an no longer developed plus your bug may not exist in the latest version.

[sirhc]

Narrative of event:
multiple instances of input under-voltage, load shedding, 24v bus under voltage, resulting in multiple Netonix port failures and two damaged Nanobeams

This simply is not possible and if it was a defect in the design of the switch many people would be screaming.

If you have ports being damaged I would look into your grounding.

first this is a DC switch so I hope you have the Earth ground lug attached to a ground bus that is attached to the tower ground bus?

Please explain what is powering this site.

Is it off grid or on grid?

Please explain what you have for grounding?

Are you using ESD Ethernet Cable with drain wire and ESD ends?

Here are some good posts on grounding:
viewtopic.php?f=30&t=1816
viewtopic.php?f=30&t=188
viewtopic.php?f=30&t=1429

[sirhc]

Ports that only run at 100M means that either pair 3 or pair 4 of the Ethernet transformer is damaged.

Not all transformer damage can be detected this way but most of the time you can. A simple test is turn the POE off on the port and remove the cable from the port and then run the cable diagnostics.

If you get a anything other than OPEN on all four pairs that means the pair coil is damaged, however sometimes the pair coil can be burned open which means it will always report open and never work but more commonly it will show short or abnormal termination.

Ethernet transformers do not fail they get damaged and the most common reason they get damaged is listed below:
1) Shorted cable
2) Ground potential difference/shift which is pulling excessive current through the port
3) Applying the wrong POE option such as 48V or 48VH or 24VH to an airMAX device
4) Surge or excessive voltage from ESD, static, or strike

[jermudgeon]

Diagnostics with no cable attached:
Load
Port
1
2
3
4
Status

OK this makes no sense?

You say diagnostic with no cable but if that is to test the port to make sure your Ethernet transformer is not damaged why would you say/indicate a "Load"? If you have no cable in the port and your testing the port then POE should be turned OFF?

Control (no voltage)
5
open
open
open
open
good

OK so I get this one, port 5 has no cable and POE is OFF so you should get open on all 4 pairs

24v
7
open
open
short
short
100!

As I said this makes no sense as you said at the top you were testing ports with no cable but OK lets move past this.
Look any UBNT 10/100 airMAX device where data is carried on pairs 1 and 2 will report open on pairs 1 and 2 when not powered and pairs 3 and 4 will report shot as UBNT shorts them to carry power but there is no way you can have a 100M link with pair 1 and 2 open???

When you apply power to a 10/100 airMAX radio pair 1 and 2 should report OK?

24v
9
open
open
open
short
100!

Once again if power is ON for an airMAX device you should see OK on pair 1 and 2 when powered up and short on part 3 and 4.

The fact that you see open on pair 3 means you have a bad crimp end or a cable with one of the wires in pair 3 broken. Pair three is the Blue and Blue White wires.

Once again you are saying you have 100M link but not possible with pair 1 and 2 open?
--

10
open
open
open
open
good

OK, nothing plugged into the port I guess and all open?

24v
11
open
open
open
open
good

Not sure why you have 24V applied and no cable but all open OK, as it should be with nothing plugged in.

48v
12
open
open
open
open
100!

Once again why 48V applied if nothing is plugged in which is indicated by all opens yet you show a 100M link?

I am confused, maybe go back and edit your post?

Will edit. It was a table that posted badly, and I should have cleaned it up first.

All tests were done with PoE off and no cable attached. The load indicated was the load at the time of the incident, not the load during testing. More in reply to your other replies. Thanks.

[jermudgeon]

But you need to upgrade to v1.4.2 please.

Our policy is to not look into bug reports unless your running the latest code as that version is closed an no longer developed plus your bug may not exist in the latest version.

Can't upgrade until tomorrow, but will do.

[jermudgeon]

Narrative of event:
multiple instances of input under-voltage, load shedding, 24v bus under voltage, resulting in multiple Netonix port failures and two damaged Nanobeams

This simply is not possible and if it was a defect in the design of the switch many people would be screaming.

If you have ports being damaged I would look into your grounding.

first this is a DC switch so I hope you have the Earth ground lug attached to a ground bus that is attached to the tower ground bus?

Please explain what is powering this site.

Is it off grid or on grid?

Please explain what you have for grounding?

Are you using ESD Ethernet Cable with drain wire and ESD ends?

Here are some good posts on grounding:
viewtopic.php?f=30&t=1816
viewtopic.php?f=30&t=188
viewtopic.php?f=30&t=1429

Drain wire and ESD ends. It is on grid, with anything having a ground lug grounded. Inside equipment (Netonix, Mikrotik, surge protection) tied to all-metal building ground, and outdoor equipment (this is a grain leg) bonded as well.

Power is a 48v battery system with mains charging plus (via diodes) a Meanwell AC/DC converter. When batteries are floating, powered by Meanwell. Additional 24v and 12v busses (off batteries). Suitable fuses and breakers. Can furnish electrical diagram on request.


We agree about the grounding. After posting, we did more post mortem.

We believe either +48v or -48v got shorted to ground at the top of the tower. NOT through the Netonix, but on a separate device that shares the ground, a Telrad ENB.

It didn't make sense to us either that an under voltage could be handled like this by the Netonix -- that's what it's job is.

Looks like we definitely have three ports with blown Ethernet transformers, and some Nanobeams likewise, but everything else checks out.

Normally we clamp black to ground. We didn't, in this instance, because we weren't sure the Netonix could handle that. Would that have made a difference? From what we could see of Netonix documentation, black isn't internally isolated from ground (at least when powered). The ENB calls for -48v, but doesn't actually require it; and the usual -48v setup doesn't work with Netonix, unless we are missing something obvious. Advice?

[jermudgeon]

Ports that only run at 100M means that either pair 3 or pair 4 of the Ethernet transformer is damaged.

Not all transformer damage can be detected this way but most of the time you can. A simple test is turn the POE off on the port and remove the cable from the port and then run the cable diagnostics.

If you get a anything other than OPEN on all four pairs that means the pair coil is damaged, however sometimes the pair coil can be burned open which means it will always report open and never work but more commonly it will show short or abnormal termination.

Ethernet transformers do not fail they get damaged and the most common reason they get damaged is listed below:
1) Shorted cable
2) Ground potential difference/shift which is pulling excessive current through the port
3) Applying the wrong POE option such as 48V or 48VH or 24VH to an airMAX device
4) Surge or excessive voltage from ESD, static, or strike

We think option 2 was the cause, assuming ground is supposed to = 0vDC. However, none of the devices powered by the Netonix were shorted. It was externally on the same DC supply and grounding.

Do you recommend that ground be bonded to negative?

[sirhc]

Normally we clamp black to ground. We didn't, in this instance, because we weren't sure the Netonix could handle that. Would that have made a difference? From what we could see of Netonix documentation, black isn't internally isolated from ground (at least when powered). The ENB calls for -48v, but doesn't actually require it; and the usual -48v setup doesn't work with Netonix, unless we are missing something obvious. Advice?

The important thing is that the Electric Service Ground is bonded to the tower grounding. However if AC power is being supplied from a distance you are better to not use the service Earth Ground at all and just use the tower Earth Ground which you can read about in this post: viewtopic.php?f=17&t=1786&start=30#p13447

The other thing that is important is to make sure there is a dedicated Earth ground from all radios to Earth Ground that is a shorter and less resistant path than the Ethernet cable path which is where adding a larger service loop sometimes does wonders to save equipment. Likewise make sure the equipment has a shorter less resistant path to the same Earth Ground as the radios use so the equipment does not try and get Earth Ground through the Ethernet Cables.

As far as bonding DC negative to Earth ground my gut tells me no but I will ping Dave to answer that.

Looks like we definitely have three ports with blown Ethernet transformers, and some Nanobeams likewise, but everything else checks out.

If you RMA the switch we will put a NEW switch board in it for $150.00