WS-12-250-AC Overvoltage - Killed all radios attached to it!

[twotincans]

Hello,

This last weekend (5-19-2017) I had a tower go down after a storm. Long story short, the power went off at the site, my battery backup kicked in just fine, I restored power and noticed no link lights on the Netonix switch even though the power lights where on. I consoled into the switch and found the following log:


Dec 31 19:00:27 switch[866]: Detected cold boot
Dec 31 19:00:28 switch[866]: PoE enabled on port 1, PoE Smart is starting cable check
Dec 31 19:00:28 switch[866]: PoE enabled on port 3, PoE Smart is starting cable check
Dec 31 19:00:28 switch[866]: PoE enabled on port 4, PoE Smart is starting cable check
Dec 31 19:00:28 switch[866]: PoE enabled on port 5, PoE Smart is starting cable check
Dec 31 19:00:28 switch[866]: PoE enabled on port 6, PoE Smart is starting cable check
Dec 31 19:00:29 switch[866]: PoE enabled on port 7, PoE Smart is starting cable check
Dec 31 19:00:29 switch[866]: PoE enabled on port 8, PoE Smart is starting cable check
Dec 31 19:00:29 system: starting ntpclient
Dec 31 19:00:31 switch[857]: !48V is out of range (58.7V)
Dec 31 19:00:31 switch[857]: !24V is out of range (28.4V)
Dec 31 19:00:31 switch[857]: !3V is out of range (4.2V)
Dec 31 19:00:33 switch[866]: Port 1 cable check results: Open, Open, Short, Open
Dec 31 19:00:33 switch[866]: Port 1 passed PoE Smart cable check, turning on power
Dec 31 19:00:33 switch[866]: Port 3 cable check results: Open, Open, Open, Open
Dec 31 19:00:33 switch[866]: Port 3 passed PoE Smart cable check, turning on power
Dec 31 19:00:33 switch[866]: Port 4 cable check results: Open, Open, Open, Open
Dec 31 19:00:33 switch[866]: Port 4 passed PoE Smart cable check, turning on power
Dec 31 19:00:33 switch[866]: Port 5 cable check results: Open, Open, Open, Open
Dec 31 19:00:33 switch[866]: Port 5 passed PoE Smart cable check, turning on power
Dec 31 19:00:33 switch[866]: Port 6 cable check results: Open, Open, Open, Open
Dec 31 19:00:33 switch[866]: Port 6 passed PoE Smart cable check, turning on power
Dec 31 19:00:33 switch[866]: Port 7 cable check results: Open, Open, Open, Open
Dec 31 19:00:33 switch[866]: Port 7 passed PoE Smart cable check, turning on power
Dec 31 19:00:33 switch[866]: Port 8 cable check results: Open, Open, Open, Open
Dec 31 19:00:33 switch[866]: Port 8 passed PoE Smart cable check, turning on power
The time/date is off because of it can't connect via NTP.

So I have to ask, if the switch detects out of range voltage, and a short on port 1, why in the world would it start turning on power to the ports?

Wouldn't it make sense to disable all POE if voltage is detected out of range?

Also, nothing is plugged in (ethernet wise) to the switch when this log was pulled.

[sirhc]

This last weekend (5-19-2017) I had a tower go down after a storm

Maybe your issue is more with your storm?

I think your post title is wrong.

WS-12-250-AC Overvoltage - Killed all radios attached to it!

Obviously the switch was also damaged from the storm as it is impossible for a WS-12-250-AC to generate more than 50V +/- as there is no way to STEP UP the power being supplied by the power supply.

That power supply in that switch will not even put out 58.7V

So when your tower went down the switch was already apparently damaged along with your radios as it is reading voltages incorrectly.

Dec 31 19:00:31 switch[857]: !48V is out of range (58.7V)
Dec 31 19:00:31 switch[857]: !24V is out of range (28.4V)
Dec 31 19:00:31 switch[857]: !3V is out of range (4.2V)

A short on Pair 3 and or 4 is pretty normal with passive POE devices, especially UBNT so we ignore shorts on pairs they are not a problem.

It is cross shorts which are shorts between pairs (polarity) that are a problem

A short on a pair will not damage anything, CROSS SHORTS ARE A PROBLEM.

There are annoucement posts at the top of each forum, you should read them.

viewtopic.php?f=6&t=1178#p8809

Question: Why do I see short on pairs 3 and 4 on a of devices especially UBNT airMAX 10/100 gear.

Answer: Because every manufacturer does not follow the proper design specs and use a 4 channel Ethernet Transformer designed for POE with a center tab for each pair. Since pair 3 and 4 is not needed to carry data with 10/100 only links they use them for power only which allows them to use a cheaper 2 channel transformer without a center tap for pairs 1 and 2. They then connect the 2 wires from each pair to the board skipping the need for a transformer with a center tap so the 2 wires from pair 3 are connected directly to the board shorted together for positive and the 2 wires from pair 4 are shorted together directly on the board for negative.

Question: I see a cross-short, is that bad?

Answer: YES, never turn POE on any port where you see a cross short as that means 1 wire from one pair is shorted to another wire of another pair which is a DEAD SHORT and will fry the switch port. Cross-Short is about the only real bad situation that will surely fry a port.

[sirhc]

Open up the switch chassis and put a meter on the power supply output which should be 48V-52V - Typical is 52V without a load and 50V with load.

It is not capable of 58V without changes being made to it.

And there is nothing on the switchboard capable of stepping up the power beyond what is supplied to it.

We take the input voltage to the switchboard and step it DOWN to produce 24V and the 24V DC2DC power supply will not output 28V as it is out of its range

The 3.3V if it was 4+ volts the board would be fried and no longer boot.

So it is obvious that the board is damaged and no longer reporting voltages correctly.

[twotincans]

I think your post title is wrong.

No, my post title is based on facts.

Obviously the switch was also damaged from the storm as it is impossible for a WS-12-250-AC to generate more than 50V +/- as there is no way to STEP UP the power being supplied by the power supply.
That power supply in that switch will not even put out 58.7V
So when your tower went down the switch was already apparently damaged along with your radios as it is reading voltages incorrectly.

I agree that the switch was also damaged from the storm but with that said, why would it provide power to ports if it's reading voltages incorrectly, in my case they are showing over voltage.

Also, I want to point out that after I took this switch out of production, I took a working test antenna, plugged it in, powered the port and it killed the test antenna (magic smoke was released). The switch shouldn't be providing power if the voltages are not in range, period.

[sirhc]

I am telling you it is impossible for the switch to put out these voltages:

Dec 31 19:00:31 switch[857]: !48V is out of range (58.7V)
Dec 31 19:00:31 switch[857]: !24V is out of range (28.4V)
Dec 31 19:00:31 switch[857]: !3V is out of range (4.2V)

Now the switch MOSFET circuit for a port can be damaged so that it is always putting out 48V all the time for that port but the switch is damaged and no longer has the ability to prevent it from happening as it can not control fried circuits and no amount of software will be able to turn that power off or prevent it from putting out 48V....IT IS DAMAGED.

Do some testing:

Open the chassis and read the voltage coming from the power supply to the switch which should be 50V+/-. There is no way this voltage is 58.7V as the switch does not have the ability to step this up to 58.7V (it's a false reading as a result of damage)

The 28.4V again is next to impossible but is more likely a result of sensor damaged on the board which throws everything else off. My guess is the 3.3V reading is off which affects the calculated 24V and 48V reading as they are based on the v3.3V reading as a reference voltage.

There are voltage test points on the board clearly labeled so you can use a meter to test each voltage at its test point.

My guess is you will see components damaged on the board when you open the chassis, and possibly on the bottom of the switch board.

You can also turn OFF POE on all ports then use a cat5 jumper with one end cut off then take the brown pair and wire it to your negative probe of your meter and the blue pair wired to positive probe of your meter (making sure they and no other wires touch and short out).

Now with POE turned off for all ports in the UI check each port and see if POE voltage is coming out any ports.

Then try setting a port to 24V POE and use your test rig and measure the voltage, there is no way to get 28.4V

But an easier way is to simply put a probe on the 24V test point on the switchboard as there is only a single 24V supply for the entire board.

So in closing there is no way that switch can put out 58.7V or 28.4V, simply not possible or at least highly improbable with the hardware in the switch.

It is more likely that the switch board is DAMAGED and producing incorrect voltage readings but it has no way to put out 58V or 28V, simply not possible or at least very highly improbable.

It is possible that a port MOSFET circuit can be damaged and put out 48V POE all the time on that port which will fry a 24V POE device plugged into that port but the MOSFET is damaged and no amount of software changes can prevent this as the circuit is damaged and would no longer respond to any form of control.

But as I said do some testing and find out if any ports are always putting out 48V-50V POE which would be what fried your 24V devices.

[twotincans]

First, I want to point out the my airfiber and all 6 of my sector antennas where taken out by this.

Today I took apart the switch as sirhc advised, I didn't spend much time on testing the hardware as I don't have the time to spend on this.

I do see damage on the top of the board but I don't see any damage on the bottom of the board, please see attached

CLICK IMAGE BELOW TO VIEW FULL SIZE

[sirhc]

I edited your post to display your picture the correct way.

Please read my post signature on the proper way to upload and display pictures in a post.

Did you remove the switch board to inspect the bottom, that is the only way to inspect the bottom.

Then when you get time follow my instructions on how to test it.

Then you should open an RMA to have the unit repaired as we will repair any unit for a modest fee even if it is out of warranty period and damaged which is not covered by normal warranty.

If it requires a NEW board the cost is $150 but if the board can be repaired it could be a lot less than that. Since the board boots there is a good change the board can be repaired.

[sirhc]

I can see if the switch is damaged it can put out 48V POE to a 24V port and fry a 24V POE device but I want to stress there is NO WAY IT CAN PUT OUT MORE THAN 50V+/- let alone 58V so there is no way it can fry a 48V device such as an AF unit.

If you lost your AF radios then you took a hit and the switch survived but the AF did not.

The switch can not change polarity on PINs or magically generate more voltage than the power supply can put out so it is impossible for the switch to fry a 48V device.

It can as I said if the MOSFET is damaged put out 48V on a port when it shouldn't or when it is told to put out 24V but there is no way to put out more voltage than the power supply sends the switch board or magically put different polarity on different PINs so there is no way for a switch to fry a 48V device like an AF.

You lost the tower in a "storm" but you are insistent that it is the switches fault and it simply is not possible for the switch to damage an AF radio which uses 48VH as the switch can not put more power out than the power supply generates and the AF is already expecting power on all 4 pairs so there is no way to do any harm to an AF radio.

Since the switch is powering up and if you verify the power supply is indeed supplying the board 50V+/- that tells you the surge did not come in through the power supply or it would be dead so the tower took either ESD, or there was significant ground current flow between the tower and the electrical service ground rods which would mean you do not have your tower ground rods properly bonded to your electrical service ground rods.

Do you have a heavy wire like #2 running between your tower ground rods and your electrical service ground rods bonding them together?

You might also want to add additional ground rods to your electrical service ground rods and make sure all connections are corrosion free.

Ground current is the #1 cause of WISP equipment damage.

Also you can NOT use Ethernet surge protector on "passive" POE switches like the WISP Switch and the ToughSwitch or any of the Edge switches if you have the port configured to "passive" POE which all 24V and 48VH is always "passive" on any Edge Switch. WISP Switches and Tough Switches are ONLY passive POE on all ports. "Active" POE is when the switch identifies the device and turns on 48V by itself. Active POE ports can deal with all 8 wires being clamped together but no passive POE port on the market can deal with this without damage to the port.

Ethernet surge protectors can only be used on "ACTIVE" POE ports and POE Bricks but never "passive" POE ports.

But 99.9% of damage is not from surges on Ethernet port but rather ground current which an Ethernet surge protector is useless to protect you from as EThernet Surge protectors work by clamping all 8 wires together along with the ESD drain wire (ground) which creates a DEAD SHORT and will damage the switch port.

The dangers of using Ethernet Surge Protectors have been discussed many times on this Forum, the UBNT Forum, WISPA Email lists, and WISP Talk group on Facebook.

[twotincans]

1) You removed my 2nd picture showing the close up of the damage of the switch. Also, I did take out the board and look, and I said that the bottom of the board had no damage in my last post.

2) The tower and building is grounded. It's at a FM radio station that done a lot of work making sure everything is grounded correctly. The building has 8 foot grounding rods in each corner of the building and are strapped together using a 4" copper strip. This also ties into the electrical service grounding rod. The tower is also a part of this grounding loop and adds 2 more grounding rods. The grounding loop is also attached to the equipment racks to ensure everything is on the same grounding setup.

[sirhc]

I did not remove your second picture I simply did not see it so I could not insert it properly.

Please edit that post with the missing picture and upload and attach the second picture and then insert that picture into the post below the first picture.

To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

As I said though there is no way the switch can damage a 48V POE device except in this rare situation:
Say you have a Titanium Rocket in port 4 and the port is set to 48V but since that port is capable of 48VH and if the switch port 4 MOSFET circuit takes damage and applies 48VH instead of 48V to port 4 the Titanium Rocket can not deal with 48VH so it would be fried. But an AF does accept 48VH (power on all 4 pair) so it can take power on all 4 pairs as that is what it wants and the switch can not do a Jesus think and turn 48V into a voltage higher than the AF can use. Besides I think the AF radios can handle up near 60V anyway but this is a mute point because the switch can NOT put out that much voltage it is impossible.

There is nothing that can allow the switch to send more than 50V+/- to a 48V POE device so there is no way to over voltage damage it.

There is also no way for the switch to produce more than 24V+/- but if a port MOSFET circuit is fried it can output 48V on a 24V port which will fry a 24V POE device.

Your tower went down in a "storm"......the switch sort of survived.

Now you can determine if some of the ports or all of the ports are sending out 48V (50V+/-) instead of 24V or even when told not to send any POE which obviously 48V will fry a 24V device.

So I can see 24V devices that maybe survived what ever happened here then got fried by the switch which also took damage and is sending them 48V but there is no way the switch could damage a 48VH device, simply not possible.

I am very interested when you get time to test the switch out which ports are stuck at 48V POE.
I am also interested in you to confirm that 24V is 24V and 48V is 48V (50V) and that the switch is simply reporting voltage incorrectly in the UI due to damage most likely from the 3.3V sensor which is used as a reference voltage to calculate the 24V and 48V readings. So if the 3.3V is incorrectly reading high it will make the 24V and 48V readings high like you see them being reported but they are not actually high.