Strange Power Issue on WS-12-250-AC

[MyThoughts]

We have been having a reoccurring problem with the WS-12-250AC. We have had it happen three times, on two different units, running in completely different locations.

The issue is that the WISP switch crashes or reboots (not do to power) for an unknown reason, when it powers back up the unit sends power to all the devices, but the switch portion is non functional, no ethernet links on any of the powered equipment.

If it happens I will see our AF24 backhaul and AF5 backup go offline and come back online, but everything past them will still be down. When I login to the units I will see that the uptime is short, just as if the switch was rebooted. BUT there will be no ethernet connections at all on any of the PoE ports for the WISP switch. It just so happens that at one of the locations that experienced the problem I have a second WS-12-250AC running from the same power system, that has a UBNT mPower 8-port powerbar. I was able to login to the mPower via a sector antenna connected to through the 2nd WISP Switch. I was able to verify that there was no loss of power or power issue at the location via the PowerWare 9130 UPS at the tower site. By using the mPower I was able to turn off the problematic wisp switch and turn it back on. After the power cycle everything comes back on normally.

Prior to the first time this happened we did not have the newest firmware on the WISP switch, if it is not broke don't fix it... After it happened we updated all the WISP switches in our network (2 x WS-12-250A RevB, 11 x WS-12-250AC RevD) to v1.3.9.

We assumed it was a firmware issue, but this past two weeks it has happened to a 2nd unit in a completely different location, and once more on the until that originally had the problem. Both locations are using double conversion sine-ware UPS systems, and both are grounded using a ring ground topology.

First occurrence was February 2nd, a light flurry passed through, maybe a small amount of upper atmosphere lightning, but nothing major. WISP switch rebooted remotely via mPower.

Second occurrence was March 27, this was a major electrical storm, worst lightning storm I've seen in recent memory. We did not take any direct hits, and we lost no equipment anywhere on the network during the event. I had to do a truck role to the tower that went offline due to the WISP switch problem, after physically power cycling the WISP switch everything came online without issue, and has been running fine since.

Third occurrence was April 6th, light rain, no lightning that I was aware of, same switch that had the problem February 2nd. WISP switch rebooted remotely via mPower.

Has anyone seen or experienced anything like this? Are we seeing faulty hardware, or is this a software bug?

I need to get these problems sorted out as as I am sure everyone here will understand, internet clients are not patient nor understanding people. God forbid a 15 minute outage.

[sirhc]

I had experienced this type of issue on our Bareville water tank two years ago.

viewtopic.php?f=6&t=1547&p=11426&hilit=bareville#p11426
viewtopic.php?f=6&t=1547&p=11420&hilit=bareville#p11420
viewtopic.php?f=17&t=733&p=5427&hilit=bareville#p5427

I want to stress that this same switch is still in service today.

We beefed up the grounding.
Eliminated the Ethernet cable connection to another ground potential (customer on property).
Made sure our INTENDED grounding system was cleaned and serviced.

Read more about grounding systems here:
viewtopic.php?f=30&t=188
viewtopic.php?f=30&t=1429

Really read those posts above. High winds around radios/antennas can cause a Static discharge to come down your Ethernet cable and can cause the switch to reboot FAST in a state that is weird.

Ground Potential shift can do similar things.

If power is switched on and off too fast you switch will start to depower but before it is discharged power is restored and the switch does not boot properly.

It is really easy to replicate this on a bench, especially with a WS-6-MINI and a barrel adapter. Boot the switch up and have it power a couple devices on the bench. Now pull the barrel adapter and then plug it back in only after about 1-2 seconds, you will see the switch in a weird state.

I never tried this with an AC model so not sure how long to depower it has the power supply had large caps and hold some amount of power so the power cycle may need to be slightly longer?

[sirhc]

I really think your issue is power related or grounding related.

Poor grounding can cause the switch to reboot from static charges which is more common then you think, when you have a piece of metal moving fast through air (wind) and the humidity is just right a massive static charge can be created and if you do not have an intended ground path for the discharge to take it will follow your Ethernet cable down to the switch.

Also if you have a power blip and your UPS is too slow kicking in the switch does a partial shut down then tried to boot again.

The evidence of something like this is right in front of you. The MOSFET circuit that controls POE power is active driving the POE powering to the device but the switch is locked up. This can only happen if the switch losses power for a second and the MOSFET circuits do not have time to power off (open) before power is restored and the switch does not boot properly (locked up).

Static charges and ground potential shifts can do the same thing to occur.

I would look at your UPS, test it by cutting power and putting power back several times real fast and see if your UPS handles it smoothly or you end up with the same condition.

Next I would look at your grounding system and compare to what I explain in the posts in the post above. Too many people spend too little time and money on grounding and have soo many issues.

I use ZERO Ethernet surge suppressors in my network and I have only lost 1 radio in over 5 years now. And the one radio I lost is described in the post above because Sprint destroyed the grounding system when they left the tower.

You need an intended ground path and not the steel tower as towers are much higher resistance than your Ethernet cable the surges would rather ride your less restive Ethernet cable.

Ethernet cable runs need to be longer than intended ground paths by at least 10% so the Ethernet cable looks like a less attractive path to ground.

Poor grounding will also cause damage to your switch by frying your current sensors or ESD protection diodes. If suddenly your switch is reporting 12 watts to power an AF24 then your sensors took damage. RMA it for repair and upgrade your grounding system.

[sirhc]

First occurrence was February 2nd, a light flurry passed through, maybe a small amount of upper atmosphere lightning, but nothing major. WISP switch rebooted remotely via mPower.

Second occurrence was March 27, this was a major electrical storm, worst lightning storm I've seen in recent memory. We did not take any direct hits, and we lost no equipment anywhere on the network during the event. I had to do a truck role to the tower that went offline due to the WISP switch problem, after physically power cycling the WISP switch everything came online without issue, and has been running fine since.

Third occurrence was April 6th, light rain, no lightning that I was aware of, same switch that had the problem February 2nd. WISP switch rebooted remotely via mPower.

All three of these events occurred with atmospheric conditions. If it was a faulty unit it would happen on warm sunny days with ZERO wind or rain. All three days could have also happened during a power blip.

The condition you are describing is one I experienced as you can read above in the linked posts from last year and we solved with improved grounding, change dumb ass cable run to different building to fiber (man this was dumb on our part), and replacing the UPS inverter charger.

Not sure which one fixes it which was also dumb as I should have done them one at a time but just wanted it to stop happening so we did all three at once.

[MyThoughts]

Just an update.

As for the grounding, I agree and take grounding at our locations very serious. The last time we lost equipment at a tower was due to a lighting strike on the nearby electrical utility lines, even then the only damage we suffered was a couple 16 port switches, and this was because a fault in a TrippLite UPS that passed the incoming electrical surge out the SNMP webcard. The UPS was sent to Tripplite and they confirmed the internal protection failed. I have had the privilege of having had a few conversations with 'Sirhc' on the phone, and more then once it was mentioned to ensure good grounding practice. I attribute the lack of problems in the past at our towers due to our use of proper grounding techniques, many of which we adopted many years ago after first talking to 'Sirhc'

Now I am not ruling it out as a possibility, but both locations that have experienced the problem were inspected for any grounding issues after the first occurrences and we found no apparent faults. And one of the locations has multiple WS-12-250AC units and it has been the same one experiencing the problem (not conclusive but it is the same grounding system for both).

That one unit has had another blip, its third, it just rebooted itself. This time it restarted without requiring manual intervention, and it happened on a calm day without any rain/storm/etc, although it is unfortunately overcast and not sunny..
We have replaced it, and will be keeping an eye on the system. Problem is of course finding the cause, it was 2 months from the first occurrence to the second, so not exactly easy to reproduce.

As for our power backup, we use double conversion on-line UPS systems at our tower sites and as such there is no transfer time during any utility interruptions, our tower grounding system is bonded to the electrical ground. When the netonix switch rebooted the UPS did not report any power issues.

We have never had these issues with any other equipment in the 12 years of running our ISP, it may just be a slowly dying unit. With a new unit in place, it is now wait and see game.

[sirhc]

Well keep us posted.

It is always good to be able to swap equipment out to eliminate it but as you said it is a waiting game in your case.