When APs reboot, Switch freezes and requires power cycle

[JoshuaResound]

Hello all!

We've recently identified an "issue" where our Netonix WS-26-400-IDC switches will become unresponsive when rebooting multiple APs that are connected to them.

It's not a major issue, but it's given us quite the scare twice between yesterday and today.

We are having to reboot some rockets from ubiquiti for firmware upgrades, and if the AP count is about 6-8, then the switch is fine. If the AP count is 10-13 and we do them one by one, then the switch is fine. If we do them all simultaneously, then the switch has a stroke and UI, POE output and the ability to restart either via UI or SSH is gone. Restarting Lighttpd service gets us back into the UI, but there is a time bomb that's counting down. After a while, the switch will become unresponsive to ICMP pings, and needs to be power cycled. We are currently on 1.4.7, and are planning firmware upgrades either tonight or tomorrow evening. The strange part is if the UI is still accessible while in the post AP reboot state, the POE values seem to be off. Very low watts to all devices that are unresponsive, and this most recent event had all POE labelled as OFF, while the wattage readout was still reading values.

Has anyone else had this issue? Also, does anyone know if the newer firmware revisions have either identified this or have fixed this issue? I know this isn't necessarily a real "issue" as it is "user error", but I did not know if anyone has seen this or not.

[Stephen]

1.4.7 is almost 3 years old it is very possible the latest has this patched assuming it's a software problem.
But I can't personally attest to it because if this was an issue on this firmware then it was before my time.

However, I wanted to warn you though that when you upgrade to the latest firmware you might run into unusual behavior.
Generally speaking upgrading this many versions at once has been known to lead to unpredictable behavior at first. So before you perform the upgrade, if possible, I would take that switch out of operation temporarily so that you can make sure it is behaving correctly after the upgrade. You may not have to go this far but just be aware of this so you can prepare in the way you see fit. Worst case scenario you'll have to default it and rebuild the config on the switch in the UI before putting it back in operation.

To avoid this issue in the future I highly recommend using the Netonix Manager to keep your switches up-to-date easily.

And obviously after you have finished the upgrade if you're still seeing this problem please post details about it here.

[Stephen]

By the way this might also be this issue:

viewtopic.php?f=6&t=2591&hilit=CAP+common#p18176

2) Defective 3.3V CAP on WS-8-150-AC, WS-8-150-DC, and WS-6-MINI - Covered under warranty
This was from an out of spec CAP used in 2 production runs of these units. Not all units will have an issue and to our best estimates we feel this can affect 2%-5% of units. The symptom is that unit will reboot if the user is using at least 3 or more active ports. Some unit will work fine up to 5 or more ports. We have extended this issue to be covered to the life of the switch so if you don't realize you have the issue for 3 years it is still covered under warranty.

3) Defective CAP (C51) on 250 watt power supplies - Covered under warranty
This affected 1 batch of power supplies used in the WS-12-250-AC and WS-10-250-AC. The symptoms are if the power supply is not loaded with at least 50-60 watts the unit will reboot after several hours and keep rebooting quicker and quicker until eventually it gets stuck in a reboot loop. If you let it cool down it operates again as it did before for several hours. We have extended this issue to be covered to the life of the switch so if you don't realize you have the issue for 3 years it is still covered under warranty.

[JoshuaResound]

Thank you for the information. I will forward this info to the Engineers and get a game plan going. I'll scour through the firmware pages and see if I can grab each version up to the latest and step it up.

You are awesome, and I appreciate the help and insight for our problems!

I will deploy the Netonix Manager shortly to make sure this does not happen again, and if we find a definitive reason for this happening, I will post about it here. I do not believe we have any switches that are up to date, so this will make things a lot easier to manage.

Thanks again!

[Stephen]

You're welcome JoshuaResound. I'm glad that was helpful.