WS-12-250DC breaks after configuration update

[alexishh]

We've got a WS-12-250DC switch powering a tower.
We regularly make updates to the switch to add a new vlan. This is the only thing we update : adding a new entry to the VLAN table, applying then backing up the configuration.

The same issue happened twice now, where sometimes, one or two minutes after the update, communication with the switch is lost. Here are the details of what we found after a truck roll to the tower :

#1
- switch appears fine : lights are ON, PoE devices are still powered and did not reboot, but no traffic is going through the switch. FAN seem to be always blowing while usually it's only doing that from time to time, but difficult to say if it's really unusual, maybe it's warmer today or something.
- we unplug power and plug it back to the switch. It reboots but this time all ports lights stays OFF, PoE devices are no longer powered. Still no admin access to the device.
- we exchange the switch with a spare one and inject the backup configuration, and everything comes back online.
- in the lab, we reset the switch. Sadly I don't remember if I tried the soft reset first, but the switch came back after I reset while powering it up. I tested it and everything was operating as expected.
On the syslog server side, the last entry from the switch before the outage was "!Reverting to last known good configuration
".
I tested doing successively many VLAN configuration updates and the switch never had any issue with that. Could not reproduce.

#2 - happens again on the spare (brand new) switch, running up-to-date 1.4.9 firmware, and after several similar updates that went fine
- same symptoms: PoE still ON, FAN active, no traffic, no admin access (did not try the console port - we don't have anything to plug into that thing)
- this time I tried to directly reset it instead of unplugging : I pressed the reset button for a while while it was still on but nothing happened. No light show.
- I then unplugged the switch's power line, and kept the reset button pressed while turning the power back on. Switch went back online, I applied backup configuration and everything went back to normal.
Same last syslog entry before outage : !Reverting to last known good configuration.

We had auto backup feature enabled this time, and the switch did backup it's configuration successfully before (and that's the one we put back in place after resetting). The "Reverting to last..." log entry happened about 60 seconds after the auto backup, which matches with the Revert Timer.

In both cases I'm pretty sure the configuration was applied. It should not have reverted it. And even if it had to revert, it should just come back to the previous version and not lock itself like that...

Anyone ever experienced similar behavior? I don't know how to troubleshoot this further and I can't reproduce in lab (note that it's not under same PoE & traffic load while in lab). Anything I should pay attention to ?

[alexishh]

I swear that I did browse the forum for similar issues... and it's only after I finally post that I find something relevant...
See https://forum.netonix.com/viewtopic.php?f=17&t=3631

The effects are very similar. However in their case it happens after turning PoE ON/OFF while we never updated PoE status. Just VLANs.

[sirhc]

Well if it happens again use a serial console cable to see if you can access the switch and see what's going on.

[Banana Jack]

I had this same thing happen on three switches so far: 2 x WS-6-MINI and 1 x WS-8-150-DC. My feeling is that the issue isn't dependent on the specific setting being changed. On one of our WS-6-MINI switches and the WS-8-150-DC (both running 1.5.0) I added a syslog host and shortly after pressing Save/Apply, the switches died, i.e. they stopped passing traffic and became unreachable although their PoE ports remained powered. On the other WS-6-MINI I made a minor configuration change but I forget what it was (not syslog). Before the switches went offline, they sent a syslog message "!Reverting to last known good configuration".

I guess I'll try resetting the failed switches to see if they'll come back to life, but I'm not sure I trust them any more.

I mainly just wanted to add my experience 'into the melting pot' in case anyone can spot a pattern or figure out why this happens or how I could prevent it happening. It makes me nervous about making changes since if it goes wrong, it needs a truck roll. And (after taking feedback from customers) ... usually quickly!

Glenn

[sirhc]

I simply do not see this occur on any of my switches in service at my WISP.

Maybe your configs are corrupted?

Did you do any scripts or modify the config manually in any way?

[Banana Jack]

I simply do not see this occur on any of my switches in service at my WISP. Maybe your configs are corrupted? Did you do any scripts or modify the config manually in any way?

Thanks for the reply. No I didn't make any scripts or manually modify the config; I didn't think that was possible anyway since the config files look like binary when I view them. I only ever use the GUI to make config changes, and I usually deploy new switches by using a 'standard' config file which is taken from a manually-configured switch of an identical model. (I already learned the hard way that config files between different switch variants aren't interchangeable!)

I can't do anything with the two dead WS-6-MINI switches but I connected a console cable to the WS-8-150-DC and it just repeatedly says, "Segmentation fault" infinitely scrolling, (screenshot attached). If you'd like me to send you the switch for a 'post-mortem' I'd be happy to do that. Otherwise I suspect this will be one of those faults which is so difficult to reproduce that it might be impossible to fix.

On a more positive note, I still love Netonix switches! The problems are very rare in comparison with the features and advantages they offer.

Thanks
Glenn

[sirhc]

Well the segmentation fault "sometimes" can be fixed with a power on factory default: viewtopic.php?f=6&t=409#p2398

If that does not work and it has a console port then you can try a console firmware recovery: viewtopic.php?f=17&t=1073#p8073

If no love from above then RMA them as it is a simple re-flash, and we will do a checkup of the unit and fix anything else we find wrong and apply any hardware MODs for a reasonable fee.

You can import configs from other models so long as same firmware version, and if importing say a 6 port to a 12 port you may need to do some touch up in the UI such as VLANs.

Importing configs from a higher density port to a lower density port should result in losing any config for higher ports not present.

Yes the config files are JSON and some people do modify them with scripts - I suggest not as JSON is not forgiving if 1 character is out of place.