v1.5.0 Bug reports and Comments

[mhoppes]

We are also seeing switches with 1.5.0 rebooting themselves, and the graphing is showing that the memory spikes to 100% for about 5 to 10 minutes prior to a reboot. This did not happen in previous firmware versions. There is definitely something wrong with 1.5.0

[dthomas]

Yeah we have a WS-12-250B that never rebooted until putting 1.5.0 on it. I defaulted it and reconfigured as suggested but it still kept doing it. I have swapped it out with a newer WS-26-400-AC. So far it hasn't rebooted but it has only been about 10 days.

[mhoppes]

I swapped to a new switch and it's still doing the same thing.

[sirhc]

We are also seeing switches with 1.5.0 rebooting themselves, and the graphing is showing that the memory spikes to 100% for about 5 to 10 minutes prior to a reboot. This did not happen in previous firmware versions. There is definitely something wrong with 1.5.0

Then since this is an older switch anyway Matt simply downgrade to v1.4.9 with all the same configurations and confirm.

But as I said in another post you just made: viewtopic.php?f=17&t=4868#p27788

In the above post I asked you to disable all services not needed to try and find the issue and pin it down to a service.

I am not sure why people document the same issue in multiple places as it makes it hard for us to follow and to make sure all locations are updated so someone reading through the forums finds an issue with no solution.

Bad Ethernet devices can cause havoc to switches and routers as noted in this post: viewtopic.php?f=17&t=3623&p=26619&hilit=+bad+sfp#p26617

I just wondered whether there has been any progress on eliminating these 'random reboots' as we are severely suffering from it on our network at the moment. We have about 70 Netonix switches and they are all rebooting seemingly at random, causing their connected radios to be power cycled. Sometimes we can go 3-4 days with no problems, and other days we can see many tens of switches rebooting. It's turning into a disaster zone. On our remote syslog server I am seeing 'Detected cold (watchdog) boot'. Nothing is syslogged before the reboot occurs.

Last month we were running 1.4.9 on all switches apart from 2-3 units where we were trialling 1.5.0. After the first major 'reboot storm' (described in this thread) I upgraded all switches to 1.5.0 to try to prevent recurrence but the reboots seem to have got more frequent if anything, although that might be a co-incidence. We have around 7 Mikrotik switches which occasionally log packet loops ("etherX: bridge port received packet with own address as source address, probably loop") but I'm not sure whether this is an 'erroneous error' or if there is a genuine intermittent loop / storm and whether it's possibly causing the Netonix reboots.

Should I try enabling Loop Protection on all our Netonix Switches? What about the other Storm Control settings i.e. Broadcast / Multicast / Unicast / Pause Frame? Is there any downside or risk to enabling these settings?

Thanks
Glenn

Thanks sirhc - at the moment I'm suspecting that a network loop and/or bad plug/cable/radio is to blame... Today I've been enabling loop protection on selected switches and it seems to be pointing me toward a certain part of the network based on the ports on which loops are being detected. So maybe I'm making some progress.

Just to update you sirhc and anyone else who's interested for the record: this problem turned out to be an apparently faulty Mikrotik S-85DLC05D SFP module which was in our main WS-26-400-AC switch. Wireshark showed a broadcast spike (2000-3000 PPS) every minute and sometimes with additional smaller spikes in between. Swapping out the SFP module immediately cured it - screenshot attached.

The Netonix 'Loop Protection' feature was useful in showing up the problem and mitigating it in some parts of the network (to the detriment of the downstream parts), although sometimes the port that it reported as having the problem led me up some blind alleys. It was only when I tried setting Broadcast Storm Control to 2K on our main switch that the switch freaked out, crashed (with syslog kernel err FAIL: RX Alloc 1866 bytes) and when it came back up, the SFP remained down. I had to plug/unplug the SFP several times to get the link up, so then I became suspicious and swapped it out, which then cured the broadcast spikes according to Wireshark's extremely useful Statistics > I/O Graph feature. Hopefully our switches will now stop watchdog-rebooting.

Thanks
Glenn

[mnitech]

Installed a new switch (WS-12-250-AC) but forgot to turn on SNMP. I enabled it remotely but still couldn't get SNMP to respond. Finally rebooted switch and it worked.

[Stephen]

Installed a new switch (WS-12-250-AC) but forgot to turn on SNMP. I enabled it remotely but still couldn't get SNMP to respond. Finally rebooted switch and it worked.

Just tested a WS-12-250-AC with both 1.5.0 and 1.5.1rc12.
Fresh install and powered up with SNMP inactive on both versions. After enabling SNMP it worked without a reboot.

If something like that happens again, check the logs for anything suspicious before you reboot it and post it here.