Netonix Keeps Rebooting on 1.5.0

[sirhc]

My problem is when it happens it's increadibly sporadic.... .I'll see what I can catch next time it happens.

Here are some questions;

1) Does it happen on all your switches?
If the answer is no then probably something local to this Layer 2 segment

2) Did you try disabling all services not 100% essential including SNMP/Discovery/SMTP/SYSLOG and monitor ,manually?
If you have not then do so.

3) You say it did not happen with v1.4.9 and these are OLD switches so did you try downgrading to v1.4.9 leaving all configurations the same and see?

[mhoppes]

1) I have very randomly and very very sparcly see it happen elsewhere -- not often enough to comment on or be concerned with. It has happened too many times on this segment.

2) I just disabled everything this afternoon.

3) When I downgrade to 1.4.9 I start having all kinds of major momentary glitches in data flow and spanning tree flaps. On 1.5.0 that doesn't happen, but the switch will reboot from time to time. Now that I think about it... it did happen with the old switch, which was on 1.4.9 from time to time... so it's NOT something purely new on 1.5.0.

I'll keep an eye on it and let you know of any conclusions or further details I figure out.

[mhoppes]

So it happened again.. same issue. The switch is on a private IP (though not firewalled). I've disabled STP as well as all discovery protocols and enabled administrative controls on IP addresses that can access the switch.

I'm also running a continual top and tail -f /var/log/messages

Eric, where is the syslog kept? In memory? is it ever purged? Is it possible that some kind of error or attack against the switch would overwhelm the memory through syslog messages and ultimately cause the switch to reboot?

[sirhc]

Eric is out of town. But I am talking to Stephen to see if he can answer your questions regarding syslog but I think it is in memory and if not still lost on a reboot.

I still think this is a bad piece of equipment somewhere spewing packets or something.
Getting an alert of memory usage spiking and then checking port statistics and even looking at TOP would be a big help.

If there are SFP modules in this unit I would swap them out.

[mhoppes]

By Job! This issue may have started when we added a MikroTik SFP.

If that’s the issue. And it’s a big if. Why are there so many issues with SFPs with these switches? I have another one where if I even bump the copper SFP everything just goes down.

[sirhc]

By Job! This issue may have started when we added a MikroTik SFP.

If that’s the issue. And it’s a big if. Why are there so many issues with SFPs with these switches? I have another one where if I even bump the copper SFP everything just goes down.

If you go back and read the post on Facebook which linked here: viewtopic.php?f=17&t=3623&p=26619&hilit=+sfp+reboot#p26619

It is NOT a compatibility issue it is a BAD SFP module.

I also told you on Facebook that I have a fiber media converter that is bad that I can plug into "any" Layer 2 segment and it will within an few hours lock up or reboot "all" managed devices even Cisco top of the line switches.

DID YOU NOT READ THE POSTS I MADE?

[Stephen]

Just double checked to be certain.

/var/log/messages is lost on reboot but the file is not allowed to grow past 200K in memory.
After it fills up 200K it starts throwing away oldest log entries. - FIFO

Otherwise the log is lost after reboot.

[deerefanatic]

I see the same Issues with Rebooting at random... May go months and then BAM reboot.

[mhoppes]

Chris, can I get an answer from Eric on where the log file is kept? Is it kept in memory or somewhere else?

In other words -- some kind of an error or attack against a switch -- would it potentially overwhelm the onboard memory if the log fie was made to grow?

NOW that being said...since disabled STP and adding access controls the switch has not recently rebooted. I'm going to let it continue to go to see what happens. However, this is the longest I've ever seen it stay up without a reboot in recent times. I'll probably let it go to the 60 day mark before I revert any changes to see what happens.

That being said.. for the last two years I have been trying to track down some instability in a network segment that runs through this switch. For no explainable reason we would suddenly lose connectivity for a minute and then it would return. Switch logs would sometimes show nothing, other times it would show an STP hit on a port that definitely has no loop.

Since turning STP off on all three switches on this network segment the segment has run smooth as a baby's bottom without crashing and without going down.

Now riddle me this batman..... if there actually were loops wouldn't I still be seeing packet storms that would crash the network? Why is STP firing off when there are no (apparent) loops.... also loop control isn't firing off either with STP disabled.

[Eric Stern]

Chris, can I get an answer from Eric on where the log file is kept? Is it kept in memory or somewhere else?

In other words -- some kind of an error or attack against a switch -- would it potentially overwhelm the onboard memory if the log fie was made to grow?

Its stored in /tmp/log/messages, which is technically in memory because /tmp is a ramdisk. But the file is rotated once it exceeds 128kB so it will not grow to fill all of memory.