Netonix Keeps Rebooting on 1.5.0

[mhoppes]

So I have a very busy core site that the Netonix switch there keeps rebooting every few weeks. It's a 24 port that is pretty heavily utilized on port density and throughput. The switch that was there died a few weeks ago and I replaced it. The new switch is doing the same thing.

I've verified the site is not loosing power. The switch appears to be exhausting memory moments before the crash. Why is this happening and how do I prevent it from happening?

[sirhc]

Please disable all serviced not essential such as but not limited to the following:

Discovery Protocols
Discovery TAB <= Especially this one
SMTP
SNMP


Then monitor the switch manually every so many hours and watch what happens to memory.

[mhoppes]

I turned off the discovery tab.... all discovery protocols were already disabled.

If you look at the graph, this was a sudden memory increase, not something that crep up slowly over time. As of this morning, it's still down low.

[sirhc]

I turned off the discovery tab.... all discovery protocols were already disabled.

If you look at the graph, this was a sudden memory increase, not something that crep up slowly over time. As of this morning, it's still down low.

That still does nit mean another network device suddenly goes NUTS and spews a bazillion packets to the switch.

Like I said there are 70K+ switches out there all basically the same running the same firmware and only a "few" people see this.

If this was a memory leak all would see it.

Now as I said that does not mean something weird is happening at these "few" peoples network maybe similar that affects the switch and we could possibly handle it better but unless someone tells us how to recreate an event how can we fix it?

[mhoppes]

Chris,
You have multiple people who have rebooted reboot issues on 1.5.0 that have not seen reboot issues prior.

Could it be a device spewing packets? Sure..... but the switch should handle this gracefully and not go into an over memory situation.

[sirhc]

Chris,
You have multiple people who have rebooted reboot issues on 1.5.0 that have not seen reboot issues prior.

Could it be a device spewing packets? Sure..... but the switch should handle this gracefully and not go into an over memory situation.

Yes Matt out of 75,000+ people using our switches less than 5 people have reported this issue.

What does logic tell you?

And you are correct, we "possibly" could handle it gracefully "if we knew what it was". Then again not always. I have a fiber media converter that if I plug it into "any" manageable Layer 2 or Layer 3 switch it will lock it up. There are certain types of packets that are granted access to the core regardless such as broadcasts. The switch core and its code is not changeable by us.

What steps have you taken to determine what it is?

In the post link above you can see the user dug out his wire shark and started narrowing it down, turned out to be a bad SFP module causing his reboots.

Its easy to point a finger at someone and say it is your fault but there are countless times on this forum where that was done and in the end it was not our fault.

You have quite a few switches of ours, are they all experiencing the same issue?

There are 75K+ out there and only a handful of people are seeing this and they may or may not be all the same thing.

If it is something like a malformed broadcast packet there is nothing we can do to prevent this and besides the piece of equipment issuing those packets needs to be removed from a network.

Its hard to fix or even know what an issue is unless you know what it is. As you can see in the pictures below none of my switches are seeing this?

[sirhc]

We do actively investigate peoples issues that make sense or that we can actually recreate.

Here is an example, read from this post down: viewtopic.php?f=17&t=3914&start=70#p27795

Issue reported Jan 17, 2019 9:34 pm

New firmware patched released Jan 18, 2019 2:08 AM <= A FIRMWARE FIX WAS ISSUED IN 3 HOURS, WHO ELSE DOES THIS?

Stephen and I worked diligently to confirm then fix a bug that we "could" investigate.

You tell me what you have provided that I can use to work off of?

Seriously where would I start to replicate and then test your issue, not to mention it does not make sense as what you call a "memory leak" would be affecting everyone. Obviously something is differnt from your situation and all of my switches in service and I have quite a few in service not to mention everyone else who is not seeing this issue.....WHERE WOULD I BEGIN TO EVEN START DEBUGGING THIS?

[mhoppes]

I'm not saying you don't address customer needs -- you do.

[sirhc]

I'm not saying you don't address customer needs -- you do.

Give me something to work with Matt and we will be all over it, I promise.

[mhoppes]

My problem is when it happens it's increadibly sporadic.... .I'll see what I can catch next time it happens.