Switch reboots - High mem usage

[mhoppes]

Chris,
With all due respect - I don't think you are masking an issue.

It's clear something in the switch can't handle something (whether STP something or other, or broadcast from Discovery protocol). Not that there is necessarily a network issue -- but that some piece of software in the switch just isn't handling a situation properly.

In our case turning off STP seems to have resolved the issue and does not appear to have rendered any underlying issues.

[sirhc]

Chris,
With all due respect - I don't think you are masking an issue.

It's clear something in the switch can't handle something (whether STP something or other, or broadcast from Discovery protocol). Not that there is necessarily a network issue -- but that some piece of software in the switch just isn't handling a situation properly.

In our case turning off STP seems to have resolved the issue and does not appear to have rendered any underlying issues.

So you need to re-enable STP/RSTP and confirm that was it, again coincidences often lead one astray.

STP/RSTP talks to adjacency so one might also want to look at what those adjacency are.

Also if RSTP is enables on all switches and a wireless link between them is flaky or gets maxed out causing some packets to get delayed or even dropped even for a brief moment it can cause RSTP to behave abnormally as it misses the control packets that allow RSTP to function properly.

I do not run flat layer 2 segments beyond a tower so I can not say but I do run multiple switches in my office with RSTP enabled and no issues. If a wireless link is dropping packets needed for RSTP to function then things could go wrong.

[mhoppes]

Correct me if I'm wrong -- but dropped packets shouldn't cause the switch to reboot? They might cause RSTP to engage or throw errors -- but issues should never cause equipment reboot.

[sirhc]

Correct me if I'm wrong -- but dropped packets shouldn't cause the switch to reboot? They might cause RSTP to engage or throw errors -- but issues should never cause equipment reboot.

Correct, it "should" not cause memory to grow. That would be a bug on our part which we are "Trying" to narrow down the cause of the memory growth. Again hard to recreate this if it is.

Not saying it is or is not in the RSTP routines or if it is in the Discovery routines which we are looking at now.

[mhoppes]

I'm going to give it 30 days, maybe 60 before I re-enable STP to confirm the issue is actually gone. I'll let you know.

What I can tell you is I already had Discovery disabled on this switch that was rebooting.

[sirhc]

What I can say is what ever is causing the memory growth is something that does not occur normally. Now rather or not we are handling something wrong that occurs that should not occur is another story and again very hard to replicate in a LAB to see it and fix it.

[mhoppes]

Understood. This is sounding like a cancer :). What I can also tell you is I have a tail going on my problematic switch so hopefully I can catch the last dying gasp before it reboots if it does it again.

[sirhc]

You "think" you narrowed it down to disabling RSTP,now it would be a big help if you could verify this beyond the possibility of a coincidence by turning it back on, seeing the issue occur again. Turn it off problem goes away. Then repeat several times, each time reducing the percentage of a simple coincidence and narrowing our search.

[mhoppes]

I'm planning on it -- want to give it a few more days to make sure it's solid.

[wmackay]

Also coincidence often leads people to incorrect conclusions.
Each Layer 2 segment has differnt equipment and different traffic.

So to back up your hypothesis simply take a switch that us running v1.4.9 which you think is fine then upgrade to v1.5.0 or v1.5.1 or v1.5.2rcX and see if the mem issue starts there or not and let us know.

I have not posted any conclusions or hypotheses, only facts. I won't upgrade another switch to 1.5.0 because that could cause this issue to affect more of my customers. I plan to upgrade an affected switch to 1.5.1, and if that doesn't fix it, downgrade to 1.4.9.

I believe a switch needs to survive arbitrary network traffic, including traffic from buggy devices, even if this means "masking an issue". I would rather have a switch that fails to alert me to a problem, than a switch that reacts to problems in my network by rebooting.

Here are some graphs of memory usage from switches running 1.5.0:



And here are graphs from switches running 1.4.9: