Switch reboots - High mem usage

[sirhc]

Have you tried disabling "all" services including SNMP, SMTP, and so on, just the bare essentials then see if memory still increases over a few days.

If memory does not increase (checking manually daily to insure) then enable 1 service at a time every couple days to try and find the service that is consuming the memory.

I am running these switches and do not see this issue so its not like we can recreate it.

If you can narrow down the offending service it would be a big help as we could then narrow down what is differnt from your use of that service.

I am running almost all services on my switches and do not see any memory increase as shown in my screen grab in a post above but that does not mean in your configuration which is what we need to find is causing your issue.

Only other option would be to create port mappings to allow us access to a switch in question for us to look at and monitor.

[Banana Jack]

My opinion is that it could be (excessive) broadcast traffic which causes increasing memory usage. When UBNT released airMAX firmware 8.5.8, a bug in it created a lot of unnecessary discovery packets (whilst logged in to their GUI) and therefore broadcast traffic and therefore packet storms (on large flat networks).

On my own network this made my 60+ Netonix switches seriously unhappy and after several days they went super-slow and froze up / rebooted. Some even died completely although this may have been due to me trying to upgrade their firmware whilst they were in a state of unhappiness and maybe they didn't have enough spare CPU cycles/free memory to see it through.

Of course this problem might be a separate issue to the one being discussed in this thread, but if I was going to investigate the issue and I was wondering where to start then I would certainly begin by throwing a lot of broadcast traffic around and observing Netonix memory usage / reboot frequency to see if there's a correlation.

Glenn

[mhoppes]

Hi sirhc, I think you were replying to Kingsley, and I don't mean to hijack the thread, but in case you can help me too, here is a screen grab from my 'top' :

top.PNG

I'm polling with SNMP every 60 secs from PRTG. I'm using Netonix Manager 1.0.10. I'm logging in only once or twice simultaneously with the Web GUI and only on ad-hoc switches when I want to check on them. Every switch on my network is suffering from reboots (I have 69 switches).

Thanks
Glenn

There are three services using 57% of the memory... how's that work?

[sirhc]

So Stephen is working on this.

Now keep in mind 99% of users are not seeing this so there is something going on with these 5+/- people's network that the switch is not dealing with properly.

Stephen has some ideas on how to better deal with this and is as I said working on a fix that we "think" should help/fix it.

We believe it has to do with discovery packet storms.

Stephen can comment on when he plans to release a v1.5.2rcX to attempt to deal with this.

Now I am doing so many things but I think currently the way the switch core is configured it will pass discovery packets back to the Linux shell even if you have that type of Discovery turned off which he is changing so it will only send those packets associated with the desired discovery protocols enabled.

He is also changing the way the Discovery TAB works limiting the MAX number of entries, which will still be several thousands devices which in my opinion is WAY TO LARGE of a Layer 2 segment but hey who am I to judge ones network setup, everyone has their own way.

But it could be something totally differnt such as a bad device sending MILLIONS/BILLIONS of malformed broadcast packets or something along those lines?

[Stephen]

Like sirhc has said,

I have managed to recreate some of the behavior people have been reporting throughout the forum lately.
Memory increasing continuously until the switch reboots. What I've found does relate to the discovery process there is quite a bit of re-structuring in the background that needs to happen in order to protect the switches from this happening.


By the end this week or early next there will be a new rc out that you all can try.

[Banana Jack]

Great to hear you guys have decided to improve how Netonix switches respond to (discovery) packet storms. I know that large flat networks aren't great, but it's fantastic that you're taking a non-judgemental approach.

I'm almost certain that Netonix switches running current firmware (1.5.1) can be made to become slow/unstable by throwing a lot of packets around so I'm very happy to learn that there's work afoot to try to make them resilient to that. Maybe it's true that 99% of users are not seeing any problems, but when users do come across random reboots, it would be nice to have one less reason to worry about what may be causing them.

For the record, Ubiquiti airOS 8.5.8 is TERRIBLE for creating unwanted discovery packets, (once a minute whilst logged in to the GUI). 8.5.11 is no better as far as I can see. 8.5.12-RC is slightly better in that the discovery packets are only created by radios in router mode when you first log into the GUI. As far as I can tell, Ubiquiti have no plans to prevent unwanted discovery packets completely.

Zy at Ubiquiti has suggested to me that the problem could be mitigated by disabling CDP on all devices.

The relevant thread on the Ubiquiti Beta forum is here in case anyone hasn't seen it.

Glenn

[wmackay]

I am also seeing high memory usage and reboots on switches running 1.5.0. I see the same thing on graphs - memory usage increases for several weeks, then runs out, then the device reboots. My network includes many radios running Ubiquiti 8.5.8 and I have both the discovery tab and CDP enabled.

My switches running 1.4.9 are not affected, even with 8.5.8 and discovery. I have not tried firmware 1.5.1.

[sirhc]

I am also seeing high memory usage and reboots on switches running 1.5.0. I see the same thing on graphs - memory usage increases for several weeks, then runs out, then the device reboots. My network includes many radios running Ubiquiti 8.5.8 and I have both the discovery tab and CDP enabled.

My switches running 1.4.9 are not affected, even with 8.5.8 and discovery. I have not tried firmware 1.5.1.

Well read the WHOLE thread above and try and narrow it down by the suggestions provided.
Also coincidence often leads people to incorrect conclusions.
Each Layer 2 segment has differnt equipment and different traffic.

So to back up your hypothesis simply take a switch that us running v1.4.9 which you think is fine then upgrade to v1.5.0 or v1.5.1 or v1.5.2rcX and see if the mem issue starts there or not and let us know.

[Banana Jack]

For what it's worth, I found no improvement when I upgraded from 1.5.0 to 1.5.1. I had already upgraded from 1.4.9 when the dreaded airOS 8.5.8 came out so I don't know whether 1.4.9 would have been more resilient to the storms.

wmackay - I agree with sirhc that it would be interesting to see if you start to see problems if you upgrade from 1.4.9.

I tried setting Broadcast Storm Control on our core WS-26-400-AC switch to 2K but it didn't seem to have any effect. I was still getting packet per second counts way in excess of that when looking in Wireshark. However, I may be misunderstanding how the setting works or measuring it wrong.

The Netonix Loop Protection setting works well, but of course the downside is you lose connectivity for three minutes. Which is arguably worse than a small outage caused by a several-second packet storm, although maybe not worse than causing the Netonix memory to fill up and the switch to go slow or reboot. I have a spare WS-26-400-AC connected to our primary one and it has Loop Protection enabled. I use it purely to alert me if or when a packet storm happens (or if some numpty genuinely creates a network loop somehow).

[sirhc]

I would be more concerned to find what is causing the problem then having us MASK the issue.