Switch reboots - High mem usage

[sirhc]

EXCELLENT Glenn, thank you.

Diagnosing and trying to figure out what is the cause helps us fix something.

Sometimes I feel like Jordy working with the Pakleds

download.jpg (19.26 KiB) Viewed 5674 times

[Banana Jack]

Happy to try to help if I can sirhc! You help us guys, we help you guys, seems to make sense. Netonix products are a fantastic solution for a relatively small industry so we all have to do what we can. And it's kinda fun in a weird sorta way. The ultimate prize for me will be when my customers stop complaining and I can have a day off ;)

I'll keep you updated about any findings.

Thanks
Glenn

[jpaine619]

Just came across this thread.

I've been having the same issue myself. _All_ of my switches show mem usage around 112-118MB. All are running 1.5.0 firmware. I've chosen 2 for troubleshooting purposes and disabled discovery on them.. I've disabled the Tab, LLDP, Cisco, and Ubiquiti options and then rebooted. Generally I was hitting > 100MB of mem usage within a few days, so I'll let this run a week or so and then report back my findings.

According to TOP, the discovery process was using a rather large amount of resources. So we'll see what happens.

*Spontaneous reboots were not frequent across all devices, but I've had a couple that have been... problematic.. They'd reboot and then continue on along, so it hasn't been a HUGE issue. But I'd like to see the old days of +200 days uptime :)

[sirhc]

I am not experiencing this memory issue but all of my towers are routed and each AP is a VLAN to a router port so My layer 2 segments are only 1 AP or port.

You can see this here: viewtopic.php?f=30&t=452#p2961

My guess is you have LARGE flat layer 2 segments with many many defvices and the UBNT Dicovery Tab is your issue.

Or possibly SNMP polling to often or multiple UI/CLI logins.

[jpaine619]

I do not have a large L2 network.. Less than 150 devices. I have slowly been sub-netting the network, but at no point have I had larger than a class C on an particular segment.

No SNMP polling AT ALL and I am the only person who logs in to the switches. I rarely hit the log-out button, I just close the browser window, so I'm hoping that there is some bit of sanity in the switch OS and old logins are timed out after a period.

Towers aren't subnetted, per se. But individual sectors are. My switches are on a 10.x.x.x subnet, along with backhauls. The sectors themselves are 10.x.x.x on the wan side and 192.x on the customer side.

[Kingsley]

Eric, Sirhc, I have provided plenty of detail about my issue, I still have switches rebooting - Discovery has been enabled before but disabled now... Is it normal for the mem usage to increase over time, even with discovery disabled?

Here is another, rebooted tonight. 1.4.9

[Kingsley]

Why does my switch still show discovery when it's disabled?

[sirhc]

Eric, Sirhc, I have provided plenty of detail about my issue, I still have switches rebooting - Discovery has been enabled before but disabled now... Is it normal for the mem usage to increase over time, even with discovery disabled?

Here is another, rebooted tonight. 1.4.9

I have no idea why as you can see I have switches with v1.4.9 that are up almost a year.
This is not a widespread issue so I am guessing it is something your doing or unique to your configuration.
You should be on v1.5.0 or v1.5.1rc7, I am on v1.4.9 on some switches as I have versions running from v1.4.9 to v1.5.1rc7 as I have to for testing.

I would disable anything you do not NEED and then reboot and see if it stops then re-enable things slowly looking for a clue.

Also note that many of my switches are the original WS-24-400A that are 4+ years old....GOOD GROUNDING
I do have at least 1 of most models in service as well for testing.

[Banana Jack]

TL;DR: In my case, it's the latest Ubiquiti firmware to blame (8.5.8).

In more detail: I had a bit of a breakthrough today with this. I'm convinced that the Netonix watchdog reboots are a symptom of the problem and not the problem itself. I upgraded all our switches to 1.5.1rc7 and disabled Discovery Tab and Cisco Discovery Protocol on all of them. I'm sorry I can't immediately say whether this has fixed the problem of them rebooting because I'd need to wait 2-3 days to see what happens. However, I do have a single switch with Loop Protection enabled, and it has continued to detect 'loops' (or storms as I would call them).

I've found that if you access the web interface of any Ubiquiti radio running the latest firmware (8.5.8) then it will send a discovery broadcast packet roughly every minute whilst you remain logged in, thereby producing a 'spike of doom' when every radio on the network tries to respond at the same time. On a big flat layer 2 network this causes traffic / VOIP / connection dropouts to affect random customers… those using PPPoE on TP-Link routers seem especially susceptible to their connections being dropped and then it takes 3 minutes for them to re-establish.

The problem doesn't occur with 8.5.7 firmware. I proved it by using Wireshark to sniff all UDP traffic (capture filter: udp) and then applying a display filter: udp.port==10001 and (frame.len<100). Screenshot attached.

It's taken me about five weeks on and off to figure this out. I'm reluctant to declare this finding a 'win' because of many previous red herrings, including a faulty SFP module and a faulty media converter, (I now believe neither were to blame). But anyway I will let Ubiquiti know and ask for their comments.

For the record, another mitigation measure would be to not have a large flat network but rather a routed one. But that's sometimes easier to say than do.

Thanks to sirhc and Eric for all their support with this, which fundamentally isn't a Netonix problem anyway.

Glenn

** EDIT 5 Nov 2018: Four days later and I can confirm that my Netonix reboots are fully eliminated now that I've downgraded all Ubiquiti radios from 8.5.8 to 8.5.7. Netonix memory usage no longer creeps up. I reported the issue to UBNT via their airMAX AC Beta forum (here if you have access) and their employee Josh responded 'Interesting catch' so hopefully they will investigate/reproduce/fix.

[Kingsley]

Hi everyone.

This is still an ongoing issue for us and we are getting pretty frustrated with no resolution in site. Here is the latest one from today.

This unit was updated to 1.5.0 at the start of September and as you can see, memory usage was fine up until then and then slowly increased until finally rebooting today. Are you still going to argue there is no memory leak or something similar?

Regards