Memory usage

[Stephen]

Ludvik,

When you get a chance, try out wispswitch-1.5.2rc6
There have been some big changes in the way Discovery is handled and from the testing we've done here it looks like the memory issue has been resolved. But we really won't know for certain without the reported edge cases using this release. It should also be more efficient on your CPU if you have many discoverable devices on the network.

Thank you.

[Ludvik]

Unfortunately the upgrade did not help. The graph looks the same before and after.

[Stephen]

It is inevitable that there will be some increase in memory when discovery is active because discoverable device details are stored in memory and dumped to a file at regular intervals. So some increase is not avoidable. However, in my own tests, I had >1000 emulated discoverable devices and the memory went to about 65-70MB and stayed there, which was acceptable. I can see though that at least in one instance on your graph the memory reached max which is of course not acceptable - however, to be certain, on your second image on the first (and only) memory increase to 0 free Kb, was that before or after the upgrade?

[Ludvik]

Upgrade from 1.5.0 to 1.5.2rc6 is vertical line on monday (midnight).

The vertical lines at week 10 is two steps - disabled discovery and then restarting by watchdog.

In discovery tab is 36 devices.

[Stephen]

Oh I see, so the first image when the memory drops, it is because of the upgrade to the rc6 release.

On the second image it was still on 1.5.0

So is this an accurate interpretation?

Memory Usage

If so, and I am reading your graph correctly, then it means that the current memory usage is around 50-55MB?

If that is the case then it is still within tolerance. There are many different service's running that cause memory to fluctuate. If it goes above 80MB (or from your NMS point of view, if the memory free goes to less than 48 MB ) Then we still might be looking at a leak.

A few of the live switches we have been testing on have been resting at about 52-58MB for over a week and have not fluctuated after getting to that point.

One other thing, 1.5.2 is out, you can update to that if you want to get a few other feature's that have been worked on and continue the test from there if you'd like.
However, the memory management architecture for discovery hasn't changed from rc6 to the release so if you prefer to just let it run as is for this test then it is OK.
(But for all the latest fixes/features for everything else I would recommend going to 1.5.2)

[Ludvik]

It's same graphs of free RAM, difference is time axis.

Current memory usage is relatively OK ... but grow in time still. This is not one case. That's how all my netonix switches look like. The difference is in the curve angle. But it always points to zero.

Another example:

How exactly do you test discovery? MAC:IP one to one? In my network usually UBNT is in bridge mode and behind it is either a tplink router or a mikrotik - ie discovery twice with the same MAC.

[Stephen]

I do not have access to a large network in my development environment so I created a tool that lets me emulate lldp, cdp, and ubnt discovery packets.

Once I had verified within my own environment improvements. Chris then takes the firmware and tests the changes on a live network that he uses at his WISP. It has about 20 discoverable devices running on it.

Our tests showed that memory starts at around 46MB-48MB, then it slowly climbs just a bit before leveling off at around 50-55MB, there is evidence from other posts on this issue that this might be related to STP as well. However, on 1.5.0 it would have continued growing past the 50-60 range.

I have no doubt this is not isolated to one switch, but what I believe will happen is that it will stop growing now that it is near the cutoff point I found in testing. But I had to watch the live switches for over a week before I was satisfied that the growth had stopped with these updates

The software uses the mac address to identify new devices and interprets their type based on the packet structure and of course the payload for the values that make their way into the UI. The original leak was occurring when the switch received a response/broadcast from a device it had already discovered. It tried to remove the old entry and update it with the new values and unfortunately it was not clearing the information correctly so the memory was still allocated but could no longer be referenced. This update fixes that along with some other enhancements that improve the search time when checking if a device already exists from O(n) to O(log(n)) - hence the CPU usage improvements.

At this point, let's see where it's at after a week. If it goes above 75-80MB I will begin testing it again to see if I missed anything.

[sirhc]

Well I then rolled out to a tower which has a couple hundred devices, all be it a routed tower with VLANs for each AP.

But to be honest I never had any of this memory issue with older firmware which you can see in this post: viewtopic.php?f=17&t=4788&start=10#p28147

[Ludvik]

You are lucky man ... I'm not ...

Yes - memory usage grows sometimes slowly and it is no problem after years. But sometimes is problem after few months - switch rebooted by watchdog. And your software problem is my problem, users hate me.

[sirhc]

Maybe you will need to make a way for Stephen to access it remotely then notify him when memory is growing so he can look at it.