Memory usage

[Ludvik]

Yes.

---
For example upgrade 1.5.0 -> 1.5.3rc from 15. july.

[lambert]

I just found this on one of our new 8 port switches. It was at 123.1 MB used. Firmware 1.5.2. It's been in use for 12 days.

I turned off all the discovery config options and it made no difference in memory usage.

Via top, I found that about twenty processes were at 62MB resident. The "discovery threa" was one of them, despite having turned off all discovery options. I -HUP'd it. All the processes that had been showing 62MB used went to 11MB. They are now at 13MB ten minutes later. I'm guessing they are all one monolithic process with multiple threads. Linux does threads differently than I'm used to on the BSDs. So, I suppose I have no idea which thread was actually consuming the extra RAM.

Used RAM shows to be 51MB now.
These are the threads which all showed 62. I wish I had grabbed the pre-HUP top output. The pre and post HUP used mem and process RES don't exactly add up. I'm guessing another process cleared at the same time. Without the pre-HUP top output, I cannot track it down.

Code: Select all

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2175 admin 20 0 19520 13m 1012 S 0.0 10.8 0:01.05 mstp_thread
2176 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.02 lacp_thread
2177 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.00 vtss_appl
2178 admin 20 0 19520 13m 1012 S 0.4 10.8 0:01.55 vtss_appl
2179 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.12 port thread
2180 admin 20 0 19520 13m 1012 S 1.5 10.8 0:07.84 status_thread
2156 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.27 vtss_appl
2157 admin -3 0 19520 13m 1012 S 0.0 10.8 0:00.01 vtss_appl
2158 admin 20 0 19520 13m 1012 S 1.3 10.8 0:01.77 RX thread
2160 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.00 event thread
2161 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.07 discovery threa
2164 admin -2 0 19520 13m 1012 S 0.0 10.8 0:00.00 ubnt discovery
2174 admin 20 0 19520 13m 1012 S 0.0 10.8 0:00.05 erps
 

It sounds like some progress has been made, but thought I would post this as another data point. Although the release notes say a leak was fixed in 1.5.2. I don't fully understand the Netonix release / version numbering process. To me, it looks like I'm running the latest 1.5.2 code. The switches came with 1.5.2 on them. This one is ec:13:b2:d1:e9:a0.

[lambert]

This is the other recently installed switch. It's in the same subnet. 95.7MB used. ec:13:b2:d1:eb:78

Code: Select all

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2560 admin -2 0 63340 56m 1016 S 0.0 45.4 0:02.84 ubnt discovery
2570 admin 20 0 63340 56m 1016 S 0.0 45.4 0:24.87 erps
2571 admin 20 0 63340 56m 1016 S 0.0 45.4 3:01.18 mstp_thread
2581 admin 20 0 63340 56m 1016 S 0.0 45.4 0:00.02 lacp_thread
2582 admin 20 0 63340 56m 1016 S 0.0 45.4 0:00.00 vtss_appl
2583 admin 20 0 63340 56m 1016 S 0.0 45.4 15:07.08 vtss_appl
2584 admin 20 0 63340 56m 1016 S 0.0 45.4 2:34.80 port thread
2585 admin 20 0 63340 56m 1016 S 1.5 45.4 150:23.63 status_thread
2555 admin 20 0 63340 56m 1016 S 0.0 45.4 0:02.36 vtss_appl
2556 admin -3 0 63340 56m 1016 S 0.0 45.4 0:00.00 vtss_appl
2557 admin 20 0 63340 56m 1016 S 0.0 45.4 68:31.24 RX thread
2558 admin 20 0 63340 56m 1016 S 0.0 45.4 0:00.02 event thread
2559 admin 20 0 63340 56m 1016 S 0.3 45.4 194:37.91 discovery threa


I'm not going to HUP anything yet.

[Ludvik]

Next version 1.5.3rc2 have patch that may be solving our memory problem. Is too soon for conclusion, but memory seems much better.
If you not using SNMP, turn off it.

[Ludvik]

Unfortunatelly ... no change. Version 1.5.3rc2-201907151558

SNMP log is small, this bug is solved. But memory no ...

[Stephen]

I see, really thought we had it that time. Well we'll keep at it.

[zRares]

So it seems that we're having the same issue, after like 60 days of up time our WS-12-250-AC switches usage memory gets to around 110 MB out of 128 MB.
Is there any fix without rebooting the Switch ?

[Stephen]

Have you tried upgrading to 1.5.3?
There might be a few things left to fix, but we had a major leap forward in that revision for this problem.

Speaking of which, Ludvick, how do your system's look right now? Is the memory increase still steady?

[Ludvik]

My newest version is 1.5.3rc2-201907151558, no significant change.

Last spike after week 31 is power outage.

[Stephen]

Ah I see, I thought it had improved more than that. Speaking of which, Ludvick do you have any SFP module's in your switches?
Another user mentioned to us that they believed they had a leak and described similar symptom's that we've been dealing with here. They seem to believe it is related to their SFP module's.