Memory usage

[jpaine619]

I haven't tried turning off loop protection, so I cannot comment on how effective that is in reducing memory usage.

But, I did turn off Ubiquiti, LLDP, and the Discovery Tab options (I left Cisco enabled since I use /usr/bin/cdpr on my laptop) and saw a HUGE reduction in memory usage. On some of my switches Memory usage was at 120MB, now it averages about 47MB (roughly 1/3).

[Ludvik]

Disabling loopback-detection doesn't help. Memory usage growing by about 1MB daily.

Now I try disable LLDP (with other discovery options enabled), and ... after 28 hours memory usage no growed. Will I see later.

[Ludvik]

Disabled LLDP slightly help ... but memory usage still grow.

On other site I test 1.5.1.rc11 (lldp disabled). It is worse. 4MB of usage by day. It is new site, no users (very few traffic), only seven UBNT Rocket Prism AC gen2 (v 8.5.11) and four Siklu EH-710TX (only one with data) and RB450Gx4 router.

I must revert firmware of all my switches to 1.4.9. Newer version is unusable, unstable.

[sirhc]

Disabled LLDP slightly help ... but memory usage still grow.

On other site I test 1.5.1.rc11 (lldp disabled). It is worse. 4MB of usage by day. It is new site, no users (very few traffic), only seven UBNT Rocket Prism AC gen2 (v 8.5.11) and four Siklu EH-710TX (only one with data) and RB450Gx4 router.

I must revert firmware of all my switches to 1.4.9. Newer version is unusable, unstable.

I am running v1.5.0 and v1.5.1rc11 and I am not seeing this memory growth issue.

There is no way we can address your issue without knowing the cause.

You need to make a way we can access one of these switches or put a switch on your network, set it up the same but does nothing real. Then verify you are seeing this memory growth and make that switch accessible to us.

[Ludvik]

I can't help you. It is misterious. Memory usage grows on every switch, slowly and rapidly ... I don't see coherence.


This is 1.5.0 and many users on it (49 lines in discovery) - UBNT radios and mostly tplink's routers (sometimes mikrotik). 47 days uptime.
After all discovery disabled. Memory usage has dropped significantly. But still grow, slowly, but grow.



----


This is 1.5.1rc11 and only one client ... (plus UBNT access points - 12 lines in discovery). 12 days uptime. Reboot is visible (not by watchdog - site was off by electric power).


This is 1.4.6 (uptime 1 year and 102 days). A little number clients - 10 in discovery.


This is 1.4.9, 22 lines in discovery. 158 days uptime.

----

Sorry, I can't help you more. I am not developer. I don't have lab for this testing. And testing need big time.
However, memory leakage is proven.

I do not use STP, ERPS, LAG. No ping watchdog nor port bounce. IPv6 disabled.

If it is slow, uptime may be years without problems (reboot is solved by electric company aproimatelly once in two years). But time to time is memory leak too fast, switch is unstable. And it is shame.

Code: Select all

 

[sirhc]

No, memory leak is not proven.

60,000 switches in service and only 5 people have reported an issue.
So there is some condition that causes this and we have no idea what it is.

Post up screen grabs of all your TAB (complete TAB)

[Ludvik]

You don't see graphs? It is not proof? What is proof?
You have tens thousand and no problem - i have only tens your switches and memory leak is on every.

The Ethernet switch must be stable in every situation. If not, it's a developer's fault, not an owner.

As I tell you - if is leak slow and network monitoring worse, admin know nothing.

If you do not want to do a complete code review and look for a bug, create a internal watchdog. If the free memory is less than about 10 (or 20) MB, restart the loopback-detection and all discovery processes (respect config). It can help for now.

https://uloz.to/!5vYAYJVhhK7A/switch-conf-nfg-7z
password is SirhC

It is sanitized config (no credentials, no radius). Typical config in our network (same position as 10.106.48.94).

Code: Select all

 #top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2535 admin 20 0 49416 42m 1004 S 0.3 34.4 92:48.45 port thread
2523 admin 20 0 49416 42m 1004 S 0.0 34.4 0:07.12 vtss_appl
2524 admin -3 0 49416 42m 1004 S 0.0 34.4 0:00.01 vtss_appl
2525 admin 20 0 49416 42m 1004 S 0.0 34.4 44:58.63 RX thread
2526 admin 20 0 49416 42m 1004 S 0.0 34.4 0:00.02 event thread
2527 admin 20 0 49416 42m 1004 S 0.0 34.4 472:41.82 discovery threa
2528 admin -2 0 49416 42m 1004 S 0.0 34.4 0:00.05 ubnt discovery
2529 admin 20 0 49416 42m 1004 S 0.0 34.4 0:54.04 erps
2530 admin 20 0 49416 42m 1004 S 0.0 34.4 0:10.48 mstp_thread
2531 admin 20 0 49416 42m 1004 S 0.0 34.4 0:00.02 lacp_thread
2532 admin 20 0 49416 42m 1004 S 0.8 34.4 646:27.12 loop_protection
2533 admin 20 0 49416 42m 1004 S 0.0 34.4 0:00.00 vtss_appl
2534 admin 20 0 49416 42m 1004 S 0.0 34.4 75:30.68 vtss_appl
2536 admin 20 0 49416 42m 1004 S 2.9 34.4 1566:17 status_thread
 812 admin 20 0 39292 3196 2456 S 0.0 2.5 1:04.27 php-fcgi
 797 admin 20 0 8256 2928 1104 S 0.0 2.3 1:08.11 lighttpd
 795 admin 20 0 7796 2592 1116 S 0.0 2.1 18:58.50 lighttpd
 796 admin 20 0 7216 2188 1104 S 2.1 1.7 9:12.50 lighttpd
 813 admin 20 0 39292 1944 1324 S 0.0 1.5 0:00.34 php-fcgi
 788 admin 20 0 39144 1624 1048 S 0.0 1.3 0:00.69 php-fcgi
 786 admin 20 0 39144 1616 1040 S 0.0 1.3 0:00.72 php-fcgi
 787 admin 20 0 39144 1612 1040 S 0.0 1.3 0:00.77 php-fcgi
1055 admin 20 0 5544 1560 688 S 0.0 1.2 3:13.98 snmpd
 824 admin 20 0 3360 1164 580 S 10.0 0.9 18155:31 switch
 831 admin 20 0 3360 1164 580 S 0.0 0.9 40:41.72 watchdog thread
 833 admin 20 0 3360 1164 580 S 0.0 0.9 152:23.13 poe thread
 830 admin RT 0 3360 1164 580 S 0.0 0.9 0:00.00 switch
 832 admin RT 0 3360 1164 580 S 2.6 0.9 6085:49 i2c thread
 811 admin 20 0 39144 776 196 S 0.0 0.6 0:00.00 php-fcgi
3463 admin 20 0 3152 748 448 S 0.0 0.6 0:00.06 switch
 785 admin 20 0 6340 704 448 S 0.0 0.6 0:00.02 lighttpd
3487 admin 20 0 2128 648 452 R 1.3 0.5 0:00.55 top
3429 admin 20 0 2060 608 448 S 0.8 0.5 0:02.48 dropbear
3465 admin 20 0 2688 556 460 S 0.0 0.4 0:00.01 ash
 840 admin 20 0 2696 548 444 S 0.3 0.4 249:34.47 sh
3455 admin 20 0 2688 528 432 S 0.0 0.4 0:00.03 ash
 1 admin 20 0 2692 488 392 S 0.0 0.4 0:01.21 init
3464 admin 20 0 2688 464 376 S 0.0 0.4 0:00.01 sh
2217 nobody 20 0 2268 428 344 S 0.0 0.3 0:00.09 shellinaboxd
3017 admin 20 0 2680 412 332 S 0.0 0.3 0:00.00 syslogd
 498 admin 20 0 2688 396 304 S 0.0 0.3 6:35.04 watchdog
 292 admin 20 0 2680 388 300 S 0.0 0.3 0:00.03 klogd
 793 admin 20 0 1524 372 244 S 0.0 0.3 1:46.41 stats.fcgi
3613 admin 20 0 2680 356 276 S 0.3 0.3 0:00.01 sleep
 415 admin 20 0 1988 344 264 S 0.0 0.3 0:00.15 dropbear
2223 nobody 20 0 2268 324 240 S 0.0 0.3 0:00.00 shellinaboxd
 282 admin 20 0 2692 292 196 S 0.0 0.2 0:00.00 init
 792 admin 20 0 1452 244 192 S 0.0 0.2 0:00.00 stats.fcgi
 304 admin 20 0 1132 244 192 S 0.0 0.2 0:00.04 hotplug2
 791 admin 20 0 1452 244 192 S 0.0 0.2 0:00.04 stats.fcgi
 48 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
 58 admin 15 -5 0 0 0 S 0.0 0.0 0:02.20 mtdblockd
 61 admin 15 -5 0 0 0 S 0.0 0.0 0:07.35 spi-vcoreiii.0
 269 admin 30 10 0 0 0 S 0.0 0.0 0:00.15 jffs2_gcd_mtd6
 2 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd
 3 admin 15 -5 0 0 0 S 0.0 0.0 0:00.23 ksoftirqd/0
 4 admin RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
 5 admin 15 -5 0 0 0 S 0.0 0.0 65:10.29 events/0
 6 admin 15 -5 0 0 0 S 0.0 0.0 0:00.03 khelper
 21 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
 45 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
 46 admin 20 0 0 0 0 S 0.0 0.0 0:01.13 pdflush

[Stephen]

We have had a few interesting discoveries recently about the supported discovery protocols.

In the config you provided, it is still enabled.

There is work being done to improve it's behavior and prevent memory growth from becoming problematic and there will be a new release that reflects that soon.

However, in the meantime I recommend turning it OFF on all switches you are concerned about.

[Ludvik]

In sended config - yes, discovery is enabled.

But in this switch is disabled all discovery protocols plus tab (and loopback-detection too). At moment you see spike ... (uptime is 60 days). Problem still persist. But memory usage grow slowly.

[sirhc]

Not saying I do not believe you but I am not able to recreate this on any of my WISP's Netonix switches.

What we need to find is what is different about your network than mine and most other users as obviously this is not widespread but limited to about 5 people.

We have found one in the past that turned out to be a bad piece of equipment (an SFP module) that caused his issue.

Also not saying that we need to make the unit deal with adverse and bad conditions better but they are hard to relictae.

Maybe you can work with Stephen to give him access to one of these switches?