v1.5.15rc1 Bug Reports and Comments

[summit-nathan]

We are also seeing the sfp ports drop every time any config change is applied. Also seeing slow increase in memory usage over time. We'll add some details once we have a bit more runtime.

[jpmoller]

Actually, you will note the far right ramp starts with higher memory available then the previous two ramps.....memory usage will grow with use, but new code should limit that...would be good to see it further on..

but in the meantime can you share below:

- switch model
- your switch configuration
- which oids you are querying on snmp
- share /var/log/messages and your snmp log.

Extra starting memory is cool, but I've already lost 12% of memory in the first 7 days of deployment. If the trend continues, the extra memory has gained us a few days, at most.

To answer your questions:

- switch model
WS-6-MINI

- your switch configuration

ws-6-mini-config.txt

WS-6-MINI-config

(15.81 KiB) Downloaded 1162 times

- which oids you are querying on snmp

walk:
- 1.3.6.1.4.1.46242.2
- 1.3.6.1.4.1.46242.3
- 1.3.6.1.4.1.46242.4
get:
- 1.3.6.1.2.1.1.1.0
- 1.3.6.1.4.1.46242.1.0
- 1.3.6.1.4.1.46242.6.0

- share /var/log/messages and your snmp log

Code: Select all

/www# cat /var/log/messages
Jan 1 00:00:09 kernel: All bugs added by David S. Miller <davem@redhat.com>
Jan 1 00:00:09 kernel: VFS: Mounted root (squashfs filesystem) readonly.
Jan 1 00:00:09 kernel: Freeing unused kernel memory: 116k freed
Jan 1 00:00:09 kernel: Please be patient, while OpenWrt loads ...
Jan 1 00:00:09 kernel: Algorithmics/MIPS FPU Emulator v1.5
Jan 1 00:00:09 kernel: mini_fo: using base directory: /
Jan 1 00:00:09 kernel: mini_fo: using storage directory: /jffs
Jan 1 00:00:09 sysinit: killall: udhcpc: no process killed
Jan 1 00:00:12 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Jan 1 00:00:12 kernel: switch: 'Luton26' board detected
Jan 1 00:00:13 kernel: vtss_port: Loaded port module on board Luton26, type 5
Jan 1 00:00:14 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Jan 1 00:00:14 system: Setting MAC address from flash configuration: EC:13:B2:92:22:3A
Jan 1 00:00:14 kernel: i2c /dev entries driver
Jan 1 00:00:14 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Jan 1 00:00:15 sysinit: Loading defaults
Jan 1 00:00:16 sysinit: Loading synflood protection
Jan 1 00:00:16 sysinit: Adding custom chains
Jan 1 00:00:16 sysinit: Loading zones
Jan 1 00:00:17 sysinit: Loading forwarding
Jan 1 00:00:17 sysinit: Loading redirects
Jan 1 00:00:17 sysinit: Loading rules
Jan 1 00:00:17 sysinit: sh: bad number
Jan 1 00:00:17 sysinit: Loading includes
Jan 1 00:00:17 root: adding lan (eth0) to firewall zone lan
Jan 1 00:00:18 dropbear[819]: Running in background
Jan 1 00:00:19 netonix: 1.5.15rc1 on WS-6-MINI
Jan 1 00:00:22 UI: web config OK
Jan 1 00:00:25 kernel: eth0: no IPv6 routers present
Jan 1 13:00:35 syslogd exiting
Jan 1 13:00:36 syslogd started: BusyBox v1.19.4
Jan 1 13:00:36 syslogd exiting
Jan 1 13:00:37 syslogd started: BusyBox v1.19.4
Jan 1 13:00:38 root: removing lan (eth0.100) from firewall zone lan
Jan 1 13:00:42 root: adding lan (eth0.100) to firewall zone lan
Jan 1 13:00:43 root: removing lan (eth0.100) from firewall zone lan
Jan 1 13:00:46 root: adding lan (eth0.100) to firewall zone lan
Jan 1 13:00:51 root: adding lan (eth0.100) to firewall zone lan
Jan 1 13:00:52 dropbear[1739]: Failed listening on '22': Error listening: Address already in use
Jan 1 13:00:52 STP: msti 0 set port 1 to discarding
Jan 1 13:00:52 dropbear[1748]: Running in background
Jan 1 13:00:53 STP: msti 0 set port 2 to discarding
Jan 1 13:00:53 sysinit: killall: telnetd: no process killed
Jan 1 13:00:53 STP: msti 0 set port 3 to discarding
Jan 1 13:00:53 sysinit: killall: ntpclient: no process killed
Jan 1 13:00:53 sysinit: killall: ntpclient: no process killed
Jan 1 13:00:53 system: starting ntpclient
Jan 1 13:00:55 sysinit: 1970-01-01 13:00:54: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Jan 1 13:00:55 sysinit: 1970-01-01 13:00:54: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Jan 1 13:00:55 sysinit: 1970-01-01 13:00:54: (log.c.216) server started
Jan 1 13:00:56 STP: msti 0 set port 3 to learning
Jan 1 13:00:56 STP: msti 0 set port 3 to forwarding
Jan 1 13:00:56 STP: msti 0 set port 2 to learning
Jan 1 13:00:56 STP: msti 0 set port 2 to forwarding
Jan 1 13:00:57 STP: msti 0 set port 1 to learning
Jan 1 13:00:57 kernel: eth0.100: no IPv6 routers present
Jan 1 13:00:57 STP: msti 0 set port 1 to forwarding
Jan 1 13:00:59 switch[1811]: temp sensor version 3
Jan 1 13:00:59 switch[1814]: Detected warm boot
Oct 25 11:11:39 system: time set by NTP server
Oct 30 08:50:11 Port: link state changed to 'down' on port 2
Oct 30 08:50:11 STP: msti 0 set port 2 to discarding
Oct 30 08:50:14 Port: link state changed to 'up' (100M-F) on port 2
Oct 30 08:50:14 STP: msti 0 set port 2 to discarding
Oct 30 08:50:16 STP: msti 0 set port 2 to learning
Oct 30 08:50:16 STP: msti 0 set port 2 to forwarding
Oct 30 08:50:38 Port: link state changed to 'down' on port 2
Oct 30 08:50:38 STP: msti 0 set port 2 to discarding
Oct 30 08:50:42 Port: link state changed to 'up' (1G) on port 2
Oct 30 08:50:42 STP: msti 0 set port 2 to discarding
Oct 30 08:50:45 STP: msti 0 set port 2 to learning
Oct 30 08:50:45 STP: msti 0 set port 2 to forwarding
Oct 30 09:00:40 Port: link state changed to 'down' on port 2
Oct 30 09:00:40 STP: msti 0 set port 2 to discarding
Oct 30 09:00:42 Port: link state changed to 'up' (100M-F) on port 2
Oct 30 09:00:42 STP: msti 0 set port 2 to discarding
Oct 30 09:00:43 switch[2443]: !unexpected link change on port 2 (ap-a.rofl.lw.rur) from 1G to 100M-F
Oct 30 09:00:45 STP: msti 0 set port 2 to learning
Oct 30 09:00:45 STP: msti 0 set port 2 to forwarding
Oct 30 09:01:07 Port: link state changed to 'down' on port 2
Oct 30 09:01:07 STP: msti 0 set port 2 to discarding
Oct 30 09:01:11 Port: link state changed to 'up' (1G) on port 2
Oct 30 09:01:11 STP: msti 0 set port 2 to discarding
Oct 30 09:01:14 STP: msti 0 set port 2 to learning
Oct 30 09:01:14 STP: msti 0 set port 2 to forwarding
Nov 1 15:54:55 UI: Configuration backup by root (::ffff:10.1.96.20)
Nov 1 15:58:11 dropbear[3292]: Pubkey auth succeeded for 'root' with key sha1!! 1f:2d:9e:a8:14:fd:ff:7a:1b:22:0a:a6:63:9f:c6:a3:1d:62:ff:44 from 10.1.96.20:61417
Nov 1 16:02:19 dropbear[3814]: Pubkey auth succeeded for 'root' with key sha1!! 1f:2d:9e:a8:14:fd:ff:7a:1b:22:0a:a6:63:9f:c6:a3:1d:62:ff:44 from 10.1.96.20:61501
Nov 1 16:04:36 dropbear[313]: Pubkey auth succeeded for 'root' with key sha1!! 1f:2d:9e:a8:14:fd:ff:7a:1b:22:0a:a6:63:9f:c6:a3:1d:62:ff:44 from 10.1.96.20:61519


snmp logs:

Most of snmpd.log contents is "No MemShared line in /proc/meminfo"
Here is the top of the file before that entry gets spammed

Code: Select all

 
head -n 100 /var/log/snmpd.log
Cannot find module (IP-MIB): At line 0 in (none)
Cannot find module (IF-MIB): At line 0 in (none)
Cannot find module (TCP-MIB): At line 0 in (none)
Cannot find module (UDP-MIB): At line 0 in (none)
Cannot find module (HOST-RESOURCES-MIB): At line 0 in (none)
Cannot find module (SNMPv2-MIB): At line 0 in (none)
Cannot find module (SNMPv2-SMI): At line 0 in (none)
Cannot find module (NOTIFICATION-LOG-MIB): At line 0 in (none)
Cannot find module (SNMP-VIEW-BASED-ACM-MIB): At line 0 in (none)
Cannot find module (SNMP-COMMUNITY-MIB): At line 0 in (none)
Cannot find module (SNMP-FRAMEWORK-MIB): At line 0 in (none)
Cannot find module (SNMP-MPD-MIB): At line 0 in (none)
Cannot find module (SNMP-USER-BASED-SM-MIB): At line 0 in (none)
Cannot find module (TUNNEL-MIB): At line 0 in (none)
Cannot find module (UCD-DLMOD-MIB): At line 0 in (none)
Cannot find module (IPV6-ICMP-MIB): At line 0 in (none)
Cannot find module (IPV6-MIB): At line 0 in (none)
Cannot find module (IPV6-TCP-MIB): At line 0 in (none)
Cannot find module (IPV6-UDP-MIB): At line 0 in (none)
NET-SNMP version 5.1.2
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo
No MemShared line in /proc/meminfo


[Dave]

thanks..will send info to developer

[jpmoller]

thanks..will send info to developer

Hi Dave,

I've actually got some other OIDs we also scrape. Sorry for missing that the first time around

get:
- 1.3.6.1.2.1.1.3.0
walk:
- 1.3.6.1.2.1.2
- 1.3.6.1.2.1.31.1.1
- 1.3.6.1.4.1.2021.4
- 1.3.6.1.4.1.2021.11

[Dave]

no problem..info was very helpful, new RC coming soon

[jpmoller]

This seems like it might be the same issue?
https://forum.netonix.com/viewtopic.php?f=17&t=5384

I can see the same issue happening on our 1.5.14 hosts. I just checked a switch that had 6MB of memory left and the entire snmp.log file was 65MB of "No MemShared line in /proc/meminfo"

[Dave]

probably. This issue has been resolved, developer is testing before releasing...

[summit-nathan]

no problem..info was very helpful, new RC coming soon

Is the new RC expected to resolve the issues with the SFP ports losing link any time config changes are applied?

If not please let us know if you need additional info to track down this issue.

[sirhc]

no problem..info was very helpful, new RC coming soon

Is the new RC expected to resolve the issues with the SFP ports losing link any time config changes are applied?

If not please let us know if you need additional info to track down this issue.

I don't think so. I disused this with the team and we want to investigate this first as I do not see this in my deployments.

After SNMP / memory leak is fixed we will lab it. At that time we may ask some questions and or request configs.

[sirhc]

Thread closed - rc2 released