monitor: restarting vtss_appl

[socraticmethod]

Hey all, I've done some searching on the forums for the vtss_appl logs and understand it's a system process of some sort. After (finally) pushing all of my currently active switches to a syslog, I've discovered a half dozen that are regularly restarting vtss_appl and causing packet loss and spanning tree changes. See below for more info, but I've removed the switches in question, updated firmware, hard reset, changed config and re-deployed in completely different locations and they'll eventually begin restarting vtss_appl again.

Any idea what the cause of this might be? I have a large inventory of Netonix and as I've been bringing them online and monitoring, it seems like a decent handful are experiencing this issue.

Thanks!

Code: Select all

426826h20m17000|Sep 10 10:20:16 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h20m18000|Sep 10 10:20:17 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h20m19000|Sep 10 10:20:18 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h20m19000|Sep 10 10:20:18 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 3 to learning NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 3 to forwarding NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 2 to learning NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 2 to forwarding NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 1 to learning NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 1 to forwarding NTX-Catering-2
426826h20m24000|Sep 10 10:20:23 10.6.2.9/10.6.2.9 monitor: restarting vtss_appl NTX-VetTech1
426826h20m24000|Sep 10 10:20:16 10.6.2.8/10.6.2.8 STP: set port 1 to discarding NTX-Barn4
426826h20m24000|Sep 10 10:20:17 10.6.2.8/10.6.2.8 STP: set port 1 to learning NTX-Barn4
426826h20m24000|Sep 10 10:20:17 10.6.2.8/10.6.2.8 STP: set port 1 to forwarding NTX-Barn4
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 3 to learning NTX-VetTech1
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 3 to forwarding NTX-VetTech1
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 2 to learning NTX-VetTech1
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 2 to forwarding NTX-VetTech1
426826h20m49000|Sep 10 10:20:48 10.6.2.24/10.6.2.24 Port: link state changed to 'down' on port 10 NTX-Broadcast-Comp
426826h20m51000|Sep 10 10:20:50 10.6.2.24/10.6.2.24 Port: link state changed to 'up' (10M-H) on port 10 NTX-Broadcast-Comp
426826h21m13000|Sep 10 10:21:12 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h21m14000|Sep 10 10:21:13 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h21m14000|Sep 10 10:21:13 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h21m14000|Sep 10 10:21:13 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 3 to learning NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 3 to forwarding NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 2 to learning NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 2 to forwarding NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 1 to learning NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 1 to forwarding NTX-Catering-2
426826h21m31000|Sep 10 10:21:30 10.6.2.9/10.6.2.9 monitor: restarting vtss_appl NTX-VetTech1
426826h21m31000|Sep 10 10:21:23 10.6.2.8/10.6.2.8 STP: set port 1 to discarding NTX-Barn4
426826h21m31000|Sep 10 10:21:23 10.6.2.8/10.6.2.8 STP: set port 1 to learning NTX-Barn4
426826h21m31000|Sep 10 10:21:23 10.6.2.8/10.6.2.8 STP: set port 1 to forwarding NTX-Barn4
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 3 to learning NTX-VetTech1
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 3 to forwarding NTX-VetTech1
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 2 to learning NTX-VetTech1
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 2 to forwarding NTX-VetTech1
426826h22m19000|Sep 10 10:22:18 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h22m20000|Sep 10 10:22:19 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h22m20000|Sep 10 10:22:19 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h22m20000|Sep 10 10:22:19 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 3 to learning NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 3 to forwarding NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 2 to learning NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 2 to forwarding NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 1 to learning NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 1 to forwarding NTX-Catering-2
426826h22m26000|Sep 10 10:22:25 10.6.2.9/10.6.2.9 monitor: restarting vtss_appl NTX-VetTech1
426826h22m26000|Sep 10 10:22:18 10.6.2.8/10.6.2.8 STP: set port 1 to discarding NTX-Barn4
426826h22m26000|Sep 10 10:22:19 10.6.2.8/10.6.2.8 STP: set port 1 to learning NTX-Barn4
426826h22m26000|Sep 10 10:22:19 10.6.2.8/10.6.2.8 STP: set port 1 to forwarding NTX-Barn4
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 3 to learning NTX-VetTech1
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 3 to forwarding NTX-VetTech1
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 2 to learning NTX-VetTech1
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 2 to forwarding NTX-VetTech1
426826h23m14000|Sep 10 10:23:13 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h23m15000|Sep 10 10:23:14 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h23m16000|Sep 10 10:23:15 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h23m16000|Sep 10 10:23:15 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
 

[sirhc]

What version of firmware are you using?

[socraticmethod]

1.5.0, on board Rev F

[Stephen]

As implied above definitely make sure you're running the latest software, vtss_appl is responsible for communicating with the switch-core about protocol configurations. So when vtss_appl is first started it causes STP restart as well. There are a few configuration changes that can cause vtss_appl to restart, however, in your logs it says that the 'monitor' is what is restarting vtss_appl which implies it is crashing. I've seen this happen a few times, the most common reason is a broadcast storm. Depending on your network configuration, maybe you need to enable loop protection? Again, also make sure all your switches are running the latest software.

**EDIT**
You're response came in right before I posted this so I couldn't see that you're running 1.5.0, might want to try 1.5.1rc4 as well, some work was done on vtss_appl

[socraticmethod]

Following up on this -- it seems like passing multicast/IGMP is somehow related to these issues. See screenshot from my syslog where immediately after disabling IGMP snooping for an IPTV client off the switch, the restarting vtss_appl issue began where it had been previously never showed the error. There really isn't a lot of data (<50Mbps) crossing a single port at a given time so I'm not sure how to prevent this error from restarting RSTP an interrupting traffic without disabling STP altogether.

Any ideas?

[Stephen]

Is IGMP snooping disabled on all affected switches?

We might be looking at a new bug here.

If possible could send a screenshot of the following tabs:
Status,
Configuration,
STP,
VLANs,
QoS

From a switch that is exhibiting this behavior?

[SemperFi]

Is there an update on this error ? I just upgraded 7 switches from 1.4.7 to 1.5.0 and now I'm seeing this error repeating in the log, and occasional packet loss (1% - 2%) on all switches. I've tried disabling Spanning Tree and IGMP Snooping and the problem persists.

[Stephen]

Try upgrading to 1.5.2rc3.
There has been several improvements on igmp, lacp, mstp, etc that all could potentially be responsible.

Get 1.5.2rc3 here:
viewtopic.php?f=17&t=240

[SemperFi]

Done. Unfortunately the problem persists. Packet loss is better than 1.5.0 but still ~1% pinging the switch from a directly attached router. Dropped ping replies seem to match up to 'restarting' messages in the log. This particular switch is board Rev B

Code: Select all

 Jan 1 00:00:08 sysinit: killall: udhcpc: no process killed
Jan 1 00:00:09 netonix: 1.5.2rc3 on WS-24-400A
Jan 1 00:00:11 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Jan 1 00:00:11 kernel: switch: 'Luton26' board detected
Jan 1 00:00:14 kernel: vtss_port: Loaded port module on board Luton26, type 5
Jan 1 00:00:15 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Jan 1 00:00:15 system: Setting MAC address from flash configuration: EC:13:B3:01:04:AC
Jan 1 00:00:15 kernel: i2c /dev entries driver
Jan 1 00:00:15 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Jan 1 00:00:16 sysinit: Loading defaults
Jan 1 00:00:17 sysinit: Loading synflood protection
Jan 1 00:00:17 sysinit: Adding custom chains
Jan 1 00:00:17 sysinit: Loading zones
Jan 1 00:00:18 sysinit: Loading forwarding
Jan 1 00:00:18 sysinit: Loading redirects
Jan 1 00:00:18 sysinit: Loading rules
Jan 1 00:00:18 sysinit: sh: bad number
Jan 1 00:00:18 sysinit: Loading includes
Jan 1 00:00:18 admin: adding lan (eth0) to firewall zone lan
Jan 1 00:00:26 kernel: eth0: no IPv6 routers present
Dec 31 19:00:31 syslogd exiting
Dec 31 19:00:32 syslogd started: BusyBox v1.19.4
Dec 31 19:00:32 syslogd exiting
Dec 31 19:00:33 syslogd started: BusyBox v1.19.4
Dec 31 19:00:34 admin: removing lan (eth0) from firewall zone lan
Dec 31 19:00:38 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:48 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:49 sysinit: killall: telnetd: no process killed
Dec 31 19:00:49 sysinit: killall: ntpclient: no process killed
Dec 31 19:00:49 sysinit: killall: ntpclient: no process killed
Dec 31 19:00:49 system: starting ntpclient
Feb 13 05:42:06 sysinit: 1969-12-31 19:00:50: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Feb 13 05:42:06 sysinit: 1969-12-31 19:00:50: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Feb 13 05:42:06 sysinit: 1969-12-31 19:00:50: (log.c.216) server started
Feb 13 05:42:08 dropbear[1403]: Running in background
Feb 13 05:42:12 switch[1432]: Detected warm boot
Feb 13 05:42:34 monitor: restarting vtss_appl
Feb 13 05:43:30 monitor: restarting vtss_appl
Feb 13 05:44:37 monitor: restarting vtss_appl

Copyright 2014-2018 Netonix


[Stephen]

Can you post your configuration tab, status tab, and ports tab?