Netonix Forums

Hey all, I've done some searching on the forums for the vtss_appl logs and understand it's a system process of some sort. After (finally) pushing all of my currently active switches to a syslog, I've discovered a half dozen that are regularly restarting vtss_appl and causing packet loss and spanning tree changes. See below for more info, but I've removed the switches in question, updated firmware, hard reset, changed config and re-deployed in completely different locations and they'll eventually begin restarting vtss_appl again.

Any idea what the cause of this might be? I have a large inventory of Netonix and as I've been bringing them online and monitoring, it seems like a decent handful are experiencing this issue.

Thanks!

Code: Select all

426826h20m17000|Sep 10 10:20:16 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h20m18000|Sep 10 10:20:17 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h20m19000|Sep 10 10:20:18 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h20m19000|Sep 10 10:20:18 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 3 to learning NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 3 to forwarding NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 2 to learning NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 2 to forwarding NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 1 to learning NTX-Catering-2
426826h20m21000|Sep 10 10:20:20 10.6.2.15/10.6.2.15 STP: set port 1 to forwarding NTX-Catering-2
426826h20m24000|Sep 10 10:20:23 10.6.2.9/10.6.2.9 monitor: restarting vtss_appl NTX-VetTech1
426826h20m24000|Sep 10 10:20:16 10.6.2.8/10.6.2.8 STP: set port 1 to discarding NTX-Barn4
426826h20m24000|Sep 10 10:20:17 10.6.2.8/10.6.2.8 STP: set port 1 to learning NTX-Barn4
426826h20m24000|Sep 10 10:20:17 10.6.2.8/10.6.2.8 STP: set port 1 to forwarding NTX-Barn4
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 3 to learning NTX-VetTech1
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 3 to forwarding NTX-VetTech1
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 2 to learning NTX-VetTech1
426826h20m28000|Sep 10 10:20:27 10.6.2.9/10.6.2.9 STP: set port 2 to forwarding NTX-VetTech1
426826h20m49000|Sep 10 10:20:48 10.6.2.24/10.6.2.24 Port: link state changed to 'down' on port 10 NTX-Broadcast-Comp
426826h20m51000|Sep 10 10:20:50 10.6.2.24/10.6.2.24 Port: link state changed to 'up' (10M-H) on port 10 NTX-Broadcast-Comp
426826h21m13000|Sep 10 10:21:12 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h21m14000|Sep 10 10:21:13 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h21m14000|Sep 10 10:21:13 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h21m14000|Sep 10 10:21:13 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 3 to learning NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 3 to forwarding NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 2 to learning NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 2 to forwarding NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 1 to learning NTX-Catering-2
426826h21m16000|Sep 10 10:21:15 10.6.2.15/10.6.2.15 STP: set port 1 to forwarding NTX-Catering-2
426826h21m31000|Sep 10 10:21:30 10.6.2.9/10.6.2.9 monitor: restarting vtss_appl NTX-VetTech1
426826h21m31000|Sep 10 10:21:23 10.6.2.8/10.6.2.8 STP: set port 1 to discarding NTX-Barn4
426826h21m31000|Sep 10 10:21:23 10.6.2.8/10.6.2.8 STP: set port 1 to learning NTX-Barn4
426826h21m31000|Sep 10 10:21:23 10.6.2.8/10.6.2.8 STP: set port 1 to forwarding NTX-Barn4
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 3 to learning NTX-VetTech1
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 3 to forwarding NTX-VetTech1
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 2 to learning NTX-VetTech1
426826h21m34000|Sep 10 10:21:33 10.6.2.9/10.6.2.9 STP: set port 2 to forwarding NTX-VetTech1
426826h22m19000|Sep 10 10:22:18 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h22m20000|Sep 10 10:22:19 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h22m20000|Sep 10 10:22:19 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h22m20000|Sep 10 10:22:19 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 3 to learning NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 3 to forwarding NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 2 to learning NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 2 to forwarding NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 1 to learning NTX-Catering-2
426826h22m23000|Sep 10 10:22:22 10.6.2.15/10.6.2.15 STP: set port 1 to forwarding NTX-Catering-2
426826h22m26000|Sep 10 10:22:25 10.6.2.9/10.6.2.9 monitor: restarting vtss_appl NTX-VetTech1
426826h22m26000|Sep 10 10:22:18 10.6.2.8/10.6.2.8 STP: set port 1 to discarding NTX-Barn4
426826h22m26000|Sep 10 10:22:19 10.6.2.8/10.6.2.8 STP: set port 1 to learning NTX-Barn4
426826h22m26000|Sep 10 10:22:19 10.6.2.8/10.6.2.8 STP: set port 1 to forwarding NTX-Barn4
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 3 to learning NTX-VetTech1
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 3 to forwarding NTX-VetTech1
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 2 to learning NTX-VetTech1
426826h22m30000|Sep 10 10:22:29 10.6.2.9/10.6.2.9 STP: set port 2 to forwarding NTX-VetTech1
426826h23m14000|Sep 10 10:23:13 10.6.2.15/10.6.2.15 monitor: restarting vtss_appl NTX-Catering-2
426826h23m15000|Sep 10 10:23:14 10.6.2.14/10.6.2.14 STP: set port 1 to discarding
426826h23m16000|Sep 10 10:23:15 10.6.2.14/10.6.2.14 STP: set port 1 to learning
426826h23m16000|Sep 10 10:23:15 10.6.2.14/10.6.2.14 STP: set port 1 to forwarding
 

What version of firmware are you using?

1.5.0, on board Rev F

As implied above definitely make sure you're running the latest software, vtss_appl is responsible for communicating with the switch-core about protocol configurations. So when vtss_appl is first started it causes STP restart as well. There are a few configuration changes that can cause vtss_appl to restart, however, in your logs it says that the 'monitor' is what is restarting vtss_appl which implies it is crashing. I've seen this happen a few times, the most common reason is a broadcast storm. Depending on your network configuration, maybe you need to enable loop protection? Again, also make sure all your switches are running the latest software.

**EDIT**
You're response came in right before I posted this so I couldn't see that you're running 1.5.0, might want to try 1.5.1rc4 as well, some work was done on vtss_appl

Following up on this -- it seems like passing multicast/IGMP is somehow related to these issues. See screenshot from my syslog where immediately after disabling IGMP snooping for an IPTV client off the switch, the restarting vtss_appl issue began where it had been previously never showed the error. There really isn't a lot of data (<50Mbps) crossing a single port at a given time so I'm not sure how to prevent this error from restarting RSTP an interrupting traffic without disabling STP altogether.

Any ideas?

Is IGMP snooping disabled on all affected switches?

We might be looking at a new bug here.

If possible could send a screenshot of the following tabs:
Status,
Configuration,
STP,
VLANs,
QoS

From a switch that is exhibiting this behavior?

Is there an update on this error ? I just upgraded 7 switches from 1.4.7 to 1.5.0 and now I'm seeing this error repeating in the log, and occasional packet loss (1% - 2%) on all switches. I've tried disabling Spanning Tree and IGMP Snooping and the problem persists.

Try upgrading to 1.5.2rc3.
There has been several improvements on igmp, lacp, mstp, etc that all could potentially be responsible.

Get 1.5.2rc3 here:
viewtopic.php?f=17&t=240

Done. Unfortunately the problem persists. Packet loss is better than 1.5.0 but still ~1% pinging the switch from a directly attached router. Dropped ping replies seem to match up to 'restarting' messages in the log. This particular switch is board Rev B

Code: Select all

 Jan 1 00:00:08 sysinit: killall: udhcpc: no process killed
Jan 1 00:00:09 netonix: 1.5.2rc3 on WS-24-400A
Jan 1 00:00:11 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Jan 1 00:00:11 kernel: switch: 'Luton26' board detected
Jan 1 00:00:14 kernel: vtss_port: Loaded port module on board Luton26, type 5
Jan 1 00:00:15 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Jan 1 00:00:15 system: Setting MAC address from flash configuration: EC:13:B3:01:04:AC
Jan 1 00:00:15 kernel: i2c /dev entries driver
Jan 1 00:00:15 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Jan 1 00:00:16 sysinit: Loading defaults
Jan 1 00:00:17 sysinit: Loading synflood protection
Jan 1 00:00:17 sysinit: Adding custom chains
Jan 1 00:00:17 sysinit: Loading zones
Jan 1 00:00:18 sysinit: Loading forwarding
Jan 1 00:00:18 sysinit: Loading redirects
Jan 1 00:00:18 sysinit: Loading rules
Jan 1 00:00:18 sysinit: sh: bad number
Jan 1 00:00:18 sysinit: Loading includes
Jan 1 00:00:18 admin: adding lan (eth0) to firewall zone lan
Jan 1 00:00:26 kernel: eth0: no IPv6 routers present
Dec 31 19:00:31 syslogd exiting
Dec 31 19:00:32 syslogd started: BusyBox v1.19.4
Dec 31 19:00:32 syslogd exiting
Dec 31 19:00:33 syslogd started: BusyBox v1.19.4
Dec 31 19:00:34 admin: removing lan (eth0) from firewall zone lan
Dec 31 19:00:38 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:48 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:49 sysinit: killall: telnetd: no process killed
Dec 31 19:00:49 sysinit: killall: ntpclient: no process killed
Dec 31 19:00:49 sysinit: killall: ntpclient: no process killed
Dec 31 19:00:49 system: starting ntpclient
Feb 13 05:42:06 sysinit: 1969-12-31 19:00:50: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Feb 13 05:42:06 sysinit: 1969-12-31 19:00:50: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Feb 13 05:42:06 sysinit: 1969-12-31 19:00:50: (log.c.216) server started
Feb 13 05:42:08 dropbear[1403]: Running in background
Feb 13 05:42:12 switch[1432]: Detected warm boot
Feb 13 05:42:34 monitor: restarting vtss_appl
Feb 13 05:43:30 monitor: restarting vtss_appl
Feb 13 05:44:37 monitor: restarting vtss_appl

Copyright 2014-2018 Netonix


Can you post your configuration tab, status tab, and ports tab?