v1.5.15rc3 Bug Reports and Comments

[summit-nathan]

Hi Team,
v1.5.15rc3 seems better (SFP ports no longer drop) but LACP still drops when vlans are changed. STP is disabled on this one.

Code: Select all

 
Nov 8 09:12:32 UI: Configuration changed by admin
Nov 8 09:12:32 UI: VLAN 31 Enable: changed from 'Disabled' to 'Enabled'
Nov 8 09:12:45 LACP: starting negotiation with partner 48-A9-8A-CC-10-11
Nov 8 09:12:47 LACP: LACP changed state to Active on port 8 (key 1)
Nov 8 09:12:47 LACP: LACP changed state to Active on port 7 (key 1)
Nov 8 09:13:55 UI: User 'admin' logged in from ***


[KBrownConsulting]

Really happy to see development and rapid releases happening again!

Since we're addressing bugs, here's a list of the bugs I've been tracking. (Meaning things reported by others that I haven't noticed any fixes for yet.)

• jQuery 1x or 2x security vulnerability (Dave commented back in 2022 that "it is on the list to be fixed when we release another round of firmware for the WS line of products.")
• DHCP snooping changes require reboot to take effect
• SMTP bug - switch forges its HELO string - triggering spam filters
• LAG bug (In the very next comment Stephen commented that "I do intend to tackle the LAG issue before 1.5.7 is complete", but unless I missed it, I haven't noticed any mention of this being addressed in any of the changelogs since 1.5.7)
• QinQ and Double VLANs options sometimes missing (I say sometimes because I personally tested several different switches and was not able to reproduce this bug, and the original reporter later stated, "I have had mixed results so far with the issue being present and not being present", but since sirhc stated he confirmed the bug & it would be fixed in the next release I figured it's worth bringing up again.
• SNMP issue - (requesting xxx.1.6.1 from the Netonix switch gets 1.7.1 in response)

I am really happy that Netonix is still around and appreciate the hard work that goes into development. Hoping a few more of the items above can be addressed as part of the current development momentum.

Cheers

[summit-nathan]

Hi Netonix Team,

Memory usage on this RC looks much better than 1.5.14. It is certainly not trending down quickly.

We'd be quite happy to run this version in production if it didn't renegotiate LACP every time VLAN config is changed.

At the moment we are a bit stuck as we have stock of switches that will not allow us to downgrade to 1.5.5 (must run 1.5.12 or newer) but our standard deployment uses LACP on the SFP ports and every version we have tried from 1.5.12 onwards has issues with SFPs and/or LACP when VLAN config is changed.

* Is the LACP issue being worked on?
* Do you need an example config to replicate the issue?

Please let us know if there is anything we can help with to progress resolution of the LACP issue.

Cheers,
Nathan

[sirhc]

Hi Netonix Team,

Memory usage on this RC looks much better than 1.5.14. It is certainly not trending down quickly.

We'd be quite happy to run this version in production if it didn't renegotiate LACP every time VLAN config is changed.

At the moment we are a bit stuck as we have stock of switches that will not allow us to downgrade to 1.5.5 (must run 1.5.12 or newer) but our standard deployment uses LACP on the SFP ports and every version we have tried from 1.5.12 onwards has issues with SFPs and/or LACP when VLAN config is changed.

* Is the LACP issue being worked on?
* Do you need an example config to replicate the issue?

Please let us know if there is anything we can help with to progress resolution of the LACP issue.

Cheers,
Nathan

So the issue with LACP would not have changed since v1.0, what did change probably was it being reported in the log. Many errors were not reported in the log initially.

Anytime there is a config to the switching functions meaning anything like STP RSTP LACP LAG VLAN MTU QOS and so on requires the reprograming of the core. You can not change 1 aspect of the core you basically push the new core config and restart the core process.

You should hardly notice a lost ping or few from this process? Depends on the LACP configuration (LACP or Static) and what equipment your trying to reestablish the LAG link with. Static LAGs are fastest which is what I use.

To fix this would require kernel changes and VTSS changes both of which were written originally by VITESSE now owned by micro chip. We do have the core code but really hate messing with that if at all possible.

[jpmoller]

Today we experienced another SFP related issue. Our uplink is via an SFP port (14)

We had a WS-12-250-DC that was on firmware version 1.5.14 (Board revision F). Using the Netonix Manager, we scheduled a reboot and then a firmware upgrade to 1.5.15rc3. Switch rebooted fine, then proceeded to upgrade fine. After post-upgrade reboot, the switch disappeared.

Here are the Netonix Manager logs:

Netonix Manager Logs

We still had physical link on our router, but there was no RX traffic coming from the switch. One of our technicians visited site and moved uplink from SFP port to one of the copper ports. Management was restored immediately. Looking at the logs after reboot, port 14 never came online.

Switch logs:

Code: Select all

 
Jan 1 00:00:09 kernel: mini_fo: using storage directory: /jffs
Jan 1 00:00:09 sysinit: killall: udhcpc: no process killed
Jan 1 00:00:12 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Jan 1 00:00:12 kernel: switch: 'Luton26' board detected
Jan 1 00:00:13 kernel: vtss_port: Loaded port module on board Luton26, type 5
Jan 1 00:00:14 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Jan 1 00:00:14 system: Setting MAC address from flash configuration: EC:13:B2:83:0F:D8
Jan 1 00:00:14 kernel: i2c /dev entries driver
Jan 1 00:00:14 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Jan 1 00:00:15 sysinit: Loading defaults
Jan 1 00:00:16 sysinit: Loading synflood protection
Jan 1 00:00:16 sysinit: Adding custom chains
Jan 1 00:00:16 sysinit: Loading zones
Jan 1 00:00:17 sysinit: Loading forwarding
Jan 1 00:00:17 sysinit: Loading redirects
Jan 1 00:00:17 sysinit: Loading rules
Jan 1 00:00:17 sysinit: sh: bad number
Jan 1 00:00:17 sysinit: Loading includes
Jan 1 00:00:17 root: adding lan (eth0) to firewall zone lan
Jan 1 00:00:18 dropbear[819]: Running in background
Jan 1 00:00:19 netonix: 1.5.15rc3 on WS-12-250-DC
Jan 1 00:00:22 UI: web config OK
Jan 1 00:00:25 kernel: eth0: no IPv6 routers present
Jan 1 13:00:36 syslogd exiting
Jan 1 13:00:37 syslogd started: BusyBox v1.19.4
Jan 1 13:00:37 syslogd exiting
Jan 1 13:00:38 syslogd started: BusyBox v1.19.4
Jan 1 13:00:39 root: removing lan (eth0.100) from firewall zone lan
Jan 1 13:00:44 root: adding lan (eth0.100) to firewall zone lan
Jan 1 13:00:44 root: removing lan (eth0.100) from firewall zone lan
Jan 1 13:00:48 root: adding lan (eth0.100) to firewall zone lan
Jan 1 13:00:57 root: adding lan (eth0.100) to firewall zone lan
Jan 1 13:00:58 kernel: eth0.100: no IPv6 routers present
Jan 1 13:00:59 dropbear[1797]: Failed listening on '22': Error listening: Address already in use
Jan 1 13:00:59 dropbear[1804]: Running in background
Jan 1 13:00:59 sysinit: killall: telnetd: no process killed
Jan 1 13:00:59 sysinit: killall: ntpclient: no process killed
Jan 1 13:01:00 system: starting ntpclient
Jan 1 13:01:00 sysinit: killall: ntpclient: no process killed
Jan 1 13:01:02 STP: msti 0 set port 3 to discarding
Jan 1 13:01:02 sysinit: 1970-01-01 13:01:01: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Jan 1 13:01:02 sysinit: 1970-01-01 13:01:01: (network.c.272) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
Jan 1 13:01:02 sysinit: 1970-01-01 13:01:01: (log.c.216) server started
Jan 1 13:01:02 STP: msti 0 set port 4 to discarding
Jan 1 13:01:03 STP: msti 0 set port 5 to discarding
Jan 1 13:01:03 STP: msti 0 set port 6 to discarding
Jan 1 13:01:03 STP: msti 0 set port 7 to discarding
Jan 1 13:01:03 STP: msti 0 set port 8 to discarding
Jan 1 13:01:04 STP: msti 0 set port 9 to discarding
Jan 1 13:01:04 STP: msti 0 set port 10 to discarding
Jan 1 13:01:04 STP: msti 0 set port 11 to discarding
Jan 1 13:01:04 STP: msti 0 set port 12 to discarding
Jan 1 13:01:06 switch[1871]: temp sensor version 3
Jan 1 13:01:07 switch[1872]: Detected warm boot
Jan 1 13:01:10 STP: msti 0 set port 12 to learning
Jan 1 13:01:10 STP: msti 0 set port 12 to forwarding
Jan 1 13:01:10 STP: msti 0 set port 11 to learning
Jan 1 13:01:10 STP: msti 0 set port 11 to forwarding
Jan 1 13:01:10 STP: msti 0 set port 10 to learning
Jan 1 13:01:10 STP: msti 0 set port 10 to forwarding
Jan 1 13:01:10 STP: msti 0 set port 9 to learning
Jan 1 13:01:11 STP: msti 0 set port 9 to forwarding
Jan 1 13:01:11 STP: msti 0 set port 8 to learning
Jan 1 13:01:11 STP: msti 0 set port 8 to forwarding
Jan 1 13:01:11 STP: msti 0 set port 7 to learning
Jan 1 13:01:11 STP: msti 0 set port 7 to forwarding
Jan 1 13:01:11 STP: msti 0 set port 6 to learning
Jan 1 13:01:11 STP: msti 0 set port 6 to forwarding
Jan 1 13:01:11 STP: msti 0 set port 5 to learning
Jan 1 13:01:11 STP: msti 0 set port 5 to forwarding
Jan 1 13:01:11 STP: msti 0 set port 4 to learning
Jan 1 13:01:11 STP: msti 0 set port 4 to forwarding
Jan 1 13:01:12 STP: msti 0 set port 3 to learning
Jan 1 13:01:12 STP: msti 0 set port 3 to forwarding
Jan 1 13:01:14 system: starting ntpclient
Jan 1 13:01:28 system: starting ntpclient
Jan 1 13:01:41 system: starting ntpclient
Jan 1 13:01:55 system: starting ntpclient
Jan 1 13:02:08 system: starting ntpclient
Jan 1 13:02:21 system: starting ntpclient
Jan 1 13:02:35 system: starting ntpclient
Jan 1 13:02:48 system: starting ntpclient


We then moved the uplink back from the copper port to the SFP port to see if it would spark port up. No dice. No entries in the logs for the port move. Upon a physical switch reboot, the switch was now accessible via the SFP port again.

[sirhc]

Leave switch on, pull the SFP module and then re-insert it, I bet it comes on line.

This issue was discussed for years

viewtopic.php?f=6&t=293&p=11025&hilit=SFP+pull+insert#p11025

viewtopic.php?f=7&t=2849&p=19636&hilit=SFP+pull+insert#p19636

[RebusCom]

Really happy to see development and rapid releases happening again!

Since we're addressing bugs, here's a list of the bugs I've been tracking. (Meaning things reported by others that I haven't noticed any fixes for yet.)

I don't see the mass reboot issue (propagating across the network) of switches running RSTP, which I've reported multiple times. Has that been resolved?

[Stephen]

RebusCom,

The fixes present in this rc address several general stability issue's.
If you can try this one out and let us know if you still see the problem, that would be helpful.

[Wifimax]

Amazing. I will test this on a few sites that we have with fiber and see if it fixes it.

The only other outstanding bug that I've seen with the WS2 line is if I have two links on a site with one port in shutdown. If I shut down one port and bring up the other, all the other side sees is untagged traffic.

I have to disable one port, enable the other and reboot the switch for traffic to pass again.

Thank you very much!

I just wanted to add in here that I see the exact same issue on my Netonix switches. I have dual PTP links headed to a tower with the backup link disabled at the netonix port. If the main link has issues, I disable the ethernet port at the netonix, and enable the backup port but it will not pass traffic unless the netonix switch is rebooted. Once the Netonix is rebooted everything works fine unless I need to switch back to the main backhaul, then another reboot is needed.

Same VLAN, Layer 2 network very simple. I have had this happen on 1.5.14 and at least as far back as 1.5.8. STP is disabled.

[sirhc]

if this is related to an SFP port leave switch on, pull the SFP module and then re-insert it, I bet it comes on line.

This issue was discussed for years

viewtopic.php?f=6&t=293&p=11025&hilit=SFP+pull+insert#p11025

viewtopic.php?f=7&t=2849&p=19636&hilit=SFP+pull+insert#p19636

If this is the case replace the SFP module but test the new module by doing several software reboots. Some SFP modules have a tolerance +/- in timing issue even if other SFP modules of same type worked fine.