v1.5.0 Bug reports and Comments

[tmporter]

we encountered the same issue on all 5 switches. logs below.

Dec 31 19:00:04 sysinit: killall: udhcpc: no process killed
Dec 31 19:00:06 netonix: 1.5.0 on WS-12-250-AC
Dec 31 19:00:07 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Dec 31 19:00:07 kernel: switch: 'Luton26' board detected
Dec 31 19:00:09 kernel: vtss_port: Loaded port module on board Luton26, type 5
Dec 31 19:00:09 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Dec 31 19:00:10 system: Setting MAC address from flash configuration: EC:13:B2:63:7B:BE
Dec 31 19:00:10 kernel: i2c /dev entries driver
Dec 31 19:00:10 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Dec 31 19:00:11 sysinit: Loading defaults
Dec 31 19:00:11 system: starting ntpclient
Dec 31 19:00:11 sysinit: Adding custom chains
Dec 31 19:00:12 sysinit: Loading zones
Dec 31 19:00:12 sysinit: Loading forwarding
Dec 31 19:00:12 sysinit: Loading redirects
Dec 31 19:00:12 sysinit: Loading rules
Dec 31 19:00:12 sysinit: Loading includes
Dec 31 19:00:12 ydim: adding lan (eth0) to firewall zone lan
Dec 31 19:00:15 syslogd exiting
Dec 31 19:00:16 syslogd started: BusyBox v1.19.4
Dec 31 19:00:25 STP: set port 9 to discarding
Dec 31 19:00:25 sysinit: 1969-12-31 19:00:24: (log.c.216) server started
Dec 31 19:00:28 STP: set port 9 to learning
Dec 31 19:00:28 STP: set port 9 to forwarding
Dec 31 19:00:28 dropbear[818]: Running in background
Dec 31 19:00:30 switch[848]: Detected cold boot
Dec 31 19:00:32 system: starting ntpclient
Dec 31 19:00:43 system: starting ntpclient
Dec 31 19:00:55 system: starting ntpclient
Dec 31 19:01:06 system: starting ntpclient
Dec 31 19:01:17 system: starting ntpclient
Dec 31 19:01:28 system: starting ntpclient
Dec 31 19:01:39 system: starting ntpclient
Dec 31 19:01:51 system: starting ntpclient
Dec 31 19:02:02 system: starting ntpclient
Dec 31 19:02:14 system: starting ntpclient
Dec 31 19:02:25 system: starting ntpclient
Dec 31 19:02:36 system: starting ntpclient
Dec 31 19:02:47 system: starting ntpclient
Dec 31 19:02:59 system: starting ntpclient
Dec 31 19:03:10 system: starting ntpclient
Dec 31 19:03:21 system: starting ntpclient
Dec 31 19:03:32 system: starting ntpclient
Dec 31 19:03:44 system: starting ntpclient
Dec 31 19:03:55 system: starting ntpclient
Dec 31 19:04:06 system: starting ntpclient
Dec 31 19:04:17 system: starting ntpclient
Dec 31 19:04:29 system: starting ntpclient
Dec 31 19:04:40 system: starting ntpclient
Dec 31 19:04:51 system: starting ntpclient
Dec 31 19:05:02 system: starting ntpclient
Dec 31 19:05:14 system: starting ntpclient
Dec 31 19:05:25 system: starting ntpclient
Dec 31 19:05:36 system: starting ntpclient
Dec 31 19:05:47 system: starting ntpclient
Dec 31 19:05:59 system: starting ntpclient
Dec 31 19:06:10 system: starting ntpclient
Dec 31 19:06:21 system: starting ntpclient
Dec 31 19:06:32 system: starting ntpclient
Dec 31 19:06:44 system: starting ntpclient
Dec 31 19:06:55 system: starting ntpclient
Dec 31 19:07:06 system: starting ntpclient
Dec 31 19:07:17 system: starting ntpclient
Dec 31 19:07:28 system: starting ntpclient
Dec 31 19:07:39 system: starting ntpclient
Dec 31 19:07:51 system: starting ntpclient
Dec 31 19:08:02 system: starting ntpclient
Dec 31 19:08:13 system: starting ntpclient
Dec 31 19:08:24 system: starting ntpclient
Dec 31 19:08:36 system: starting ntpclient
Dec 31 19:08:47 system: starting ntpclient
Dec 31 19:08:58 system: starting ntpclient
Dec 31 19:09:09 system: starting ntpclient
Dec 31 19:09:20 system: starting ntpclient
Dec 31 19:09:32 system: starting ntpclient
Dec 31 19:09:43 system: starting ntpclient
Dec 31 19:09:54 system: starting ntpclient
Dec 31 19:10:05 system: starting ntpclient
Dec 31 19:10:16 system: starting ntpclient
Dec 31 19:10:28 system: starting ntpclient
Dec 31 19:10:39 system: starting ntpclient
Dec 31 19:10:50 system: starting ntpclient
Dec 31 19:11:01 system: starting ntpclient
Dec 31 19:11:12 system: starting ntpclient
Dec 31 19:11:24 system: starting ntpclient
Dec 31 19:11:35 system: starting ntpclient
Dec 31 19:11:47 Port: link state changed to 'down' on port 9
Dec 31 19:11:47 STP: set port 9 to discarding
Dec 31 19:11:47 system: starting ntpclient
Dec 31 19:11:54 Port: link state changed to 'up' (1G) on port 9
Dec 31 19:11:54 STP: set port 9 to discarding
Dec 31 19:11:57 STP: set port 9 to learning
Dec 31 19:11:57 STP: set port 9 to forwarding
Dec 31 19:11:58 system: starting ntpclient
Dec 31 19:12:09 system: starting ntpclient
Dec 31 19:12:21 system: starting ntpclient
Dec 31 19:12:32 system: starting ntpclient
Dec 31 19:12:43 system: starting ntpclient
Dec 31 19:12:54 system: starting ntpclient
Dec 31 19:13:05 system: starting ntpclient
Dec 31 19:13:17 system: starting ntpclient
Dec 31 19:13:28 system: starting ntpclient
Dec 31 19:13:39 system: starting ntpclient
Dec 31 19:13:50 system: starting ntpclient
Dec 31 19:14:01 system: starting ntpclient
Dec 31 19:14:13 system: starting ntpclient
Dec 31 19:14:24 system: starting ntpclient
Dec 31 19:14:35 system: starting ntpclient
Dec 31 19:14:46 system: starting ntpclient
Dec 31 19:14:57 system: starting ntpclient
Dec 31 19:15:07 Port: link state changed to 'down' on port 9
Dec 31 19:15:07 STP: set port 9 to discarding
Dec 31 19:15:09 system: starting ntpclient
Dec 31 19:15:15 Port: link state changed to 'up' (1G) on port 9
Dec 31 19:15:15 STP: set port 9 to discarding
Dec 31 19:15:17 STP: set port 9 to learning
Dec 31 19:15:17 STP: set port 9 to forwarding
Dec 31 19:15:20 system: starting ntpclient
Dec 31 19:15:31 system: starting ntpclient
Dec 31 19:15:42 system: starting ntpclient
Dec 31 19:15:54 system: starting ntpclient
Dec 31 19:16:05 system: starting ntpclient

[LRL]

Okay, it's official, I've decided whatever was changed in the I2C starting in 1.5.0 has really broken something with the SFP's we use. We use 10Gtek SFP-GE-T modules are pretty much all of our sites. We do this because after trying numerous manufacture's modules they appeared to work the best without hiccup with Netonix. But since upgrading to 1.5.0 I've had nearly a dozen switches have SFP issues. The switch reverts to claiming the SFP is unknown and while in this state STP appears to be in a flapping state. Disabling STP leads to small momentary loops being detected by the tower router (there are no loops). It seems as if the error-ed state of the SFP is leading the module or switch to repeat traffic more like a hub.

Downgrading to 1.4.9 solves the problem, upgrading to 1.5.1rc7 does not.

I don't know if this is specific to the 10Gtek module or not, but prior to 1.5.0 they've had years of great results.

[Stephen]

Hey LRL,

Could you let us know which switch model's you're seeing this problem on?

[LRL]

Ws-12-250 ac and dc. More often port13 has an issue more than 14, but typically we use 13 as untagged on the mgmt vlan and 14 just on tagged vlans.

My largest issue now is that winter has presented itself and access to some of these sites is not as simple anymore.

[Eric Stern]

LRL,

I got a 10Gtek SFP-GE-T for the lab and it seems to work fine in a WS-12-250-AC.

Do you have a switch in the lab you can test with? If so, can you try this?

Remove the SFP module. Run these commands from the CLI:
cmd
/etc/init.d/vtss_appl stop
vtss_appl -l -m -s -v 4
<wait for about 5 seconds until most of the messages stop>
<insert the SFP>
<hit CTRL-C>
/etc/init.d/vtss_appl start

Then copy the messages on the screen and send it to us. It should look something like this:

Code: Select all

7.441(8201): i2c_read: i2c_read port 24 i2c_addr 50 addr 0 cnt 60
7.454(8201): sfp_detect: Port 25: Vendor: 10Gtek
7.455(8201): sfp_detect: Port 25: Part: SFP-GE-T
7.455(8201): sfp_detect: Port 25: Rev:
7.556(8201): sfp_detect: Port 25: host side Aneg is 1000BASE-X mode
7.556(8201): i2c_read: i2c_read port 24 i2c_addr 56 addr 0 cnt 10
7.559(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.560(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.561(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.562(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.562(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.563(8201): sfp_detect: Port 25: sfp_rom = PORT_SFP_1000BASE_T
7.666(8201): sfp_detect_if: Port 25: sfp_type:PORT_SFP_1000BASE_T, cisco_aneg:1
7.667(8201): sfp_detect_if: Port 25: sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO
7.667(8201): sfp_detect_if: Port 25: sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO
7.671(8201): sfp_detect_if: Port 25: sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO, sfp_type:PORT_SFP_1000BASE_T
7.671(8201): port_setup: Port 25: port_mac_sfp_interface
7.674(8201): port_mac_sfp_interface: Port 25: port.mac_sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO
7.675(8201): port_setup: Port 25: mac_if:VTSS_PORT_INTERFACE_SGMII_CISCO
7.675(8201): port_mac_sfp_interface: Port 25: port.mac_sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO
7.687(8201): port_setup: Port 25: if_type = VTSS_PORT_INTERFACE_SGMII_CISCO, speed:VTSS_SPEED_1G FDX OBEY GENERATE 0
7.696(8201): port_setup: port is not PHY
7.697(8201): port_mac_sfp_interface: Port 25: port.mac_sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO
7.697(8201): port_setup: Set port 24 to SGMII mode
7.699(8201): i2c_read: i2c_read port 24 i2c_addr 56 addr 2 cnt 4
7.749(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.750(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.751(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.752(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.753(8201): i2c_write: i2c_write port 24 i2c_addr 56 cnt 3
7.854(8201): port_setup: exit, port_no: 24
7.854(8201): port_thread: Port 25: sfp_if:VTSS_PORT_INTERFACE_SGMII_CISCO, port.mac_sfp_if[port_no]:VTSS_PORT_INTERFACE_SGMII_CISCO
7.855(8201): port_thread: Port 25 SFP changed: old: 0, new: 1


[LRL]

Eric,

I've got a production switch in a hard to get to spot that's having this and shows error on PS firmware version ws-12-250-dc. Interested in poking at it?

So far to date, it always appears to be port13 that's having issues. 13 is mad and 14 is fine.

[jpaine619]

I'm seeing random reboots on _all_ of my switches on 1.5.0. They've been upgraded at various times over the last month and a half. (Always with a reboot first). But switches that saw huge uptimes on 1.4.9 are now rebooting before hitting 30 days uptime. The only switches I have that aren't rebooting (cold) randomly are the couple I still have on 1.4.9

Anyone else seeing this? I highly doubt it's hardware related, because most of these switches are pretty new.. Less than 2 years old at the oldest, and some are less than 6 months old.

Also, once the switch reboots, it works just fine, but it's killing POE to all the ports.

[sirhc]

I'm seeing random reboots on _all_ of my switches on 1.5.0. They've been upgraded at various times over the last month and a half. (Always with a reboot first). But switches that saw huge uptimes on 1.4.9 are now rebooting before hitting 30 days uptime. The only switches I have that aren't rebooting (cold) randomly are the couple I still have on 1.4.9

Anyone else seeing this? I highly doubt it's hardware related, because most of these switches are pretty new.. Less than 2 years old at the oldest, and some are less than 6 months old.

Also, once the switch reboots, it works just fine, but it's killing POE to all the ports.

I am at a loss as I am not seeing this at all?

I run a WISP as well and I simply am not seeing it?

[sirhc]

Went through all my towers and they are all up as long as they should be, here is another one.

[LRL]

Chris,

I'm starting to think STP may have something to do with it, do you STP enabled on your switches? SFPs? If so what brand model? I'd be curious to see what features you're using verses others of us that are seeing this.