WS-6-Mini - STP Discarding On Active Ports?

[orders@xytel.net]

We have two WS-6-Mini's located inside the same nema enclosure at one of our sites.
Yesterday morning around 8:00AM one of them changed the state of all the enabled ports to Discarding.

This dropped two of our sectors offline and took several people offline.

I was able to reboot the switch, which fixed it temporarily, however it did the same thing several hours later.
I disabled STP entirely and prayed to god we didn't have any loops, everything has been fine since... No loops. :)

We have several WS-12-250-DC models with STP enabled and have never had this issue.

[sirhc]

What firmware version?

Did you review the logs and see why it did this?

[orders@xytel.net]

Code: Select all

Jan 1 00:00:05 netonix: 1.4.2 on WS-6-MINI
Jan 1 00:00:09 system: Setting MAC address from flash configuration: EC:13:B2:91:43:B6
Jan 1 00:00:12 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:24 admin: removing lan (eth0.100) from firewall zone lan
Dec 31 19:00:29 admin: adding lan (eth0.100) to firewall zone lan
Dec 31 19:00:32 admin: adding lan (eth0.100) to firewall zone lan
Dec 31 19:00:33 STP: set port 2 to discarding
Dec 31 19:00:33 STP: set port 3 to discarding
Dec 31 19:00:33 STP: set port 4 to discarding
Dec 31 19:00:33 STP: set port 5 to discarding
Dec 31 19:00:33 STP: set port 6 to discarding
Dec 31 19:00:34 STP: set port 2 to learning
Dec 31 19:00:34 STP: set port 2 to forwarding
Dec 31 19:00:34 STP: set port 6 to learning
Dec 31 19:00:34 STP: set port 6 to forwarding
Dec 31 19:00:36 dropbear[1251]: Running in background
Dec 31 19:00:39 switch[1278]: Detected cold boot
Dec 31 19:00:39 Port: link state changed to 'down' on port 2
Dec 31 19:00:39 STP: set port 2 to discarding
Dec 31 19:00:40 Port: link state changed to 'down' on port 4
Dec 31 19:00:40 Port: link state changed to 'down' on port 3
Dec 31 19:00:40 Port: link state changed to 'down' on port 5
Dec 31 19:00:41 switch[1278]: PoE enabled on port 2, PoE Smart is starting cable check
Dec 31 19:00:41 switch[1278]: PoE enabled on port 3, PoE Smart is starting cable check
Dec 31 19:00:41 switch[1278]: PoE enabled on port 4, PoE Smart is starting cable check
Dec 31 19:00:41 switch[1278]: PoE enabled on port 5, PoE Smart is starting cable check
Dec 31 19:00:46 switch[1278]: Port 2 cable check results: Open, Open, Open, Open
Dec 31 19:00:46 switch[1278]: Port 2 passed PoE Smart cable check, turning on power
Dec 31 19:00:46 switch[1278]: Port 3 cable check results: Open, Open, Open, Open
Dec 31 19:00:46 switch[1278]: Port 3 passed PoE Smart cable check, turning on power
Dec 31 19:00:46 switch[1278]: Port 4 cable check results: Open, Abnormal termination, Open, Open
Dec 31 19:00:46 switch[1278]: Port 4 passed PoE Smart cable check, turning on power
Dec 31 19:00:46 switch[1278]: Port 5 cable check results: Open, Open, Open, Abnormal termination
Dec 31 19:00:46 switch[1278]: Port 5 passed PoE Smart cable check, turning on power
Dec 31 19:00:49 Port: link state changed to 'up' (1G) on port 5
Dec 31 19:00:49 STP: set port 5 to discarding
Dec 31 19:00:49 Port: link state changed to 'up' (100M-F) on port 4
Dec 31 19:00:49 STP: set port 4 to discarding
Dec 31 19:00:49 Port: link state changed to 'up' (100M-F) on port 3
Dec 31 19:00:49 STP: set port 3 to discarding
Dec 31 19:00:52 STP: set port 3 to learning
Dec 31 19:00:52 STP: set port 3 to forwarding
Dec 31 19:00:52 STP: set port 4 to learning
Dec 31 19:00:52 STP: set port 4 to forwarding
Dec 31 19:00:52 STP: set port 5 to learning
Dec 31 19:00:52 STP: set port 5 to forwarding
Dec 31 19:01:03 Port: link state changed to 'down' on port 5
Dec 31 19:01:03 STP: set port 5 to discarding
Dec 31 19:01:06 Port: link state changed to 'up' (1G) on port 5
Dec 31 19:01:06 STP: set port 5 to discarding
Dec 31 19:01:07 STP: set port 6 to discarding
Dec 31 19:01:07 STP: set port 5 to learning
Dec 31 19:01:07 STP: set port 5 to forwarding
Dec 31 19:01:07 Port: link state changed to 'down' on port 3
Dec 31 19:01:07 STP: set port 3 to discarding
Dec 31 19:01:07 STP: set port 6 to learning
Dec 31 19:01:07 STP: set port 6 to forwarding
Dec 31 19:01:09 Port: link state changed to 'down' on port 4
Dec 31 19:01:09 STP: set port 4 to discarding
Dec 31 19:01:09 Port: link state changed to 'up' (1G) on port 3
Dec 31 19:01:09 STP: set port 3 to discarding
Dec 31 19:01:12 Port: link state changed to 'up' (1G) on port 4
Dec 31 19:01:12 STP: set port 4 to discarding
Dec 31 19:01:12 STP: set port 3 to learning
Dec 31 19:01:12 STP: set port 3 to forwarding
Dec 31 19:01:14 STP: set port 4 to learning
Dec 31 19:01:14 STP: set port 4 to forwarding
Dec 31 19:01:15 STP: set port 3 to discarding
Dec 31 19:01:17 STP: set port 4 to discarding
Dec 31 19:02:11 Port: link state changed to 'up' (1G) on port 2
Dec 31 19:02:11 STP: set port 2 to discarding
Dec 31 19:02:13 STP: set port 2 to learning
Dec 31 19:02:13 STP: set port 2 to forwarding
Dec 31 19:02:31 Port: link state changed to 'down' on port 2
Dec 31 19:02:31 STP: set port 2 to discarding
Dec 31 19:02:33 Port: link state changed to 'up' (1G) on port 2
Dec 31 19:02:33 STP: set port 2 to discarding
Dec 31 19:02:36 STP: set port 2 to learning
Dec 31 19:02:36 STP: set port 2 to forwarding
Dec 31 19:02:55 STP: set port 5 to discarding
Dec 31 19:15:41 UI: Configuration changed by admin (192.168.1.1)
Dec 31 19:15:41 UI: STP_Enable: changed from 'Enabled' to 'Disabled'
Dec 31 19:15:43 STP: set port 2 to discarding
Dec 31 19:15:43 STP: set port 6 to discarding


[aistech]

I just had a similar issue on a WS-12-250-DC, but it only affected one port. Any thoughts on the cause? See log output below...

Nov 23 01:13:43 dropbear[511]: Child connection from ::ffff:###.###.###.###:44308

Nov 23 01:18:43 dropbear[1079]: Child connection from ::ffff:###.###.###.###:29076

Nov 23 01:23:43 dropbear[1640]: Child connection from ::ffff:###.###.###.###:26677

Nov 23 01:27:44 STP: set port 2 to discarding

Nov 23 01:27:44 kernel: link state changed to 'down' on port 2

Nov 23 01:27:46 STP: set port 2 to discarding

Nov 23 01:27:46 kernel: link state changed to 'up' on port 2

Nov 23 01:27:49 STP: set port 2 to learning

Nov 23 01:27:49 STP: set port 2 to forwarding

Nov 23 01:28:18 STP: set port 2 to discarding

Nov 23 01:28:18 STP: set port 2 to learning

Nov 23 01:28:18 STP: set port 2 to forwarding

Nov 23 01:29:02 dropbear[2221]: Child connection from ::ffff:###.###.###.###:14430

Nov 23 01:33:44 dropbear[2742]: Child connection from ::ffff:###.###.###.###:34399

Thank you.

[sirhc]

I am hoping all of you have upgraded to the latest firmware before asking for help? The latest firmware is v1.4.5

PLEASE UPGRADE YOUR FIRMWARE AND SEE IF THE PROBLEM IS STILL THERE AND IF SO MAYBE INVESTIGATE THE ISSUES AS OUTLINED BELOW.

In both examples above you can clearly see that the link state on the port goes down which triggers STP to go to Discarding

Then the link state goes to UP which forces STP to go into Learning mode to make sure there are no loops

Then finally STP changes to Forwarding after it determines there is no loop.

Possible Causes:
1) The device is rebooting taking the link down and up but I doubt this as the time from down to up is so quick

2) You have the port speed forced to a speed/duplex and errors on the line force the link to drop as it can not modulate down to compensate. I NEVER force link speed/duplex as it is just a bad idea.

3) There is a bad cable or end that loses momentary connection and the link drops then comes back up triggering STP

4) You have a bad port in the device or switch which can be tested by swapping units to find a bad unit.

5) There is interference around the 100 MHz range interfering with Ethernet communications causing the link to drop and if the port link/speed is forced this can be aspirated.

You can simply disable STP on these ports and monitor your log and my guess is you will still see those ports dropping out just no STP action will be taken.

Disable STP on those ports in the STP Tab by unchecking the box in front of the port in question.

Always take a complex problem and break it down into simple steps and attempt to isolate the issue. Being an IT guy is like being a detective as none of us were handed a crystal ball when we decided to get into IT work.

Above I came up with all the possible causes I could think of now the next step is to narrow down the list by doing some testing.