Switch downgrades links all at once

[LRL]

Anyone ever seen this before? Everythign is working great, then suddenly the switch downgrades nearly all links. Links at 1g go to 100 and links at 100 go to 10. Links restore to their prior state, but at that point spanning tree freaks out and leaves several ports in the discarding state. Below is the log, any help would be appreciated.

I should also note that the switch refuses to take config changes after it happens. I can't bounce a port, turn poe off or reboot/reload from software. The only fix is to physically power cycle it.

ws-12-250-dc running 1.4.7

Aug 25 20:30:44 UI: Configuration backup by admin (*.*.*.*.)
Aug 25 21:39:23 Port: link state changed to 'down' on port 10
Aug 25 21:39:23 STP: set port 10 to discarding
Aug 25 21:39:23 Port: link state changed to 'down' on port 11
Aug 25 21:39:23 Port: link state changed to 'down' on port 5
Aug 25 21:39:23 Port: link state changed to 'down' on port 6
Aug 25 21:39:23 STP: set port 6 to discarding
Aug 25 21:39:23 Port: link state changed to 'down' on port 7
Aug 25 21:39:23 Port: link state changed to 'down' on port 3
Aug 25 21:39:23 Port: link state changed to 'down' on port 9
Aug 25 21:39:23 Port: link state changed to 'down' on port 1
Aug 25 21:39:23 STP: set port 1 to discarding
Aug 25 21:39:25 Port: link state changed to 'up' (10M-F) on port 6
Aug 25 21:39:25 STP: set port 6 to discarding
Aug 25 21:39:25 Port: link state changed to 'up' (10M-F) on port 11
Aug 25 21:39:25 Port: link state changed to 'up' (10M-F) on port 10
Aug 25 21:39:25 STP: set port 10 to discarding
Aug 25 21:39:25 Port: link state changed to 'up' (10M-F) on port 7
Aug 25 21:39:25 Port: link state changed to 'up' (10M-F) on port 5
Aug 25 21:39:25 switch[1512]: unexpected link change on port 6 (Port 6 AP2 South UT) from 100M-F to 10M-F
Aug 25 21:39:26 switch[1519]: unexpected link change on port 11 (Port 11 AP3 South LT) from 100M-F to 10M-F
Aug 25 21:39:26 Port: link state changed to 'up' (100M-F) on port 9
Aug 25 21:39:26 Port: link state changed to 'up' (100M-F) on port 3
Aug 25 21:39:26 Port: link state changed to 'up' (100M-F) on port 1
Aug 25 21:39:26 STP: set port 1 to discarding
Aug 25 21:39:26 switch[1524]: unexpected link change on port 10 (Port 10 AP5 AHS LT) from 100M-F to 10M-F
Aug 25 21:39:27 switch[1530]: unexpected link change on port 7 (Port 7 AP6 South UT) from 100M-F to 10M-F
Aug 25 21:39:27 switch[1534]: unexpected link change on port 5 (Port 5 AP1 NCr UT) from 100M-F to 10M-F
Aug 25 21:39:27 STP: set port 10 to learning
Aug 25 21:39:27 STP: set port 10 to forwarding
Aug 25 21:39:27 STP: set port 6 to learning
Aug 25 21:39:27 STP: set port 6 to forwarding
Aug 25 21:39:27 switch[1539]: unexpected link change on port 9 (Port 9 AC14 Low) from 1G to 100M-F
Aug 25 21:39:28 switch[1545]: unexpected link change on port 3 (Port 3 AP7 North LT) from 1G to 100M-F
Aug 25 21:39:28 switch[1549]: unexpected link change on port 1 (Port 1 temp AC AP SOLAR) from 1G to 100M-F
Aug 25 21:39:28 STP: set port 1 to learning
Aug 25 21:39:28 STP: set port 1 to forwarding
Aug 25 21:39:43 Port: link state changed to 'down' on port 9
Aug 25 21:39:45 Port: link state changed to 'down' on port 3
Aug 25 21:39:45 Port: link state changed to 'up' (1G) on port 9
Aug 25 21:39:45 Port: link state changed to 'down' on port 1
Aug 25 21:39:46 STP: set port 1 to discarding
Aug 25 21:39:48 Port: link state changed to 'up' (1G) on port 1
Aug 25 21:39:48 STP: set port 1 to discarding
Aug 25 21:39:48 Port: link state changed to 'up' (1G) on port 3
Aug 25 21:39:49 Port: link state changed to 'down' on port 6
Aug 25 21:39:49 STP: set port 6 to discarding
Aug 25 21:39:49 Port: link state changed to 'down' on port 7
Aug 25 21:39:49 Port: link state changed to 'down' on port 10
Aug 25 21:39:49 STP: set port 10 to discarding
Aug 25 21:39:49 Port: link state changed to 'down' on port 11
Aug 25 21:39:49 Port: link state changed to 'down' on port 5
Aug 25 21:39:50 STP: set port 1 to learning
Aug 25 21:39:50 STP: set port 1 to forwarding
Aug 25 21:39:51 Port: link state changed to 'up' (100M-F) on port 10
Aug 25 21:39:51 STP: set port 10 to discarding
Aug 25 21:39:51 Port: link state changed to 'up' (100M-F) on port 7
Aug 25 21:39:51 Port: link state changed to 'up' (100M-F) on port 5
Aug 25 21:39:51 Port: link state changed to 'up' (100M-F) on port 11
Aug 25 21:39:51 Port: link state changed to 'up' (100M-F) on port 6
Aug 25 21:39:51 STP: set port 6 to discarding
Aug 25 21:39:53 STP: set port 6 to learning
Aug 25 21:39:53 STP: set port 6 to forwarding
Aug 25 21:39:53 STP: set port 10 to learning
Aug 25 21:39:53 STP: set port 10 to forwarding
Aug 25 21:40:07 sSMTP[1525]: Sent mail for REMOVED outbytes=570
Aug 25 21:40:07 sSMTP[1528]: Sent mail for REMOVED outbytes=566
Aug 25 21:40:07 sSMTP[1543]: Sent mail for REMOVED outbytes=552
Aug 25 21:40:07 sSMTP[1535]: Sent mail for REMOVED outbytes=566
Aug 25 21:40:07 sSMTP[1520]: Sent mail for REMOVED outbytes=566
Aug 25 21:40:07 sSMTP[1550]: Sent mail for REMOVED outbytes=560
Aug 25 21:40:07 sSMTP[1553]: Sent mail for REMOVED outbytes=568
Aug 25 21:40:07 sSMTP[1540]: Sent mail for REMOVED outbytes=562
Aug 25 21:55:23 Port: link state changed to 'down' on port 5
Aug 25 21:55:23 Port: link state changed to 'down' on port 6
Aug 25 21:55:23 STP: set port 6 to discarding
Aug 25 21:55:23 Port: link state changed to 'down' on port 7
Aug 25 21:55:23 Port: link state changed to 'down' on port 10
Aug 25 21:55:23 STP: set port 10 to discarding
Aug 25 21:55:23 Port: link state changed to 'down' on port 11
Aug 25 21:55:23 Port: link state changed to 'down' on port 9
Aug 25 21:55:23 Port: link state changed to 'down' on port 1
Aug 25 21:55:23 STP: set port 1 to discarding
Aug 25 21:55:23 Port: link state changed to 'down' on port 3
Aug 25 21:55:23 Port: link state changed to 'down' on port 4
Aug 25 21:55:25 Port: link state changed to 'up' (10M-F) on port 7
Aug 25 21:55:25 Port: link state changed to 'up' (10M-F) on port 5
Aug 25 21:55:25 Port: link state changed to 'up' (10M-F) on port 6

[sirhc]

Have never seen that but why are you not using v1.4.8rc7 ?

There is an important FIX for all smart DC switches in v1.4.8rcX for Fan control to prevent overheating.

FIXED/CHANGED
- Fixed SNMP support for Q-SWITCH-MIB, LLDP and CDP - RC7
- Changed temp sensor readings on DC models to effect fan speeds - RC2
- Grey out unavailable bulk poe options - RC3
- Upper out of voltage range for IDC is now 54V - RC4

ENHANCEMENTS
- Added ERPS protocol implementation is g.8032v2 - Ring Protection - RC1
- Added BETA note to ERPS - RC3
- Added port name to "show interface status" CLI command - RC3
- ADD an input voltage calibration for DC models - RC4
- Improved Q-BRIDGE-MIB support - RC5
- improved fan control on DC models - RC6
- improved stability of ERPS- RC6

I would swap the unit out with a spare and see if the issue repeats itself.

If it does not happen with spare unit then write up a very detailed RMA descriptions and RMA.

If it does repeat itself with the spare unit then add your config TABs and as much information about site as possible to this thread.

[LRL]

Have never seen that but why are you not using v1.4.8rc7 ?

There is an important FIX for all smart DC switches in v1.4.8rcX for Fan control to prevent overheating.

FIXED/CHANGED
- Fixed SNMP support for Q-SWITCH-MIB, LLDP and CDP - RC7
- Changed temp sensor readings on DC models to effect fan speeds - RC2
- Grey out unavailable bulk poe options - RC3
- Upper out of voltage range for IDC is now 54V - RC4

ENHANCEMENTS
- Added ERPS protocol implementation is g.8032v2 - Ring Protection - RC1
- Added BETA note to ERPS - RC3
- Added port name to "show interface status" CLI command - RC3
- ADD an input voltage calibration for DC models - RC4
- Improved Q-BRIDGE-MIB support - RC5
- improved fan control on DC models - RC6
- improved stability of ERPS- RC6

I'm a little gun shy lately about RCs , but I will upgrade.

Most of the issue persisted with a replacement switch. The only difference was the ports didn't remain in the discarding state. The only recent change to the site was the addition of 6 rocketacPrism Gen2s. We're early in the implementation phase so they were all placed in the managed bridge vlan. It smells like a STP/loop/broadcast storm issue but very odd behavior to downgrade links.

[sirhc]

*sigh* - RC does not mean Release Candidate with us.
After we go to v1.4.8 there will not be any more RC versions

viewtopic.php?f=17&t=2931&p=19985&hilit=the+next+day#p19985

[mike99]

It smells like a STP/loop/broadcast storm issue but very odd behavior to downgrade links.

STP always broadcast and would see the loop immediately, not only when the problem happen. A strange problem like this could be power or ground related.

Edit: After looking at your log, you can see that the all link goes by discarding (happen automaticly when link is down) to learning (listening to know if it detect a loop) and forwarding again so that not an STP or loop problem elle learning would be followed by discarding again. The initial discarding is because of the state down of the port, not a detected loop.

[LRL]

I'd normally agree, but it appears when this happens the switch management UI goes unresponsive and upon the links coming back up it clearly causes a loop by what's seen on adjoining switches. There is an intentional loop (managed by STP) between the switches at the site. We found water in an old tough cable that we replaced and I'm hoping that was the culprit.

[LRL]

Let me digress here. I believe you're right, just very odd behavior for the switch. I understand when dealing with power issues things get unpredictable. I believe whatever the exact cause it caused management over the switch chip to be loss and as a result the strange stuff happened. I guess this is better as a feedback for how could we prevent this in the future or what happens if the management process fails.

I believe sudden down grade of multiple ports coupled with the inrush of packets that couldn't possibly be handed off as quickly as received may have caused the management issue, but I'm just grasping here.

[Dawizman]

We have a similar issue with a WS-12-250-AC.

Are your devices rebooting when this happens?

[LRL]

Check your cables. Ours was a water inside a cable issue.