Link Dropping for 5 minutes

[MarkTTU]

Periodically, and almost always in bad weather, we have one port on our WS-12-250-AC that drops the link for almost exactly 5 minutes (give or take 1-2 seconds each time). It's the last port I'd expect to see an issue with as it's the uplink that goes to the core router two racks over.

The setup is pretty simple. Port 1 is an AirFiber, port 2 is a Bullet M5 Titanium, and port 12 is the uplink to the core router at our ISP (which is literally two racks over from where the Netonix switch sits). The AirFiber and Bullet are linking back to a switch at our HQ about 3 miles away. Just using STP to make the Bullet a backup to the AirFiber.

When I look at the Netonix logs I don't see any issues with port 1 or port 2, but port 12 will go down and then 5 minutes later come back up. Below is the log from yesterday and today (everything from yesterday though now).

I asked one of the data engineers at our ISP to see what their logs show and he said he saw two physical drops yesterday, but that the physical interface shows to be running clean and the history shows a single CRC error. His first thought was bad cable/end/port, but if that were the case he'd expect to see more drops and errors. I tend to agree with him, if it were cabling it should have more errors and not seem to be weather related.

Firmware is 1.4.6 which is old, but the switch isn't easy for me to access (it's in the ISP's core routing room) so I've been hesitant to upgrade it. I don't see anything in the firmware notes that sounds like this problem, but I'm the first to admit I could've overlooked something.

Anyone have any thoughts on what would cause a single port to drop and then re-establish 5 minutes later?


May 23 18:32:11 Port: link state changed to 'down' on port 12
May 23 18:32:11 STP: set port 12 to discarding
May 23 18:32:12 STP: set port 2 to discarding
May 23 18:32:12 STP: set port 2 to learning
May 23 18:32:12 STP: set port 2 to forwarding
May 23 18:32:13 STP: set port 2 to discarding
May 23 18:32:15 STP: set port 2 to learning
May 23 18:32:16 STP: set port 2 to discarding
May 23 18:32:18 STP: set port 2 to learning
May 23 18:32:18 STP: set port 2 to forwarding
May 23 18:32:19 STP: set port 2 to discarding
May 23 18:37:13 Port: link state changed to 'up' (1G) on port 12
May 23 18:37:13 STP: set port 12 to discarding
May 23 18:37:14 STP: set port 1 to discarding
May 23 18:37:14 STP: set port 12 to learning
May 23 18:37:14 STP: set port 12 to forwarding
May 23 18:37:14 STP: set port 1 to learning
May 23 18:37:14 STP: set port 1 to forwarding
May 23 18:37:17 STP: set port 2 to learning
May 23 18:37:17 STP: set port 2 to forwarding
May 23 19:17:37 STP: set port 1 to discarding
May 23 19:17:38 Port: link state changed to 'down' on port 12
May 23 19:17:38 STP: set port 12 to discarding
May 23 19:17:38 STP: set port 1 to learning
May 23 19:17:38 STP: set port 1 to forwarding
May 23 19:17:38 STP: set port 2 to discarding
May 23 19:22:39 Port: link state changed to 'up' (1G) on port 12
May 23 19:22:39 STP: set port 12 to discarding
May 23 19:22:40 STP: set port 1 to discarding
May 23 19:22:40 STP: set port 12 to learning
May 23 19:22:40 STP: set port 12 to forwarding
May 23 19:22:40 STP: set port 1 to learning
May 23 19:22:40 STP: set port 1 to forwarding
May 23 19:22:43 STP: set port 2 to learning
May 23 19:22:43 STP: set port 2 to forwarding
May 24 01:46:45 STP: set port 1 to discarding
May 24 01:46:45 STP: set port 1 to learning
May 24 01:46:45 STP: set port 1 to forwarding
May 24 14:06:48 STP: set port 1 to discarding
May 24 14:06:49 Port: link state changed to 'down' on port 12
May 24 14:06:49 STP: set port 12 to discarding
May 24 14:06:52 STP: set port 1 to learning
May 24 14:06:53 STP: set port 1 to discarding
May 24 14:06:55 STP: set port 1 to learning
May 24 14:06:56 STP: set port 1 to discarding
May 24 14:06:56 STP: set port 1 to learning
May 24 14:06:56 STP: set port 1 to forwarding
May 24 14:06:58 STP: set port 1 to discarding
May 24 14:06:59 STP: set port 2 to discarding
May 24 14:06:59 STP: set port 1 to learning
May 24 14:06:59 STP: set port 1 to forwarding
May 24 14:11:50 Port: link state changed to 'up' (1G) on port 12
May 24 14:11:50 STP: set port 12 to discarding
May 24 14:11:52 STP: set port 1 to discarding
May 24 14:11:52 STP: set port 12 to learning
May 24 14:11:52 STP: set port 12 to forwarding
May 24 14:11:52 STP: set port 1 to learning
May 24 14:11:52 STP: set port 1 to forwarding
May 24 14:11:55 STP: set port 2 to learning
May 24 14:11:55 STP: set port 2 to forwarding

[sirhc]

Well when you say bad weather are you talking rain?

Some thoughts to investigate:
Older firmware not v1.5.2 or v1.5.3rcX as there were fixes to STP/RSTP and the AF goes down and RSTP has a root negotiation issue. Even if this is v1.5.2 or newer this could still be an RSTP issue when the AF goes down? Disable STP and see.

Rain can also cause ground current surge/spikes and ground current can interfere with Ethernet Communications, ground current not enough to cause damage but is messing with links.

Ports speeds are hard set instead of AUTO which should always be used.

Could be the router port and not the switch at all????

"Many" things at play here, you need to investigate.

[MarkTTU]

Great thoughts. It is older firmware (currently 1.4.6) so we'll try updating to the latest firmware first and see if that helps. If that doesn't fix it we can disable STP and see if that fixes it.

I thought about the ground surges, but I'd expect to see that affect the AF or bullet radio port and not the one going to the core router. We've got pretty good surge/lightning protection at the base of the tower, but I don't think it does much for ground spikes on the shield.

All port speeds are auto.

Could possibly be the port on the router, but it's a relatively new router (our ISP just replaced their core a year ago) and if it's bad it's the only port. Still even our ISP isn't discounting that it could be on their end. We'll try the firmware upgrade first, then disabling STP, and if that still doesn't get it we'll get them to try us on a new port.

Thanks for the assist!