Dropping ports on new WS, what is wrong with my setup?

[adairw]

Oooh. and I also remember that I had another switch in a whole different part of the network lock up about a month ago.

Power cycle would fix it for a few minutes but it would lock right back up.
The site had a DSL line adapter plugged in to it feeding a customer in another building. It wasn't until I unplugged the DSL adapter that the switch would reboot and be ok. As soon as I would plug the ethernet back in it would lock up again. Once I power cycled the DSL adapter life was good.

At what point did STP start being turned on by default? I just looked and it was on, on this switch. That's annoying.

[nickwhite]

The most recent example of this is a leg of our network with no possible loops. There is a redundant link (which had spanning tree running and was disabled), plus a link back between #1 and #21, but it was also disabled. The fiber link in the drawing was also down during this.

This started Friday night at #5 in the drawing. First thing we did was swap the switch (WS-12-250-AC) with a new one on Saturday morning at #5. It ran for anywhere from 45 minutes to several hours. I turned on RSTP on all switches in case there was a loop or something odd going on - spanning tree at #10 started blocking towards #5. Finally had a guy onsite last night while it was happening (for the 4th time over the weekend) and had him start pulling cables one at a time out of the Netonix. It stopped randomly on a RocketM5 AP with 14 clients. Plugged it back in and no other issues. We did this on Saturday as well, and it started passing when we unplugged the backhaul to #91 at #5. I turned on ping watchdog on the switch at #5 to ping back to our core and reboot almost everything when it stops responding - not a solution, but it was me from dispatching a guy.

Some things to note in case they matter:
- We don't run Discovery on our switches. We're working on disabling it on our radios as well
- The strange 8Mbps TX/RX, lots of PPS issue started happening on our Router port at #5. Bouncing the SFP remotely didn't help (screenshot below)
- We had an issue a few weeks ago where all UBNT AC radios in AP mode on the same management VLAN that had Discovery turned on basically crashed within 15 minutes of each other... UBNT. Had to do with the infctld process and discovery.
- #5 is a very RF noisy POP... several towers up there that range from TV, Radio, other WISPs, First Responders, etc...
- RSTP was initially only on between #91 and #22 for the backup link (#22 is solar, and we shut the AF5 primary off when it drops below a certain input voltage).
- This first started happening about 6 months ago.

[nickwhite]

One other thing to note:

About a month ago, we had this happen on the opposite side of the network. When it happened, it was an AF24 and AF5 link on the same Netonix switch showing a flat-lined 15Mbps of RX traffic each. I was able to get into the radios from the other side, and they showed effectively zero traffic (about 30Kbps of management traffic from being logged in is all).

[nickwhite]

What would be awesome would be to know what this traffic is? Knowing what this traffic is could help. Would need to mirror port and use Wire Shark.

I actually tried this once. I mirrored the port to a second port on a Mikrotik running packet sniffer. It reported almost zero packets... Really bizarre.

[nickwhite]

My pop #5 is doing this right now again. Here's what the port on #10 shows. It's actually showing STP blocking on Port 2 (towards #5), but still shows 17Mbps of RX traffic.

[nickwhite]

And apparently the AF24 on Port 4 is doing the same thing... to a different POP. Very strange. No STP blocking on that port though.

And port 5 as well which is a Powerbeam M5

[Dave]

arrrggg..we are discussing issue internally...it is so hard to fix when we can't replicate issue ourselves so it is hard to determine if it is us directly, or something on network we are not expecting....

[adairw]

Thanks Dave. I know problems like this can be hard to track down. Everyone does everything different.

[Eric Stern]

My pop #5 is doing this right now again. Here's what the port on #10 shows. It's actually showing STP blocking on Port 2 (towards #5), but still shows 17Mbps of RX traffic.

Note that the TX rate on port 2 is 0, because STP is blocking. So the AF is blasting 17Mbps at the switch even though its never getting any traffic back.

[adairw]

My pop #5 is doing this right now again. Here's what the port on #10 shows. It's actually showing STP blocking on Port 2 (towards #5), but still shows 17Mbps of RX traffic.

Note that the TX rate on port 2 is 0, because STP is blocking. So the AF is blasting 17Mbps at the switch even though its never getting any traffic back.

So it appears. but I'm not sure that's what is really happening.
When we saw this the radio showed no traffic, even when the switch port did.