Dropping ports on new WS, what is wrong with my setup?

[nickwhite]

My pop #5 is doing this right now again. Here's what the port on #10 shows. It's actually showing STP blocking on Port 2 (towards #5), but still shows 17Mbps of RX traffic.

Note that the TX rate on port 2 is 0, because STP is blocking. So the AF is blasting 17Mbps at the switch even though its never getting any traffic back.

So it appears. but I'm not sure that's what is really happening.
When we saw this the radio showed no traffic, even when the switch port did.

Ditto. As I noted a few posts up, I was able to get into an AF24 on a different occurrence of this and it showed zero traffic, even though the switch port showed a ton.

[nickwhite]

I have a tech heading onsite to #5 now. If anyone has anything they want me to run or check, speak now.

I can get remote access as well if anyone is available from Netonix and wants to investigate.

[adairw]

I have a tech heading onsite to #5 now. If anyone has anything they want me to run or check, speak now.

I can get remote access as well if anyone is available from Netonix and wants to investigate.

I'm not from netonix, but I'd be interested to see if the log says anything useful. also detail on what brings it back to life.

like, all you had to do was unplug this one cable and everything else started working, reboot or not, etc.

[nickwhite]

Yesterday when it was happening, I had my tech try getting in on two of the Untagged management ports with his laptop and he couldn't. So we resorted to unplugging things. LOL. I don't know that unplugging that one AP actually "fixed it" or that that is the root cause. I suspect that port is also seeing the high packets and unplugging triggers it to start passing. I'm seeing this on the #10 switch right now... two of the other backhauls exhibited this behavior and rebooting both of them and turning RSTP back on Port 2 "fixed" them.

I think I'll have my tech connect to one of those ports and run wireshark for a few minutes before we reboot anything.


Side Request: Any chance we can get better debug/logging options for Spanning Tree? Like being able to see the MAC that is being detected when there is a loop.

[nickwhite]

Tech is onsite... I was able to enable the backup link between #1 and #21 and get into #5. Ports 1 (AF24 to #10), Port 12 (SFP to Router), and then Port 4 that my tech plugged his laptop into are all maxed at 8Mbps TX. I'm having him download and run wireshark now.


EDIT: He was plugged into Port 4, and after unplugging, it was still showing TX of 150Kbps and 300PPS... wtf? Port showed down.

[nickwhite]

Still going through the wireshark, but we got 9500+ packets in 80 seconds. And 99% of it is ARP it looks like... still going through it.

EDIT:
Looks like whatever weirdness was showing on the interface isn't actually making it to his laptop... He just ran another wireshark for about the same length of time and got about 10,000 packets. No weirdness on the switch though during this one... we rebooted the switch before this second capture.

So wireshark isn't going to solve this one it looks like. My guess is maybe console cable when it's happening to get in to the cli/cmdline.

[nickwhite]

This switch is running 1.4.0rc12 BTW and I have broadcast and multicast storm set really low - 64.

I'm wondering if rolling back to 1.3.8 would help.

[nickwhite]

I'm not from netonix, but I'd be interested to see if the log says anything useful. also detail on what brings it back to life.

like, all you had to do was unplug this one cable and everything else started working, reboot or not, etc.

The only thing interesting in the logs was that STP started discarding on Port 2 (BH to #91) then went to forwarding. Then Port 1 (BH to #10) started discarding. That's when it went down. Other than that, there was nothing odd.

[nickwhite]

This just happened again a few minutes ago. I bounced the AF24 at #10, turned up the backup between #1 and #21 and then downgraded the switch at #5. when I got into the switch at #5, it was flatlined on the port for the Router and the AF24 to #10. Downgrading it to 1.3.8

[nickwhite]

And it just happened again. That was on 1.3.8, rolling it back to 1.4.0rc12.

Any thoughts? Anything I can do? This is happening pretty frequently, so if you guys want access to troubleshoot this, I can provide it.

It did look like the CPU was running at 98% the last two times this was happening. I didn't catch what process was taxing it though. I will try and catch it the next time.