Dropping ports on new WS, what is wrong with my setup?

[WisTech]

I am really curious to see if v1.4.0rc12 fixes this.

Please let me know.

If v1.4.0rc12 does not fix it try disabling every service not needed under the Device/Configuration Tab such as:
IGMP Snooping
Discovery
UBNT Discovery
LLDP
Cisco Discovery

Everything then if that fixes it enable 1 at a time until you find it.

But I am hoping v1.4.0rc12 fixes you up.

I've got 1.4.0rc12 loaded in both the 24 and the 6 mini on this end. The NxN radios are disabled, so now the 6 mini on the roof is only powering a single 500AC.

[sirhc]

I've got 1.4.0rc12 loaded in both the 24 and the 6 mini on this end. The NxN radios are disabled, so now the 6 mini on the roof is only powering a single 500AC.

Thanks for loading v1.4.0rc12 Tony

When you say NxN radios are you referring to the AFX 2X (2 radios)? If so that is an interesting thought.

[WisTech]

The combiners with two 5x's on a single antenna GPS synced. =)

[sirhc]

The combiners with two 5x's on a single antenna GPS synced. =)

Yea that is what I thought.

I do not think this is the problem but it is a good idea to eliminate as many possibilities as possible and narrow it down.

[adairw]

Just crashed and burned again. almost exactly 24 hours from when it did it last time (last night).
Nothing came through is syslog. I think I will upgrade all the switches before we start yanking everything.

I haven't had a chance to sit down and gather all my screen shots but I will try to do that later.

[sirhc]

Just crashed and burned again. almost exactly 24 hours from when it did it last time (last night).
Nothing came through is syslog. I think I will upgrade all the switches before we start yanking everything.

I haven't had a chance to sit down and gather all my screen shots but I will try to do that later.

Adair we want to help have you done as I asked to help narrow it down?

Upgrade to latest version v1.4.0rc12
If this does not help then swap out switch in question and disable features specified under Device/Configuration

[adairw]

Chris,
I have to. I just update updated all 8 switches last night after 1 am to 1.4.0rc12.
I'm not holding my breath that the firmware alone is going to fix it since every other switch in the network is running 1.3.9 without issue.
I also already provided that I had turned on syslog on all the switches on this leg and it didn't report ANYTHING.

Again this is a leg of the network that is six towers sites that are the most recent tower's we've built.
The problem happens with the switches circled.

Dumas has Fiber>Mikrotik router>WS12>WS6>AF5xLinktoMorton>WS12>AF5xLinktoStinnet (and so on down the line)
It used to be Fiber>Mikrotik router>WS12>AF5xLinktoMorton>WS12>AF5xLinktoStinnet (no WS6 in the middle) but the cable run was a little long and we needed to run another sector down on the other end of this elevator. The AF5X and it's cable at Dumas changed along with the WS12. Since the problem was there before the WS6 was installed than I suspect it's not the problem AND when the problem happens (16K pps and 8Mb traffic) we see it as incoming traffic to the WS6 on the port with the AF5X hooked to it.
If I was watching for the problem and noticed the traffic jump, and I was logged in to the WS6 running the AF5x and disabled the port, the WS6 and the WS12 at Dumas would not lock up.

This makes me pretty sure the problem is at Morton, as it and Dumas are the only ones that ever have to get rebooted by the netbooter's.
What I'm not sure of is if the problem is with Stinnett, blowing traffic towards Morton and Dumas. I just know it doesn't lock up.

We will focus on Morton next.
Do you still want screen shots of the switches?

[sirhc]

What would be awesome would be to know what this traffic is? Knowing what this traffic is could help. Would need to mirror port and use Wire Shark.

when the problem happens (16K pps and 8Mb traffic) we see it as incoming traffic to the WS6 on the port with the AF5X hooked to it.

Also note that Tony's "possible" issue is also related to a WS feeding a WS-6-MINI feeding (2) AF5X with NxN and RSTP. But I have MANY AF5X radios in service with no issue other than they seem to generate FAR more Ethernet CRC errors then any other piece of UBNT gear? - weird

Common between you two is WS-12/24 => WS-6 => AF5X

The only thing I can think of which is why I had you upgrade to v1.4.0rc12 was there was a fix with UBNT Discovery.

So if this happens again since you upgraded to v1.4.0rc12 I want you to go in and disable all non-essential stuff like UBNT Discovery, CDP, LLDP, IGMP Snooping (if enabled)

I think you are saying all of your other WISP Switches elsewhere are working fine so this has to be something unique to this location or configuration?

I understand your frustrated Adair, I get that, I am a WISP too and when shit happens we get anxious and very upset but we need to dissect this 1 piece at a time. If in the end we find something we are doing wrong we will fix it as fast as possible.

Now we are working on a new v1.4.0rcX release but it has more to do with I2C issues mostly for the SMART DC switches and false fan failure alerts but some other stuff as well but that is a couple days out.

[adairw]

Chris,
From my previous post where I showed that there didn't used to be a WS6 in line with the AF5X and this was still a problem.
I thought the long cable and the errors the port was taking were causing the problem. The port(s) no longer take errors now with the WS6 inline but the problem still occurs.

It's worth mentioning I attempted to capture this traffic via a port mirror and wireshark when I was on site and I didn't see anything the looked out of the ordinary. (I have the capture file if you want to see it) Just arp traffic and other traffic the PC running wireshark was generating.
What's odd is that when I had the mirror up, I didn't see equal traffic coming out of the mirror port.

For instance, I had port 3 set to mirror to port 7 where my laptop was plugged in. While port 3 saw 8Mb and 16K pps, port 7 did not. It's like the switch was just eating the traffic..

There is nothing different about these site configurations than any other site we have. I've gone through and verified the switches and router configs multiple times.

And yes, we are a little frustrated with the issue. It's obviously customer affecting as there is 8-10 minutes of down time every time they lock up, reboot and all the links reestablish.

The bigger part of my frustration was coming from your attitude towards the issue, honestly. You can't just throw up a wall and say we have 50,000 switches out there, this isn't a problem! When clearly there is. I appreciate you taking the time to work with us on this finally. I'm not trying to sound ungrateful. But we need a solution.

[sirhc]

Adair,

My attitude is there has to be something with this site that is differnt that is causing the issue and we need to find out what it is. By your own words this is setup like all the rest of your network. You have how many switches that do NOT have an issue and they are setup the same?

There is nothing different about these site configurations than any other site we have. I've gone through and verified the switches and router configs multiple times.

If your setup is the exact same everywhere else and this is the only place your having issues then something is amiss:
Maybe it is a bad switch which I doubt as you say the traffic is coming into the switch but I asked you to swap it out to rule that out.
Maybe it is a bad AF5X that is spitting out garbage? Maybe it is not the one on this side but the other one sending that unexpected traffic across the link?

Maybe it is another piece of the puzzle like a router somewhere or whatever?

When you see this traffic you can not explain hitting the switch did you log into the AF5X radio and see if it is showing the traffic coming across from the other side of the link?

Did you look at the switch on the other side of the link and see if the traffic was leaving that switch to the AF5X over there?

This garbage traffic is coming from somewhere you would think?

when the problem happens (16K pps and 8Mb traffic) we see it as incoming traffic to the WS6 on the port with the AF5X hooked to it.

If we can determine what is causing the issue we can address it. - If it is indeed our problem, as this has yet to be determined?

My statement was and is that if this was an inherent hardware or firmware issue you would think many more people would see it including you at all your other sites which you say are setup the same way.

There is nothing different about these site configurations than any other site we have. I've gone through and verified the switches and router configs multiple times.

Now to be clear so we are both on the same page, this is the only place you are having this issue, none of your other sites that are setup the same way are having this issue?

Now with your explanation of the the MIRROR there are some types of traffic that do not get mirrored such as BPDU packets and some types of broadcast packets. The only way to get those would be to dig up an old HUB (not a switch) and plug that in the middle along with your laptop and WireShark and capture it that way.