Dropping ports on new WS, what is wrong with my setup?

[tma]

It "appears" that for whatever reason the airFIBER radios start sending THOUSANDS of pause frames per second at the switch port connected to the airFIBER. It does not lock up the switch but instead it basically leaves the switch inaccessible from that port and what ever port was sending the data stream to that port as they are now paused indefinitely.

I'm not sure this is conclusive. If an AF causes a pause frame flood, these packets would be consumed by the switch (port) and not be spilled out onto other ports, which is what I had observed. Pause frames are not "switchable".

If I remember correctly, someone reported that the switch unblocked one port when another port was unplugged. In my case, it was the plugging into another port that helped. And all my 4 encounters with this problem did not have an Airfiber anywhere nearby. Well, that may indicate one of two things: (a) I'm talking about something else and only the 8 Mbps/15 Kpps flood is common to both or (b) I saw the same problem, but then it happened between two Netonixes and nothing else - which means that the AF should not be the source of the flood ... it may still be the trigger somehow.

[sirhc]

Think of it this way.

If Data is coming in Port 1 from a router and exiting Port 6 to an AF radio but the AF sends THOUSANDS of pause frames to the switch thus Port 6 will see Rx Pause Frames then when the buffer fill up Port 1 will send Tx Pause frames to the source of that stream. Also these are shared buffers so if Port 6 is in a constant state of Pause all the switch buffers will fill up and stuff just goes downhill from there fast.

The question is "WHY" is the switch receiving thousands of pause frames per second from the device which in this case seems to be predominantly airFIBER radios?

Now I am not sure what firmware Adair is using but Tony is using Beta firmware to deal with the NxN combiner which you would think they may have tweaked the Ethernet drivers to deal with 2 radios working in tandem? This is why we are requesting people to report the firmware version they are using on their AF radios and also tell us if disabling Flow Control on the switch port facing the AF radio causes the problem to go away.

Another test would be to not disable Flow Control on our switch port facing the AF but turn Flow Control OFF on the AF radio which "should" then not be able to generate Pause Frames to send to the switch unless there is a bug and when you turn Flow COntrol OFF it is really still on which would be the opposite of the previous problems with some UBNT devices where you would turn Flow Control ON but in reality it was OFF?

[tma]

Do you know if you had Flow Control enabled or not?

I can't remember exactly but I think FC must have been enabled - we were installing these switches on a new site and almost everything was done by the book ( of Chris ;-). Later, in the lab setup, the switches were at their defaults for 1.3.2 which is "FC on" I believe.

[sirhc]

Do you know if you had Flow Control enabled or not?

I can't remember exactly but I think FC must have been enabled - we were installing these switches on a new site and almost everything was done by the book ( of Chris ;-). Later, in the lab setup, the switches were at their defaults for 1.3.2 which is "FC on" I believe.

Now keep in mind Thomas one problem you were having at that time was with LAG and RSTP or in your case not wanting to use RSTP with LAG as it masked a problem and through your persistence you were able to prove that bug to us and Eric fixed it.

Just make sure not confuse that problem with this problem, similar maybe, but not the same.

[tma]

If Data is coming in Port 1 from say a router is exiting Port 6 to an AF radio and the AF sends THOUSANDS of pause frames to the switch Port 6 will see Rx Pause Frames then when the buffer fill up Port 1 will send Tx Pause frames to the source of that stream.

Okay, I see what you mean. So FC frames aren't switched by themselves but they may cause a domino effect. Would the switch consider these pause frames for the graph, though?

[sirhc]

Configuration wise, I run a complete layer 3 network. No possible vlan loops can happen. It's not a spanning-tree problem.

Example of config on a switch facing an airfiber

Code: Select all

 switchport trunk encapsulation dot1q
switchport trunk allowed vlan 13,975,976,992
 switchport mode trunk
 flowcontrol receive desired
hold-queue 512 in
 hold-queue 512 out

The bogus traffic coming in on the airfiber port happens with flow-control enabled and high peak time. I can't reproduce this problem with Rocket M5s, SAF Luminas and Ceragon microwaves units with the same switch config.

Total capacity of this link is 357/140mbps. I'm doing at most 170/40 at night, so it's not a problem with congestion.

The UBNT guys did see a lot of pause frames on the AF radios its self. I have the input/output queue increased on my switches to help with flows. Interface statistics on the switch shows dropping no packets from buffer starvation.

I've been back and forth with UBNT's engineers on this problem, still no solution.

Shutting down the port and bringing it back up or a power cycle of the AF unit will stop the bogus RX traffic and restore the normal flow.

I have a similar problem with Cisco and Airfiber units.

Having flow-control and RSTP will cause the interface to randomly lockup, shutting it down and bringing it back up is the only way to get it to work again. The interface will report 0 TX packets and lots of RX packets. Disabling flow-control & spanning-tree fixed the problem but that's not a good solution.

Perhaps the Airfiber units are sending some funky traffic when a certain situation occurs?

It's a shot in the dark but the situation looks similar. I just use Netonix for access points & PoE switches up towers when height is an issue.

Well this has been tossed around internally, we are not sure if this is us or something else "possibly" AF radios in certain configurations.

Thanks you for your input, this is the type of information we need to nail this down. Please provide as much detail as you can about your situation.

It does appear that most reported cases involve an airFIBER radio but not certain that all cases have been AF radios and then what the exact configurations were.

I would also like to point out that mayheart is able to produce pretty much the event and he is not using our switch but rather a Cisco in his lab.

[sirhc]

If Data is coming in Port 1 from say a router is exiting Port 6 to an AF radio and the AF sends THOUSANDS of pause frames to the switch Port 6 will see Rx Pause Frames then when the buffer fill up Port 1 will send Tx Pause frames to the source of that stream.

Okay, I see what you mean. So FC frames aren't switched by themselves but they may cause a domino effect. Would the switch consider these pause frames for the graph, though?

That is an excellent question Thomas and the answer I can give you is "I THINK SO"??

Since I do not see this problem I am going off of user posts and I think they say the switch graphs do in fact show the pps and traffic which makes sense as they are packets all be it very small packets with no data payload but they have been reported to be from 9K to 17K pps so that is a TON of small pause frame packets.

[Uintadave]

I found what looked like a switch lockup, but after further investigation it was a flood of about 15,000 pause frames a sec from my AF5x. Its FW is 3.2

With this many pause frames hitting my Mikrotik router it caused the switch chip on it to reset constantly. Every port in that switch group would start flapping and losing registration. as soon as I disabled FC on the Mikrotik it now worked and I could access other links and access the switch again. The only way i could get the AF5x to stop sending all the pause frames was to reboot it.

I have 2x 12 port netonix at this site and was lucky to have remote access to see this problem. if there was just one switch at the site I would have no way to access it.

This also brings up watchdog... which does not start or work if you have NTP enabled and the NTP server is unreachable....

[tma]

Now keep in mind Thomas one problem you were having at that time was with LAG and RSTP or in your case not wanting to use RSTP with LAG as it masked a problem and through your persistence you were able to prove that bug to us and Eric fixed it. Just make sure not confuse that problem with this problem, similar maybe, but not the same.

The problem that was fixed was another problem - it was easily reproducible from a simple recipe. The problem I had seen 4 times but then failed to reproduce was never solved (obviously). It looked very similar to the one in this thread: Same 8 Mbps / 15 Kpps symptom. But as said, similarity may end at this point, because in my case no AF was involved and no STP, just two Netonixes connected by one cable. The trigger may have come in from the port I was connected to, but even after I unconnected that port, they continued sending packets to each other - again: on an island of two Netonixes connected by one cable. (But when I connected to the other Nextonix, it suddenly stopped.)

Anyway, my (unfixed) problem did not reoccur since many months and because it could be that only the visible part looks alike, I'll leave it at that ... for the moment ;-)

[Uintadave]

Also as a side note there was exactly 0 pps being transmitted to the AF5x