airFiber changes cause switch to stop passing traffic

[WisTech]

I originally thought it was something to do with a failing 5x on one end point where if I made any change within the UI or via ssh, it would cause the switch to stop passing traffic between all ports for 10 seconds or so, and then resume normally. I also noticed it was enough to reset the 11FX link uptime, although I had been working on a 5X link attached to the switch behind the 11FX.
This last week, I upgraded an 8 DC switch from 1.4.6 to 1.4.7 and the switch core locked up and needed a power cycle. Hung a 5x and replaced some older rockets while I was up there. Went to make a change on the new 5x link and bam, the main hilltop it connects to now stopped passing traffic, and reset the 11FX link uptime again.
Wondering what in the world was going on, I tried adjusting a pair of 5x links here on the roof of our office powered by a 24 port netonix, and experienced the same exact thing. Change a 5x, and bam, even my 3x to my house stopped passing traffic entirely momentarily.

I spoke with Alex in the airFiber team about this issue and he said he has never heard anything like this and would like to know what exactly is causing it. I'm just wondering what kind of packet is being passed from the airFiber to/through the Netonix that would do this. I'm honestly not sure where to start, other than possibly a tcpdump on a device connected to said switch and then try and make a change.

[sirhc]

I would turn off Flow Control on an airFIBER and airMAX AC radios or the switch port that faces them or both as they can send Pause Frame Packet Storms that packet lock the switch.

This issue has been discussed many times on this forum and UBNT forums.

[WisTech]

I've done this, but will see packet loss from clients traversing these links/switches also.

[sirhc]

I've done this, but will see packet loss from clients traversing these links/switches also.

I don't know Tony, I would have to see a network diagram, your config Tabs and so on to even guess what is going on.

[WisTech]

Let me see what I can come up with man. I hate to bug you directly on this. Thought someone else had possibly come across the exact issue. I've seen the Gig-E to FE problem, but I run FC enabled on everything to prevent any type of packet loss, except when I make a change (even to power, which doesn't drop the link).

[sirhc]

The problem with Flow Control with some wireless devices like UBNT AF and AC radios is they start sending unsolicited Pause Frames as fast as they can generate and send them (Packet Storm).

So essentially the radio tells the switch to hold the packets the radio can not accept but if the flow of packets fills up the switch buffers then all of the sudden the switch can become packet locked if that packet path is shared by other radios or other radio in the switch then send a single Pause frame it just gets ugly.

[WisTech]

Two QoS Routing Linux boxes for both sides of the network --> 24 Port Netonix --> 2X/3X/5X/11FX radios powered, each in their own VLAN

Remote end of 11FX:
3x 12 Port DC switches, ports attaching switches 1 and 2, and 2 and 3, but 3 is not physically attached to 1, except through switch 2.
Powering two 5x links, and 14 other AC and M5 access points, each AP (or group of APs) is in it's own VLAN also.

Remember, this network is fully bridged (because the boss said so), so it's a mess to work with. I have flow control enabled on all radios and ports on every Netonix on the network.

[sirhc]

Two QoS Routing Linux boxes for both sides of the network --> 24 Port Netonix --> 2X/3X/5X/11FX radios powered, each in their own VLAN

Remote end of 11FX:
3x 12 Port DC switches, ports attaching switches 1 and 2, and 2 and 3, but 3 is not physically attached to 1, except through switch 2.
Powering two 5x links, and 14 other AC and M5 access points, each AP (or group of APs) is in it's own VLAN also.

Remember, this network is fully bridged (because the boss said so), so it's a mess to work with. I have flow control enabled on all radios and ports on every Netonix on the network.

DISABLE Flow control on all airFIBER and airMAX AC radios as they have a known issue with Flow Control.

When you make a change to an AF or AC radio that break communications or modulates it down below needed capacity then the radio sends unsolicited Pause Frames at the switch as fast as it can but packets keep coming into the switch destined to that port so the switch attempts to HOLD those packets and quickly fills up the switch buffers causing a packet lock on all ports. This is NOT a bug on Netonix as it handled queued packets the same way all switches do and when buffers fill up bad things happen.

Also epmp 2000 radios I would suggest disabling Flow Control as there is an issue with that radio as well.

The switch is not really locked up and if you simply remove the cable from the port to the offending radio it will cause the switch to drop all held packets for that port as the link is down and the switch will start passing packets again.

This has been discussed many times on this forum and on the UBNT forum.

[WisTech]

I'll give it a shot again. What happens if/when you have 100M-F devices operating on the same switch fed by an airFiber radio at 1Gbps with FC disabled?

[WisTech]

I'd love to see an example of where/when it's suggested to use flow control on a wireless network. I keep researching, and cannot find a definitive answer, just 'every network is different, so your needs/recommendation may be different than another user's'.