LACP goes down briefly when radio loses power?

[sbyrd]

If you leave it on and have Pause Frame Storm Protection Enabled it will protect you for the most part.

That is for the most part true, but even with it enabled in my situation it was killing traffic for all the other backhauls if one of the slaves went down and was interfering with the LAG. It did not seem that the switch Pause protection was sufficient or fast enough in this specific circumstance to stop it.

[sbyrd]

Here is the latest explanation from UBNT-Chuck on this issue for those that care.

Hi,

In general, flow control is a good thing if you have a network that is operating normally and your purpose is to regulate the interarrival time of packets. It allows devices (such as airFiber) to minimize the size of their hardware buffers and minimize buffer bloat.

Flow control can cause issues when you have devices that:
never dump packets operate in large layer 2 only networks operate in overloaded condition
As a example, say you are using airFiber radios that are connected directly router ports. Because routers are designed to drop packets when they become overloaded, Ethernet flow control will allow the airFiber radio to regulate in inbound packet interarrival times without affecting the remainder of the network. Any overload condition will be contained in the router. Ultimately, if packets are dropped by the router, the associated streams TCP, RTP, etc will modulate their traffic load on the system to balance out the networks operation. Only the streams carried by the overloaded link are affected.

In contrast, let's consider two new systems where a layer 2 network and several airFiber radios are connected to a single switch:
System 1: All switch ports are configured to generate and respond to flow control packets System 2: All switch ports are configured to only respond to flow control packets
System 1 is the classic layer 2 flow control issue. Say you have a main feed into this switch (P1) that then switches traffic to the airFiber links. If one airFiber link goes into overload (a heavily loaded link moves to a lower modulation rate or stops passing traffic), it will send flow control BDPUs to the switch. Once the switch buffer fills, the switch will send pause BDPUs out the upstream port (P1) which has the effect of flow controlling all data going to ALL airFiber links from P1. Not good.

System 2 does not have this issue. Since it is only responding to flow control requests, when it overflows it will start to dump packets as they are queued to the destination port (the overloaded airFiber). This effectively contains the overload condition to the overloaded airFiber radio.

As currently designed, airFiber does not drop packets. As a result, if you are using flow control, it is important to make sure that there are devices that will drop packets between the airFiber and any potential layer 2 bottleneck (P1 in the case of our example). We are considering design changes to allow airFiber to drop packets when an overload is detected. I do not have a ETA on that feature, but, we are actively looking into it.

This may be an over simplification, but, I hope it makes things clearer.

Chuck

[sbyrd]

Sirhc - Thoughts on what UBNT-Chuck told me for my setup.

What I said.

I am still slightly confused. Besides turning flow control completely off, the way to eliminate the issue in my setup where an AF link goes offline (remote side) and the Master (local side) sends 60kpps (30Mbps) of pause frames to the switch which then interupts all traffic on the switch and temporarily breaks the dynamic LAG (uplink to router) would be to:
1. Set every AF facing port on the switch to Obey pause frames only. Not Generate.
2. Set the router facing LAG ports on the switch to Obey pause frames only. Not Generate
or
1. Set every AF facing port on the switch to Generate pause frames only. Not Obey.
2. Set the router facing LAG ports on th switch to Obey pause frames only. Not Generate

All connections are 1G on the switch.

Chuck's response:

Hi,
I think the simplest answer to this is that you should set all ports on your switch to obey flow control requests but to never generate flow control. Depending upon the network topology, there may be other options but this is the best one-size-fits-all advice.
Doing this allows the switch to do some buffering for the radios but ensures that the switch will drop packets when needed (rather than flow controlling the source).
Chuck

[sirhc]

Either set the port facing the AF to "Generate" or "OFF" but this event will rarely or never happen where the switch sends a Pause Frame.
The ports facing the LAG I would leave as "BOTH"

Or you can try what Chuck suggested, it has merit.

Hi,
I think the simplest answer to this is that you should set all ports on your switch to obey flow control requests but to never generate flow control. Depending upon the network topology, there may be other options but this is the best one-size-fits-all advice.
Doing this allows the switch to do some buffering for the radios but ensures that the switch will drop packets when needed (rather than flow controlling the source).
Chuck

Ideally they need to fix their firmware to be aware of the link state changes and act accordingly as far as generating "solicited" pause frames.

[sbyrd]

Why such a discrepancy between what you say is best in regards to flow control and what Chuck is saying?

I seem to recall in the past you said that flow control should be on every link to a routed interface where then the normal L3 control mechanisms can take over, but Chuck is saying to isolate flow control to just the AF facing port and not take it all the way back to the router when you have multiple devices connected to an L2 switch.

[sirhc]

Why such a discrepancy between what you say is best in regards to flow control and what Chuck is saying?

I seem to recall in the past you said that flow control should be on every link to a routed interface where then the normal L3 control mechanisms can take over, but Chuck is saying to isolate flow control to just the AF facing port and not take it all the way back to the router when you have multiple devices connected to an L2 switch.

Seth, you are taking select bits and piece of what I said to make your argument which is not correct.

Depending on your setup (topography) Flow Control is GOOD to use, especially when you have a router at every tower and not a large flat network.

Also if you take care as to not have backhaul and uplink ports combined, this is achieved with creating midspan injectors with the switch as seen in my video.

Go back and watch my 1.5 hour video and re-read my posts on flow control. I am not going to type out an hour long post to recap what I have said already, sorry.

What Chuck is saying has merit as far as his suggestions to deal with their Flow Control Pause Frame Storm (WHICH SHOULD NOT HAPPEN) is all I said.

You could do what he suggested and/or try the other option I gave you and see what works best for you.

But personally I would like to see them FIX this issue with their firmware so as not to send millions of unsolicited Pause Frames when their buffer is full because a modulation rate changed drastically reducing its capacity or their link dropped.
Stupid to send pause frames when the link is DOWN, they should DROP the packets. THis is what is causing you the most problems at this point.

This is not Rocket Science and if you sit back and think about or even draw it out on a chalkboard you will understand what is happening.

Picture the switch in your case.

Port 1,2 is the uplink for all ports with an LACP LAG

Port 3,4,5 are PTP links

If port 3 get slammed with Pause Frames and packet are still coming in port 1 or 2 destined to Port 3 they get held in the switch buffers until they fill up and then the switch says I am not allowed to drop them so I need to tell what ever is sending the packets in Port 1 or 2 to Pause so it send Tx Pause Frames out port 1 or 2 but WAIT A MINUTE, when it do this all links including ports 4,5 no longer get data and the switch also no longer is getting data on either port 1 or 2. Port 1 and 2 belong to a LACP lag so the absence of the BPDU packets which are also paused cause the LACP to drop that link from the LACP LAG.
JUST THiNK ABOUT WHAT THE CAUSE AND EFFECTS ARE.

Instead of trying to ask me what is best for your network try to understand what is going on. It may take you awhile but they you will have a better understanding of what is going on and how to build your topography.

We do NOT have a bug in the way we handle Flow Control or LACP.
You know what the AF will do when the link is under needed capacity from modulating down, simply sending ti too much data to pass, or the link drops.

Now draw it out on a board or in your head and wrap your mind around what is happening. It really is not to hard to figure this out and understand it.

[Andy Hanson]

My apologizes if this has already been answered in a different post but I am currently having the same issues I have tried my air fiber on none both obey and generate and nothing has really worked well for me I have three tower sights with air fiber back hauls. You explain having a router at each tower site I currently have a peplink at my main site thats feed by an upstream fiber then from there it goes into a netonix switch and powers my rockets and air fiber then deploys out to my other two sites which both have the netonix switch. Is it better to have a router before the netonix? Just trying to figure whats best for flow control we have done a ton of upgrades to handle more speeds but seems to lag at the switch.