Flow Control..... take 2, 3, 4?

[mhoppes]

Further,
It appears that the backhaul device (microwave) must support pause frames/flow control, or else the flow control doesn't do anything. If this is true, this is very bad for many wireless networks.... and is a good reason to use something like SAF or airFiber for your backhaul radios!

[sirhc]

There are two sides to the Flow Control function which we could implement separately per port but for now we just combine them into 1 option to turn both on.

The 2 behaviors are "obey" and "generate"
Obey means if it is told to pause it will pause transmissions for 50 microseconds.
Generate means that if it's own buffers are about to be full to generate a pause frame in the direction of the offending flow.

Enabled FC on the airFibers and I'm seeing nothing incrementing on the switches.

I am not sure how your setup is so let me guess

(Internet<===>Port1[1G]-Router-Port3[1G]<===>[1G]AF5~~~AF5[1G]<===>[1G]Port1-Switch-Port5[100M]<===>AP~~~CPE

In this case traffic backing up on switch Port5[100M] will fill it's buffers so the switch should be sending a Pause Frame from Port1 to the AF5 which instructs the AF5 to stop for 50 microseconds which will of course overflow that AF5 Ethernet port buffers pretty quickly.

What I do not know is how the AF5 handles Pause Frames beyond that, 3 possible scenarios below:

1) The AF5 receives a Pause frame on it's Ethernet port connected to switch Port1 so it pauses 50 microseconds which fills it's buffers up quick so the wireless side sends a pause frame across the wireless links to tell the other radio to pause 50 microseconds and then in turn it fills up it's buffers and then in turn sends a Pause Frame out it's Ethernet port to the router port1 telling it to Pause????

2) The AF5 radio receive the pause frame on it's Ethernet port from the switch and transparently passes the pause frame to the far side and issue the pause frame to the routers port2 feeding the AF5 on the far end? <= THIS IS HOW IT SHOULD BE HANDLED - I WOULD ASK CHICAGO HOW THEY DEAL WITH IT.

3) None of the above?

See the wireless side and the Ethernet side of a radio are normally 2 ports that are bridged but with the airFIBER I am not sure how it is? THen it comes down to if each side of the radio has it's own buffers and how do they handle pause frames and so on and so on.

So the answer is I do not know how the airFIBER handles Pause Frames. Since I have a router at all of my towers so that pause frames make it to a routed way point where they can be effectively handled with normal TCP mechanisms slowing down independent streams as needed not pausing the entire pipe.

Now do not get me wrong a star topology as shown above will work "if" the backhaul handles Pause Frames in either way described above (PREFERABLY METHOD 2)

[mhoppes]

And that's where the Cisco switches appear to be fundamentally broken. They DO NOT send Pause Frames... I'm in a heated conversation (polite but heated) with a Cisco tech right now about this.

Cisco switches will RECEIVE but not TRANSMIT FC pause frames.

Apparently the solution at this point looks like use Netonix switches and Cisco routers.... *sigh*.

[mhoppes]

But yes, that's pretty much how the network is setup, except it's AF24s... but with that setup the Cisco is never receiving pause frames.

[mhoppes]

So I may have muddied the water here... There are multiple networks at play here (I do consulting for people). The first network was fixed by turning flow control on on Netonix switches and airFibers.

The second network is running Cisco switches with airFibers and turning flow-control on on the airFiber and flow-control receive on on the Ciscos yields nothing useful and no counter increments. Speeds are still bad.

When I said "Cisco" I meant the Cisco ME switches.

I'm thinking we may have to swap a $1,500 ME switch for a $500 Netonix because the Cisco is fundamentally not able to function properly in a microwave environment! 8-O

[sirhc]

So yea, here is one of my tower switches port details, now keep in mind that I STATIC LAG the switch to the Cisco Router so this is but ONE of TWO ports feeding the AP's on that tower. I do this to spread the pause frames across 2 ports. That way the number of people being affected by or causing Pause Frames is cut in half!

Note the number of Tx Pause Frames being sent back to the router, these pause frames are being generated because the ports feeding APs on this tower have their buffers full quite often.

CLICK IMAGE BELOW TO SEE FULL SIZE

avatar.jpg (215.98 KiB) Viewed 9774 times

[sirhc]

You know, this thread and the following two threads should be mandatory reading for new WISPs:
CLICK LINK => A COMPLETE HOW TO ON GOOD GROUND PRACTICES
CLICK LINK => A GOOD SUGGESTION ON HOW TO SETUP TOWERS (ROUTER / SWITCHES / VLANs / LAGs)

There are a ton of Threads about the benefits of RF Armor over on their forums somewhere.
And tons of Posts I made on Proper Jumper Treatments to prevent "Standing Wave"
And then the posts that Justin made like the O-Tard.
And your posts.
You know there is a wealth of knowledge that the "Community" provided over the years, UBNT should have hired someone to do nothing but compile the information into an organized How To.

Do you think UBNT would allow me to copy these threads over to their forums? These threads would be of a good benefit to the community but a lot of people to not even know this forum exists.

If we ever get big and could afford to pay someone I would do that.

[mhoppes]

Chris,
So have you completed removed all Cisco switches in your network at this point?

[sirhc]

No Cisco switches in service anywhere in my network!

Cisco 2951 Routers at every tower and 2 Cisco Routers at head end, does that count?

If UBNT would add BFD and Flow Control to their routers I might switch to their routers?

[mhoppes]

This stinks... big time.