An after, with a before this time.

[adairw]

I'm almost ashamed to post this. This was our very first tower before we were a business and just some ham radio guys playing around with wireless.. Since then it's been added to and added to and never any time was spent on keeping it clean. Now we do much nice work and keep things clean.

So here is the after pic.

Osage better.jpg (293.51 KiB) Viewed 21912 times

And a couple of before pics....

Osage Crap 2.JPG (723.25 KiB) Viewed 21912 times

I have more I want to do to manage the cables better. Namely I'm going to add rack mount surge suppression to terminate all the out door cables to and just use short patch cables to the switch. Another day. :)

Here are some of how it's configured

osage config 1.PNG (127.06 KiB) Viewed 21912 times

osage config 2.PNG (85.81 KiB) Viewed 21912 times

[sirhc]

Looking good Adair, big difference, the before an after pictures. As WISPs we all have our cable snakes in some closet somewhere.

How is the new v1.1.0rc18 doing for you?

Do you have Flow Control turned of in your switch?

I have found though extensive testing that when I feed airMAX AP's I turn FC on the 100M ports to the radios and the 1G port feeding those ports from the router at that location (I run a routed network).

Those with flat networks would want to turn FC on all the way back to their routers which means if fed from an airFIBER you would turn FC on the 1G port in the WS from the AF, turn FC on both AF units and turn FC all the way back to the router.

Flow control is a mess and I hate the concept which is why Rory and Tom will be playing with some other option for the WS as well as our routers and we will post those results and possible other solutions as we figure them out but for now people should try turning FC on and testing with and without it to see what works best for their configuration.

Thanks for posting!

[adairw]

Thanks Chris.

I am not running flow control on any ports right now. I might tick it on and just see if we notice anything.
These AP's only push 25-30Mb (10mhz channels) and segmented back to the tower router so I never thought about it because they are not close to passing 100Mb.

[sirhc]

That is what I thought so I played with Flow Control and saw a very good improvement. Now I am lucky that at my house I connect like all my other customers into a shared AP. This allowed me to stay up late at night and play with the network all the way back to the fiber to find the best configuration.

Since we run a routed network with a router in each tower I found that just turning FC on in the tower switch on the 1G port(s) from the router to the switch and the 100M ports to each AP was the best configuration for my network. If I turn FC off my downloads vary depending on AP load between 10 and 30 but with flow control on I get between 50 and 90. Now keep in mind I am on a 30MHz wide channel airMAX M5 using a pretty clean channel with only 4-5 miles to tower.


Trust me Rory HATES FC and really so do I but if it works better I will use it until we find a better solution.

I do not run FC on my AF24 back hauls between routers but turning it on had no ill effect it just did not cause any pause frames to occur on those interfaces anyway so I just leave it off on those ports.

We had done LAG trunking and ran our back hauls and AP through the same LAG on the switches to the router but found that since we used FC on the AP ports and the LAG ports to the router for better AP throughput that those pause frames caused by the 100M ports on the 1G ports to the tower router was a bad idea so we are now using the WS as a mid-span injector for the airFIBERS and the only thing going though the LAG is the AP's and private PTP commercial links. Also since the AP's go through a LAG to get to the router the pause frames caused but the airMAX 100M devices get divided up across the three ports in the Static LAG which means less 50 micro second pauses per port. It has been WORKING GREAT!

[adairw]

So you're basically running FC on all ports? Trunks and AP's?

[sirhc]

So you're basically running FC on all ports? Trunks and AP's?

Yes.

SO here is what happens.

Packets go rushing to the AP and not all from one stream (user).

The AP gets stuck with a bad connection and gets delayed or even the fact that the wireless link for all intensive purposes is seen by the network as a 100M FD link but in reality is at best with a 20 Mhz wide channel 70 Mb HD with tons of retrys and errors even under the best circumstances.

So the AP gets over run by packets and fills it's own buffers so it wants to issue a pause frame to the Ethernet port that is feeding it.

That port also only has so many buffers so when it gets full it wants to issue a pause frame back to the uplink port that is feeding the switch.

The uplink port buffers get full so it issues a pause frame to the router.

Flow control works OK when each port has one user or device but the AP is a bridge to MANY devices and many different streams of packets, some small and very numerous.

TCP has mechanisms to deal with congestion but the problem is they all work around a destination device telling the sending device to slow down in the event the destination can not handle the packets for what ever reason. This is broken for WISPs because the bottle next is usually the AP not the final destination so therefor the mechanisms built into TCP to combat this become broken or ineffective as the final destination device (the customer computer or router) is not under any distress handling the incoming packets so no state of distress is ever sent to the sending device to slow down and the AP is just a simple bridge with no way of acting as a proxy for the final destination device. It would be so great if the airOS would act as proxy for its clients and interject for them to the sending device to slow down and thus no pause frames would need to be sent.

Pause Frames SUCK because they penalize the entire interface not just the stream to the device struggling to get packets from a wireless connection but for now I have found that FC is better then no Flow Control. It is treating the symptoms and not the cause but it is the best solution FOR NOW.

Now 1G Ethernet ports have far more buffers than 100M ports which is a GREAT help, as it is not so much volume but timing that gets messed up when a HD wireless bridge is in the path.

I have recently came up with an idea that I want to discuss with my guys Rory and Tom where we put the AP in routing mode (NOT NAT) this way the AP may be able to better use the mechanisms built into TCP to deal with congestion as it becomes a destination within the chain. As I said it is just an IDEA, could be crappy idea but I want to explore it.

[adairw]

Opps, I lied. I am running flow control since I guess it's on by default on all ports? I was thinking it wasn't...

I just layered on Port isolation, see how that goes.

[sirhc]

Opps, I lied. I am running flow control since I guess it's on by default on all ports? I was thinking it wasn't...

I just layered on Port isolation, see how that goes.

Well Flow Control is on by default in some of the 1.1.0 versions but originally it was not.

If you upgrade a switch that originally had a firmware with Flow Control disabled by default then it will still be off unless you turn it on for each port you want it on or default the switch with a newer firmware version that has it on by default.

People should check and experiment what works best for them.

Rory also has some ideas with some features that the switch core offers that we are not using but I have asked him to concentrate on the existing feature set currently implemented until we have a STABLE FINAL RELEASE of v1.1.0 then I will allocate time for him to play with other features.

[mhoppes]

Chris,
I'm still trying to wrap my head around FC. I've got it wrapped... but I don't really understand how using it makes anything work better in high-throughput networks like we have.

If a pause frame gets sent, doesn't it back everything up to the last point of contention? (e.g. your router?)

So wouldn't this cause your entire tower to breathe when one client on one AP gets backed up?

[sirhc]

Chris,
I'm still trying to wrap my head around FC. I've got it wrapped... but I don't really understand how using it makes anything work better in high-throughput networks like we have.

If a pause frame gets sent, doesn't it back everything up to the last point of contention? (e.g. your router?)

So wouldn't this cause your entire tower to breathe when one client on one AP gets backed up?

Pause Frames SUCK because they penalize the entire interface not just the stream to the device struggling to get packets from a wireless connection but for now I have found that FC is better then no Flow Control. It is treating the symptoms and not the cause but it is the best solution FOR NOW.

As I said they SUCK but the alternative is 100M interface buffers are small and fill up quickly which throws the timing off which causes dropped packets which cause re transmissions which feeds itself like trying to put out a fire with gasoline.

I think I made a pretty long and somewhat detailed post on what is happening?

I explained that pause frames work their way back your FLAT network until a point of destination which is routed which in my case is my tower router. Once a point of destination or routed way point is reached between the original source and final destination is issued a pause frame that can then use the TCP mechanisms to slow down the appropriate stream(s).

Flow Control is "my" best solution between the AP and a routed device (my flat segment). Once the router is reached it can use TCP mechanisms that can deal with packet contention.

I do not run a flat network so I only have to back up pause frames to the tower router whereas you have to push it back over a back haul but then into a router so you need to run FC all the way back to your router.

But as I also said everyone should test with it on and off and see what works best for their network.

With my network I have a router at each tower so I like

WWW====Router=====airFIBER======Router====FC====SWITCH====FC====AP====CPE


With your network which is a Flat Network I would try this

WWW====Router===FC====airFIBER====FC=====SWITCH====FC====AP======CPE

In my opinion pause frames are better then dropping packets which is what happens when you run out of buffers on flat segments of your network.

Routers deal with packet over flows differently as they can use TCP mechanisms to slow down individual stream but a layer 2 network or flat does not have the ability to do so.