MORE IS NOT ALWAYS BETTER: Flow Control and Routing

[mhoppes]

I experienced a very interesting issue the other day. A network that I administer is fed by a 200megabit Fiber connection. It is a primarily bridged network (although bridged with VLANS... not ten thousand devices on one broadcast domain). Even with flow control and Netonix switches in the network we occasionally see buffer overruns if the network is very very busy. For this reason, we are working on moving everything to routed at the main towers.

Generally, however, people see 35-40megabits at the end device. The network usually runs about 150-180megabits on the fiber drop. Now... one would think, HEY! If I upgrade the fiber drop to 300, 400, 500 megabits service should be even better for the end users because of more headroom right?


WRONG!!!

With the fiber running near capacity, there is only around 20-50megabits of additional overhead for "new requests" during prime time. However, with the fiber upgraded to 1 Gigabit, ALL TRAFFIC FROM THE INTERNET NOW FLIES IN AT 1 GIGABIT! This means that buffers in the switches are QUICKLY overloaded many times faster than they were with a 200megabit connection.

The end result is the end user experience is WORSE as buffers fill up faster, and everything slows down with literally millions of pause frames being generated on a switch interface per hour and over 3,000 pause frames per second on off peak hours!!!!

Screen Shot 2015-12-29 at 12.16.31 PM.png (22.7 KiB) Viewed 16641 times

[yoder]

So what's the solution? Does flow control fix this?

[mhoppes]

Well sort of. Without Flow Control the network would hardly be working. With Flow Control it mostly works. The solution is to use routers at sites to better buffer and issue TCP STOP requests to the data streams. Also, throttling at the head-end as opposed to the CPEs should help (I'm working on that too).

[sirhc]

I basically walk you through my setup in the 1.5 hr video on YouTube

My setup uses a router at all my main towers.

Smaller micro pop towers use vlans back to a main tower but every radio is on a virtual routed interface on the router via vlans from the tower router through the switch to the radio

Many people have visited my network and witnessed the speeds we provide.

There is a second video on YouTube of me doing some speed tests from a typical client location

[mhoppes]

I just thought it was interesting enough to post how increasing the head-end bandwidth actually caused a very noticeable decrease in network performance!

[adairw]

This is why I feel like head end traffic shaping is really the only way to go.
Those that disagree have small networks and..well you know what else...

Not to high jack the thread but I've been trying to resolve how our network handles this. Head end traffic shaped, all ospf routed except for small nodes off main towers (vlans) with VPLS tunnels to the core. SO the network is routed, but the customer traffic is layer 2.
When I look around at switches and routers I don't see any pause frames anywhere, most of the time.
Soo does that mean, we are good or are they wrapped inside of a VLAN or VPLS tunnel somewhere??

[adairw]

Opps. spoke too soon.
Been testing a couple different configs.
Where one would look like this
Edge > Customer block(s) router, (VLAN per /24) > traffic shaper (transparent) > MPLS router (accepts VLAN(s) in, bridged with multiple VPLS tunnels to each tower). The reverse happens at the tower, VPLS tunnel bridged with VLANs and AP ports.

The other would just be straight VLAN's all the way out.
Edge > customer router vlans > traffic shaper> switch with back haul radio's > vlans / ap's customers.

The second setup is producing pause frames on the backhaul ports.

The first setup is not producing ANY pause frames.

So it seems that the traffic is being treated much differently when it's passed around the network and VLAN layer 2 vs VPLS layer 2.

Does this make sense to anyone? I know it's probably confusing to understanding our topology.

[mike99]

Are you sure flow control is a good idea on a flat network ? I never tryed it but it's not recommended. On a flat network where pause frame affect every one, it's recommended to let tcp do is job and leave flow control off. The only time flow control is recommended is on a PTP link like Chris do, a ethernet port for every backhaul.

[sirhc]

I use flow control from AP back to router.

Watch my video on YouTube I tour my towers in service and explain.

[mike99]

Yes, I know, a router port shared for every AP on a single tower and it's seem to work better for WISP, at less with Ubnt AP, but flow-control is normally use between only a single device and the router. You normally don't want to have the router port paused to multiple device while the're only a single device congestionned. ECN is a better solution to this problem since the "pause frame" is send from end to end instead of device to device like flow-control but it's not democratized enough wet. Anyway, on a flat network, flow-control should not be enabled. Flow-control on a flat network generaly lead to