Flow Control..... take 2, 3, 4?

[petecarlson]

I would think that the "pop over" would be the same as the cable diag "pop over" which works fine... I'll have my web guy take a look at it and figure out what is going on.

Back to the subject at hand. I'm interested in thoughts on flow control when terminating at a MPLS VLPS circuit. I am moving from a fully routed network to a MPLS VPLS core. I n the following example, what happens with TX Pause?

OSPF Edge Router <--> MPLS VPLS circuit across core <--> Netonix SW <--> AF24 <--> Customer OSPF router.

Are the pause frames transmitted on each VLAN so that they would travel transparently across the VPLS circuit?

[sirhc]

Pause frames are sent to the device connected on that port. I do not think pause frames are encapsulated, or VLAN aware, they are on the physical layer.

However you are aware that as I understand it UBNT EDGE Routers do not "currently" support Pause Frames which I find a little silly as the idea is to use Pause Frames [YUK] to a routed way point where better TCP congestion mechanisms can come into play.

I hope UBNT fixes/changes this soon as a lot of WISPs use their routers these days.

Look I do not like Pause Frames but at this point in the game they are needed in the wireless industry and if the manufacturer implements it correctly it works a whole lot better better than no Flow Control.

[ste]

Hi sirc,

I'm lookin into flow control now to clean up some hickups in our network. Mainly customers who do not get the rate they should without no apparent reason. Talked to our backhaul supplier (SAF) to hear their approach implementing FC.

Some of your statements sound not appropriate in this light:

As a packet is sent from 1 router to the next router that router acknowledges the packet was received OK and sends a small packet back requesting the next packet. This is the normal TCP mechanism that deals with how many packets from a fat pipe can fit into a small pipe.

TCP works end to end. The TCP/IP stacks at both ends do acknowledgment and sliding window handling. The routers in between never acknowledge packets if they are not the source or destination of the communication.


Your approach is to enable FC in the whole Layer2 network up to the routers.

This can lead to buffer bloats as you blow up buffers along the path back to the router and then send packets out. So latencys might go up and fluctuate on a congested network and voip may suffer from this. FC does not take into account the TOS fields of the packets so it stops all packets not only bulk.

In my eyes and that is what the SAF technician recommended to me is to use FC only to overcome buffer overflows from bursts and drop packets as soon as the overflow condition is longer. You have to start dropping packet at some time when a link is overloaded anyway.

So his recommendation is to enable FC on the ethernet side of the link and on the directly connected switch (which should have a reasonable sized buffer space 2MB) and turn off FC further into the network. This is for Luminas which only have 128KByte Buffers in their internal switches. With Integras the size of the buffers is configurable and much higher. There FC is not neccesary. As they can handle longer bursts on their own.


Regards,
Stefan

[sirhc]

Yes there is a condition called "Buffer Bloat" which Matt Hoppes and I have discussed on numerous occasions but me saying you need Flow Control from your AP back to a routed point does NOT cause buffer bloat in itself.

Buffer Bloat is caused by too large of buffers, most often from really high end data center switches and telecommunications devices with enough memory to hold a full second (time) or sometimes much more of data per port which is the main culprit of buffer bloat. Yes Buffer Bloat can also occur from the accumulative effect of buffers of many devices in a flat layer 2 network but I also do not feel this is the case either that WISP's are experiencing. There is a good article or 2 on Wikipedia about Buffer Bloat and there has been a consortium tackling the issue for more then a decade but I do NOT feel that what WISPs are experiencing is a symptom of Buffer Bloat.

I agree with you that TCP is an end to end point communication protocol, and no, the routed way points (routers in the middle of the route) do not talk to the the endpoints or have any influence in that aspect and yes it is the responsibility of the end points to determine if a "dropped" packet is to be resent, that is not what I am saying, maybe I am doing a poor job of expressing what I mean.

But as far as you saying that no communications exists between routers along a route I do not agree with. I need to go do some reading/homework and sharpen my pencil but what I am referring to is a router receives a packet from one router that it must send to the next router in the chain/route (store and forward). There is communications between those routers in the handling of that packet. If router B received the packet OK from router A and there was no CRC or other issues it stores that packet until it can deal with it and then when router B is ready to forward it to router C and router C says I am ready to receive it the packet is forwarded but if router C gets a CRC on the packet transmission from router B that packet is just not dropped as router B is still holding that packet and will attempt to resend that packet to router C. It is this inter router communications that I am talking about.

Router A does not send packets to router B until router B says I am ready to receive a packet and then router B inspects the packet to insure what it received is correct. Also at this point if the packet TTL is hop oriented it reduces the value by 1 or if the TTL is time based it inspects it and if either condition warrants it then the packet is dropped. This dropped packet is an example of a packet the the endpoint determines if it is re-sent or not. Packet transmission from one router to another that results in a dropped packet do to a CRC or other type error can be resent from that router to the next without talking to either endpoints. Packets that are dropped from buffer over flows are also not resent by the sending router but would require the end points to issue the need for it to be resent. This is one reason in wireless topologies like WISP's having dropped packets in the middle due to buffer over flows are really bad. This is also where your window you mentioned comes into play in a bad way as it may require more than just the single packet that was dropped to be resent. Also if this is TCP verses UDP the receiver(destination) may have to hold good packets that are now out of sequence until the missing packets that were dropped make it to the final destination.

There is a whole lot more going on with packet handling from one router to another then I care to get into in this post. Packets are inspected and checked each time they are handed from one physical device to another. The way these packets are handled is dependent on the the type of device. The way routers handle packets are different then the way Layer 2 switches handle them. One simple difference is the way TTL is handled, routers care about a packets TTL whereas Layer 2 switches do not.

[LRL]

No... oh man no. Go Cisco if you don't go EdgeRouter.

Ok, I have to ask. What's the Mikrotik turn off?

Coming from being very deeply immersed in the Cisco world I'm very impressed with what mikrotik has done.

[rkelly1]

We use all Mikrotik routers from lower end for a simple pop to the high end for our head routers. We couldn't be happier. We also have a few edge routers installed, however we run MPLS which I think is not supported on ER.

I can't see any reason to move from Mikrotik to something else. Our provider uses Cisco and he is amazed at what tools we have available in MT RouterOS.

No turn off here. Just Love.

[ste]

We use all Mikrotik routers from lower end for a simple pop to the high end for our head routers. We couldn't be happier. We also have a few edge routers installed, however we run MPLS which I think is not supported on ER.

I can't see any reason to move from Mikrotik to something else. Our provider uses Cisco and he is amazed at what tools we have available in MT RouterOS.

No turn off here. Just Love.

We use MT everywhere, too. But our love has some problems. Software Quality!!

- We had LDP issues where some routes are not propagated once a device restarted.
- Autonegotiation of FlowControl on SFP an a CCR does not work

In the past we had big troubles with OSPF which matured. Just look at the forum and see the problems which arise from not testing new SW Releases enough before spreading it.

[josh]

We use all Mikrotik routers from lower end for a simple pop to the high end for our head routers. We couldn't be happier. We also have a few edge routers installed, however we run MPLS which I think is not supported on ER.

I can't see any reason to move from Mikrotik to something else. Our provider uses Cisco and he is amazed at what tools we have available in MT RouterOS.

No turn off here. Just Love.

I take it your provider isn't MetroE trained with that level of gear?

That's a whole different class of telco-style equipment

[tma]

But as far as you saying that no communications exists between routers along a route I do not agree with. I need to go do some reading/homework and sharpen my pencil but what I am referring to is a router receives a packet from one router that it must send to the next router in the chain/route (store and forward). There is communications between those routers in the handling of that packet. If router B received the packet OK from router A and there was no CRC or other issues it stores that packet until it can deal with it and then when router B is ready to forward it to router C and router C says I am ready to receive it the packet is forwarded but if router C gets a CRC on the packet transmission from router B that packet is just not dropped as router B is still holding that packet and will attempt to resend that packet to router C. It is this inter router communications that I am talking about.

Sorry Chris to come up late with this while the thread has become stale, but FC is such an important topic ... but for all I know I think you are wrong about this communication magic between routers. If a CRC occurs, the receiving router interface will drop the packet and that's it. It will not somehow communicate back that there was a CRC and a drop, and the sending router will not repeat anything for failures on layer 1 and layer 2. Even on layer 3 (IP) there is no handshaking and "please repeat" feature. So we come to layer 4 and even here there is nothing in UDP ... it is in TCP (and derivates) and it is end-to-end, not router-to-router. The only thing a router can do is send FC pause frames, but it won't do that for a bad packet, only for buffer overflow reasons (on the interface). And yes, I know there are some mechanism like ECN, but these communicate back to the TCP originator, too.

I'm not saying that routers won't make a difference in an otherwise flat network, but the difference is only in that they have bigger buffers usually, the contention is split to the individual interfaces and that, if you wish, interfaces could do traffic shaping which avoids bursts down into the flat network (but adding to the buffer bloat problem as a negative side effect, of course).

Other than that, there is really much knowledge in this thread that everybody should have read who needs to deal with lossy half-duplex links, i.e. everything in 802.11. For the sake of completeness, I'd like to add this, though:

- If you run Iperf with, say, 10 TCP streams you approximate the behavior of UDP on the same link, i.e. you can find out what the capacity of the link is. But that doesn't say much about how the end-user will perceive his internet feed (well, unless he shares it with some co-workers). Put it that way: You could say, you're getting 50 megs while he could claim my download tops out at 20 megs - and both would be right. You would be right in saying 50 megs for multiple parallel downloads (from multiple persons in his office) and he would be right for his downloads that he observes. Speedtest.net downloads are somewhere in between, because they sometimes measure with more than one stream - may configurable on the server (don't know). On leased lines, fiber or even DSL, there is almost no difference between single-stream and multiple-stream TCP performance. And that's what we are compared with.

- Yes, the Netonix switches are great and provide a lot of value over ToughSwitches. And yes it's very good to know you listen to us while others ... well, okay. The Netonix buffers are not that much larger that FC could be kept turned off, though. Now, to be fair, UBNT did say in their very long thread about TS poor downstream performance that FC must be turned on. That advise, given quite early, was often ignored and the thread went on discussing similar (and also interesting) things for a long time. At the end of the day, though, it is the same issue here and there ... FC is the magic. And, if you turn on FC on TSes, the net performs much better too. So to some extent, if you consider typical user behaviors, the problem on the TSes seems to be that FC is off by default.

[mhoppes]

Except even on the TS if you turned FC on it still wouldn't work right often. Plus there were other issues related to heat and firmware corruption.