LAGs reloaded, static and with LACP

[tma]

While I was trying to set up the constellation pictured in my post about "LAGs and RSTP may cause broadcast storm" I failed to make my setup work with static LAGs. It worked when I changed it all to LACP LAGs. Of course, this seems to prove what Chris said (he said: use LACP) but I want to get my point documented anyway.

Before I go into the details, I'd like to emphasize that I really like the Netonix switches - or I wouldn't do this testing and reporting. I like them for their design tailored to passive PoE guys as we all are and the many features not available if I had to use standard 802.3af/at stuff. The software has been improved to the point where I would fully trust it in a single switch scenario and that's what we've started to use them for. I want them to work in our redundant setup on core sites, though, and that's why I'm testing LAGs and RSTP ...

Now about static vs. LACP'ed LAGs: Basically one could say that LACP only adds two features. It detects a failed leg when this is not obvious because something keeps the link state up - this could be a media converter or an Airfiber if you wanted to LAG to another site. Secondly, it would prevent a loop in case someone would plug in two cables between switches on ports that are not prepared to be in a LAG statically.

However, a loop must not occur on ports that are configured statically to be in a LAG, with or without STP, and LACP is also not necessary to redistribute traffic to the remaining legs of a LAG if one leg fails - static LAG will do that just as well, by definition and tested on Netonix by me ;-). Also LACP is not necessary to get switches into an agreement about the distribution algorithm used as this is solely the decision of each switch. So in case it's really only cables between the switches and they are plugged right, static LAG isn't worse than a LAG with LACP.

Back to the issue: When I set up my 4-switch-LAGed-ring-thing pictured in the other post, static loops would work if all switches are DLINK. When I replaced S3 and S4 to be Netonix, I would not be able to ping a switch unless I was directly connected to one of its ports. Actually, that's not the full thruth, as I was *sometimes* able to ping across a LAG but that never worked for a long time as if there was some MAC caching or ARP timeout involved. It didn't investigate further because when I modified all switches to use LACP, everything started to work (nothing else changed).

Now, like I was thinking to myself, you may say that whatever makes it work should be kept and that's it. However, by definition, LACP should not be required for what I wanted to achieve and in fact it is not if it's only between DLINKs. Well, take it or leave it - it's not a showstopper and I will have it LACP'ed to get the switches into production (once the loop thing is resolved).

[sirhc]

Why do you refuse to enable RSTP when using LAG?

I run LACP and Static LAGs on these switches all over my network with no issues but I do enable RSTP on my switch ports that are in a LAG.

Our firmware works perfectly with LAGs even better between our switches than most other switches I tested as far as detecting an LACP port drop and picking it back up faster with less ping drops.

Just enable RSTP on ports that use LAG, problem solved. Is it a matter of principle or what is your adversity to just simply enabling RSTP on ports using LAGs?

[tma]

Here is the drawing that helps you understand the setup I was referring to:



As I mentioned in the other post, please imagine the cables on the Netonix switches are plugged in on ports 21-24. And in case you are wondering why do it like this:

It basically all boils down to the fact that not all switches last forever (only Netonix will :-). We have our core routers and Ceragon links connected to the DLINK S1/S2 switches. One upstream is connected to S1, the other to S2. Likewise, we try to distribute sectors and links on the Netonix S3 and S4 switches in a way that, if a switch fails, there is a good chance we can still remote control the site while customers can fail over between adjacent sectors, minimizing the impact of a failure.

To go one step further, the S1/S2 and S3/S4 pairs are configured identically (except for the management IP address and VLAN) and cables are only plugged into the odd numbered ports on S1 and S3, and into the even numbered ports on S2 and S4. In case of failure, there's no need to prepare a replacement - anybody living close to the site can go there in the middle of the night, move all cables from S1 to S2 (or from S3 to S4) or the other way and the site will immediately be fully operational again. This concept has helped us a couple of times and the good news for manufactures is that we buy twice as many switches for the same purpose :-).

[tma]

Why do you refuse to enable RSTP when using LAG?

I run LACP and Static LAGs on these switches all over my network with no issues but I do enable RSTP on my switch ports that are in a LAG.

Our firmware works perfectly with LAGs even better between our switches than most other switches I tested as far as detecting an LACP port drop and picking it back up faster with less ping drops.

Just enable RSTP on ports that use LAG, problem solved. Is it a matter of principle or what is your adversity to just simply enabling RSTP on ports using LAGs?

A LAG MUST NOT form a loop ever, so no STP needed. The reason to avoid STP when it's not needed is because it briefly interrupts traffic on other ports if a port goes down or up which cannot normally be part of a loop. So you would have to disable RSTP for all ports where it seems unnecessary (which is no help then if a cable is plugged wrongly). Secondly, this effect of short interrupts in traffic tends to influence the next site because BPDUs get there, unless STP is disabled carefully by port. These short interrupts then tend to go buy undetected by your monitoring until customers yell at you. We've had all that. We actually do filter BPDUs in Airmax gear for that reason, but we obviously cannot in Airfiber or Ceragons.

Anyway. That was not the reason for my posting. The posting is about LACP being required although it shouldn't be needed. In the setup described, RSTP is in fact enabled because it is required to prevent a loop over all four switches.

[sirhc]

A LAG MUST NOT form a loop ever, so no STP needed. The reason to avoid STP when it's not needed is because it briefly interrupts traffic on other ports if a port goes down or up which cannot normally be part of a loop.

One port on a switch changing RSTP state (Discarding/Learning/Forwarding) has no affect on traffic on other ports on the switch unless that ports traffic is going through the port that the RSTP state is changing?

There are two ideologies to handling ports in a defined LAGs, one would be not to allow any traffic to pass across the LAG unless the LAG is complete and functioning which some manufacturers do. The other way is to allow the ports to pass traffic as a standalone ports in the event the LAG is not yet functional for some reason which is what we do.

I could go on and on about the pros and cons with this behavior but we felt in the event a LAG fails down to a single port between our switches we would prefer traffic to flow across the ports as a standard port thus with our switches you are required to use RSTP if using LAGs between two of our switches.

However if our switch has a LAG to another device such as Cisco, HP or what have you if that device will not allow a port on their device that belongs to a defined LAG to pass traffic unless the LAG is up then this prevents traffic from passing to our switch anyway.

This is by design not a bug.

[tma]

A LAG MUST NOT form a loop ever, so no STP needed. The reason to avoid STP when it's not needed is because it briefly interrupts traffic on other ports if a port goes down or up which cannot normally be part of a loop.

One port on a switch changing RSTP state (Discarding/Learning/Forwarding) has no affect on traffic on other ports on the switch unless their traffic is going through that port?

That would be nice if it was always true. But there is this rule that RSTP has to block all traffic (on the ports it controls) when a topology change has been noted and until that topology change has been agreed upon by all switches. RSTP got that process down to just a few seconds, which is still not acceptable in a world with RDP sessions being common. For that reason, some implementations let you declare some ports to be "edge" ports, i.e. they have RSTP enabled but are considered not to be part of the topology until BPDUs come in on that port. The Netonix switches only let me turn off RSTP by port completely. So if a cable is (mis)plugged on a RSTP disabled port and a loop occurs, RSTP will not be there to prevent it.

Thus, my thinking is: use RSTP when it's needed in a loop topology and keep it off when it's not needed. And it should definitely not be needed to prevent a loop in a LAG.

But I have to remember you once more: I was talking about IP connectivity failing on a static LAG while it started to work once LACP was enabled. RSTP was enabled all the time because it was required anyway.

Admittedly, I did not fully test the influence of the "RSTP on LAGs" switch. I had it active because that seemed to be safe. But I don't understand what it is supposed to do: will it exempt LAG ports from STP entirely if it's off or will STP treat them as if they weren't in a LAG?

[sirhc]

I do not know Thomas, I just ran a test on my office switches remotely where my desktop computer is connected to port 8 pinging non stop to our cash register on port 12 of the same switch (WS-24-400A).

All ports have RSTP Enabled.

This WS-24-400A has an LACP LAG to another WS-24-400A both using ports 25 & 26.

Disabled Port 25 and then re-enabled it which forced RSTP State "on that port only" to Learning/Forwarding then when I disabled the port it went to Blocking and so on and I lost no pings from my desktop to the cash register and only half the time lost 1 ping to a computer on the other switch through the LACP LAG (Port 25 & 26) which is expected as my traffic may have been on that port at that time.

Did this at least 6 or 7 times same result each time, traffic on other ports were not affected, did not lose a ping?

[tma]

It may be that - although it can't be configured - your implementation handles all ports as "edge" ports by default. In that case, they wouldn't trigger a topology change unless the port has seen BPDUs before. If so, I would indeed extend bad experiences from other switches to yours.

[tma]

But please note that I've tried to move this discussion about RSTP for LAGs into the thread where it belongs. I'm afraid the purpose of the original post in this thread is completely lost now.

[sirhc]

It may be that - although it can't be configured - your implementation handles all ports as "edge" ports by default. In that case, they wouldn't trigger a topology change unless the port has seen BPDUs before.

I will ask Eric if indeed this is indeed what we are doing or why our RSTP responds the way it does. Personally that is the way I would want RSTP to behave?

If so, I would indeed extend bad experiences from other switches to yours.

Not sure I follow you here Thomas, normally your German/English is excellent but this one I am not sure?