Dropping ports on new WS, what is wrong with my setup?

[tma]

Anyway, before I leave for home ... Like almost every day in this nice part of the year, we deployed another AF5X link today, together with a WS-24-400A. Being so happy about the performance of both devices! They are an incredible combination - no cable mess, good insight into performance and power consumption. Just lovely.

[sirhc]

Anyway, before I leave for home ... Like almost every day in this nice part of the year, we deployed another AF5X link today, together with a WS-24-400A. Being so happy about the performance of both devices! They are an incredible combination - no cable mess, good insight into performance and power consumption. Just lovely.

Thanks and just remember to disable FC on the AF and your good (as a cludge workaround) but we will continue to work on this until a resolution is found.

At least the next firmware we release has a cludge to catch the Pause Frame Storm and take action to disable FC on the switch port to hopefully prevent a truck roll.

For now Eric has set this trigger point at 10K Pause Frames per second and the switch will disable FC on the port.

I felt 10K was too high and may not catch it all the time but he feels it is correct pps count, if we are too high we will adjust it down on the next release.

It will send an SMTP alert and log the event in the switch log.

[tma]

Thinking about a question IntL-Daniel had asked at the UBNT forum on this topic, I did a "clear counters" and created this screenshot after some seconds. Note there are 6915 pause frames sent for 38276 packets received. I've even seen the TX FC counter to go up while the traffic was as low as 1-2 Mbps as seen on the left half of the total throughput graph.

This is the setup for VLAN 100 which connects port 1 and port 5 (nothing else):

AF24 (GbE) <> port 1 (GbE) Netonix port 5 (FE) <> (FE) Router

Traffic then uses other VLANs from other router ports to other 100 meg devices.

Yes, there's a 1000->100 transition from port 1 to port 5. If the logic presented in the best practices video applies, TX pause frames on port 1 must have been caused from buffers filling up towards port 5, because all other devices are in separate VLANs and the router talks to them 100->100 or 100->1000. In fact the FC counters on these other ports in other VLANs are all 0 even without clearing them.

Which leaves me with this question: is it to be expected that 2 to 25 Mbps of traffic on a 1000->100 transition causes the TX FC counter to go up? Even going up while there was only 2 Mbps coming in?

One reason could be that an AF5X, being half-duplex, causes traffic to arrive in bursts because it must buffer downstream packets on the far side while it is transmitting upstream and, once it gets around to sending downstream and having a capacity of 215 Mbps, it sends whatever has been buffered into the switch rather quickly.

Still, I wouldn't have expected 2 Mbps of traffic, even it was made bursty by the half-duplex nature of an AF5X, to cause any device on the switch to generate pause frames. And in fact they didn't, except the switch did on port 1 towards the AF5X.

[sirhc]

Your excellent word "bursty" explains it all. Leave it to the Germans to create a new English word!

Even if it is shown on graph as 2 Mbps which is an average over 1 second (on our switch most other devices are 5 second averages) what is important is how fast the "bursty" is. Buffers are filled and emptied in micro-seconds and even smaller increments of time.

Good idea with maybe the HD nature of AF5X play a roll but people have reported the issue with AF24 I believe (although not as much mind you).

[sirhc]

It has something to do with switch to switch with AF in between as I still have never seen this behavior on any of my switches where I have a router at every tower so I have this

MY SETUP:
Cisco Router + WISP Switch -<AF LINK>- WISP Switch + Cisco Router

Now this goes like this 8 tpwers in main ring and another 7 towers in side rings inter hooking into main ring

Now technically there is a switch on each side of the AF link between the AF and the router with the switch set up as a mid-span injector using U on the 2 ports and E on all other ports.

[tma]

Like you, I believe in L3 and OSPF. So my setup is:

Linux Router + WISP Switch -<AF LINK>- WISP Switch + Linux Router

The major difference is (probably) that our routers for smaller sites are still having 100 meg FE interfaces. On all these sites I see a ~ 55% FC rate from Netonix to AF5X. This is not the case for all 1000->1000 situations I checked. So yes, it believe it has to do with 1000->100, but I wouldn't have expected it to occur at 2 Mbps - because it would mean that in every graph second, there's a 20 ms peak of at least 100 Mbps (or else no reason to send a pause frame), and no packet for the remaining 980 ms. Seems to prove that there's more burstiness around than we can imagine.

BTW #1, while looking around I found at least one Netonix-AF combination with AF5 3.2 firmware, where FC is indicated OFF by the switch although it is configured ON on both the AF5 and the switch (verified that). On that site of course, there's no TX FC packets. I remember I had set the link state to disabled (on the switch) during deployment, so it seems they failed to negotiate FC when the link state was set to enabled finally.

BTW #2, the word bursty (and burstiness) must have been invented by someone before me. Google finds it 559.000 times on this planet, not only in Germany ;-)

[sirhc]

Yes, Flow Control is a function of AUTO NEGOTIATION and hard coding speed / duplex will disable Flow Control on "most" devices.

I think i warned people about this on this forum already?

viewtopic.php?f=6&t=714&p=5289&hilit=disables+flow+control#p5289
viewtopic.php?f=17&t=539&p=4106&hilit=disables+flow+control#p4106
viewtopic.php?f=6&t=714&p=5317&hilit=disables+flow+control#p5317

[TheHox]

The weird issue for me, is my original post I had the 16k pps and 8mbps traffic, but this was in an MDU with NO AF's at all. I did have 18 UAP-AC-Lites I was firing up.
Mikrotik router - netgear switch - netonix switch - UAP-AC-Lite

I took the net gear switch out and my issue went away.

The similarities I had was more than 1 switch, one being a netonix with the end device being a UBNT device.

[tma]

Yes, Flow Control is a function of AUTO NEGOTIATION and hard coding speed / duplex will disable Flow Control on "most" devices.

I think i warned people about this on this forum already?

viewtopic.php?f=6&t=714&p=5289&hilit=disables+flow+control#p5289
viewtopic.php?f=17&t=539&p=4106&hilit=disables+flow+control#p4106
viewtopic.php?f=6&t=714&p=5317&hilit=disables+flow+control#p5317

In general this is correct, but in this case the port was set from Disabled to Enabled with speed being Auto on both sides. So they should have auto-negotiated at that point, and they actually did and agreed on 1000F, but w/o flow control. It could be the AF or the switch but it's a minor issue and off-topic.

The more interesting sentence in my comment was this:

"I wouldn't have expected it [FC frames] to occur at 2 Mbps - because it would mean that in every graph second, there's a 20 ms peak of at least 100 Mbps (or else no reason to send a pause frame), and no packet for the remaining 980 ms."

I still can't believe that a 2 Mbps stream - even if it has some peaks that don't show in the graph - would cause FC action. Again, this may be off-topic but at least it's somehow closer to the subject. Consider this: What if the FC TX and RX counters are reversed in the popup? Then the AF5X would send this amount of FC frames to the switch and the AF5X is the bad guy. But if they weren't reversed, there's something wrong on the switch, I'd say, on paths that involve a 1000->100 transition.

[tma]

The weird issue for me, is my original post I had the 16k pps and 8mbps traffic, but this was in an MDU with NO AF's at all. I did have 18 UAP-AC-Lites I was firing up.
Mikrotik router - netgear switch - netonix switch - UAP-AC-Lite

I took the net gear switch out and my issue went away.

The similarities I had was more than 1 switch, one being a netonix with the end device being a UBNT device.

Thank you for bringing this up again - I remembered vaguely to have seen a post where no AF was involved but didn't want to read thru this long thread again. Now this is very similar to the 4 incidents I had, one where it happened like this.

myself <---> Netonix1 <---> Netonix2

After Netonix2 had finished a firmware upgrade, I could not reach it anymore, but I saw the the 16k pps and 8mbps traffic while looking at Netonix1, on the port that connected it to Netonix2. I then disconnected from Netonix1 because I wanted to connect to Netonix2 and check if I can find out what's going on there. While disconnected, the activity continued (as judged by LED flickering). When I connected "myself" to Netonix2, though, it suddenly stopped. After that, I failed repeat it.

So, as in your case, it was between two switches and no AF. The way it stopped was somewhat unexpected and that's why I've asked the forum to unplug/plug unrelated ports before unplugging the port with the AF on it.