Dropping ports on new WS, what is wrong with my setup?

[tma]

Keep in mind they are running a software driven switch/bridge not a core.

My impression is that the data path is directly coupled to the wireless side. If inband management is configured, it is coupled with a bridge to an internal interface of the data path - only that can be seen in the CLI:

# brctl show
bridge name bridge id STP enabled interfaces
br0 ffff.0618d6e3959c no eth0
air0

Also, if the data path is not a software bridge, your ideas become more likely, because a bridge would probably not forward pause frames. For that reason it would take something more low level than a PC with a bridge sitting between the AF and the switch to really find out what's going on.

Right now, UBNT-Chuck answered my post about the 600 Mbps flow I had seen. Need to clarify a bit, over there:
http://community.ubnt.com/t5/airFiber/P ... -p/1561048

[sirhc]

Well there is a wireless portion the radio which is basically seen as an interface

Then they have 2 Ethernet ports one for data and one for management if desired.

Regardless if the unit is set to in-band or not DATA/PAYLOAD packets have to be bridged from the wireless interface to the Ethernet interface.

If in-band management is selected or the management interface is used this is just which interface the IP is assigned to and the CPU/UI/CLI is looking for communication on.

If in-band management is not selected the IP address is not advertised on the data interface and the CPU is not looking for communications to the UI/CLI.

But either way DATA/PAYLOAD packets have to be bridged between the 2 physical interfaces.

All 3 interfaces have a different MAC address which I complained about that when using in-band management the MAC address reported for the unit does not match what is shown in the UI nor does the reported MAC show up in any MAC lookup data base as being a UBNT piece of equipment.

Personally I would have done it differently so that when in-band is selected the management port is dead and move the MAC addresses around so the MAC reported on the DATA port matches what the UI reports and it can be found in a MAC lookup data base to belong to a UBNT device.

With our switches every port has a different MAC address but all MAC addresses we report will look up as owned by Netonix.

Although in the beginning there was a mes-sup between us and the MAC people and we mistakenly used EC-13-B3 instead of EC-13-B2 but we fixed that in all newer models. We are looking into owning/purchasing EC-13-B3 as well.

[tma]

UBNT-Chuck pointed out that in my post viewtopic.php?f=17&t=1654&start=160#p12686 a ranging error shows (reported distance = 0 meters) on both AF5X that may have caused the 600 Mbps flood I had seen, indirectly or "somehow". So please don't take my report as the ultimate proof that an AF5X can/will regularly produce this 600 Mbps flood.

[sirhc]

UBNT-Chuck pointed out that in my post viewtopic.php?f=17&t=1654&start=160#p12686 a ranging error shows (reported distance = 0 meters) on both AF5X that may have caused the 600 Mbps flood I had seen, indirectly or "somehow". So please don't take my report as the ultimate proof that an AF5X can/will regularly produce this 600 Mbps flood.

"I" personally never noticed that range of 0 feet before, interesting!!!!

Did you notice it before Thomas?

Nice clue, you deserve a Scooby Snack! Do Germans watch Scooby Doo?

[tma]

Never had this 0 meters/feet range before. I'm sure you would notice when it happens because it had serious side effects on the wireless link. Even after properly aiming the link one direction was doing 8x and the other 1x only. There was a firmware mismatch due to an oversight when preparing the units - one was 3.2b5 and one was 3.2 - and a firmware upgrade finally fixed the link. (A cold start might have done too, but I haven't tried.)

BTW, while this was an unusual situation, I still got that feeling that a misaligned AF link (or whenever capacity is lower than traffic) will produce FC frames, maybe in a flood. I.e. UBNT-Chuck is right in saying this one doesn't proof anything, but I'll look for it next time we set up a new link.

Scooby Doo? It's strange ... I know the name like I know Mickey Mouse but I'm sure I haven't seen the cartoons. I spent my youth in Austria (where I'm born) and at that time we had 2 TV channels only ...

[yahel]

Hiya'll,

This thread is a bit too long so I may have missed some important posts --- TL;DR.
That said, I've just (2:45AM) upgraded 20 Netonix switches to RC19 (using the Netonix manager) - and all went well, but I'm seeing the "Excessive flow control pause frames" on 6 ports of 5 switches.

I must clarify that we've not seen any problems with this network, none-whatsoever, for many months and it has a perfect uptime with no loss, etc...

(Perhaps upgrading the firmware was a bad idea? We'll probably know in the morning -- I hope it'll be fine).


Here's the list of switches and ports for which we got the "Excessive flow control pause frames" message, and for which the FC got disabled, and what's on the other side of that port (which you'll find interesting - not all are AirFibers - in-fact, two are other Netonix switches!!!).

(Will FC ever re-enable itself for these ports automatically? Upon loss of link-state? )..

Switch ----- Port -- What's-on-the-other-side of that cable
---------- ------- -----------------------------------------------------------------
RJ-12 -------- 4 - Netonix-Mini (6 ports) switch.
TenMiles ---- 1 - AF5
Skyway ------ 1 - AF5
Skyway ----- 13 - Fiber to another Netonix switch !!! (half a mile away).
SandDune -- 1 -- AF3X (AirFiber-X).
Bean -------- 2 -- AF5
===========================================

Hope this helps with this mystery.
It's late, so take whatever I say with a grain of salt (and I just got off a 17 hours flight).

Thanks,

Yahel.

[sirhc]

Not sure I would have upgraded any firmware even to a final version before or after a flight, but you west coast people do like to live on the edge.

The Pause Frame issue with AF radios in this thread is not new to v1.4.0rcX so upgrading from v1.3.X should not cause it to start happening?

We are not sure who or what is at fault with this issue however it was not reported until after UBNT fixed FC in their AF firmware which was AF firmware v3.2 RC or Beta I think?

v1.4.0rc19 does have a KLUDGE FIX though in an attempt to prevent the issue from requiring a truck roll.

THE KLUDGE FIX:
If the switch detects 10K per second of Pause Frames on a port it will disable Flow Control on that port and report it.
Once it disables FC on a port you manually have to turn it back ON in the UI/CLI for that port.

Maybe our KLUDGE FIX falsely turned off FC, it could have a bug or the threshold is too low? Maybe there is an initial burst of Pause Frames sent when a link first comes up after the switch reboots that is normal that we did not account for?

You can try and re-enable FC on a port and see if it trips the KLUDGE FIX again and look at the port stats before and after and note Pause Frame Count.

What version of firmware were you running prior?

[sirhc]

This is our typical tower setup and we have never seen the Pause Frame issue?

You will notice that Flow Control is turned on and ACTIVE as indicated by the 2 little arrows to the right of link speed.

You will also notice the dreaded CRC errors on the AFX radio - gerrrrr
No mater what we do we can not seem to get rid of them.

CRC errors will crop up on the AF24 ports if there is a heavy rain event and especially if the links drop out from rain.

We also see a lot CRCs on another tower with an AF24 with a Dragonwave 24GHz link that is 25 feet away at a 90 angle (not our Dragonwave Link). No matter what we do on that link/tower we can not get the CRCs to stop and they are consistent but the link does push 500 to 600 Mbps reliably with little to no packet loss so we just live with it.

You might also take note that Flow Control is NOT active on the MIMOSA B5 link on Port 4 even though we have it turned on in their UI and in the switch (their Flow Control no workie yet)


CLICK IMAGE TO VIEW FULL SIZE

[yahel]

Noon update...

Received 8 emails for additional ports seeing the FC flood and got FC disabled:
"Excessive flow control pause frames received"


Definately see some minor packet loss across the network - 5% to 8% in some places!
Not good... didn't had that prior to upgrade last night!
Worried about evening traffic which may cause increase in loss due to FC being disabled almost on all core links now.

Also - received two strange emails from switches, which I've never seen before the upgrade:

From "PALH-Netonix" switch: "unexpected link change on port 6 1G - from PALH-Netonix (172.17.0.20)"
From "MES-Netonix" switch: "unexpected link change on port 1 1G - from MES-Netonix (172.17.0.100)"

Any idea what's that about?


We'll have to live through the evening and hope for the best... but might be forced to downgrade firmware to avoid the "Kludge fix" and re-enable FC on the core.


Worried!

Yahel.

[sirhc]

Noon update...

Received 8 emails for additional ports seeing the FC flood and got FC disabled:
"Excessive flow control pause frames received"


Definately see some minor packet loss across the network - 5% to 8% in some places!
Not good... didn't had that prior to upgrade last night!
Worried about evening traffic which may cause increase in loss due to FC being disabled almost on all core links now.

Also - received two strange emails from switches, which I've never seen before the upgrade:
From "PALH-Netonix" switch: "unexpected link change on port 6 1G - from PALH-Netonix (172.17.0.20)"
From "MES-Netonix" switch: "unexpected link change on port 1 1G - from MES-Netonix (172.17.0.100)"
Any idea what's that about?

We'll have to live through the evening and hope for the best... but might be forced to downgrade firmware to avoid the "Kludge fix" and re-enable FC on the core.
[/color]

People asked for an alert if port links speeds changes on its own, that is what those messages are from.

You know when a link decides to downgrade from 1G to 100M or 100M to 10M on its own from either bad cabling or bad device or interference in 100 MHz range.

In this case it is probably because Flow Control was turned off and Eric should make it not alert as FC is being forced OFF by the KLUDGE FIX.

The Kludge Fix that turned off Flow Control at 10K Pause frames per second either did it because it is working properly and the threshold is too low but 10K is a LOT?

Or the KLUDGE FIX is tripping below 10K per second. Can you try enabling it and then clear the stats on the interface and sort of keep an eye on it and see if it is maybe tripping at a lower number?

Still not sure why you would roll out rc19 to all switches especially after a long flight and your tired? - crazy brave!