Sudden excessive pause frames and link speed change

[cwachs]

Netonix WS-12-250-AC has been running great for months. It currently has firmware 1.4.6. Ports 4 and 5 have Cambium ePMP 2000 APs attached to them. Until today, both were running fine and linked up at 1G with flow control on.

Today, around the same time, port 5 gets downgraded to 100M-F and port 4 has a ton of excessive pause frames in the log:


Jan 24 14:11:35 switch[1372]: got excessive pause frames on port 4 (16380), count = 10
Jan 24 14:11:36 switch[1372]: Excessive flow control pause frames received on port 5 (ePMP South 2k (red/white)) (ePMP South 2k (red/white)), disabling flow control

Traffic on both APs is fairly light (under 50 Mbps). I am going to take the switch to 1.4.7rc4 tonight during a maintenance window but I'm curious what might cause this suddenly and both at the same time. They are on different legs of the tower and both cables are in conduit so I am ruling out tower climbers causing problems - plus it's snowing today so I don't think anyone is up there...

No other ports are reporting any issues (another ePMP 1000 AP, Mimosa B5 and an AF24). Both cables check out OK to the exact same length on all pairs respectively.

I also can't force port 5 back to 1G. When I try (from auto), it goes up and down constantly.

Lastly, I turned off FC and STP on the ports and they continue to throw excessive pause frames.

[sirhc]

Well a firmware upgrade on our switch is NOT going to fix the radios from sending Pause Frame Storms.

As far as the modulation down to 100 you know the possible causes and ways to debug it but it is not our firmware.

The switches are built on switch CORES which means the packet switching is handled inside the core we do not write that code and these Cores/PHYs are mature products found in many other switches including Cisco. All we do is write the interface and stats and such.
If you do not have the tower grounds bonded to electrical service grounds and the ground freezes or gets wet it can change ground rod potential and for ground current across Ethernet cables.

viewtopic.php?f=6&t=2456#p17155

viewtopic.php?f=6&t=2456#p17158

There are known BUGS with UBNT AF and AFX firmware in respect to how it handles Flow Control which has been discussed openly on both forums.

It has also been reported that airMAX AC radios also have the same issue with Flow Control.

There has also been at least 1 report that epmp 2000 radios cause Pause Frame Storms.

[cwachs]

Perhaps a radio reboot tonight is in order. I've manually shut off flow control on both ports. Just very odd this started on 2 of 5 radios at the same time today. Temp has been pretty constant for days and we do have electrical and tower grounds bonded - I did that myself up there.

We did get a freezing fog that came through... How about ice build up on antenna horns? Can that cause the pause frames?

Still not sure about the Ethernet issue - might be getting time to run a strand of fiber up there and use a transceiver at the top.

[sirhc]

Well yea, if wireless conditions degrade less data gets through the wireless side and gets backed up on the Ethernet side and when buffers fill up the device sends Pause Frames to the switch. WHich unless you have been following can cause weird behavior on other ports as switches use a shared pool of memory for packet buffers.

If you are uploading all radio ports from router through a single port on the switch and one radio starts sending a Pause Frame Storm to the switch eventually the switch will start sending Pause Frames to the router affecting all traffic coming through that port to the switch.

This is why I always seperate backhaul radios so they never share the same port to the router as local tower radios.

The newer firmware will protect against Pause Frame Storms unless you disable it.