RX Thread - high CPU

[sporkman]

Howdy.


Yes, this switch is running 1.4.0rc25, but that's pretty damn close to FINAL. And I can't upgrade it in the middle of the day anyhow...

I have little visibility into what happened, but the switch upstream of this saw the port bounce 3 or 4 times and then this switch rebooted. I have a few log lines collected by syslog while this was going on:

Code: Select all

May 26 15:43:46 192.168.3.48 monitor: restarting vtss_appl
May 26 15:43:51 192.168.3.48 STP: set port 4 to learning
May 26 15:43:51 192.168.3.48 STP: set port 4 to forwarding
May 26 15:44:19 192.168.3.48 monitor: restarting vtss_appl
May 26 15:44:41 192.168.3.48 monitor: restarting vtss_appl
May 26 15:45:14 192.168.3.48 monitor: restarting vtss_appl
May 26 15:45:47 192.168.3.48 monitor: restarting vtss_appl


After the reboot, the switch seems to be working, but the "RX Thread" process is gobbling from 80-100% CPU. This unit will get an upgrade late tonight, but I'd like to know if anyone has seen this behavior and what it might be (bug vs. some ubnt gear freaking out). It's pretty lightly loaded - one link down to a cisco switch, one link to an AP with 1 customer, one link to another AP with 3 customers.

See attached pics for cpu usage snapshot.

[sirhc]

Upgrade to v1.4.0 FINAL as soon as possible and then report if the issue is still there.

This is what people should do when they have a problem:
If you have a problem upgrade to the latest FINAL version, in this case v1.4.0

If problem still persists upgrade to the latest RC version, in this case there is no RC version

If the problem still persists then read that firmwares version "BUG and COMMENTS THREAD" thread to see if the problem has been reported so far and if there is a work around or what is up but if no mention then of your problem then report the issue in that thread.

viewtopic.php?f=17&t=1722

[sporkman]

edit: moving to the 1.4.0 thread.

[sporkman]

Upgraded to 1.4.0. Same story. RX Thread taking up most of the CPU. What can I look at?

[sirhc]

Upgraded to 1.4.0. Same story. RX Thread taking up most of the CPU. What can I look at?

If you have a large flat network Disable Discovery Tab on the Device/Configuration Tab and restart the switch.

If that is the case let us know as we are looking for someone with this issue to let us debug it.

[sporkman]

I wouldn't call it very large. There are barely a dozen devices in the bridge domain.

So if I turn off discovery, then one of my APs (Rocket 5AC Prism) stops working unless I turn off STP. So that's interesting. Attaching a rough diagram, but in summary:

Cisco 3550 with a native VLAN and VLAN 101 on the port that goes to the problematic WS-6-Mini

On that Mini:

Port 1 to the cisco, with the Mini set to have VLAN 1 untagged, VLAN 101 tagged
Port 3 to Rocket 5AC Prism in bridge mode with mgmt IP in VLAN 1, also with VLAN 1 untagged and VLAN 101 tagged
Port 4 to Rocket M5 in bridge mode with mgmt IP in VLAN 1, VLAN 1 untagged

The Rocket M5 only has one client associated. The client CPE is in router mode with a mgmt IP in the same subnet as all other devices above.

The Rocket 5AC has VLAN 1 and VLAN 101 each with their own bridge with WLAN and LAN bridged (not to each other, the Rocket is just bridging tagged 101 and untagged 1).

There is a Nanobeam AC associated with that, also in router mode with a management IP in the same subnet as all other devices above.

There is also a PowerBeam AC associated with the Rocket 5AC. It bridges VLAN 1 and VLAN 101 and is connected to another WS-6-Mini.

The WS-6-Mini has VLAN 1 untagged and VLAN 101 tagged on port 3.

Port 1 has VLAN 101 untagged and a customer has a single device (Fortinet FW) plugged in there. The port is limited to 20/20.

Port 4 has another Rocket 5AC Prism. It's in bridge mode, no VLANs and has no clients associated with it.

So that's the topology...

I can tinker with this in the late evenings, these are all business customers that are generally not using the circuit at night.

[sporkman]

Thoughts? Why would disabling discovery cause the port the Rocket AC is on to stop forwarding?

[sirhc]

Thoughts? Why would disabling discovery cause the port the Rocket AC is on to stop forwarding?

It would not do that, you need to dig deeper and find out what is happening.

However you are aware of the Pause Frame Storm Bug in the airMAX AC and airFIBER firmware?

http://community.ubnt.com/t5/airFiber-Beta/640-Mbps-FC-storm-while-in-1-4x-SISO/m-p/1564144/highlight/true#M3756

http://community.ubnt.com/t5/airFiber/Possible-Flow-Control-Bug/m-p/1556296/highlight/true#M28857

http://community.ubnt.com/t5/airMAX-AC-Beta/AC-gear-with-flow-control/m-p/1568041/highlight/true#M8233

http://forum.netonix.com/viewtopic.php?f=17&t=1654


But if you are running v1.4.0 we have a Pause Frame Storm protection routine that will disable Flow Control on a port if it detects a storm on it.

There is also a bug in v1.4.0 with Loop Protection not disabling but that would show up in your switch log if that is the case.

[sporkman]

This is what I've got in the log from last night. Note there's no way for there to actually be a loop on the problematic port - single AP with a single client.

Code: Select all

May 26 22:13:28 STP: set port 3 to discarding
May 26 22:13:30 STP: set port 3 to learning
May 26 22:13:30 STP: set port 3 to discarding
May 26 22:13:32 STP: set port 3 to learning
May 26 22:13:32 STP: set port 3 to discarding
May 26 22:13:34 STP: set port 3 to learning
May 26 22:13:34 STP: set port 3 to discarding
May 26 22:13:36 STP: set port 3 to learning
May 26 22:13:37 STP: set port 3 to discarding
May 26 22:13:45 STP: set port 3 to learning
May 26 22:13:45 STP: set port 3 to forwarding
May 26 22:13:45 STP: set port 3 to discarding
May 26 22:13:49 STP: set port 3 to learning
May 26 22:13:49 STP: set port 3 to forwarding
May 26 22:13:49 STP: set port 3 to discarding
May 26 22:14:03 UI: Configuration changed by bwayadmin (x.x.x.x)
May 26 22:14:03 UI: STP_Enable: changed from 'Enabled' to 'Disabled'
May 26 22:14:04 STP: set port 1 to discarding
May 26 22:14:04 STP: set port 4 to discarding


I'm assuming both of those last log entries are bogus, as the ports remained active.

What's the connection between discovery and STP? Should I re-test this again tonight to see how repeatable it is (enable discovery, see if port 3 comes up, disable discovery, see if port 3 goes into STP crazies)?

I can also just leave STP off and forget about it and call it fixed... :)

edit: regarding pause frames, not seeing any on that port, attached screenshot

port 3 stats

[sirhc]

Well I want to back track it might be possible that the way Eric was queuing packets with a large flat network Discovery could mess with STP

Will put up a fix v1.4.1rc2 with a fix for the STP and Loop Protection not disabling in a little bit