WS-6MINI Lockup after swapping AP?

[sbyrd]

The fact that it lasted 4 months leads me towards the idea that there may be some other issue at work here, but I won't know until I get a first-hand look. I take it
there were no other changes to your deployment, I.E. you didn't plug anything else in or make any configuration changes?

There were not any changes to the deployment. It recently has gotten colder and the switch is in an outdoor box. The temp here the last few days has been colder in the 40deg range. The switch connected to UPS for power, but nothing else on the same UPS - WS-12-250-AC, Mtik Router, Airfiber5X - rebooted. Switch is powered using the barrel connector and not the PoE in port.

Basically I got interface flapping notifications from my Mikrotik router at the tower and after the port stopped flapping the switch showed it had just rebooted along with all the APs powered by the switch.

It happened at 3:12pm, 3:13pm, 3:21pm, 3:49pm until I replaced just the switch at 4:10pm yesterday.

Sometimes the router would show the port flapping a couple of times which looks like a boot sequence.

Capture.PNG (10.93 KiB) Viewed 7682 times

Other times it would maybe flap twice as much before the switch rebooted.

I will fill out an RMA form and get it sent back. Based on this what do you think some of the possible causes could be?

[Julian]

So the sequence is UPS -> power brick -> switch -> radios. Continued function of other connected devices rules out UPS, so, could be the power brick, the switch, or a radio failing..

I know, helpful, but not being there, tough for me to come to any conclusion.

[sbyrd]

So the sequence is UPS -> power brick -> switch -> radios. Continued function of other connected devices rules out UPS, so, could be the power brick, the switch, or a radio failing..

I know, helpful, but not being there, tough for me to come to any conclusion.

Well since I replaced the switch and the issue stopped (so far), I am leaning towards switch. I was going to replace the power supply as well, but I brought a PoE Brick and forgot it was powered by the barrel plug on the back. If the new switch starts doing it then it must be something else.

I filled out an RMA form already, should I have waited?

[sirhc]

You might not be locking up the switch you may be experiencing Flow Control packet locks.

Try disabling Flow Control on all ports and see what happens.

This Flow Control Pause Frame Storm Packet lock is NOT a bug with Netonix but rather has been proven time and time again to be issues on the radio side.

There are many posts on this forum and UBNT forum you can search for and read

Could be a defective WS-6-MINI as well which you can test by swapping out with a new one.

There was a know issue a while back where we had a batch of WS-6-MINI and WS-8-150-AC units that had defective 3.3V caps that could result in a lock up. If it is the defective 3.3V CAP we extended the warranty to that issue to life of the unit.

[sbyrd]

You might not be locking up the switch you may be experiencing Flow Control packet locks.

Try disabling Flow Control on all ports and see what happens.

This Flow Control Pause Frame Storm Packet lock is NOT a bug with Netonix but rather has been proven time and time again to be issues on the radio side.

There are many posts on this forum and UBNT forum you can search for and read

Could be a defective WS-6-MINI as well which you can test by swapping out with a new one.

There was a know issue a while back where we had a batch of WS-6-MINI and WS-8-150-AC units that had defective 3.3V caps that could result in a lock up. If it is the defective 3.3V CAP we extended the warranty to that issue to life of the unit.

Well I can rule out Flow Control as I do not think FC packets could cause a switch to crash and reboot along with rebooting all the APs attached to it? Also I have the ports facing the APs set to Obey only and the uplink port of the switch is also set to Obey only.

This switch was purchased from the same place at the same time as 2 other Minis that I have RMA'd for the 3.3V cap issue so I am leaning in that direction at the moment. The reason I did not RMA this switch with the other 2 was because on a 24hr bench test the 3.3V seemed to be stable. My speculation is that it just took 4 months for whatever flaw is in the 3.3V to deteriorate enough that the switch started having this problem.

The switch that I replaced it with yesterday is one of the repaired ones that have the 3.3V cap replaced, so if it starts rebooting then it is 100% something else.

[sirhc]

Well I can rule out Flow Control as I do not think FC packets could cause a switch to crash and reboot along with rebooting all the APs attached to it? Also I have the ports facing the APs set to Obey only and the uplink port of the switch is also set to Obey only.

Actually no your flow control settings would not protect you from a packet lock, in fact your logic is backwards and more easily creates a packet lock.

Think about it, the switch has tons of packets to send to the AP but the switch port is obeying Pause Frames from the AP telling the switch to HOLD the packets which starts filling up buffers on the switch. And to make things worse the switch can not even tell the uplink or the source of these packets that it can not get rid of to stop sending packets.

Once the switch buffer memory is FULL you have a packet lock until the buffer can finally start sending packets.

Now you are correct that a packet lock would not cause the radios to reboot BUT if you have a ping watchdog running on the radio which can no longer reach its ping destination then the radio watch dog will reboot the radio.

Or if you have a watchdog on the switch pinging the AP it can not reach the radio due to a packet lock and you have the switch rebooting the radio.

Look, break it down to simple steps.
1) Make sure all firmware is up to date on switches and radios
2) Disable watchdogs
3) Disable Flow Control

If it is still happening then it is possible you have the 3.3V CAP issue so swap unit with a WS-6-MINI recently purchased which you know does not have this possible defect. If the problem goes away RMA unit. If the problem stays then there is something deeper going on.

[sbyrd]

Well I can rule out Flow Control as I do not think FC packets could cause a switch to crash and reboot along with rebooting all the APs attached to it? Also I have the ports facing the APs set to Obey only and the uplink port of the switch is also set to Obey only.

Actually no your flow control settings would not protect you from a packet lock, in fact your logic is backwards and more easily creates a packet lock.

Think about it, the switch has tons of packets to send to the AP but the switch port is obeying Pause Frames from the AP telling the switch to HOLD the packets which starts filling up buffers on the switch. And to make things worse the switch can not even tell the uplink or the source of these packets that it can not get rid of to stop sending packets.

Once the switch buffer memory is FULL you have a packet lock until the buffer can finally start sending packets.

Now you are correct that a packet lock would not cause the radios to reboot BUT if you have a ping watchdog running on the radio which can no longer reach its ping destination then the radio watch dog will reboot the radio.

Or if you have a watchdog on the switch pinging the AP it can not reach the radio due to a packet lock and you have the switch rebooting the radio.

Look, break it down to simple steps.
1) Make sure all firmware is up to date on switches and radios
2) Disable watchdogs
3) Disable Flow Control

If it is still happening then it is possible you have the 3.3V CAP issue so swap unit with a WS-6-MINI recently purchased which you know does not have this possible defect. If the problem goes away RMA unit. If the problem stays then there is something deeper going on.

I ping everything, switch, APs, and CPEs attached to the APs once a minute. I assume the packet lock you are saying would be easily apparent and show as loss?

Anyway not only are the APs rebooting, but the switch is also rebooting which is what I expect is causing the APs to reboot, but its hard to know what is rebooting first.

I have already replaced the switch and the new switch does not have the 3.3v defect and is running the same configuration while on the latest RC FW.

[sirhc]

So is the switch you swapped in also rebooting (same behavior)?

If so please disable all watchdogs that can cause anything to reboot, and disable FC on all ports.

If it is still rebooting then we need to look at power, could still be a defective switch.

Please post up Status, Ports, Device/Configuration, and Device/Status TABs of switch in service if problem is still occurring since the swap.

[sbyrd]

So is the switch you swapped in also rebooting (same behavior)?

If so please disable all watchdogs that can cause anything to reboot, and disable FC on all ports.

If it is still rebooting then we need to look at power, could still be a defective switch.

Please post up Status, Ports, Device/Configuration, and Device/Status TABs of switch in service if problem is still occurring since the swap.

Sorry, somehow I must have not been very clear. Yesterday a mini started rebooting randomly and constantly. I replaced it late yesterday with another fixed mini. The replacement has not exhibited the same issue.

[sbyrd]

Slightly OT question, but about FC. Correct me where wrong if you want.

For an AP switch where all the port on the switch feed Access Points, the best configuration of the uplink port to the tower router would be with FC set to Both. That way when the AP tells the switch to hold packets the switch can tell the router to pause if the switch buffers then get full due to the packet hold. Of course when the switch tells the router to pause this will pause traffic for all customers on all APs, but is better than getting full buffers as, depending on your setup, may only affect a few hundred customers.

For an BH switch where all the port on the switch feed Backhauls, the best configuration of the uplink port to the tower router would be with FC set to off. Reason being if one Backhaul tells the switch to hold packets you don't want the switch telling the router to pause the traffic as this could now affect, depending on your setup, several thousand customers on different towers downstream with packet pauses. This would be especially true with Airfibers that have an issue with excessive pause frames during power loss to the Slave or a drastic drop in modulation due to weather or atmosphere. In my case these Airfiber storms caused my LAG to drop to the tower router briefly which drastically affected traffic to all towers fed by this switch when the uplink ports to the tower were set to Generate or Both on FC.