WS-6MINI Lockup after swapping AP?

[sbyrd]

First I will post my config as requested by Chris.

[sbyrd]

My issue is as follows.

I have had for over a year a WS-mini-6 powering 4 Rocket M5 APs. The APs are named 54E, 54E2, 54SW, 54NW.

On 5-22-17 I replaced 54E and 54E2 with Rocket AC-Prism APs. No issues.

Today 5-31-17 I decided to also replace 54SW and 54NW with Ac-Prisms. Now I have this issue.

I started by replacing 54SW. After 5 minutes or so I lost connectivity to the switch and all devices connected to it. I go and look at the switch and all the AP ports show power but all the ethernet links are slowly going on and off. This included the switches uplink port.

I then hard power cycle the switch. After another 15 minutes it happens again. This time I decide to put the 54SW M5 AP back and move on to 54NW.

I replace 54NW with AC-Prism and it runs fine. I then do some other work on the tower and maybe after an hour or so (I did not keep track of time) I decide to go back and try the AC-Prism on 54SW again.

I put the AC-prism back and after some time the switch locks up again. I reboot the switch and decide to turn on my phone stop watch to see if it is happening at the same time or not. I also take 54SW and power it with a passive PoE injector and turn off the power on 54SW port.

After around 15-20 minutes the switch locks up. I then take my laptop and plug directly into 54SW which is now powered by the passive POE. It connects right away and I see all the clients are connected to the AP and look perfectly fine.

I then at Chris's suggestion replace 54SW with another known good AC AP. This time I put an AC-PtMP AP that I had running perfectly fine at another tower for almost a year powered by a WS-12-250AC switch. After 30 minutes the switch locked up again.

Now either I have a bad switch or something about an AC device (1G eth link) in Port 5 is causing the switch to lockup. The only other thing would mean the M5 clients connected to the AC AP are sending bad traffic that locks up the switch, but does not happen when the same M5 clients are connected to an M5 AP??

[Julian]

Is this switch up a tower? What I'd like to know is if the problem only occurs on port 5, or if it's stable when you leave any one port unplugged.

[sbyrd]

It is stable right now for the last 4 hours with me putting back the RocketM5 in port 5 (54SW) instead of an AC AP.

[sbyrd]

Well it took almost 5 hours but the switch locked up again. So the issue is not caused by having an AC AP in port 5.

This time the switch was locked up for 5 minutes and then rebooted itself. Here is what my tower router showed on the interface connected to the switch during the last lockup.

At this point I still plan to replace the switch and power supply to hopefully figure out if it is the switch or not that is the problem.

As it is almost dark here my issue now becomes is it stable enough to wait until tomorrow morning to do the replacement or not.

If it can got about 4-5 hours between lockups then I think I can wait till the morning. If it starts doing it every 30 minutes again I guess it will be time to get out the head lamp and climb.

[sbyrd]

In a possibly vain attempt to stabilize the switch I have set port 6 (54NW) to auto 100m negotiation. My thought is that before today both ports 5/6 were 100M due to using Rocket M5s that could only do 100F. As the new AC radios can do 1G on the PHY interface maybe there is more power draw at 1G on the switch and setting back to 100F might stabilize it until tomorrow.

Now I do not know what PHY chips you use in the WS-6Mini, but looking at the technical documentation for another 10/100/1000 PHY chip from TI it does show the 1000F link uses more Amps than when the PHY is running at 100F. It is very minor, but if my 3.3V caps are faulty a little bit of power savings might extend the time between lockups or eliminate them completely.

When both port 5/6 were at 1000F it seemed the regularly lockup at no more than 30 minutes. With only port 6 at 1G I got 4.5 hours. I am reaching I know, but just hoping to avoid a tower climb tonight and also explain why this switch worked perfectly fine when both ports 5/6 were Rocket M5s and not Rocket ACs.

[Julian]

You're on the right track, I believe. Our phys are monolithic to the SOC, which is a vitesse vsc742* series,

There is a slightly higher power draw on the 2.5V supply when linking at a gigabit, and a set of circumstances can occur which will cause the excess current on 2.5 to drag down the 3.3V rail, which, long story short, causes a soft reboot. PoE will stay up, but traffic will stop while the system gets stuck in a boot loop.

Reach away-nobody likes climbing in the dark.

This got past test, since it takes a good while to present, and under fairly specific conditions. The only thing I can say for sure is that it presents in a percentage of WS-6 and 8 models, and root cause is, in essence, component tolerance.

If you're handy with an iron, we can send you a DIY repair kit at no charge, and if not, it's covered under warranty, and usually I will have it fixed and on the way back same or next day.. My apologies for the trouble, we had no indication there was an issue. We have since revised our test plan to address this use case, so there's at least that.

[sbyrd]

Well since changing port 6 to be 100F instead of 1G I now am looking at ~11 hours of up time with no lockups.

I will be changing the switch out today for another one I have. It is the same board revision 'E' as the one that is locking up. Should I be concerned that this one might be bad as well?

I have had the replacement switch on the bench all night with 1G links in all 6 ports and ports 3-6 also doing 24V PoE to UBNT radios. So far in 14hrs it has no locked up. However I do show in the switch logs (1.4.2rc6) that the 3.3V has fluctuated from 3.1 to 3.5V. Does this indicate this switch might be bad as well?

[Julian]

3.1v is marginal. On-board power reset circuit asserts RESET* to CPU once 3.3v falls to 3.0 - this is to ensure a tidy shutdown during normal depower, 3.1v is a bit close for my taste.

Short version, it probably is affected, but to a lesser degree than the one on the tower. In my testing here, it can take up to 24 hours for the oscillation to reach the critical voltage and start boot looping, but once it starts it will continue to cycle at a more rapid pace until it can no longer finish a boot sequence.

The fact that you're sustaining 6 gigabit links might tend to tell me that you'll be okay in your particular application, but no, your switch isn't right. Let me know what I can do to help, I'm definitely at your service via here, or email.

[sbyrd]

Well all I have on my shelf are 2 minis that came from the same production run as I got them both shipped to me as part of repair for grounding damage I caused. So in your opinion should I replace the tower switch with the one I bench tested and then RMA the tower switch along with the other one I have on my shelf, or maybe put something like a UBNT toughswitch 5 in their temporarily and RMA all 3 switches at the same time?