Switch rebooting periodically......

[jjonsson]

I have Netonix switches at all my sites/towers. I'm very satisfied...

Unfortunately I have 2 sites where the switches reboot periodically. We have grounded everything and we have used Ohmmeter to measure that grounding is working correct.
What we have found is that at both sites, owners has solar cell system. Could it be something with the inverters for the solar system causing the switches to reboot randomly ?
If yes, how can we measure if that is the case, and how can we prevent it from happen ?
Of course this is only a wild guess, everything else has been tried without luck. And again grounding is checked by independent electricians....

[sirhc]

Model?
MAC?
Firmware version?

The easiest way to determine if there is an issue with the switch itself is to swap it out with another spare unit and see if the problem continues or stops.

[sirhc]

There are some known issues that occurred with a few model which require RMA to fix. Not saying this is your issue but explaining why we require more information "than my switch reboots". - LOL

WS-6-MINI, WS-8-150-AC, and to a limited extent WS-8-150-DC had a production run awhile back (about a year ago +/-) where several hundred units had incorrect 3.3V CAPs which caused reboots. The reboot issue was more prone to occur the more ports that had links (USING POE OR NOT HAD NO EFFECT ON ISSUE SIMPLY THE LINK ITSELF) . Some units worked fine with all 100M links then acted up when radios were upgraded to 1G links. This issues was talked about a long time ago and we offered to cover this issue for the life of the unit so if you discover today or next year we will fix it free.
viewtopic.php?f=17&t=2975&p=20206&hilit=CAP#p20206
viewtopic.php?f=17&t=2796&p=19345&hilit=CAP#p19345


WS-12-250-AC units had one production batch about a year ago +/- where the power supply manufacturer had a bad CAP at C51 position on control board which would cause the switch to reboot. Usually occurred if you had a light POE load such as 30-60 watts or less. It started occurring after being powered up anywhere from 1 hour to 12+ hours. Usually this issues starts rebooting and gets worse until the switch locks up but f it cools down (the power supply) it works again as normal for 1 to 12+ hours then starts again. Once again we told people what to look for and said this issue is covered for life of unit.

viewtopic.php?f=17&t=2737&p=19310&hilit=C51#p19310
http://forum.netonix.com/viewtopic.php? ... C51#p18205

[jjonsson]

Model?
MAC?
Firmware version?

The easiest way to determine if there is an issue with the switch itself is to swap it out with another spare unit and see if the problem continues or stops.

Site 1 has had 5 switches swapped. It has been running for 1-2 month without problems and suddenly starts to do it...
It's a WS-8-150-AC
MAC Address ec:13:b2:d1:13:70
Firmware Version 1.4.7
Board Rev F
Site 2 has got WS-6-MINI installed. Has been running for 2-3 weeks without problems and suddenly starts to randomly reboot.
MAC Address ec:13:b2:91:6b:a6
Firmware Version 1.4.7
Board Rev F
For both sites it started out of the blue the other day. We have had really sunny days (cold weather), and that's why I suspect the solar inverters maybe to generate some kind of noise what ever, since both sites has solar systems installed probably generating full power because of the sun...

[jjonsson]

Here's uptime graph for site A with WS-8-150-AC switch.....

Start week 34 we started to see problems with the switch rebooting. It was swapped (switch no. 3), but problems was still there, switch rebooted randomly, Again in week 40 a new switch (no. 4) replaced the old.
In week 41 we found that grounding was not sufficient and fixed that. In week 41/42 we replaced again (switch no. 5) to make sure it was not the switch.
As you can see it ran without problems from week 42 til now where it suddenly started to reboot again....
That's why we don't expect the switch to be the problem, except if all 3 switches was from same batch (I actually don't know if they are ?)....
Switch 1 and 2 was replaced because of water in cable (Cable has been removed and new installed) 1½ years ago...
All cables has been checked for water, no water at all, cables are dry, so is switch/switchports etc....

Can you see from MAC address (above post) if switch is from bad batch ?

[sirhc]

Well you can test the WS-6 and WS-8-150-AC units to see if they have the 3.3V CAP issue which can cause reboots.

Setup on your bench

links "all" ports at 1G (no POE is required but does not affect the test) no real traffic is needed just links.

Let the suspected unit run all night and see it reboots, if it does not reboot it is fine.

I will post up a MAC range tomorrow that "could" be affected by this defective CAP issue but nothing made in the at least the past 6 months no longer is at risk.

[jjonsson]

I will post up a MAC range tomorrow that "could" be affected by this defective CAP issue but nothing made in the at least the past 6 months no longer is at risk.

I'll wait for that list and check....

[MyThoughts]

We have over the years had numerous WS-12-250-AC that reboot randomly as well.
Many times we have just replaced them with newer revisions. We are starting want to know what is going on as prior to gigabit based equipment we used 24V PoE midspan smart control devices from a competitor. We have only had one failure from those products in over 10 years of operating them and it was the flash on the unit that failed. We still have a few places where they are still in operation.

We have quite a few Netonix 12 port PoE switches deployed (>20) and the rate of replacement due to odd not easily explained issues is starting to bother us. I believe we have replaced around 6 units do to unexplained issues.

Just today we had another switch in operation less then 1.5 years randomly reboot.
The switch in question is a RevD and was recently upgraded this past Wednesday from v1.3.9 to v1.4.8.
Upgrading from v1.3.9 to v1.4.8 did solve one issue we were seeing, and none of our other switches that were upgraded to v1.4.8 have experienced any reboots so we are not sure if it is a coincidence or not.

Nothing looks out of the ordinary on the unit, board temp is 46, cpu is 57, and phy is 64. Voltages all check out, CPU usage is <30% and average throughput is around 300 Mbps. it has 175W of equipment that it is powering.

It is powered from a online double conversion UPS and when the issue has occurred only the switch (and PoE devices on the switch) go out, all the other equipment stays online. Logs from the WS-12-250-AC just report a cold boot.

If there is a known issue with certain boards do to a bad run I would definitely like to see a list of affected MAC addresses.

[sirhc]

Well since you upgraded to latest version see what happens.

v1.3.8 is 2+ year old firmware.

Of course the log is blank when it reboots as the log is kept in memory which is erased on reboot.

If you want to retain logs you use log servers.

[MyThoughts]

Reboot issue occurred again ~1 hr ago, at of course 4AM.

The reboot issue has only occurred since upgrading to v1.4.8.

I have other switches that were upgraded a few days ago to v1.4.8 and they do not experience this reboot issue.

Aside from a coincidence is there anything that could be causing this behavior?

Otherwise I will have to toss this unit in the pile of other Netonix switches with unexplainable service impacting issues and replace it.

We have shutoff one of our backup AF24 units that is powered by this switch to see if lower heat/power usage have any affect.