WS-12-400-AC Dropping several ports simultaneously

[sirhc]

I don't wish to drag this off topic but the NEC required minimum for service ground is intended to draw enough current on a ground fault to throw your average 15A breaker but there is no actual test to verify intent. In some cases the resistance on the service ground added to the resistance on the circuit under fault might not actually trip a breaker. Remember that the code intent is not to protect sensitive electronic equipment but rather to save lives and buildings.

Also, code requirement for bonding is #6 wire, not that puny bare copper in your 12/2 feed.

I have always suggested when bonding service ground rods to tower ground rods to use #2

As you said those standards are not intended to make sure there is not more than 5 Ohms of ground potential to protect sensitive equipment.

But I still say in his application do not worry about bonding the service and tower grounds as they as so far apart, simply do not use the service ground and only use the tower grounding system as his electrical earth ground.

I do this at many of my sites and it stopped my problems. I was not the genius that came up with this I happen to be located on a cell tower owned by an electrical service company and they were providing me AC power from 200+ feet away and I was bonding the 12-2 ground wire to my ground bus in my box bonded to the tower ground system. Their service grounds at their shop were poor and the ground current was flowing across that 200+ feet of small copper trying to get to the GOOD ground halo ring the cellular companies put in. That 12-2 wire ground wire heated up and caught fire. THey came out and told me how stupid I was and explained why I did not want to use their ground just use their Hot and Neutral and use the tower Earth ground as my Electrical Earth ground and problems went away.
So that is an example how Ground Current can cause damage, in this case it was trying to push so much ground current across that 200' of cable that it heated up the ground wire so hot it caught the wire on fire. Luckily it tripped their breaker in their shop and the wire fire was nowhere near anything else that burned so no 3 alarm fire ensured.

[Dawizman]

Alright, so I took your advice. We removed the ground coming in from the electrical service. I measured between that wire, and our ground buss, and there was about 0.75V, so there was definitely a potential difference. We also replaced the switch with another WS-12-400-AC. Since then, we have still recorded reboots of every single 24V device on the tower, and all at the exact same time. Only the AF5X has not rebooted.

I'm not sure where to go from here... Are the radios all in need of replacement? Has the switch been damaged somehow? It seems like too much of a coincidence that two switches would be doing the exact same thing. I have the first switch here on my bench, loaded up with 24V radios. I'm going to see if it has an issue here.

[sirhc]

You did upgrade to v1.4.7rc18 right, when we are helping with a problem we like to be on the latest version as a baseline starting point?

Next set up your LAB as you suggested and test on bench, great idea.

Do you have a record on how often this occurs or does it always happen at the same times?

On your port Descriptions it is always a good idea to say what the radios is (just a suggestion). Like "RM5 10.0.0.0" or "RACL 10.0.0.0" or "NB500 10.0.0.0"
Would be nice if I knew what radios are involved.

Now you say "ALL" 24V device reboot at the same time?

So since all 24V POE ports share the same 24V power supply on the board you could have one of the following:
1) A bad radio that pulls way too much current and dips the 24V power supply

2 A bad cable or end that shorts out when it is moving maybe in the wind. This can be caused by a strap rubbed through to bare wires. If this happens often enough you could try turning POE OFF on a group of ports for the duration to see if it stops to narrow down the offending port.

[Dawizman]

Yes, both switches are running 1.4.7rc18. Our staff are aware what the radios are based on their IP addresses. In this instance it is as follows:

Port 1: Airefiber 5X
Port 2: Rocket M5
Port 3: Powerbridge M5
Port 4: Rocket M2
Port 5: Rocket M2
Port 6: Rocket M2
Port 7: Rocket 5AC
Port 8 : Rocket 5AC
Port 9: Rocket 5AC
Port 10: Rocket M365
Port 11: Rocket M365
Port 12: Rocket M365

As for when it occurs, it is random. Anywhere from 30s to 42 hours. Here is the uptime chart for one of the AP's from PRTG:

[sirhc]

And all 24V radios reboot at the exact same time? ( verifying )

They simply reboot once then 30-40 hours rinse and repeat?

[Dawizman]

Most of the time they all reboot at the same time, however I see a few occasions where one or two of the radios did not reboot. It's anywhere from 30 seconds to 40 hours, most commonly around 25 minutes or so.

The switch on the bench here definitely has two blown Ethernet ports. The radios power, but get no Ethernet link. My guys replaced the switch at the tower before I was able to get out there to make changes to the grounding, so I'm wondering if the replacement switch got damaged in that time. I may throw another switch in, and see if it works, and then send these two in for repair.

Both switches are rev D. I'm wondering if the ground current mod would have prevented this issue? Can you guys perform the ground current mod if we send these switches in for repair?

[sirhc]

Yes the ground current MOD will help a lot against ground current and YES we can and do apply all hardware MODs to any switch that comes in for RMA free of charge making them basically the same as what the newest boards are.

Submit an RMA request for those 2 units.

In "most" cases blown ports are cheap to fix and you get the latest MODs added.

1-2 Blow port is $50
3-4 Ports $75

A whole new board for that model is $150 but if the switch still works that is rare that it needs a new board, depends if a trace is blown off and no repairable or a PHY in the SOC is damaged.

[sirhc]

You by chance are not using Ethernet Surge protectors are you?

Ethernet Surge protectors are BAD with passive POE devices as they basically clamp all wires to ground creating a dead short. Active POE can deal with this but not passive POE it usually results in damage to the port and would explain all the radios rebooting as it would be shorting out the 24V power supply.

viewtopic.php?f=5&t=2525&p=18239&hilit=+passive+ethernet+surge#p18239

viewtopic.php?f=14&t=2578&p=18052&hilit=+passive+ethernet+surge#p18052

viewtopic.php?f=17&t=2379&p=16718&hilit=+passive+ethernet+surge#p16718

This topic has been discussed over on Facebook WISP group as well.

[Dawizman]

We do not use surge protectors at most of our sites. Unfortunately, some Telco sites we are at require us to use surge protectors. The grounding there is done incredibly well however, and we have yet to ever have a problem.

I am on my way out to replace the switch as soon as the firmware updates on the new switch on my desk. I will be sending the other two units in for repair. They are definitely worth repairing over throwing out, and we have had our fair share of RMA's (techs that didn't follow instruction for the most part).

[Dawizman]

I'm curious Chris... where do you ground ubiquity sectors, and rocket dishes? I am wanting to start grounding at the top of the towers as well as the bottom. Air fibers are easy, but there is no dedicated ground point on the rocket radios. We use all ubiquity sectors and dishes with RF armor if that makes a difference.

Thanks