Power out of range? (3V)

[jakematic]

And so it begins...

Code: Select all

 Aug 21 03:03:55 ws8b.jakematic.lan switch[1330]: !3V is out of range (3.1V) 

A little background.
I'm not a WISP, but am spinning up a business for residential customers and looking for something superior to ToughSwitches and off the shelf crapola.
My own install is meant to be the showcase to show prospective clients.

What I have is is not atypical from most layouts.
Is there a ground issue? Maybe. I don't know. I don't want to spend thousands on testers or electricians.
Adding that cost to customer installs is out of the question.
I spent 35 years in a family owned dealership, am now in technical pre-sales, and have a good grasp of what customers will/won't swallow.

Right now I've got 2 dead and 1 dying WS8-150-AC
- one back under RMA to be analyzed
- one I take full responsibility for killing and am waiting on RMA (powered my uplink port on the bench by mistake, fried a port on a Netgear, killed the WS8)
- one that is dying in production

Also have a WS12-250-AC that is chugging along for 3 days with no problems.
And a TS5 that's been in service for several years showing no signs of any issue.

The only thing that I can come up with is to electrically disconnect the two switches.
To do that I'd put in another WS-12-250-AC, run fibre beween the WS-12s, and pull all the PoE cables back to the new WS-12 thus eliminating a switch.

Does this sound reasonable, and is it the only way forward for other installs ?

What kind of concerns me is there are no posts about the W8-150-AC other than a similar problem: viewtopic.php?t=1905

I can't keep throwing money at this.
Either there is a problem with the WS8, or for residential networks fibre backbone is a better idea, or both.
Dunno, but I'm going to have to run fibre and install another WS12 at this point to replace the dying WS8.

[sirhc]

What I have is is not atypical from most layouts

I must admit when I said give me a rough drawing maybe I chose my words poorly and you took the words rough to extreme as I can not figure out your previous posted drawing but that is OK we will address below.

Is there a ground issue? Maybe. I don't know. I don't want to spend thousands on testers or electricians.
Adding that cost to customer installs is out of the question.
I spent 35 years in a family owned dealership, am now in technical pre-sales, and have a good grasp of what customers will/won't swallow.

There is no need to use electricians but simply understand what NOT to do with Ethernet.

THINGS NOT TO DO:
1) Never connect Ethernet devices that are on 2 different Earth Ground Potentials such as having a tower grounding system that is not properly bonded to the electrical ground system. The further the electric service ground rods are from the tower ground rods the heavier the ground bonding cable needs to be or if more than 30 to 50 feet one should opt to only use the Tower Earth Ground system only using the Hot and Neutral from the Electrical service and using the Earth Ground from the tower as described in this post: viewtopic.php?f=17&t=1786&start=30#p13447

2) Never connect Ethernet devices that are each located on different electrical services or even different sub-panels as they will have different ground potentials and it will try to equalize across your Ethernet cable and switch, in these cases use fiber. This is discussed in this post: viewtopic.php?f=30&t=1816

3) Never plug any device onto an Ethernet network that is not AC surge protected as that would be like locking all your doors and windows in your house except the back door before going on vacation. An AC surge can come in on any device and run across the Ethernet network and damage random equipment.

Right now I've got 2 dead and 1 dying WS8-150-AC

OK, lets look at the units in play

- one back under RMA to be analyzed

I am going to send you your repaired unit (new board installed) back tomorrow and air ship the damaged board to Dave (our Engineer) in Canada for an in-depth analysis to determine what is going on. He can do this much better than I can do here as I do not have the equipment to do such and analysis and this is above my pay grade anyway. Now this unit was exposed to AC without a surge suppressor so it "could" have been damaged by an AC surge?

- one I take full responsibility for killing and am waiting on RMA (powered my uplink port on the bench by mistake, fried a port on a Netgear, killed the WS8)

OK this one is most definitely your fault, I will get you an RMA number tomorrow on this to send in for repair.
DO NOT BLAME THE SWITCH FOR THIS ONE.

- one that is dying in production

This one I would leave run as is and see what happens until we can diagnose the issue on the first unit you sent back.

Also have a WS12-250-AC that is chugging along for 3 days with no problems.

So leave this one run as is.

And a TS5 that's been in service for several years showing no signs of any issue.

Noted.

The only thing that I can come up with is to electrically disconnect the two switches.
To do that I'd put in another WS-12-250-AC, run fibre beween the WS-12s, and pull all the PoE cables back to the new WS-12 thus eliminating a switch.

Does this sound reasonable, and is it the only way forward for other installs ?

Well I could better answer this question if you take the time to make a better drawing and provide details on the Electric service or services at this location.

What kind of concerns me is there are no posts about the W8-150-AC other than a similar problem: https://forum.netonix.com/viewtopic.php?t=1905

What that person has in that post is most definitely 2 ground potentials. There is nothing we can do to protect against this behavior and is a common issue when using "any" Ethernet devices not just ours with 2 ground potentials as outlined in this post which is quote from another Engineer from another company preaching the same things I am trying to educate people on: viewtopic.php?f=30&t=1816

I can't keep throwing money at this.

Either there is a problem with the WS8, or for residential networks fiber backbone is a better idea, or both.

Dunno, but I'm going to have to run fibre and install another WS12 at this point to replace the dying WS8.

Well I would really like to determine if this is a ground potential issue which would require a better drawing from you and a better explanation of the electrical service or services involved with this setup. If you make a better drawing maybe using MS Paint and post it and then PM me your cell number I can call you later today and discuss.

So what we are looking at as possible causes:
1) You do have a ground potential issue here and we will help you find it if possible.

2) We have a faulty component on some of the boards, as it does happen. You get reels of components and some percentage of the components are faulty sometimes which is why we have batch numbers to track this.

3) We are also discussing the possibility of a software issue as we may not be properly sampling/averaging the 3.3V readings but Dave wants to determine if the 3.3V is actually fluctuating once he receives your board and looks at it.

[jakematic]

First, thank you very much for taking the time to look at this.

Hopefully the drawing attached is a bit better.
As you can see I'm no artist !

It is very possible there is a ground issue.
Have worked in data centers for decades and although I'm no electrical engineer am familiar with them... just didn't expect to see one in this type of setting.
If there is, I just have to work around it and try to avoid them at all costs for other installs I do.

This experience will lay the ground work for future setups and the price is better paid now than later.

You are correct the one you have now wasn't sure protected, totally my fault.
Everything is now.

There are no sub-panels, and if you have any questions on the picture let me know.
EDIT: That main ethernet trunk may be closer to 75' now that I think about.

Thanks again for your time - hopefully this is something simple.... but I rarely get those

[sirhc]

OK this drawing is AWESOME - Thanks

The only possible ground issues could be that the Electric service ground rods need attention or another ground rod added which is easy to do. If your ground rod or rods are not good then there is no place for surge protectors to shunt excess current to and this excess current is trapped within your electric service. For WISPs this is often seen when they encounter radios at customer houses with blown Ethernet ports but the radio portion still works fine.

Please examine your electric service ground rods and ground wire connection in your panel to insure they are SOLID in the ground and connections are free of corrosion and tight.

You may also wish to consider adding an additional new ground rod about 4 feet away from existing rod or rods and bond them to the old ground rods with #6 copper and 2 acorn connectors.

At my house 17 years ago when I first got into being a WISP I had a poor ground rod and often lost radios here every time it rained. Once I added a new ground rod the problem went away and I have never lost any additional equipment since then. You can get a ground rod, 2 Acorns, and some #6 copper wire at Lowes or Home Depot.

Tomorrow I will send you back your WS-8-150-AC you RMA'd

I would like to take that TS5 out of the picture for a couple reasons but mainly because the TS5 is powered by a barrel connector and has no access to Earth Ground.

Once you get the WS-8-150-AC returned do you have enough switches to have all switches Netonix? If not you could purchase another one and I will ship it to you with the repaired RMA unit and if in the end we can not solve this issue I will take it back as a return.

But please check your ground rods and please consider adding a new ground rod before we replace the existing switches.

[jakematic]

Will re-check the ground rounds and it never hurts to add another as per your description.

Yes it will be Netonix end to end.
My original plan was to have the 2 WS8 on the far end along with a spare WS8, and the WS12 was already on the way to complete the office end.

At least bill me for shipping on the RMA unit. If I killed it due to no surge protection I'll gladly pony up the repair costs.

Off to Lowe's...

[jakematic]

Additional ground rod is in and bonded with #6 to existing grounds.

[sirhc]

Additional ground rod is in and bonded with #6 to existing grounds.

Did you drive it all the way in?

Was it 8' long?

Is it 4' from existing ground rods?

Did you verify all the connections are clean and tight?

You also might want to check that the wire that runs from the ground rods to the service panel is not cut and has a tight connection in the service panel. If you are not confident in opening the panel cover seek assistance here.

[jakematic]

All the way in. Which was fun in Badin-Urban soil profile...
It is an 8' - not sure they even sell anything else, and they wouldn't be to code anyway.

6' from the existing, as per code.

Emery papered the rods, the cable at the connection, added a new acorn to existing, cleaned up the existing again since I was already there, the whole works.
We used to have a construction company too... not my first rodeo.
Although I sure as hell can't swing a sledge like I used to... getting old sucks.

Panel run is good too.


ps- we also had several good size farms and dairies as a family only operation.... got out and went into computing since I'd already spent 3 lifetimes of physical labor by age 18

[sirhc]

Eric and Dave will be investigating the software tomorrow to make sure we are properly dealing with erroneous readings.

[jakematic]

Cool, thanks.

Did notice there is no Watts column on the main status page of the WS-8-150-AC like there is on the WS-12-250-AC/

Is this architectural differences ?


WS8 is running 1.4.3rc6 and WS12 is running 1.4.2