i2c errors

[jbaird]

We have a WS-12-250-DC running 1.3.7 that just started 'freaking' out (see attached logs/screenshots). Several 'i2c' errors and other errors as well (see logs). In this state, the switch was still forwarding packets, but interface errors started to increment (at least on the UI) once this started happening. We typically see 0 errors on all interfaces.

Screenshots also attached as well as a full log after a warm (remote) reboot. The errors and everything persist after the warm reboot. The switch appeared to have recovered after a cold reboot.

Has anyone seen this? At this point, I'm thinking it may be due to a power event at the site? This particular site has an AC UPS and a DC power supply. We use this same setup at several of our sites with these switches. The switch is getting +48VDC. It had been running fine for over a week until this event.

[sirhc]

My guess is the switch was damaged from a surge or ground potential shift from poor grounding.

All the sensors share the I2C bus.

I am at a loss as to why you would use a WS-12-250-DC in this type of setup as you are obviously supplying the switch a constant voltage of 48V from an AC to DC power supply so why not use the WS-12-DC instead of the WS-12-250-DC?

The WS-12-250-DC is designed to hook to batteries direct and then a charger to keep the batteries topped off. When AC power goes out or during the night on OFF Grid sights using solar panels or non windy days with a windmill the WS-12-250-DC can continue to operate on different voltages as the batteries discharge and maintain constant POE voltages until the batteries start charging again.

Plus if you are at a site with AC grid power why not use an AC model like the WS-12-250-AC and then use a product like the Tripp Lite Power Inverter Charger as seen here: https://www.google.com/webhp?sourceid=c ... e%20aps750

Your using the wrong model for your application? Why would you pay the extra money for the WS-12-250-DC instead of the WS-12-DC if your going to feed it conditioned constant 48V???

But as I said why not use an AC model as you have grid power?

Anyway lets determine what may have damaged this unit.

Please describe your tower and box where your equipment is located.

Please describe where you get AC power from and how.

Please describe your grounding setup on your tower.

Can you post some pictures of the inside of the box and the site including the box and tower.

Here are some good posts on grounding:
http://forum.netonix.com/viewtopic.php?f=30&t=1816
http://forum.netonix.com/viewtopic.php?f=30&t=188
http://forum.netonix.com/viewtopic.php?f=30&t=1429
http://forum.netonix.com/viewtopic.php? ... =30#p13447

[lligetfa]

At this point, I'm thinking it may be due to a power event at the site? This particular site has an AC UPS and a DC power supply..

Can you better describe this power arrangement? If you suspect a power event, I'm guessing there is/was some sort of transfer lag time. A momentary power sag as when a UPS transfers from line to inverter can leave the switch in an indeterminate state.

[jbaird]

We use the WS-12-250-DC everywhere and have standardized on it. I don't want to have to deal with multiple models. This switch works at all of our sites (both 24, 48, off-grid, etc).

This particular site used to be all AC, but has been converted to DC beyond the AC UPS that was already in place. Once the AC UPS needs to be replaced, we will convert it to a full DC site (with a charger and batteries at the bottom).

This site has the AC UPS and DC power supply at the bottom. We run 12AWG DC and fiber up the tower to where the switch is mounted. The DC has surge suppression. The switch is bonded to the tower ground at the top of the tower (as is the charger, etc at the bottom of the tower). All radios go through surge suppression (GIGE-APC-HV) before being cabled to the switch. We get AC power from utility which is also bonded to the common/tower ground. It's a water tank site.

I don't have pictures handy, but will try to get some.

[jbaird]

Can you better describe this power arrangement? If you suspect a power event, I'm guessing there is/was some sort of transfer lag time. A momentary power sag as when a UPS transfers from line to inverter can leave the switch in an indeterminate state.

There is an APC XL unit (with external batteries). The AC/DC power supply is connected directly to the APC XL unit (AC). The rest of the site is DC, and get's +48VDC from fused distribution on the load side of the AC/DC power supply (Traco TSP). The APC did log that it (very quickly) had power input problems (3 or so in ~5 seconds) but it quickly recovered. We normally don't see transfer lag on these units, but this one COULD be faulty in some way.

[sirhc]

These are the specs on your switch:
MAC Address: EC:13:B2:81:0A:3A
Power Supply SN: 144952-0321
Switch Board SN: 145337-0408
Manufacturing Date: 09/29/2015

First remove the unit from service and test it back at your shop on the bench but my guess is it will not do any differently as far as reporting sensor telemetry.

If it does not act differently on bench then I give you permission to open this unit up and remove the switch board and take picture of the front and back of the switch board and post them up here.

My guess is you will see damage similar as seen below:

CLICK IMAGE TO VIEW FULL SIZE

[sirhc]

Now if your power source did malfunction and sent a surge of greater than 80V even for a second it would have blown out a current sensor which depending on how it blows out the sensor it can cause the entire I2C bus to fail which means all sensors would report ERROR.

Now in the new Rev F boards we put in protection DIODE circuits to protect the sensors from excessive voltages out of the speced range in hopes to prevent this type of damage from excessive surges as we had several people report that their off grid chargers malfunctioned and sent 80V+ to the batteries for what ever reason bowing out the sensors.

Now keep in mind the sensors are rated to 80V and we only originally spec'd this unit to accept 9V-60V then later 62V with a firmware upgrade.

We are now shipping with Rev F Switch Boards and DC2DC Power Supply Rev B where we allow 9V-72V which is still below the rated voltage of the sensors we use.

We could not use higher rated sensors and still be anywhere near accurate at lower voltages/current that we use.

[jbaird]

Well, as I said, the switch is functioning properly after a cold reboot. It's now reporting proper board voltages and not logging any errors. So, I'm hoping there is no permanent damage.

[sirhc]

Well, as I said, the switch is functioning properly after a cold reboot. It's now reporting proper board voltages and not logging any errors. So, I'm hoping there is no permanent damage.

OH, I missed that!

Well as Les said if there was a power blip it could have caused the switch or the power supply micro to half ass power cycle meaning it was powering down but did not discharge before power was restored so some components were in a weird state. I have seen this many times.

You can easily recreate this on a bench with a WS-6-MINI buy pulling power for a second or two then re-applying the power and now the WS-6-MINI is in a messed up state and sensors and MOSFETS are screwed up.

Powering the WS-6-MINI down for 15 seconds then powering it up restore sanity.

It is never good to cycle power fast to electronics.

In fact power cycling devices fast several times often corrupts the flash basically bricking it. We have gotten several of these back for RMA and all we have to do is re-flash the unit and it is OK and we return it.

[lligetfa]

I am thinking there was more likely a sag than a surge and that the sag left the switch in an indeterminate state that a hard power cycle cleared.

I don't care for standby type of UPS that transfer from line to inverter. If not running a DC system with batteries, I prefer to use a true online UPS that always run from the inverter.

Since you use a switch that can take unregulated DC, your best bet would be to convert over to a battery banked DC plant that avoids transfer lag.