We have a WS-12-250-DC on a tower and are having issues where
a) In the status section the temperature and power sections all flip to ERROR off and on.
b) Some units are losing connection and then being rebooted by watchdog occasionally
We are running the latest firmware 1.3.8. We've replaced the DC power supply twice, and the actual netonix unit twice. Still having the same issue.
Any ideas what might be going on?
WS-12-250-DC ; Error reading temp sensor
-
zhall - Member
- Posts: 19
- Joined: Mon Dec 01, 2014 2:01 pm
- Has thanked: 0 time
- Been thanked: 0 time
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC ; Error reading temp sensor
Please post up screen grabs of your Tabs that have config changes including the Power Tab, Device/Status Tab, and Device/Configuration Tab
What type of site is this, meaning is this a co-located site and if so how busy is it (how much RF)? A couple people who went to put our DC switches at co-located tower sites with a LOT of RF noise had issues with the switch picking up RF noise that messed with the communications between the switch and the DC power supply. This does not affect our AC units just our DC units and we are working on a fix for those customers.
What are you powering the switch with (batteries, chargers).
As far as ping watchdog that is probably a separate issue so why not turn that off for now and lets figure out the other stuff first.
What type of site is this, meaning is this a co-located site and if so how busy is it (how much RF)? A couple people who went to put our DC switches at co-located tower sites with a LOT of RF noise had issues with the switch picking up RF noise that messed with the communications between the switch and the DC power supply. This does not affect our AC units just our DC units and we are working on a fix for those customers.
What are you powering the switch with (batteries, chargers).
As far as ping watchdog that is probably a separate issue so why not turn that off for now and lets figure out the other stuff first.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
zhall - Member
- Posts: 19
- Joined: Mon Dec 01, 2014 2:01 pm
- Has thanked: 0 time
- Been thanked: 0 time
Re: WS-12-250-DC ; Error reading temp sensor
http://imgur.com/a/1Zwx2
It's a 250ft tower (I think) with our equipment around the 200ft mark. At the top are 2 sets of those black Batarang shaped FM broadcast antennas. In the building the rack unit for the FM station says "forward power 191 watts".
At first we were powering it using an industrial-style "48V 5A" power supply, somewhat like this one: http://www.ebay.com/itm/AC-110V-220V-to ... 1555738541
Today we replaced the power supply with an Iota DLS-5413: http://www.iotaengineering.com/dls5413.htm
Replacing the power supply did not change the behavior.
There are no batteries on the DC side.
We have FTP Cat-6 connected to the power supply outputs, using orange/blue for positive, green/brown for negative, with the drain wire connected to the power supply's ground lug. 200 feet up the tower in a polycarbonate box we have the other end of the Cat-6, with the pairs connected to the screw terminals on the WS-12-250-DC, and the drain wire connected to its ground screw.
It's a 250ft tower (I think) with our equipment around the 200ft mark. At the top are 2 sets of those black Batarang shaped FM broadcast antennas. In the building the rack unit for the FM station says "forward power 191 watts".
At first we were powering it using an industrial-style "48V 5A" power supply, somewhat like this one: http://www.ebay.com/itm/AC-110V-220V-to ... 1555738541
Today we replaced the power supply with an Iota DLS-5413: http://www.iotaengineering.com/dls5413.htm
Replacing the power supply did not change the behavior.
There are no batteries on the DC side.
We have FTP Cat-6 connected to the power supply outputs, using orange/blue for positive, green/brown for negative, with the drain wire connected to the power supply's ground lug. 200 feet up the tower in a polycarbonate box we have the other end of the Cat-6, with the pairs connected to the screw terminals on the WS-12-250-DC, and the drain wire connected to its ground screw.
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC ; Error reading temp sensor
Please install v1.3.9rc8 and see if that helps and let me know.
v1.3.9rc8 has some code to help mitigate RF noise from broadcast towers.
If the firmware does not help we have an experimental shielded cable that i may need to send you but that will not happen until after the New Year.
v1.3.9rc8 has some code to help mitigate RF noise from broadcast towers.
If the firmware does not help we have an experimental shielded cable that i may need to send you but that will not happen until after the New Year.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
Dave - Employee
- Posts: 726
- Joined: Tue Apr 08, 2014 6:28 pm
- Has thanked: 1 time
- Been thanked: 158 times
Re: WS-12-250-DC ; Error reading temp sensor
zhall
If I understand your comment correctly, it sounds like the Netonix switch is not very far away from two FM transmitter Antennas. This could be a case of to much RF interference from an outside source (the FM antennas).
I want to make sure I understand correctly, is the Netonix switch indeed about 50 feet from the two FM antennas? And if so, how is the Netonix switch mounted (i.e. nema box, etc)?
Thanks.
Dave
It's a 250ft tower (I think) with our equipment around the 200ft mark. At the top are 2 sets of those black Batarang shaped FM broadcast antennas. In the building the rack unit for the FM station says "forward power 191 watts".
If I understand your comment correctly, it sounds like the Netonix switch is not very far away from two FM transmitter Antennas. This could be a case of to much RF interference from an outside source (the FM antennas).
I want to make sure I understand correctly, is the Netonix switch indeed about 50 feet from the two FM antennas? And if so, how is the Netonix switch mounted (i.e. nema box, etc)?
Thanks.
Dave
-
JayDee - Member
- Posts: 18
- Joined: Wed Jan 06, 2016 9:12 pm
- Location: Harrisonburg, VA
- Has thanked: 1 time
- Been thanked: 0 time
Re: WS-12-250-DC ; Error reading temp sensor
Hi, I've been working with zhall on this issue.
As I understand it, the problem is thought to stem from uncorrected bit errors on the i2c bus, introduced by the nearby RF field.
Do you guys have a way to control the i2c bus frequency from software? We may find that a lower (or higher) bus frequency could sidestep the issue.
Also, some kind of sanity checking on the received values might help the software tolerate such errors.
As I understand it, the problem is thought to stem from uncorrected bit errors on the i2c bus, introduced by the nearby RF field.
Do you guys have a way to control the i2c bus frequency from software? We may find that a lower (or higher) bus frequency could sidestep the issue.
Also, some kind of sanity checking on the received values might help the software tolerate such errors.
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC ; Error reading temp sensor
v1.3.9rc12 has some additional enhanced I2C error correction over v1.3.8
v1.3.8 was the beginning of our attempt to correct for this and have implemented several enhancements through the v1.3.9rcX versions which has greatly improved things for these HEAVY AM/FM/TV RF sites.
We are also working with a couple people who are on HEAVY AM/FM/TV broadcast towers that are trying an experimental shielded I2C cable between the switch board and the DC2DC power supply. see picture below
Notice the white ground/shield wire that connects to the chassis via the switchboard screw, but for this to work the user MUST have the chassis grounded to earth ground via the earth ground screw on the front of the chassis since this is a DC switch the switch NEEDS an earth ground which should also be bonded to the tower ground system. If the switch does not have an earth ground then the ESD shielding on the Ethernet is useless.
That is an interesting idea, changing the I2C communication Hz, but I do not think that is an option.
It is not a matter of sanity checking an erroneous value returned it is more of a matter that the I2C device being polled errors and does not respond and hangs the I2C bus which is why other I2C values which all share the same I2C bus stop responding. We have implemented I2C recover routines which work and do recover the the I2C but if the RF is constant it will constantly error out the I2C so you will see ERROR come back as the RF interference cycles.
v1.3.9rc12 is MUCH better and does recover but does not completely mask the issue as the I2C recovery takes so many ms to occur and we are polling everything every second so repeated error correcting can cause a bottleneck which shows up as ERROR.
v1.3.8 was the beginning of our attempt to correct for this and have implemented several enhancements through the v1.3.9rcX versions which has greatly improved things for these HEAVY AM/FM/TV RF sites.
We are also working with a couple people who are on HEAVY AM/FM/TV broadcast towers that are trying an experimental shielded I2C cable between the switch board and the DC2DC power supply. see picture below
Notice the white ground/shield wire that connects to the chassis via the switchboard screw, but for this to work the user MUST have the chassis grounded to earth ground via the earth ground screw on the front of the chassis since this is a DC switch the switch NEEDS an earth ground which should also be bonded to the tower ground system. If the switch does not have an earth ground then the ESD shielding on the Ethernet is useless.
That is an interesting idea, changing the I2C communication Hz, but I do not think that is an option.
It is not a matter of sanity checking an erroneous value returned it is more of a matter that the I2C device being polled errors and does not respond and hangs the I2C bus which is why other I2C values which all share the same I2C bus stop responding. We have implemented I2C recover routines which work and do recover the the I2C but if the RF is constant it will constantly error out the I2C so you will see ERROR come back as the RF interference cycles.
v1.3.9rc12 is MUCH better and does recover but does not completely mask the issue as the I2C recovery takes so many ms to occur and we are polling everything every second so repeated error correcting can cause a bottleneck which shows up as ERROR.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
JayDee - Member
- Posts: 18
- Joined: Wed Jan 06, 2016 9:12 pm
- Location: Harrisonburg, VA
- Has thanked: 1 time
- Been thanked: 0 time
Re: WS-12-250-DC ; Error reading temp sensor
Hmm.
We've got 1.3.9rc12 on the affected switch now, and it does not appear to behave any differently than 1.3.8 did.
http://i.imgur.com/I7x3tj4.png
As a group, all the "Power Supply" values together usually show "Error", and will show all their correct values maybe 1 in 10 times.
"Board Temp" and "CPU Temp", independently of one another will sometimes show the correct value, sometimes an erroneous value, and sometimes "Error".
"Fan Speed" will occasionally show an erroneous (high) value, but never "Error". Usually the correct value is shown.
All the other values on the Device Status page appear to always be shown correctly.
We've got 1.3.9rc12 on the affected switch now, and it does not appear to behave any differently than 1.3.8 did.
http://i.imgur.com/I7x3tj4.png
As a group, all the "Power Supply" values together usually show "Error", and will show all their correct values maybe 1 in 10 times.
"Board Temp" and "CPU Temp", independently of one another will sometimes show the correct value, sometimes an erroneous value, and sometimes "Error".
"Fan Speed" will occasionally show an erroneous (high) value, but never "Error". Usually the correct value is shown.
All the other values on the Device Status page appear to always be shown correctly.
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC ; Error reading temp sensor
We are aware that at HEAVY RF sites like those with AM/FM/TV broadcast towers we may not be able to fix this with just firmware which is why we are experimenting with a shielded I2C cable as seen above. We are waiting for some feedback from those people that are trying them.
Yes, when the I2C cable picks up noise it disrupts the I2C bus which will affect Temp, voltage, and current sensors but it has no affect on the FAN RPM sensors to my knowledge.
I want to make a point that those people who are having issues at these HEAVY RF sites are a very low minority as there are now thousands of DC switches out there in service with only a handful of people at these difficult RF sites that are having any issues.
For the most part with the latest v1.3.9rc version the issues is primarily that during the interference period from external RF sources the switch just loses telemetry information which is indicated by the UI reporting ERROR for those sensor readings, it does not affect the switching functions of the switch.
There is 1 customer in Panama who has the switch installed on top of a volcano on a tower with AM/FM/TV broadcasts that suspects the prolonged interference has caused his DC switch to power cycle but he is running an older version of firmware, I think v1.3.4 which is before we implemented a lot of fixes. He waiting to try the upgrade until he is at the tower site as it is a 3-4 hour journey to get to the switch.
Yes, when the I2C cable picks up noise it disrupts the I2C bus which will affect Temp, voltage, and current sensors but it has no affect on the FAN RPM sensors to my knowledge.
I want to make a point that those people who are having issues at these HEAVY RF sites are a very low minority as there are now thousands of DC switches out there in service with only a handful of people at these difficult RF sites that are having any issues.
For the most part with the latest v1.3.9rc version the issues is primarily that during the interference period from external RF sources the switch just loses telemetry information which is indicated by the UI reporting ERROR for those sensor readings, it does not affect the switching functions of the switch.
There is 1 customer in Panama who has the switch installed on top of a volcano on a tower with AM/FM/TV broadcasts that suspects the prolonged interference has caused his DC switch to power cycle but he is running an older version of firmware, I think v1.3.4 which is before we implemented a lot of fixes. He waiting to try the upgrade until he is at the tower site as it is a 3-4 hour journey to get to the switch.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
JayDee - Member
- Posts: 18
- Joined: Wed Jan 06, 2016 9:12 pm
- Location: Harrisonburg, VA
- Has thanked: 1 time
- Been thanked: 0 time
Re: WS-12-250-DC ; Error reading temp sensor
sirhc wrote:I want to make a point that those people who are having issues at these HEAVY RF sites are a very low minority as there are now thousands of DC switches out there in service with only a handful of people at these difficult RF sites that are having any issues.
Indeed, we have a dozen or more other Netonix switches deployed at various sites, including at least one other WS-12-250-DC. Even at this same site in this same tower box we have two WS-6-MINIs. None of them have shown any significant issues.
sirhc wrote:For the most part with the latest v1.3.9rc version the issues is primarily that during the interference period from external RF sources the switch just loses telemetry information which is indicated by the UI reporting ERROR for those sensor readings, it does not affect the switching functions of the switch.
Well, we do still see things like this happening sometimes on 1.3.9rc12:
!Disabling PoE power on port 12 due to Power Supply Thermal out of specification (controller is 30 C)
Who is online
Users browsing this forum: sirhc and 50 guests