PROBLEMS with DC switches at or near FM tower locations

[lligetfa]

It is in a lift motor room.

Do you mean in an elevator penthouse?

Decades ago I was having a really weird issue with a network going down about 4 times a day, morning noon, 1:00PM, and quitting time. Turned out the installers ran the wires too close to the lift motor power feed in the elevator shaft. Whenever the elevator car was full to capacity the current on the motor was enough to take down the network.

[wisp-wand]

Yep , but if its the lift / elevator gear, why does the switch issues seem to occur out of hours when no one is using the lift/elevator. Switch has no trouble when lift is busy in the morning and evening rush, but then does play up when building is closed.
Odd

Wayne

[wisp-wand]

OK this switch caused an outage overnight, it went down at 02.45am and I decided to attend in order to check it out.
So this is the second unit I have, DC power supply was fine, voltage normal, but fan not running, no power light, I could see the internal LED on though. No ports were lit up, no POE or link lights.
Power cycled the unit and it came straight back up. Wonder if it was in hibernation. Have swapped back to the original switch, as they both exhibit the same issues.
I have enabled syslog, as we didn't have that on before, so I can't see what led up to the switch shutting down, I could guess its going to be the usual voltage/temp out of tolerance messages though.
While I was there I took a medium wave / AM radio to allow me to listen to any electrical noise, this has been a great way to find issues in the past. Surprisingly there is little electrical noise present, unless I remove the earth wire connected to the switch!!
The lift gear, creates little electrical noise, I guess they are all soft start now. The florescent tubes cause the usual interference and so I tried turning them on and off a few times to provoke the switch to fault. I also turned the power on and off to the PSU 5 times, but the batteries just took the load, and nothing logged on the switch. I might change this site back to the previous DC-DC config with Cisco SG300-10 and POE injectors.

Can Chris or anyone in support suggest anything? Did you take onboard my suggestions to stop the switch having knee jerk reactions to spurious PSU values?

[sirhc]

I have asked to review your posts.

Something has to be different here with your setup as there are several thousand WS-12-250-DC units in service.

I am sorry we do not have an answer here yet.

Please post up your remote sys log when you get them.

[sirhc]

I see you are using 24V battery bank but is there any possible way your charging unit could have a spurious output voltage above 60V?

If the power supply detects voltage above 60V it will shut down.

v1.4.0rc4 does increase this to 61.5V.

Do you have another charging unit you could swap with? I know I am grasping at straws here.

But on our front Eric is supposed to review his code based on the weird log entries on voltage and temperature.

The only other thing would be some type of intermittent RF interference in the FM band but you seem pretty confident this is not possible?

Really scratching my head here.

[wisp-wand]

It is a 24v SMPSU charger/PSU with 2 x 12v 7ah batteries, the output is the float voltage, so can't see how it can get above 30 volts without some serious psu and/or battery damage.

Its just disabled the backup link again, then brought it back in, funny though the controller is 33c now, so I guess the spike was so quick the temperature logged was not the one that caused the alarm in the first place, as 33c seems normal. Current ambient temp is 18c.
Its clear there is something upsetting the link between the PSU and switch software. I can't tie up any cause for the problem, but I do think the software needs to be changed, maybe a tick box to say 'ignore PSU environment alarms' or some hysteresis built in, perhaps voltage and temperature values are averaged over a few readings taken with say 5 seconds apart.

Mar 20 20:18:51 switch[803]: Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
Mar 20 20:18:51 kernel: link state changed to 'down' on port 2
Mar 20 20:18:56 switch[803]: Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
Mar 20 20:18:59 kernel: link state changed to 'up' on port 2
Mar 20 20:19:04 kernel: link state changed to 'down' on port 2
Mar 20 20:19:05 kernel: link state changed to 'up' on port 2
Mar 20 20:19:22 kernel: link state changed to 'down' on port 2
Mar 20 20:19:24 kernel: link state changed to 'up' on port 2
Mar 20 20:19:28 kernel: link state changed to 'down' on port 2
Mar 20 20:19:30 kernel: link state changed to 'up' on port 2

There are no FM tx's nearby. There is a low power AM TX about 3 miles distant. There is a large sub station about 500 metres away. It has an open cross bar type setup. On the same site as the substation is a cellular tower.

[wisp-wand]

This switch caused another outage again this morning, when we got to site, the switch was off again, with the internal LED on, as yesterday.
This time we have the syslog but its not helpful (port 2 is NOT where the syslog goes via, so disabling port 2 should not cause syslog to stop, port 4 is the port syslog would go via which is not POE, so the switch seems to just disappear.


2016.03.21-03:20:43 <10.0.x.x>: <140>Mar 21 03:20:39 switch[791]: !Power Supply Output Voltage is out of range (29.0V)
2016.03.21-05:11:44 <10.0.x.x>: <140>Mar 21 05:11:39 switch[803]: !Disabling PoE power on port 2 due to Power Supply Thermal out of specification (controller is 33 C)
2016.03.21-05:11:45 <10.0.x.x>: <15>Mar 21 05:11:40 kernel: link state changed to 'down' on port 2
2016.03.21-05:11:49 <10.0.x.x>: <140>Mar 21 05:11:44 switch[803]: !Enabling PoE power (24V) on port 2 due to Power Supply Thermal within specification
2016.03.21-05:11:51 <10.0.x.x>: <15>Mar 21 05:11:47 kernel: link state changed to 'up' on port 2
2016.03.21-05:11:56 <10.0.x.x>: <15>Mar 21 05:11:51 kernel: link state changed to 'down' on port 2
2016.03.21-05:11:58 <10.0.x.x>: <15>Mar 21 05:11:53 kernel: link state changed to 'up' on port 2
2016.03.21-05:12:15 <10.0.x.x>: <15>Mar 21 05:12:10 kernel: link state changed to 'down' on port 2
2016.03.21-05:12:17 <10.0.x.x>: <15>Mar 21 05:12:12 kernel: link state changed to 'up' on port 2
2016.03.21-05:12:20 <10.0.x.x>: <15>Mar 21 05:12:16 kernel: link state changed to 'down' on port 2
2016.03.21-05:12:22 <10.0.x.x>: <15>Mar 21 05:12:18 kernel: link state changed to 'up' on port 2

Is was just under an hour later from the last syslog before the switch disappeared
2016.03.21-06:05:24 <10.0.x.x>: Service ping on BG_Netonix_sw1 is now down

I thought of another possible cause of EMI - there is an electric train track nearby, less than 200 metres away, it goes over a bridge so similar height to the roof. the times don't match up though, as most issues occur overnight, but the train will be busiest during the day.

For now I have bypassed the Netonix.

[Dave]

wisp-wand.

Thanks for your detailed logs & comments. I believe we figured out why we are getting these false reports in code, and will be making
a test version for you to try.

Many Thanks.

Dave

[wisp-wand]

Hi Dave, great thanks look forward to testing it.

Regards


Wayne

[fiberpipe]

Hey Guys

I am running into the same issues, twice in the past two days. Device is been up 4 months now with firmware 1.3.8. Where do i locate 1.3.9 for download.

Thx