Reboot disables PoE on some ports

[flameproof]

Hi all,

I have deployed five WS-12-250A switches. In some cases, when AC power fails over to backup battery power via an inverter/charger, the switch will reboot (as it's being power-cycled). What I find is that out of the three ports I have PoE enabled to supply NanoStation M5s, one or two (it seems at random) will have PoE turned off, with no reason given in the log.

This forces me to access the switch every time and manually turn PoE on again on the affected ports. Running 1.3.9rc13.

Any ideas?

[sirhc]

Hi all,

I have deployed five WS-12-250A switches. In some cases, when AC power fails over to backup battery power via an inverter/charger, the switch will reboot (as it's being power-cycled). What I find is that out of the three ports I have PoE enabled to supply NanoStation M5s, one or two (it seems at random) will have PoE turned off, with no reason given in the log.

This forces me to access the switch every time and manually turn PoE on again on the affected ports. Running 1.3.9rc13.

Any ideas?

This was a bug that was in an older versions but was fixed in version v1.3.8 not sure what this is of you are running v1.3.9rcX

Please replicate power failure and then look at and copy your log into this post so I can see the log.

[flameproof]

Here is the log after a power failure:

Code: Select all

 Jan 1 00:00:05 netonix: 1.3.9rc13 on WS-12-250A
Jan 1 00:00:09 system: Setting MAC address from flash configuration: EC:13:B3:XX:XX:XX
Jan 1 00:00:11 system: starting ntpclient
Jan 1 00:00:12 admin: adding lan (eth0) to firewall zone lan
Jan 1 00:00:20 dropbear[738]: Running in background
Jan 1 00:00:25 system: starting ntpclient
Jan 1 00:00:29 switch[809]: Detected cold boot
Jan 1 00:00:30 switch[809]: PoE enabled on port 1, PoE Smart is starting cable check
Jan 1 00:00:30 switch[809]: PoE enabled on port 2, PoE Smart is starting cable check
Jan 1 00:00:30 switch[809]: PoE enabled on port 4, PoE Smart is starting cable check
Jan 1 00:00:36 switch[809]: Port 1 cable check results: Ok, Ok, Ok, Ok
Jan 1 00:00:36 switch[809]: Port 1 passed PoE Smart cable check, turning on power
Jan 1 00:00:36 system: starting ntpclient
Jan 1 00:00:37 switch[809]: Port 2 cable check results: Ok, Ok, Abnormal termination, Abnormal termination
Jan 1 00:00:37 switch[809]: Port 2 passed PoE Smart cable check, turning on power
Jan 1 00:00:37 switch[809]: Port 4 cable check results: Ok, Ok, Abnormal termination, Cross-pair short to pair 1
Jan 1 00:00:37 switch[809]: Port 4 failed PoE Smart cable check (DEAD SHORT DETECTED), disabling PoE
Jan 1 00:00:40 kernel: link state changed to 'up' on port 1
Jan 1 00:00:41 kernel: link state changed to 'up' on port 2
Jan 1 00:00:48 system: starting ntpclient
Jan 1 00:00:59 kernel: link state changed to 'down' on port 2
Jan 1 00:00:59 system: starting ntpclient
Jan 1 00:00:59 kernel: link state changed to 'down' on port 1
Jan 1 00:01:00 kernel: link state changed to 'up' on port 2
Jan 1 00:01:02 kernel: link state changed to 'up' on port 1
Jan 1 00:01:10 system: starting ntpclient
Jan 1 00:01:21 system: starting ntpclient
Jan 1 00:01:32 system: starting ntpclient
Jan 28 09:52:23 kernel: link state changed to 'down' on port 1
Jan 28 09:52:26 kernel: link state changed to 'up' on port 1
Jan 28 09:52:45 kernel: link state changed to 'down' on port 1
Jan 28 09:52:48 kernel: link state changed to 'up' on port 1
Jan 28 10:12:45 UI: Configuration changed by 172.16.0.1
Jan 28 10:12:45 UI: Port 3 PoE: changed from 'Off' to '24V'
Jan 28 10:12:45 UI: Port 4 PoE: changed from 'Off' to '24V'
Jan 28 10:12:47 switch[809]: PoE enabled on port 3, PoE Smart is starting cable check
Jan 28 10:12:47 switch[809]: PoE enabled on port 4, PoE Smart is starting cable check
Jan 28 10:12:52 switch[809]: Port 3 cable check results: Ok, Ok, Abnormal termination, Abnormal termination
Jan 28 10:12:52 switch[809]: Port 3 passed PoE Smart cable check, turning on power
Jan 28 10:12:52 switch[809]: Port 4 cable check results: Ok, Ok, Abnormal termination, Abnormal termination
Jan 28 10:12:52 switch[809]: Port 4 passed PoE Smart cable check, turning on power
Jan 28 10:12:55 kernel: link state changed to 'up' on port 4
Jan 28 10:12:55 kernel: link state changed to 'up' on port 3
Jan 28 10:13:13 kernel: link state changed to 'down' on port 4
Jan 28 10:13:13 kernel: link state changed to 'down' on port 3
Jan 28 10:13:15 kernel: link state changed to 'up' on port 4
Jan 28 10:13:15 kernel: link state changed to 'up' on port 3

So, two issues: a dead short is detected on Port 4, which is later brought up correctly (the cable is fine, not sure why a short gets detected). The second issue is that even though Port 3 is also setup to 24V PoE before the power cut, and running normally, when the switch reboots it's not getting powered (you can see how I manually changed the setting to PoE ON in the log).

[sirhc]

Are you using some sort of Ethernet surge protector that is causing the cross short detection?

You can turn off POE Smart for each port on the Ports Tab under the column "PS" but be warned short damage is not covered under normal warranty.

I would figure out what is causing the cross-short detection that is forcing POE Smart to disable the POE on the port.

[flameproof]

No, these are straight connections to NanoStation M5 devices, via direct cable, not even a patch panel. We will check the cabling again, I'm not inclined to turn POE Smart off just yet :-)

I can try reverting to 1.3.8 on this switch and see what happens if that helps.

[sirhc]

Also I am confused why the switch is rebooting if you have a UPS system?

If your UPS is not kicking in fast enough to keep the switch up then the switch could boot in a weird state if power is cut then restored to quickly before all the circulates completely discharge.

[sirhc]

No, these are straight connections to NanoStation M5 devices, via direct cable, not even a patch panel. We will check the cabling again, I'm not inclined to turn POE Smart off just yet :-)

I can try reverting to 1.3.8 on this switch and see what happens if that helps.

There was no changes with POE Smart between v1.3.8 and v1.3.9 so that is not it.

[flameproof]

So, we have an inverter/charger with a large battery bank, so we can run off grid for a few hours. A normal UPS that does this can cost 15x the setup we have. The inverter/charger does have a switch-over time of 500ms to 1 second, so a brown-out could be happening. I'm considering placing a dirt-cheap UPS just to keep the line up while the inverter/charger switches over.

[sirhc]

We use these https://jet.com/product/detail/defb667c ... Hwod8UkAxA

These inverter chargers switch fast enough the switch does not blip.

It is your brown out that is your fault here. You can not reboot a switch that fast.

And we use (4) Optima AGM Batteries (Blue TOP)

And this charger http://mk.factoryoutletstore.com/detail ... -440d.html

My towers stay up for 12-24 hours.

Some sites I have converted to WS-12-250-DC directly connected to battery bank where I hook the 4 batteries in series to 48V and use the same charger and those towers will stay up for 2-3 days