WS-26-500-DC on 1.5.0 rebooted - thoughts?

[createcoms]

Hi team,

The WS-26-500-DC (running production 1.5.0 firmware) on our biggest site rebooted last night causing a few minutes of outages for 200+ customers.

Log shows:

Code: Select all

Oct 7 18:51:03 sysinit: Error: Write failed
Oct 7 18:51:04 sysinit: Error: Write failed
Oct 7 18:51:05 sysinit: Error: Read failed
Oct 7 18:51:05 sysinit: Error: Read failed
Oct 7 18:51:06 sysinit: Error: Write failed
Oct 7 18:51:06 sysinit: Error: Write failed
Oct 7 18:51:07 sysinit: Error: Read failed
Oct 7 18:51:07 sysinit: Error: Read failed
Oct 7 18:51:08 sysinit: Error: Write failed
Oct 7 18:51:08 sysinit: Error: Write failed
Oct 7 18:51:09 sysinit: Error: Read failed
Oct 7 18:51:09 sysinit: Error: Read failed
Oct 7 18:51:10 sysinit: Error: Write failed
Oct 7 18:51:10 sysinit: Error: Write failed
Oct 7 18:51:11 sysinit: Error: Read failed
Oct 7 18:51:11 sysinit: Error: Read failed
Oct 7 18:51:12 sysinit: Error: Write failed
Oct 7 18:51:12 sysinit: Error: Write failed
Oct 7 18:51:13 sysinit: Error: Read failed
Oct 7 18:51:13 sysinit: Error: Read failed
Oct 7 18:51:14 sysinit: Error: Write failed
Oct 7 18:51:14 sysinit: Error: Write failed
Oct 7 18:51:15 sysinit: Error: Read failed
Oct 7 18:51:15 sysinit: Error: Read failed
Oct 7 18:51:16 sysinit: Error: Write failed
Oct 7 18:51:16 sysinit: Error: Write failed
Oct 7 18:51:17 sysinit: Error: Read failed
Oct 7 18:51:17 sysinit: Error: Read failed
Oct 7 18:51:18 UI: i2c error setting 0x47 12 110
Oct 7 18:51:18 sysinit: Error: Read failed
Oct 7 18:51:18 sysinit: Error: Write failed
Oct 7 18:51:19 sysinit: Error: Write failed
Oct 7 18:51:19 sysinit: Error: Read failed
Oct 7 18:51:20 sysinit: Error: Read failed
Oct 7 18:51:20 sysinit: Error: Write failed
Oct 7 18:51:21 sysinit: Error: Write failed
Oct 7 18:51:21 sysinit: Error: Read failed
Oct 7 18:51:22 sysinit: Error: Read failed
Oct 7 18:51:22 sysinit: Error: Write failed
Oct 7 18:51:23 sysinit: Error: Write failed
Oct 7 18:51:23 sysinit: Error: Read failed
Oct 7 18:51:24 sysinit: Error: Read failed
Oct 7 18:51:24 sysinit: Error: Write failed
Oct 7 18:51:25 sysinit: Error: Write failed
Oct 7 18:51:25 sysinit: Error: Read failed
Oct 7 18:51:26 sysinit: Error: Read failed
Oct 7 18:51:26 sysinit: Error: Write failed
Oct 7 18:51:27 sysinit: Error: Write failed
Oct 7 18:51:27 sysinit: Error: Read failed
Oct 7 18:51:28 sysinit: Error: Read failed
Oct 7 18:51:29 sysinit: Error: Write failed
Oct 7 18:51:29 sysinit: Error: Write failed
Oct 7 18:51:30 sysinit: Error: Read failed
Oct 7 18:51:30 sysinit: Error: Read failed
Oct 7 18:51:31 sysinit: Error: Write failed
Oct 7 18:51:31 sysinit: Error: Write failed
Oct 7 18:51:32 sysinit: Error: Read failed
Oct 7 18:51:32 sysinit: Error: Read failed
Oct 7 18:51:33 sysinit: Error: Write failed
Oct 7 18:51:33 sysinit: Error: Write failed
Oct 7 18:51:34 sysinit: Error: Read failed
Oct 7 18:51:34 sysinit: Error: Read failed
Oct 7 18:51:35 sysinit: Error: Write failed
Oct 7 18:51:35 sysinit: Error: Write failed
Oct 7 18:51:36 sysinit: Error: Read failed
Oct 7 18:51:36 sysinit: Error: Read failed
Oct 7 18:51:37 sysinit: Error: Write failed
Oct 7 18:51:37 sysinit: Error: Write failed
Oct 7 18:51:38 sysinit: Error: Read failed
Oct 7 18:51:38 sysinit: Error: Read failed
Oct 7 18:51:39 UI: i2c error setting 0x47 14 122
Oct 7 18:51:39 sysinit: Error: Read failed
Oct 7 18:51:39 sysinit: killall: telnetd: no process killed
Oct 7 18:51:40 sysinit: 2018-10-07 18:51:39: (log.c.216) server started
Oct 7 18:51:42 dropbear[1030]: Running in background
Oct 7 18:51:44 switch[1060]: Detected cold (watchdog) boot
Oct 7 18:51:46 Port: link state changed to 'down' on port 2
Oct 7 18:51:46 Port: link state changed to 'down' on port 3
Oct 7 18:51:46 Port: link state changed to 'down' on port 4
Oct 7 18:51:46 Port: link state changed to 'down' on port 5
Oct 7 18:51:46 Port: link state changed to 'down' on port 6
Oct 7 18:51:46 Port: link state changed to 'down' on port 7
Oct 7 18:51:46 Port: link state changed to 'down' on port 8
Oct 7 18:51:46 Port: link state changed to 'down' on port 9
Oct 7 18:51:46 Port: link state changed to 'down' on port 10
Oct 7 18:51:46 Port: link state changed to 'down' on port 11
Oct 7 18:51:46 switch[1060]: PoE enabled on port 1, PoE Smart is starting cable check
Oct 7 18:51:46 Port: link state changed to 'down' on port 12
Oct 7 18:51:46 Port: link state changed to 'down' on port 13
Oct 7 18:51:46 switch[1060]: PoE enabled on port 2, PoE Smart is starting cable check
Oct 7 18:51:46 Port: link state changed to 'down' on port 1
Oct 7 18:51:46 switch[1060]: PoE enabled on port 3, PoE Smart is starting cable check
Oct 7 18:51:46 switch[1060]: PoE enabled on port 4, PoE Smart is starting cable check
Oct 7 18:51:47 switch[1060]: PoE enabled on port 5, PoE Smart is starting cable check
Oct 7 18:51:47 switch[1060]: PoE enabled on port 6, PoE Smart is starting cable check
Oct 7 18:51:47 switch[1060]: PoE enabled on port 7, PoE Smart is starting cable check
Oct 7 18:51:47 switch[1060]: PoE enabled on port 8, PoE Smart is starting cable check
Oct 7 18:51:47 switch[1060]: PoE enabled on port 9, PoE Smart is starting cable check
Oct 7 18:51:47 switch[1060]: PoE enabled on port 10, PoE Smart is starting cable check
Oct 7 18:51:48 switch[1060]: PoE enabled on port 11, PoE Smart is starting cable check
Oct 7 18:51:48 switch[1059]: retrying i2c link to power supply
Oct 7 18:51:48 switch[1060]: PoE enabled on port 12, PoE Smart is starting cable check
Oct 7 18:51:48 switch[1060]: PoE enabled on port 13, PoE Smart is starting cable check
Oct 7 18:51:51 switch[1059]: retrying i2c link to power supply
Oct 7 18:51:51 switch[1060]: Port 1 cable check results: Open, Open, Open, Open
Oct 7 18:51:51 switch[1060]: Port 1 passed PoE Smart cable check, turning on power
Oct 7 18:51:51 switch[1060]: Port 2 cable check results: Open, Open, Open, Open
Oct 7 18:51:51 switch[1060]: Port 2 passed PoE Smart cable check, turning on power
Oct 7 18:51:51 switch[1060]: Port 3 cable check results: Open, Open, Open, Open
Oct 7 18:51:51 switch[1060]: Port 3 passed PoE Smart cable check, turning on power
Oct 7 18:51:51 switch[1060]: Port 4 cable check results: Open, Open, Open, Open
Oct 7 18:51:51 switch[1060]: Port 4 passed PoE Smart cable check, turning on power
Oct 7 18:51:51 switch[1060]: Port 5 cable check results: Open, Open, Open, Open
Oct 7 18:51:51 switch[1060]: Port 5 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 6 cable check results: Open, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 6 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 7 cable check results: Open, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 7 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 8 cable check results: Open, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 8 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 9 cable check results: Open, Open, Abnormal termination, Open
Oct 7 18:51:52 switch[1060]: Port 9 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 10 cable check results: Open, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 10 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 11 cable check results: Abnormal termination, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 11 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 12 cable check results: Open, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 12 passed PoE Smart cable check, turning on power
Oct 7 18:51:52 switch[1060]: Port 13 cable check results: Open, Open, Open, Open
Oct 7 18:51:52 switch[1060]: Port 13 passed PoE Smart cable check, turning on power
Oct 7 18:51:54 switch[1059]: retrying i2c link to power supply
Oct 7 18:51:54 switch[1059]: i2c link down to power supply (0)
Oct 7 18:51:54 switch[1059]: i2c link up to power supply (1)
Oct 7 18:51:55 Port: link state changed to 'up' (100M-F) on port 2
Oct 7 18:51:55 Port: link state changed to 'up' (100M-F) on port 1
Oct 7 18:51:55 Port: link state changed to 'up' (100M-F) on port 3
Oct 7 18:51:55 Port: link state changed to 'up' (100M-F) on port 5
Oct 7 18:51:55 Port: link state changed to 'up' (100M-F) on port 4
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 6
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 9
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 8
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 7
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 11
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 10
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 12
Oct 7 18:51:56 Port: link state changed to 'up' (100M-F) on port 13
Oct 7 18:52:15 Port: link state changed to 'down' on port 1
Oct 7 18:52:15 Port: link state changed to 'down' on port 2
Oct 7 18:52:15 Port: link state changed to 'down' on port 3
Oct 7 18:52:15 Port: link state changed to 'down' on port 4
Oct 7 18:52:16 Port: link state changed to 'down' on port 5
Oct 7 18:52:16 Port: link state changed to 'down' on port 6
Oct 7 18:52:16 Port: link state changed to 'down' on port 7
Oct 7 18:52:16 Port: link state changed to 'down' on port 8
Oct 7 18:52:16 Port: link state changed to 'down' on port 9
Oct 7 18:52:16 Port: link state changed to 'down' on port 12
Oct 7 18:52:16 Port: link state changed to 'down' on port 13
Oct 7 18:52:17 Port: link state changed to 'down' on port 10
Oct 7 18:52:17 Port: link state changed to 'down' on port 11
Oct 7 18:52:19 Port: link state changed to 'up' (1G) on port 1
Oct 7 18:52:19 Port: link state changed to 'up' (1G) on port 2
Oct 7 18:52:19 Port: link state changed to 'up' (1G) on port 4
Oct 7 18:52:19 Port: link state changed to 'up' (1G) on port 3
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 5
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 6
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 8
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 9
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 7
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 12
Oct 7 18:52:20 Port: link state changed to 'up' (1G) on port 13
Oct 7 18:52:21 Port: link state changed to 'up' (1G) on port 11
Oct 7 18:52:21 Port: link state changed to 'up' (1G) on port 10


[sirhc]

Please post the Status and Device/Status TAB (show all stats for all ports and such).

[createcoms]

[createcoms]

[sirhc]

Is this unit on a valid IP accessible to the world?

[createcoms]

Sorry it is not.

[sirhc]

Sorry it is not.

Well this is what I would do.

I would pull from service.
Do a power on factory default explained here: viewtopic.php?f=6&t=409#p2398

Upgrade to v1.5.0 or v1.5.1rc7 (you can upgrade to the same firmware v1.5.0 it just makes sure all files are correct)
Then "MANUALLY" set back up do NOT export and import config as there may be a corrupted flash *see below*

Oct 7 18:51:03 sysinit: Error: Write failed
Oct 7 18:51:04 sysinit: Error: Write failed
Oct 7 18:51:05 sysinit: Error: Read failed
Oct 7 18:51:05 sysinit: Error: Read failed

Then if it happens again I would RMA the unit as it may have a defective power supply?

Or it simply could be a power issue? Meaning it did experience some sort of power issue?

Is this hooked DIRECTLY to the batteries as designed or powered from a power supply or via some sort of charger?

[createcoms]

Thankyou sirhc I will follow your advice and mark this down for replacement when we are next visiting the site (or sooner if this repeats).

Regarding the Powerplant, the site is 24V Solar, so the switch is connected to the load output of the solar controller - as you can see in one of those screenshots the voltage is changing with day and night.

[sirhc]

My concern with it hooked this way is the charger and or controllers are not rated to 500 watts.

It is better to always hook our switches directly to the batteries (with an inline fuse or preferrably a breaker) and the controller hooked separately.

Your problem can be that the WS-26-500-DC draws too much power and the controller considers it a short or cuts out for a few seconds and reboot.

Why not hook switch to batteries and controller to batteries separately which is the way we designed the power supplies in the switches.

[createcoms]

We have active monitoring of the supply and no such event was recorded, also the router and PtP links which aren't using the Netonix did not suffer any problems (running off the same power supply).

The powerplant is most definitely capable of supplying full wattage to the Netonix.

Nonetheless I will also get the plant inspected in case something is not quite right e.g. poor termination somewhere.