i2c Errors on WS-26-500-DC Switches

[aaron@futuretk.com]

We have a number WS-26-500-DC of switches that are occasionally rebooting themselves. The only common thing we are seeing on these switches is a presence of i2c power supply errors in the log. We have upgraded to firmware 1.4.9 on all of them, but the issues are continuing. Here are the relevant log entries from a switch that had this happen today. I have since configured external logging so I hope to catch more detail the next time it happens.


Dec 31 18:00:06 netonix: 1.4.9 on WS-26-500-DC
Dec 31 18:00:15 system: Setting MAC address from flash configuration: EC:13:B2:06:1C:3C
Dec 31 18:00:17 system: starting ntpclient
Dec 31 18:00:18 admin: adding lan (eth0) to firewall zone lan
Dec 31 18:00:19 admin: Unable to query power supply
Mar 8 12:12:07 UI: i2c error setting 0x47 12 110
Mar 8 12:12:28 UI: i2c error setting 0x47 14 147
Mar 8 12:12:31 dropbear[954]: Running in background
Mar 8 12:12:34 switch[984]: Detected cold boot
Mar 8 12:12:37 switch[983]: retrying i2c link to power supply
Mar 8 12:12:40 switch[983]: retrying i2c link to power supply



Mar 8 12:12:43 switch[983]: retrying i2c link to power supply
Mar 8 12:12:43 switch[983]: i2c link down to power supply (0)
Mar 8 12:12:43 switch[983]: i2c link up to power supply (1)

Thank you
-Aaron

[aaron@futuretk.com]

We are continuing to have this issue. It seems to be impacting most of our new WS-26-500-DC switches all running 1.4.9 firmware.

Here is a log from another one I replaced yesterday which is also showing Read and Write errors:

Dec 31 18:00:04 kernel: 0x00500000-0x00b00000 : "rootfs"

Dec 31 18:00:04 kernel: mtd: partition "rootfs" set to be root filesystem

Dec 31 18:00:04 kernel: 0x00b00000-0x00f40000 : "rootfs_data"

Dec 31 18:00:04 kernel: 0x00f40000-0x00f80000 : "FIS directory"

Dec 31 18:00:04 kernel: 0x00f80000-0x00f81000 : "RedBoot config"

Dec 31 18:00:04 kernel: 0x00fc0000-0x01000000 : "Redundant FIS"

Dec 31 18:00:04 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

Dec 31 18:00:04 kernel: TCP cubic registered

Dec 31 18:00:04 kernel: NET: Registered protocol family 10

Dec 31 18:00:04 kernel: lo: Disabled Privacy Extensions

Dec 31 18:00:04 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team

Dec 31 18:00:04 kernel: NET: Registered protocol family 17

Dec 31 18:00:04 kernel: 802.1Q VLAN Support v1.8 Ben Greear

Dec 31 18:00:04 kernel: All bugs added by David S. Miller

Dec 31 18:00:04 kernel: VFS: Mounted root (squashfs filesystem) readonly.

Dec 31 18:00:04 kernel: Freeing unused kernel memory: 116k freed

Dec 31 18:00:04 kernel: Please be patient, while OpenWrt loads ...

Dec 31 18:00:04 kernel: Algorithmics/MIPS FPU Emulator v1.5

Dec 31 18:00:04 kernel: mini_fo: using base directory: /

Dec 31 18:00:04 kernel: mini_fo: using storage directory: /jffs

Dec 31 18:00:04 sysinit: killall: udhcpc: no process killed

Dec 31 18:00:06 netonix: 1.4.9 on WS-26-500-DC

Dec 31 18:00:07 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.

Dec 31 18:00:07 kernel: switch: 'Luton26' board detected

Dec 31 18:00:10 kernel: vtss_port: Loaded port module on board Luton26, type 5

Dec 31 18:00:11 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)

Dec 31 18:00:11 system: Setting MAC address from flash configuration: EC:13:B2:06:06:CE

Dec 31 18:00:11 kernel: i2c /dev entries driver

Dec 31 18:00:11 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19

Dec 31 18:00:13 sysinit: Loading defaults

Dec 31 18:00:13 system: starting ntpclient

Dec 31 18:00:13 sysinit: Adding custom chains

Dec 31 18:00:14 sysinit: Loading zones

Dec 31 18:00:14 sysinit: Loading forwarding

Dec 31 18:00:14 sysinit: Loading redirects

Dec 31 18:00:14 sysinit: Loading rules

Dec 31 18:00:14 sysinit: Loading includes

Dec 31 18:00:14 admin: adding lan (eth0) to firewall zone lan

Dec 31 18:00:15 sysinit: Error: Read failed

Dec 31 18:00:15 admin: Unable to query power supply

Dec 31 18:00:23 kernel: eth0: no IPv6 routers present

Dec 31 18:00:27 Port: link state changed to 'up' (100M-F) on port 17

Dec 31 18:00:27 sysinit: Error: Write failed

Dec 31 18:00:28 sysinit: Error: Write failed

Dec 31 18:00:28 sysinit: Error: Read failed

Dec 31 18:00:29 sysinit: Error: Read failed

Dec 31 18:00:29 sysinit: Error: Write failed

Dec 31 18:00:30 sysinit: Error: Write failed

Dec 31 18:00:30 sysinit: Error: Read failed

Dec 31 18:00:31 sysinit: Error: Read failed

Dec 31 18:00:31 sysinit: Error: Write failed

Dec 31 18:00:32 sysinit: Error: Write failed

Dec 31 18:00:32 sysinit: Error: Read failed

Dec 31 18:00:33 sysinit: Error: Read failed

Dec 31 18:00:33 sysinit: Error: Write failed

Dec 31 18:00:34 sysinit: Error: Write failed

Dec 31 18:00:34 sysinit: Error: Read failed

Dec 31 18:00:35 sysinit: Error: Read failed

Dec 31 18:00:36 sysinit: Error: Write failed

Dec 31 18:00:36 sysinit: Error: Write failed

Dec 31 18:00:37 sysinit: Error: Read failed

Dec 31 18:00:37 sysinit: Error: Read failed

Dec 31 18:00:38 sysinit: Error: Write failed

Dec 31 18:00:38 sysinit: Error: Write failed

Dec 31 18:00:39 sysinit: Error: Read failed

Dec 31 18:00:39 sysinit: Error: Read failed

Dec 31 18:00:40 sysinit: Error: Write failed

Dec 31 18:00:40 sysinit: Error: Write failed

Dec 31 18:00:41 sysinit: Error: Read failed

Dec 31 18:00:41 sysinit: Error: Read failed

Dec 31 18:00:42 sysinit: Error: Write failed

Dec 31 18:00:42 sysinit: Error: Write failed

Dec 31 18:00:43 sysinit: Error: Read failed

Dec 31 18:00:43 sysinit: Error: Read failed

Mar 15 08:23:33 sysinit: Error: Write failed

Mar 15 08:23:34 sysinit: Error: Write failed

Mar 15 08:23:34 sysinit: Error: Read failed

Mar 15 08:23:35 sysinit: Error: Read failed

Mar 15 08:23:35 sysinit: Error: Write failed

Mar 15 08:23:36 sysinit: Error: Write failed

Mar 15 08:23:36 sysinit: Error: Read failed

Mar 15 08:23:37 sysinit: Error: Read failed

Mar 15 08:23:37 UI: i2c error setting 0x47 12 110

Mar 15 08:23:37 sysinit: Error: Read failed

Mar 15 08:23:38 sysinit: Error: Write failed

Mar 15 08:23:38 sysinit: Error: Write failed

Mar 15 08:23:39 sysinit: Error: Read failed

Mar 15 08:23:39 sysinit: Error: Read failed

Mar 15 08:23:40 sysinit: Error: Write failed

Mar 15 08:23:40 sysinit: Error: Write failed

Mar 15 08:23:41 sysinit: Error: Read failed

Mar 15 08:23:41 sysinit: Error: Read failed

Mar 15 08:23:42 sysinit: Error: Write failed

Mar 15 08:23:42 sysinit: Error: Write failed

Mar 15 08:23:43 sysinit: Error: Read failed

Mar 15 08:23:43 sysinit: Error: Read failed

Mar 15 08:23:44 sysinit: Error: Write failed

Mar 15 08:23:44 sysinit: Error: Write failed

Mar 15 08:23:45 sysinit: Error: Read failed

Mar 15 08:23:45 sysinit: Error: Read failed

Mar 15 08:23:46 sysinit: Error: Write failed

Mar 15 08:23:46 sysinit: Error: Write failed

Mar 15 08:23:47 sysinit: Error: Read failed

Mar 15 08:23:47 sysinit: Error: Read failed

Mar 15 08:23:48 sysinit: Error: Write failed

Mar 15 08:23:48 sysinit: Error: Write failed

Mar 15 08:23:49 sysinit: Error: Read failed

Mar 15 08:23:49 sysinit: Error: Read failed

Mar 15 08:23:50 sysinit: Error: Write failed

Mar 15 08:23:50 sysinit: Error: Write failed

Mar 15 08:23:51 sysinit: Error: Read failed

Mar 15 08:23:51 sysinit: Error: Read failed

Mar 15 08:23:52 sysinit: Error: Write failed

Mar 15 08:23:52 sysinit: Error: Write failed

Mar 15 08:23:53 sysinit: Error: Read failed

Mar 15 08:23:53 sysinit: Error: Read failed

Mar 15 08:23:54 sysinit: Error: Write failed

Mar 15 08:23:54 sysinit: Error: Write failed

Mar 15 08:23:55 sysinit: Error: Read failed

Mar 15 08:23:55 sysinit: Error: Read failed

Mar 15 08:23:56 sysinit: Error: Write failed

Mar 15 08:23:57 sysinit: Error: Write failed

Mar 15 08:23:57 sysinit: Error: Read failed

Mar 15 08:23:58 sysinit: Error: Read failed

Mar 15 08:23:58 UI: i2c error setting 0x47 14 147

Mar 15 08:23:58 sysinit: Error: Read failed

Mar 15 08:23:58 sysinit: killall: telnetd: no process killed

Mar 15 08:23:59 sysinit: 2018-03-15 08:23:59: (log.c.216) server started

Mar 15 08:24:01 dropbear[941]: Running in background

Mar 15 08:24:04 switch[971]: Detected cold boot

Mar 15 08:24:06 Port: link state changed to 'down' on port 2

Mar 15 08:24:06 Port: link state changed to 'down' on port 3

Mar 15 08:24:06 Port: link state changed to 'down' on port 9

Mar 15 08:24:06 switch[971]: PoE enabled on port 1, PoE Smart is starting cable check

Mar 15 08:24:06 Port: link state changed to 'down' on port 10

Mar 15 08:24:06 Port: link state changed to 'down' on port 11

Mar 15 08:24:06 switch[971]: PoE enabled on port 2, PoE Smart is starting cable check

Mar 15 08:24:06 Port: link state changed to 'down' on port 17

Mar 15 08:24:06 Port: link state changed to 'down' on port 13

Mar 15 08:24:06 Port: link state changed to 'down' on port 14

Mar 15 08:24:06 switch[971]: PoE enabled on port 3, PoE Smart is starting cable check

Mar 15 08:24:06 Port: link state changed to 'down' on port 15

Mar 15 08:24:06 Port: link state changed to 'down' on port 16

Mar 15 08:24:06 Port: link state changed to 'down' on port 18

Mar 15 08:24:06 Port: link state changed to 'down' on port 1

Mar 15 08:24:07 Port: link state changed to 'down' on port 4

Mar 15 08:24:07 switch[971]: PoE enabled on port 4, PoE Smart is starting cable check

Mar 15 08:24:07 Port: link state changed to 'down' on port 5

Mar 15 08:24:07 switch[971]: PoE enabled on port 5, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 6, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 7, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 8, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 9, PoE Smart is starting cable check

Mar 15 08:24:07 switch[971]: PoE enabled on port 10, PoE Smart is starting cable check

Mar 15 08:24:07 switch[970]: retrying i2c link to power supply

Mar 15 08:24:08 switch[971]: PoE enabled on port 11, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 13, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 14, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 15, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 16, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 17, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 18, PoE Smart is starting cable check

Mar 15 08:24:08 switch[971]: PoE enabled on port 19, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 20, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 21, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 22, PoE Smart is starting cable check

Mar 15 08:24:09 switch[971]: PoE enabled on port 23, PoE Smart is starting cable check

Mar 15 08:24:11 switch[971]: Port 1 cable check results: Open, Open, Open, Open

Mar 15 08:24:11 switch[971]: Port 1 passed PoE Smart cable check, turning on power

Mar 15 08:24:11 switch[970]: retrying i2c link to power supply

[Julian]

we are actively pursuing a resolution on this issue, please monitor upcoming firmware releases.

[sirhc]

We have been testing the WS-26-500-DC sent back from Intellipop and so far no reboots up time 1 day 19+ hours.

We have changed the input voltages from 24V to 52V during this test with no reboots but the CPU Utilization is way too HIGH.

We are "suspecting" that this may be the cause. Eric has optimized the code to reduce CPU to 30% +/- which we think will greatly help.

A theory of what might be happening is the CPU is getting bogged down and possibly not responding to the watchdog which forces a reboot or other adverse issues resulting in a reboot.

The fact that it is not rebooting in our LAB even with HIGH CPU is probably that there is nothing else going on so the high CPU is not an issue.

Anyway below you can see Intellipops switch with HIGH CPU usage and under similar POE load as he had deployed it and also another unit with the new optimized code with LOW CPU utilization.

So anyway will be hopefully releasing v1.5.0rcX today or tomorrow.

v1.4.9

v1.5.0rcX

[aaron@futuretk.com]

Good to hear. Keep us posted. I have one switch in particular that is now rebooting several times a day so I'd be happy test the new firmware, and would likely know very soon if it solves our issue.

[sirhc]

Good to hear. Keep us posted. I have one switch in particular that is now rebooting several times a day so I'd be happy test the new firmware, and would likely know very soon if it solves our issue.

Will be happy to have you try this but your issue is slightly different.

You are seeing I2C errors along with reboots.

The firmware may help with the reboots - hopefully

But the I2C issue may be related to hardware issue, possible causes:
Lose, bad connection on I2C cable between switchboard and power supply
Bad I2C cable between switch and power supply
Bad component(s) on either board.

However complete I2C failure should not cause reboot (I think, need to confirm with Eric) but simply you lose telemetry from power supply and or I2C sensors on switch board.

[aaron@futuretk.com]

Can you email me a copy of this new firmware, or do I need to wait for General release?

[sirhc]

We released a new version to possibly help with issues for the WS-26-500-DC

You can download v1.5.0rcX here: viewtopic.php?f=17&t=240&p=848#p848

Please let us know if it helps.

[aaron@futuretk.com]

This did drastically reduce the CPU utilization. So far so good. I'll know more in a few hours.

[sirhc]

This did drastically reduce the CPU utilization. So far so good. I'll know more in a few hours.

So Aaron please PM me your cell number since your issue is a little differnt I want to talk to your about some things to look for and some possible fixes for I2C error.