Config corrupted?

[bmwake]

We have around 25 - 30 WS-8-150-DC out in the field, and seem to be getting fairly regular (probably every six weeks or so) issues. We lose all remote management to the switch, and when site staff restart it, it appears to have lost configuration - POE doesn't come up, management not available, etc.

Getting them back to my office and using the console port, it appears to start up normally, but when the boot process looks to be finished and I hit enter, I get the following:


Jan 1 00:00:04 kernel: mtd: partition "rootfs" set to be root filesystem
Jan 1 00:00:04 kernel: 0x00b00000-0x00f40000 : "rootfs_data"
Jan 1 00:00:04 kernel: 0x00f40000-0x00f80000 : "FIS directory"
Jan 1 00:00:04 kernel: 0x00f80000-0x00f81000 : "RedBoot config"
Jan 1 00:00:04 kernel: 0x00fc0000Jan 1 00:00:04 Jan 1 00:00:04 Jan 1 00:00:04 Jan 1 00:00:04 kernel: mini_fo: using base directory: /
Jan 1 00:00:04 kernel: mini_fo: using storage directory: /jffs
vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
switch: 'Luton26' board detected

BusyBox v1.19.4 (2018-06-07 14:48:31 EDT) built-in shell (ash)
Enter 'help' for a list of built-in commands.
nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Segmentation fault
Segmentation fault
Segmentation fault
i2c /dev entries driver
Segmentation fault
i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Segmentation fault


The seg fault message then just keeps repeating over and over. Hardware appears to be fine, factory reset and reconfiguration sets it back on its way nicely.

Any suggestions on cause/solution?

[sirhc]

What version of firmware are you running?

Always use the latest when having an issue to make sure it is still happening.

If your switches are running older versions there were security holes discovered and fixed in later versions.

Also if you switches are accessible to the public then use the Access Control list. Actually it is always a GOOD IDEA to use the access control list to limit who can access the switch.

[bmwake]

Initially 1.4.8, as I've worked on them they've gone up to 1.5.0 and some to 1.5.2. Latest three have gone to 1.5.3.
Just rather tough to get approval to bill for the time to upgrade all at once.

They're all on a management VLAN which does have limited access already.

[sirhc]

Well v1.4.8 definitely has security holes that allowed a hacker to corrupt the flash but not take control or change anything but still VERY BAD.

This was not a hole in our code but the web service [ lighttpd ] we use to run the UI.
https://en.wikipedia.org/wiki/Lighttpd

A factory default will clear it then upgrade

Access control list only helps if the switch is on a public IP not a NAT as the switch will always see the packets coming from the NAT address regardless

Personally I only put my infrastructure on non rout able IP outside my network then use Access Control list to prevent people from inside my net from attempting to access.

Only way outside can get to it would be to hack one of our devices in the allowed IP ranges and spring board into device.lighttpd

[bmwake]

Quick check, I can't find any still running 1.4.8, all 1.5.0 as minimum (from memory, did get to go through all for a major version number change)

My repair process is console to verify it's this same thing, factory, upgrade to whatever's current, and reconfigure

Had been wondering if it might be temperature, but most recent failure's replacement has been sitting around 40 - 45 max