Rebooted 1.0.9 - She's gone.....

[mhoppes]

This morning I rebooted a switch I have (12 port) that's been up for 3 days. Just a software reboot..... She's gone. Never returned. This is not good.


EDIT: Chris, this was the replacement 12 port you sent me.

[mhoppes]

Further looking appears to show the switch is providing power, as the radio attached to it is working, and the link to that radio is up at 100-Full. But I've lost management access to the switch after the reboot.

[mhoppes]

Further investigation, the switch is passing traffic... but all management/control of it has been lost.

[sirhc]

This morning I rebooted a switch I have (12 port) that's been up for 3 days. Just a software reboot..... She's gone. Never returned. This is not good.


EDIT: Chris, this was the replacement 12 port you sent me.

Well lets address that fact the "replacement" switch was NOT to replace a failed switch but to replace an early prototype switch you had that was rev A of the board and it did not fail I just wanted it back since Rev A boards had some issue which you knew about when I gave them to you.

As of yet we have not had a single RMA but I am sure it "will" happen sooner or later but so far we have been pretty lucky.

I would love to help but I would need to see your config and know what you find when you get on site.

It is also possible that it was just one of those rare dumb "bad" luck things that happens 1 in 1,000,000 times. Anytime I reboot a remote device I hold my breath even with Cisco stuff. And when a device is less than a month old I especially worry because if an electronic device is going to fail it usually does so in the first 30 days after that it usually runs forever unless it gets damaged

Either way please report back here what you discover, if you did get the first failure I will replace it ASAP, if it is one of those 1 in a 1,000,000 things well, it does happen.


Was the reboot just something you did?

Was this a firmware upgrade, if so from what to what?
(If this was v1.0.9 there were some important fixes here recently to do with rebooting see firmware release notes. The biggest issue was rebooting from a power failure but the reboot logic is all one part in the code)

Is the switch still functioning you just can not get to the UI (your customers are up)?

Help me out with some details.

Also did not get the pictures yet of your new shield kits you said are not fitting properly? Please send those to me today as well.

[mhoppes]

Correct... the switch wasn't swapped because of a failure... swapped because of an early prototype upgrade.... :) No failures seen yet anywhere.

I was testing the SNMP fix. Running 1.0.9 SNMP was enabled this morning so I was going to reboot to make sure I could break it before I did the upgrade to 1.0.10. However, on the reboot everything is still up and passing traffic, but the UI is unresponsive... no ping, no SSH, no web.

Shield kit pics were e-mailed about an hour ago.

[rockhead]

Passing traffic = good !
No control = scary ! LOL

I never reboot or even fettle with anything I'm not prepared to drive to.

[lligetfa]

Anytime I reboot a remote device I hold my breath...

IKWYM Reboot is a necessary evil. The longer the unit has been up the greater the risk. I've had stuff up for years not come back after a reboot. On my last job we had areas of the plant that had annual scheduled power outages. We used backup generators to avoid reboots on some gear but invariably there was always something that would not come back up.

Patching something that has been up for a long time is particularly risky as well. When I was patching servers in our DC, I always issued a reboot prior to patching and another reboot after patching. Maybe that explains things... oxygen deprived brain syndrome.

Soft reboots also are not the same as hard reboots. Seen that often.

Anyway... shit happens. Always an internal struggle as to whether to leave well enough alone or stay patched to the hilt.

[mhoppes]

I'm with Les. Reboot before and after the update. Every time. Saves me many gray hairs.

[mhoppes]

When I arrived at the site this evening I could console in but the switch's UI was not able to get out to the Internet. This is the 1 in 1,000,000 reboot issue Chris has referenced. I've never seen it before on any of the other Netonix switches I have.

The upgrade to 1.0.10 went flawless and we're back up.

The good news through this it was ONLY the UI that locked.... the UI and the switch core are completely independent -- though they talk to each other. The UI can crash (not saying it will)... but it can... and not affect the switch at all.

[WisTech]

When I arrived at the site this evening I could console in but the switch's UI was not able to get out to the Internet. This is the 1 in 1,000,000 reboot issue Chris has referenced. I've never seen it before on any of the other Netonix switches I have.

The upgrade to 1.0.10 went flawless and we're back up.

The good news through this it was ONLY the UI that locked.... the UI and the switch core are completely independent -- though they talk to each other. The UI can crash (not saying it will)... but it can... and not affect the switch at all.

Good news man. Time to play the lotto huh? hahaha did it not respond to ssh either? I had my new switch lose the UI in the office just configuring it (on rc6 at the time).