v1.4.3rcX Bug Reports and Comments

[sirhc]

Reboot of that switch is still producing the message.

Possible UI bug of somesorts?

Eric will be online soon and will help you out.

[Eric Stern]

Reboot of that switch is still producing the message.

Possible UI bug of somesorts?

Can you get a backup of the switch config and send it to me? Email to eric@netonix.com would be easiest.

[maglito]

WS-12-250-DC upgrade from RC3 to RC4, firmware upgrade failed.
The progress bar was stuck at 14% I don't remember the step (perhaps "copying to bootfs" or something).
No response from ping for over 20 minutes and traffic is mostly no longer passing through the switch, So, my partner is driving two hours to the site now.

I have been having strange power issues at site, devices which provide uplink to main site have been rebooting a few times a day but I get no SMTP alert (it is working I just sent myself a test email before the firmware upgrade). Perhaps since uplink is down and no internet access the SMTP alerts never go out (not queued until link is re-established)?

Interesting, if there was a power fluctuation during the upgrade it couldn't have been a big one because the uplink airfiberX is still powered up and the link is still up with a link uptime of 4hours and counting even though the firmware upgrade failed 45 min ago.

From the airfiberX I can ping one other device at site (a backup airfiberX) but nothing else that was connected to the switch and certainly not the switch itself.

We thought the problem was the airfiberX so we replaced that yesterday and we replaced this netonix switch with a brand new identical one a week ago to be sure it wasn't the problem.

We have this same power configuration with at least 6 or 7 of these switches setup the same way at various sites with no problems.

I haven't even had time to look at the old WS-12-250-DC to see what's wrong with it. But there must be something flaky going on with the power or ground at this site, and it is likely not due to the firmware upgrade process, just bad timing I guess.

Is there any way to recover from the firmware upgrade failure with the console port or will I need to send it in? Of course we'll try power cycling it shortly.

Just typing this up while my partner drives there since I have the time at the moment. Sorry sirhc that I don't have more information for you, I know that you hate that. I'll follow up with more soon.

*UPDATE*

Power light and fan come on the unit but nothing else.

Replaced the switch with a dumb switch and a bunch of POEs for now out of frustration.

Looks like we'll need to send both the switch we replaced from last week and this one with the failed firmware upgrade in for warranty/repair.

[sirhc]

Well I would bench test the switch and use a console cable and see if it is attempting to boot.

There is a firmware recovery via console cable procedure explained here: viewtopic.php?f=17&t=1073

[mayheart]

Reboot of that switch is still producing the message.

Possible UI bug of somesorts?

Can you get a backup of the switch config and send it to me? Email to eric@netonix.com would be easiest.

Sent.

[Eric Stern]

Reboot of that switch is still producing the message.

Possible UI bug of somesorts?

Can you get a backup of the switch config and send it to me? Email to eric@netonix.com would be easiest.

Sent.

Well I couldn't duplicate this after restoring your config onto my switch.

Two suggestions:

Use the CLI to make a trivial change to the config. ie

Code: Select all

config
location name foo
exit
 

Then reload the web interface.

If that doesn't work try restoring the configuration from the backup.

[alex.frisbie]

Two switches that I've flashed the newest firmware onto (wispswitch-1.4.3rc3) have had an issue where after a period of time it stops passing data. I cannot ping the switch or anything connected to it. When connecting to it I don't get an IP address. This has happened in three scenarios with 3 different types of switches. It happened on the WS-8-250-AC and the WS-8-150-AC and the WS-10-250-AC. The fix is rebooting the switch... I have to believe it's the new firmware, as I didn't have this problem with 1.4.2. Has anyone else experienced this?

P.S. This also happened with wispswitch-1.4.3rc4.

[sirhc]

Two switches that I've flashed the newest firmware onto (wispswitch-1.4.3rc3) have had an issue where after a period of time it stops passing data. I cannot ping the switch or anything connected to it. When connecting to it I don't get an IP address. This has happened in three scenarios with 3 different types of switches. It happened on the WS-8-250-AC and the WS-8-150-AC and the WS-10-250-AC. The fix is rebooting the switch... I have to believe it's the new firmware, as I didn't have this problem with 1.4.2. Has anyone else experienced this?

P.S. This also happened with wispswitch-1.4.3rc4.

There was a change between v1.4.2 and v1.4.3 in how we detect a Pause Frame Storm and we may have error-ed in this change?

Known radios with Flow Control issues:
AFX radios
Cambium epmp 2000


THINGS TO TRY BELOW WHEN THIS HAPPENS, PICK ONE:

Try Disabling Flow Control on all radios then every few days enable flow control on ONE radio until you find the one with the Flow Control issue.

OR

Another option to find it faster is console port into the switch with the issue and disable flow control one port at a time until the switch comes back to life.

OR

Another option is to 1 by 1 remove each Ethernet cable and count to 5 and plug it back in until the switch comes back to life. I KNOW THIS GOES AGAINST MY RULES OF NOT PLUGGING LIVE POE PORTS BUT SOMETIMES YOU NEED TO BREAK THE RULES WHEN TRACKING DOWN AN ISSUE.

I would do option #3

Once you find the device causing the issue disable Flow Control on that port and let me know.

[alex.frisbie]

Two switches that I've flashed the newest firmware onto (wispswitch-1.4.3rc3) have had an issue where after a period of time it stops passing data. I cannot ping the switch or anything connected to it. When connecting to it I don't get an IP address. This has happened in three scenarios with 3 different types of switches. It happened on the WS-8-250-AC and the WS-8-150-AC and the WS-10-250-AC. The fix is rebooting the switch... I have to believe it's the new firmware, as I didn't have this problem with 1.4.2. Has anyone else experienced this?

P.S. This also happened with wispswitch-1.4.3rc4.

There was a change between v1.4.2 and v1.4.3 in how we detect a Pause Frame Storm and we may have error-ed in this change?

Known radios with Flow Control issues:
AFX radios
Cambium epmp 2000


THINGS TO TRY BELOW WHEN THIS HAPPENS, PICK ONE:

Try Disabling Flow Control on all radios then every few days enable flow control on ONE radio until you find the one with the Flow Control issue.

OR

Another option to find it faster is console port into the switch with the issue and disable flow control one port at a time until the switch comes back to life.

OR

Another option is to 1 by 1 remove each Ethernet cable and count to 5 and plug it back in until the switch comes back to life. I KNOW THIS GOES AGAINST MY RULES OF NOT PLUGGING LIVE POE PORTS BUT SOMETIMES YOU NEED TO BREAK THE RULES WHEN TRACKING DOWN AN ISSUE.

I would do option #3

Once you find the device causing the issue disable Flow Control on that port and let me know.

Chris,

I'll try what you suggested. I do want to throw out there though, it seems to be an intermittent thing. I have a camera connected to this same switch and the archive shows it goes off at random times for random amounts of time and then comes back. It usually seems to be off a few hours. I have a UBNT NanoStation M5 connected to it and it shows that it's been up since i rebooted the switch when I last had problems. So it doesn't appear to be doing anything to the PoE functionality, just the data that passes through it. I did unplug the NanoStation M5 the other day thinking that was causing the problem, but that didn't re-enable switch functionality. Unless I didn't have it unplugged long enough. I simply moved the cable to the secondary port thinking the port on the NSM5 was bad.

I'll let you know what I find.

[sirhc]

Well cycle through all the cables as I suggested or simply disable all flow control and see if the behavior stops then we can drill down further.

Another thing to try is disable Discovery and SMTP, try to narrow down what is causing it then we can address and if our fault fix the issue.

Sorry for your troubles but we will figure this out together.

Once we narrow it down I will get Eric involved or if we do not figure it out I will get Eric involved.

When you plug directly into the switch make sure you have a static IP on the laptop in the same subnet as the switch and if the switch is not a static IP try setting it to a static IP.