v1.4.0 Bug Reports and Comments

[rgunderson]

I'll try it on a smaller pop. Still hurting from my weekend of pain from this...

[Banana Jack]

I upgraded our switches from 1.3.9 to 1.4.0 and now our wireless network has virtually ground to a halt because the switches are detecting network loops on their incoming interfaces and shutting down for 3 minutes at a time. 13 out of 17 switches are doing this and recording in their logs similar text to this:

May 26 11:37:16 Loop protection: detected loop from port 3 to port 3, disabling port 3 for 180 seconds
May 26 11:40:18 Loop protection: re-enabling port 3
May 26 11:41:08 Loop protection: detected loop from port 3 to port 3, disabling port 3 for 180 seconds
May 26 11:44:09 Loop protection: re-enabling port 3
May 26 11:44:25 UI: Configuration changed by (xxxx)
May 26 11:44:25 UI: Storm_Control_Loop_Protection: changed from 'Disabled' to 'Enabled'
May 26 11:44:46 Loop protection: detected loop from port 3 to port 3, disabling port 3 for 180 seconds
May 26 11:47:49 Loop protection: re-enabling port 3
May 26 11:48:15 UI: Configuration changed by (xxxx)
May 26 11:48:15 UI: Storm_Control_Loop_Protection: changed from 'Enabled' to 'Disabled'
May 26 11:48:36 Loop protection: detected loop from port 3 to port 3, disabling port 3 for 180 seconds
May 26 11:51:38 Loop protection: re-enabling port 3
I had previously disabled Loop Protection because we had this problem once before but on a much smaller scale and disabling the setting cured it. As you can see from the log, I tried enabling and disabling the setting again, but it didn't help. I was wondering whether there's a bug in 1.4.0 which causes it to disregard the Loop Protection setting?

I am currently trying to downgrade all our switches back to 1.3.9 unless anyone has any better ideas? I'm fearful that the firmware downgrades may catastrophically fail if an upstream switch decides to turn off its supplying port during the downgrade operation.

I don't know whether the error is genuine or not, or how to otherwise prevent it. The network had been running perfectly using 1.3.9.

Any help or comments would be much appreciated.

Thanks
Glenn

[AlexC]

Since upgrading to 1.4.0 I've been receiving emails like this:

unexpected link change on port 3 F - from (null) (Unknown) .

unexpected link change on port 2 100M-10M-H - from (null) (Unknown) .

The times and ports match that of devices i've set to reboot at odd hours of the night.

As those are expected link changes for me is there any way to prevent these particular alert emails being sent?

[sirhc]

@Banana Jack

I run different switch configurations on my WISP network to test things so I just went back and checked all the towers that had Loop Protection DISABLED and they still have Loop Protection DISABLED and those that were ENABLED are ENABLED.

So as far as I can see upgrading to v1.4.0 did not ENABLE Loop Protection for any of my towers switches that I had it disabled on?

I am not at my office this morning but when I get back I will verify Loop Protection is truly DISABLED when so selected in the UI and let you know and I have also asked Eric to verify this.

************* How firmware up/down grades work

Once the firmware is uploaded to the switch the upgrade processes is handled by the switch itself, the pretty bar graph and such that occurs on your UI has nothing to do with the processes other than provide you with status updates as to the upgrade process, the workstation is no longer needed at that point you could close your browser and walk away if you wanted to.

So once your firmware file upload is completed the process will work and if the files upload is disturbed the process will abort, no harm happens. Now powering OFF a switch during this process is BAD.

We will re-verify if Loop Protection is indeed DISABLED when it is so configured in the UI but I am pretty sure it is?

************* Advice on upgrade procedure for any equipment

Right above where you click to download the firmware the text below is present in red:

As always be careful to roll out new versions, upgrade switches that are easily accessible until you verify that the new firmware works with your network design and desired feature set. We try to test as much as possible but it is impossible to test every possible network configuration.

I own part of this company but even I follow procedures and do not just blindly trust anything including our firmware. I roll the firmware out to my office switches then to an easy tower and verify everything works as planned then a couple more then I usually use the MANAGER to roll the rest out late at night after I confirmed everything works as I intend it to.

[sirhc]

@AlexC

v1.4.0 has a new feature that notifies people when port speeds change unexpectedly. People asked for this because sometimes their radio link speed would drop from say 1G to 100M or from 100M to 10M either from bad cabling or whatever and they wanted to be alerted to this event.

Eric was supposed to make sure that it would not report port speed changes from say the user selecting port bounce, or when for the first several minutes when enabling power but he must have missed the scheduled port bounce event.

I am also wondering if he took into account ports being turned off and on by either timed events or other conditions on the Power Tab?

We will get this fixed in the next version, sorry.

We also are planning on putting in a config section soon where you can chose what events will generate an alert message to you and which ones will not be sent.

[Banana Jack]

Hi sirhc, thanks for the fast great support as always! Yes I know I should have tested the new version first, point taken! Maybe I'm just too much of an optimist ;)

The Loop Protection setting in the UI didn't get changed during the upgrade, but I just wondered whether it was really having an effect or not.

Nonetheless we are going to assume that the Netonix is alerting us to a genuine network loop problem and try to trace why that might be happening, but it doesn't seem to be an easy job. Strangely, it seems to be happening exactly every hour at the moment. In the short-term, it seems like a network-wide downgrade is the only immediate solution. If we have a network loop, it doesn't have any noticeable effect on the network, in contrast with the regular 3-minute outages :)

Thanks for your reassurance about the integrity of the firmware upgrade/downgrade process; that's great to know.

On a related subject, is it possible to do firmware upgrades/downgrades at the command line? I could then schedule them to occur out of hours in order to minimise perceived downtime.

Thanks
Glenn

** EDIT: Ah, I see you reference the "MANAGER" for scheduled updates. Is that something I can download? Sorry if I'm the only one that doesn't know this.

[AlexC]

@AlexC

v1.4.0 has a new feature that notifies people when port speeds change unexpectedly. People asked for this because sometimes their radio link speed would drop from say 1G to 100M or from 100M to 10M either from bad cabling or whatever and they wanted to be alerted to this event.

Eric was supposed to make sure that it would not report port speed changes from say the user selecting port bounce, or when for the first several minutes when enabling power but he must have missed the scheduled port bounce event.

I am also wondering if he took into account ports being turned off and on by either timed events or other conditions on the Power Tab?

We will get this fixed in the next version, sorry.

We also are planning on putting a a config section soon where you can chose what events will generate an alert message to you and which ones will not be sent.

The devices in question are doing their own internal rebooting - nothing scheduled on the switch itself so it wouldn't be filtering it. I would be quite happy to have that config section to uncheck these particular alerts - maybe just for particular ports.

[sirhc]

The devices in question are doing their own internal rebooting - nothing scheduled on the switch itself so it wouldn't be filtering it. I would be quite happy to have that config section to unchecked these particular alerts - maybe just for particular ports.

Good to know so we are not chasing our tail, thanks for reporting back.

Well at least for now you know why you are getting the alerts.

Could you change it so the switch does the reboot for you in scheduled port bounce that would then prevent those alerts if Eric did his code right and ignores port speed changes on scheduled port bounces.

Yea we need to put the configuration section in the next release.

[sirhc]

On a related subject, is it possible to do firmware upgrades/downgrades at the command line? I could then schedule them to occur out of hours in order to minimise perceived downtime.

Ah, I see you reference the "MANAGER" for scheduled updates. Is that something I can download? Sorry if I'm the only one that doesn't know this.

Damn it Glenn you noticed the MANAGER before I could tell you about it.
(good catch, at least I know your reading the posts not skimming them)

viewtopic.php?f=6&t=1549

[sirhc]

viewtopic.php?f=17&t=1459&p=12102&hilit=Loop+Protection#p12102

Loop Protection is not like RSTP.

It sends a special packet out each port every second and if the packet comes back even to the same port or another port it shuts the sending or receiving port down for 180 seconds.

So if a customer or another piece of equipment further out on your network has a leaky VLAN or a loop it will cause this to happen.

Leaky VLANs or equipment can cause false loop protections.

I do not know how you do your network but if you allow customers Layer 2 access to you network if a customer creates a loop inside their network then Loop Protection will see this and take action depending on how they have their router set up as the special packet enters then exits their location and gets pack to the sending switch.