MSTP does not handle VLAN changes

[Stephen]

david.sovereen@mercury.net

Try out the new firmware to see if the fix here resolves your issue as well. If it does not, feel free to open another thread so we can examine that one separately.

[JustJoe]

JustJoe, please take a look at the release 1.5.7rc1

I was able to replicate the issue and resolve it.

This is a bug that needs to be fixed first, BUT ... the original bug on the multi switch network still failed *sometimes* even after POR ???
...but that needs to wait for a retest when I get a version with this fix in it.

I'll keep an eye on this thread. If you still see failures after testing this release, I'll work on preparing another firmware update. First let's see how far this one takes us.

Thank you Stephen !!!

I just downloaded and will begin testing this evening. :)

[JustJoe]

Good news :) Bad news :(

So I upgraded from v1.5.6 to v1.5.7rc1, that went smoothly. Tested on the original setup.

With all ports MSTP enabled, I moved Port6 from VLAN34 (where I could ping 192.168.34.222) to VLAN32 (where I could now ping 192.168.32.222) That works perfectly now! YAY !

But then I went and tested the scenario I described in post #5 of this thread:

From the STP / MSTP / CIST instance, I Disabled port 6 from STP. That worked just fine, could still ping 192.168.32.222. YAY!

I moved Port6 from VLAN32 (where I could ping 192.168.32.222) to VLAN34 (where I could now ping 192.168.34.222) YAY!

Then I Enabled port 6 from STP. That did NOT work, I could no longer ping 192.168.34.222 NOO!

Now I again Disabled port 6 from STP. That did work, I could one again ping 192.168.34.222 YAY !

Again Enabled port 6 from STP. That did NOT work, I could no longer ping 192.168.34.222

But now tried the work-around, changed revision from "1" to "2". That fixed it, I could one again ping 192.168.34.222.
(that last lines of the log now show it correctly associated with msti 2)

See the attached log.

[Stephen]

Hey JustJoe,

Please try 1.5.7rc2

I was not able to completely replicate the issue - I was still able to ping after disabling the port, transferring it to a new VLAN and re-enabling it on the CIST. However, I was able to see that the correct msti did not cycle until after the MSTP Version was changed in the logs on my switch. The update fixes that and I suspect it should fix the pinging issue as well.

[JustJoe]

Hey JustJoe,

Please try 1.5.7rc2

I was not able to completely replicate the issue - I was still able to ping after disabling the port, transferring it to a new VLAN and re-enabling it on the CIST. However, I was able to see that the correct msti did not cycle until after the MSTP Version was changed in the logs on my switch. The update fixes that and I suspect it should fix the pinging issue as well.

Thank you Stephen, so far it looks like you have a winner, unable to reproduce any of my issues at the single switch level.

Yes, these problems seemed to require lots of specific order of previous command history. What finally helped my case was learning to understand and copy the entire log.

I am going to try playing with multiple switches to see if I can recreate that environment. But honestly I may have mis-remembered the reboot actions of the 3 switches, since back then I was in learning/testing MSTP mode, and I was not closely watching for firmware issues.

Dealing with "MSTP revisions" gave me an idea for a new feature, but I'll document that separately in a new General forum thread.

Question: Were there any other unrelated function fixes or improvements added in this RC2 image? I have v1.5.6 in production and would really like to roll out RC2 to replace it. So just wondering if I need to watch out for other changes?

[Stephen]

I'm glad that one is working. Thank you for the detailed reports.

Question: Were there any other unrelated function fixes or improvements added in this RC2 image? I have v1.5.6 in production and would really like to roll out RC2 to replace it. So just wondering if I need to watch out for other changes?

All the changes made where for the intended effect of targeting the bugs we were seeing. But I did change a few things in way of optimization (in truth, there was a little bit of OVER optimization causing part of the problems). What I think this would mean for production is just slightly longer convergence times for STP. Which should only be like 200-500ms at most.

That's the only thing I can think of that you might want to be aware of - RC2 should function fine in production.

[mcnnetops]

Some how this is posted way after you guys....

Looks like a new firmware address's this:
In RC at the time of writing this - but testing?

viewtopic.php?f=17&t=240
http://forum.netonix.com/firmware/wisps ... 5.7rc2.bin

Good luck, let us know.

[JustJoe]

Hi Stephen, bad news, found another MSTP bug. This is not new, it actually bit me once in the initial testing of enabling MSTP on a production (but physically easy to reach and POR) switch. Then it bit me again as I started rolling out v1.5.7rc2.

What made it hard to isolate at first was that I did not realizing that it stopped forwarding all VLAN ports that are covered by an instance.

So to recreate, go back to the same config as post #1.

1) Prepare for the bug recreate:
From admin PC host, one interface card connects to Port#1 (VLAN11), access the GUI on 192.168.1.222
From admin PC host, second interface card connects to Port#10 (VLAN32), then continuous ping 192.168.32.222

2) Disable all STP (uncheck [ ] Enable)
On Ports 1, 2 & 6 also uncheck [ ] Enable
Apply
3) When GUI returns, confirm ..32.222 still replying.
4) Reboot either from GUI or POR

5) ..32.222 will eventually start replying. Again access GUI from ..1.222

6) STP / CIST / Port 6 check [x] Enable
Apply

7) STP / [x] Enable version MSTP
Apply

8) At this point, ..32.222 will timeout.

Further investigation shows that none of the VLANs in MSTP instances are forwarding. Originally, although I had console access, I wasn't able to determine from the CLI what was wrong, so a Reboot was the only choice.

In this test, I unchecked Port#1* (in VLAN11) from MSTP/CIST so I could maintain continuous GUI connectivity to see what was going on. Sad thing on the original (production) occurrence, was that I was administering from over a trunk port under MSTP control, and though I lost access for over 5 minutes, it at least seemed like it did not revert to the previous setting of MSTP disabled ?

The attached log shows that port 6 was successfully enabled, but after the separate MSTP enable/apply, MSTP was not servicing the VLANs defined in the instances.

Oh, and similar to the previous bugs, if in this case where there is access to the GUI, changing something like the MSTP revision number and applying will fix it.

(Edit: *Port#1 clarification)

[JustJoe]

OK, this also is not a new bug with v1.5.7rc2 but existed before, and I was debating whether this should be a separate thread. Stephen, you are welcome to edit and move this post to a new thread if you prefer.

So in doing the previous posts debugging / data gathering, I was really stuck at some points where CLI config changes to MSTP seem to be unusable.

It seemed that often, just doing a Revision change to the region would re-synchronize MSTP. But how to do that???

I am used to using CLI "?" (especially in Cisco land) and for many of the WS features, it works fine. But in MSTP, in config mode, it takes you through:

Netonix Switch(config)# stp msti_instance

<1-7> Instance number

Correct me if I am wrong, but CIST (instance 0?) is not part of the CLI command tree (even though it lists as an Instance under sh stp output), so these fail:

Netonix Switch(config)# stp msti_instance CIST name BugReport revision 4 vlan_list 123 priority 32768
% Invalid parameter.
Netonix Switch(config)# stp msti_instance 0 name BugReport revision 4 vlan_list 123 priority 32768
% Invalid parameter.
Netonix Switch(config)#

BUT notice the rest of that line, where even IF I was trying to do a 1-7 instance, it requires a name, revision, vlan_list, and priority before it will accept a <cr> . It does set the vlan_list value when using 1-7, but then ignores the name & revision values it required. :/

The MSTP GUI takes a bit of getting used to. But it does make a distinction between what is entered for the CIST instance (basically the more global region values like name & revision) as opposed to the 1-7 instances with their VLAN lists. Whereas the MSTP CLI seems to have accidentally smooshed them together.

[JustJoe]

I'm glad that one is working. Thank you for the detailed reports.

....

Hi Stephen, any word on recreating my most recent 2 bug reports ? :)