Page 1 of 2

VLANs ... "finnicky" (1.2.2 and prior)

Posted: Mon Jun 15, 2015 6:08 pm
by josh
Just a head's up on what we have been seeing..

I think I've reported this before, but I wanted to come back and let people know we are still experiencing it.

We are currently on version 1.2.2, but we've seen this on prior versions as well. Basically, once we add a VLAN, the switch won't pass the VLAN through the trunk port until we add an ip to the vlan. Once that's done traffic immediately starts flowing, and we can remove the ip from the VLAN.

Last night I had a *really* weird one.

~ 2am I upgraded a switch from 1.1.19rc(i think?) to 1.2.2. After the upgraded, I had to add/remove ips from the vlans to get them to pass traffic... annoying, but not unusual. Around 6:30AM reports from customers started going on, and our NMS started sending us alerts that we were losing pings to customers on that VLAN. Finally all traffic on the VLANs stopped. Guess how we fixed it? You got it, add/remove IP from that VLAN.

... this is a 12/B, revB. Any tips?

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Mon Jun 15, 2015 6:43 pm
by sirhc
Well I am running VLANs on v1.2.2 and I have no IP's assigned to VLANs?

Maybe post your config Tabs so we can see what your doing and make a suggestion.

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Tue Jun 16, 2015 4:08 pm
by adairw
lots of vlans here and never seen that problem across any version I've ever used.

We trunk lots of vlans back to tower routers but we don't use the "Trunk Port - Allow all VLANs". Are you?

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Tue Jun 16, 2015 5:44 pm
by sirhc
I do not use "Trunking" either but I am pretty sure a lot of people do that requested it and I have not heard anything like this before from anyone else which is why I wanted to see his Config Tabs to try and figure out what he is seeing and what is causing it.

But the way VLANs work is once we configure them in the switch core our software does not touch them unless you change them. The switch core software, the real meat and potatoes was written by Vitesse and this product line VSC-742X) is over 3 years old and is used in over a million switches out there so a flaky VLAN would have been seen along time ago.

Pretty much any manufacturer of switches choses a Switch Core made my someone such as Vitesse, Broadcom, or Realtek and the switch 'Core' is provided on a chip and comes with it's own internal firmware. They also provide you with a reference software package that is used to configure the core but the main switching is handled by the core and that software is written by them. Yes you can modify it with what is called Hotfixes which is loaded with the drivers used to communicate with the core to configure it but we have not modified it at all.

Now if the switch was acting funny after you apply a config change or always acts funny because we are not configuring it correctly then that is our code but if it works fine and then messes up it is in the core which I find difficult to understand.

Now there are exceptions to this rule such as some functions like RSTP/LACP/Loop Protection and such as those controls require a module to be running on the external CPU and when I say external I mean from the core as we are using the MIPS 24K CPU that is built into the same package but it running Open-WRT which is what is used to run our UI and CLI interface and also run the daemons that control the higher functions like RSTP/LACP and such.

Now do not get me wrong I am not saying Josh is NOT seeing this or something I just do not understand how it is possible?

I would like to see his Config Tab, know what the switch is talking to, and how the VLANs are setup in both devices.

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Tue Jun 16, 2015 5:48 pm
by sirhc
josh wrote:~ 2am I upgraded a switch from 1.1.19rc(i think?) to 1.2.2.


Well if you are still running v1.1.19dc you are living DANGEROUSLY

I sent out WARNING after WARNING about any version prior to v1.1.8 is UNSTABLE

I also instructed everyone to reboot the switch again after the firmware is complete as the bug in the firmware prior to v1.1.8 would have been in control of the reboot after upgrading and that is where the MAJOR bug is in how we shut down the PHYs to prevent corruption or should I say we were NOT shutting the PHYs down during the reboot which was the problem and "could" cause microcode corruption in the PHY's

So upgrade your switches to v1.2.2 and make sure you reboot them after the upgrade if they were older than v1.1.8

After v1.1.8 the bugs we all MINOR!!!

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Tue Jun 16, 2015 5:54 pm
by sirhc
This is what has been posted where you Download the latest STABLE firmware v1.2.2 since April 2nd, 2015

IF YOU ARE RUNNING OLDER FIRMWARE THAN v1.1.8 YOU BETTER UPGRADE AS THERE WERE SERIOUS BUGS PRIOR TO v1.1.8

IF UPGRADING FROM A FIRMWARE OLDER THAN v1.1.8 YOU SHOULD TELL THE SWITCH TO REBOOT AGAIN AFTER THE UPGRADE AS THE BUG WAS IN THE REBOOT PROCESS SO UPGRADING TO v1.1.8 OR NEWER IS STILL REBOOTING FROM THE PREVIOUS VERSION WHICH HAD THE BUG IN REBOOT PROCESS.

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Sat Jun 20, 2015 3:01 am
by josh
I think you missed the part where I said I had already upgraded :P

We realized it was on the older firmware and had to schedule a maint window - our office runs off that switch as well as my house and a bunch of other users ;)

Anyway, it seems I can replicate this behavior on any netonix switch we have.
Get switch out of the box
upgrade firmware
reboot from GUI (just to make sure everything is in order)
add vlans... no traffic passes on anything but management...
add ip to vlans... BOOM vlans start passing through the trunk immediately as soon as an ip gets applied to that particular vlan (it doesn't matter if that ip even resides on the subnet in the vlan...)

I wouldn't have posted if we hadn't seen this over and over again. I don't know why it happens, and why others don't see it but we do, I'm simply reporting what we see.

I've been running networks for over 17 years now, if I was doing something goofy with a switch I'd be sure to tell you before-hand.

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Sat Jun 20, 2015 11:26 am
by adairw
You didn't answer my question about whether your using the Trunk All VLANs check box or not.

I've never had to put an IP on a VLAN to make it pass traffic.

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Sat Jun 20, 2015 12:49 pm
by sirhc
I can not seem to replicate this with the information provided.

Display the config Tabs and provide steps to replicate the behavior because we will fix it if we can replicate it.

Re: VLANs ... "finnicky" (1.2.2 and prior)

Posted: Mon Jun 22, 2015 6:35 pm
by josh
adairw wrote:You didn't answer my question about whether your using the Trunk All VLANs check box or not.

I've never had to put an IP on a VLAN to make it pass traffic.


No, we are not.