Switches failing to apply new VLANS

[bpnz]

I use a range of Netonix switches, and I've found I have trouble applying new VLANs to some of them.
This has happened over a range of firmwares, though currently the switches are on 1.5.5

What I've found happens is I can often create a new vlan, but it won't pass traffic on the switch until I give it an IP address on that particular vlan. If I try and apply a vlan it will sit there applying the config for 300 seconds, then default to the prior config. This happens even if I give the switch a reboot before doing the changes.

In the log file I notice it keeps saying "starting NTP client" after adding each vlan to the firewall zone

It doesn't make sense so it could be something in my config maybe causing it to get confused. What should I post to help eliminate the issue?

Below is a copy of my log after it did take almost 300 seconds, but seems to of worked!


Feb 15 02:29:34 system: starting ntpclient
Feb 15 02:29:38 switch[2869]: temp sensor version 3
Feb 15 02:29:39 switch[2870]: Detected warm boot
Feb 15 02:29:51 monitor: restarting snmpd
Feb 15 02:31:18 UI: Configuration changed by admin (::ffff:192.168.10.145)
Feb 15 02:31:18 UI: VLANs: added 'BP Test'
Feb 15 02:31:30 admin: removing lan (eth0) from firewall zone lan
Feb 15 02:31:35 admin: removing vlan41 (eth0.41) from firewall zone lan
Feb 15 02:31:38 admin: removing vlan42 (eth0.42) from firewall zone lan
Feb 15 02:31:42 admin: removing vlan43 (eth0.43) from firewall zone lan
Feb 15 02:31:46 admin: removing vlan44 (eth0.44) from firewall zone lan
Feb 15 02:31:50 admin: removing vlan45 (eth0.45) from firewall zone lan
Feb 15 02:31:53 admin: removing vlan46 (eth0.46) from firewall zone lan
Feb 15 02:31:55 admin: removing vlan47 (eth0.47) from firewall zone lan
Feb 15 02:31:58 admin: removing vlan48 (eth0.48) from firewall zone lan
Feb 15 02:32:00 admin: removing vlan60 (eth0.60) from firewall zone lan
Feb 15 02:32:01 admin: removing vlan61 (eth0.61) from firewall zone lan
Feb 15 02:32:03 admin: removing vlan100 (eth0.100) from firewall zone lan
Feb 15 02:32:05 admin: removing vlan120 (eth0.120) from firewall zone lan
Feb 15 02:32:06 admin: removing vlan121 (eth0.121) from firewall zone lan
Feb 15 02:32:09 admin: removing vlan122 (eth0.122) from firewall zone lan
Feb 15 02:32:10 admin: removing vlan123 (eth0.123) from firewall zone lan
Feb 15 02:32:21 admin: adding lan (eth0) to firewall zone lan
Feb 15 02:32:22 system: starting ntpclient
Feb 15 02:32:27 admin: adding vlan41 (eth0.41) to firewall zone lan
Feb 15 02:32:28 system: starting ntpclient
Feb 15 02:32:32 admin: adding vlan42 (eth0.42) to firewall zone lan
Feb 15 02:32:34 system: starting ntpclient
Feb 15 02:32:38 admin: adding vlan43 (eth0.43) to firewall zone lan
Feb 15 02:32:40 system: starting ntpclient
Feb 15 02:32:44 admin: adding vlan44 (eth0.44) to firewall zone lan
Feb 15 02:32:46 system: starting ntpclient
Feb 15 02:32:51 admin: adding vlan45 (eth0.45) to firewall zone lan
Feb 15 02:32:52 system: starting ntpclient
Feb 15 02:32:55 admin: adding vlan46 (eth0.46) to firewall zone lan
Feb 15 02:32:57 system: starting ntpclient
Feb 15 02:33:03 admin: adding vlan47 (eth0.47) to firewall zone lan
Feb 15 02:33:04 system: starting ntpclient
Feb 15 02:33:09 admin: adding vlan48 (eth0.48) to firewall zone lan
Feb 15 02:33:10 system: starting ntpclient
Feb 15 02:33:15 admin: adding vlan60 (eth0.60) to firewall zone lan
Feb 15 02:33:16 system: starting ntpclient
Feb 15 02:33:20 admin: adding vlan61 (eth0.61) to firewall zone lan
Feb 15 02:33:21 system: starting ntpclient
Feb 15 02:33:25 admin: adding vlan100 (eth0.100) to firewall zone lan
Feb 15 02:33:27 system: starting ntpclient
Feb 15 02:33:31 admin: adding vlan120 (eth0.120) to firewall zone lan
Feb 15 02:33:32 system: starting ntpclient
Feb 15 02:33:36 admin: adding vlan121 (eth0.121) to firewall zone lan
Feb 15 02:33:37 system: starting ntpclient
Feb 15 02:33:41 admin: adding vlan122 (eth0.122) to firewall zone lan
Feb 15 02:33:43 system: starting ntpclient
Feb 15 02:33:47 admin: adding vlan123 (eth0.123) to firewall zone lan
Feb 15 02:33:48 system: starting ntpclient
Feb 15 02:33:53 system: starting ntpclient
Feb 15 02:33:54 admin: removing lan (eth0) from firewall zone lan
Feb 15 02:33:57 admin: removing vlan41 (eth0.41) from firewall zone lan
Feb 15 02:33:58 admin: removing vlan42 (eth0.42) from firewall zone lan
Feb 15 02:34:00 admin: removing vlan43 (eth0.43) from firewall zone lan
Feb 15 02:34:02 admin: removing vlan44 (eth0.44) from firewall zone lan
Feb 15 02:34:04 admin: removing vlan45 (eth0.45) from firewall zone lan
Feb 15 02:34:06 admin: removing vlan46 (eth0.46) from firewall zone lan
Feb 15 02:34:08 admin: removing vlan47 (eth0.47) from firewall zone lan
Feb 15 02:34:10 admin: removing vlan48 (eth0.48) from firewall zone lan
Feb 15 02:34:12 admin: removing vlan60 (eth0.60) from firewall zone lan
Feb 15 02:34:13 admin: removing vlan61 (eth0.61) from firewall zone lan
Feb 15 02:34:16 admin: removing vlan100 (eth0.100) from firewall zone lan
Feb 15 02:34:18 admin: removing vlan120 (eth0.120) from firewall zone lan
Feb 15 02:34:19 admin: removing vlan121 (eth0.121) from firewall zone lan
Feb 15 02:34:21 admin: removing vlan122 (eth0.122) from firewall zone lan
Feb 15 02:34:23 admin: removing vlan123 (eth0.123) from firewall zone lan
Feb 15 02:34:34 admin: adding lan (eth0) to firewall zone lan
Feb 15 02:34:35 system: starting ntpclient
Feb 15 02:34:40 admin: adding vlan41 (eth0.41) to firewall zone lan
Feb 15 02:34:41 system: starting ntpclient
Feb 15 02:34:44 admin: adding vlan42 (eth0.42) to firewall zone lan
Feb 15 02:34:44 system: starting ntpclient
Feb 15 02:34:48 admin: adding vlan43 (eth0.43) to firewall zone lan
Feb 15 02:34:49 system: starting ntpclient
Feb 15 02:34:53 admin: adding vlan44 (eth0.44) to firewall zone lan
Feb 15 02:34:54 system: starting ntpclient
Feb 15 02:34:57 admin: adding vlan45 (eth0.45) to firewall zone lan
Feb 15 02:34:58 system: starting ntpclient
Feb 15 02:35:01 admin: adding vlan46 (eth0.46) to firewall zone lan
Feb 15 02:35:02 system: starting ntpclient
Feb 15 02:35:06 admin: adding vlan47 (eth0.47) to firewall zone lan
Feb 15 02:35:07 system: starting ntpclient
Feb 15 02:35:10 admin: adding vlan48 (eth0.48) to firewall zone lan
Feb 15 02:35:11 system: starting ntpclient
Feb 15 02:35:15 admin: adding vlan60 (eth0.60) to firewall zone lan
Feb 15 02:35:17 system: starting ntpclient
Feb 15 02:35:20 admin: adding vlan61 (eth0.61) to firewall zone lan
Feb 15 02:35:21 system: starting ntpclient
Feb 15 02:35:25 admin: adding vlan100 (eth0.100) to firewall zone lan
Feb 15 02:35:26 system: starting ntpclient
Feb 15 02:35:30 admin: adding vlan120 (eth0.120) to firewall zone lan
Feb 15 02:35:31 system: starting ntpclient
Feb 15 02:35:34 admin: adding vlan121 (eth0.121) to firewall zone lan
Feb 15 02:35:35 system: starting ntpclient
Feb 15 02:35:39 admin: adding vlan122 (eth0.122) to firewall zone lan
Feb 15 02:35:40 system: starting ntpclient
Feb 15 02:35:44 admin: adding vlan123 (eth0.123) to firewall zone lan
Feb 15 02:35:45 system: starting ntpclient
Feb 15 02:35:47 system: starting ntpclient
Feb 15 02:35:56 admin: adding lan (eth0) to firewall zone lan
Feb 15 02:35:57 admin: adding vlan41 (eth0.41) to firewall zone lan
Feb 15 02:35:58 admin: adding vlan42 (eth0.42) to firewall zone lan
Feb 15 02:35:59 admin: adding vlan43 (eth0.43) to firewall zone lan
Feb 15 02:36:00 admin: adding vlan44 (eth0.44) to firewall zone lan
Feb 15 02:36:01 admin: adding vlan45 (eth0.45) to firewall zone lan
Feb 15 02:36:02 admin: adding vlan46 (eth0.46) to firewall zone lan
Feb 15 02:36:03 admin: adding vlan47 (eth0.47) to firewall zone lan
Feb 15 02:36:05 admin: adding vlan48 (eth0.48) to firewall zone lan
Feb 15 02:36:06 admin: adding vlan60 (eth0.60) to firewall zone lan
Feb 15 02:36:07 admin: adding vlan61 (eth0.61) to firewall zone lan
Feb 15 02:36:08 admin: adding vlan100 (eth0.100) to firewall zone lan
Feb 15 02:36:09 admin: adding vlan120 (eth0.120) to firewall zone lan
Feb 15 02:36:10 admin: adding vlan121 (eth0.121) to firewall zone lan
Feb 15 02:36:12 admin: adding vlan122 (eth0.122) to firewall zone lan
Feb 15 02:36:13 admin: adding vlan123 (eth0.123) to firewall zone lan
Feb 15 02:36:14 admin: adding vlan555 (eth0.555) to firewall zone lan

[Dave]

I will ask Stephen to look at this for you.

[Stephen]

So this one needs some explanation, what you're seeing I believe is related to the fix for the SLAAC issue reported by Ludvick that was released on 1.5.5.

In the release notes:
"- IPv6 address's will no longer be assigned to interfaces via SLAAC when IPv6 is disabled which caused a security concern, instead it is now a configurable option "

What he found was that regardless of the state of IPv6 in the config, VLAN's would get an accessible IPv6 address from SLAAC that would not be protected by the firewall regardless of the configuration.
Now when applying the config, it take's it a bit more time than it use too in order to apply VLAN's in order to correct this problem. One potential issue that was noted was that the config might take more than 300 seconds to be applied and cause it to default. This shouldn't happen because the timer isn't suppose to start until after it complete's adding them to the firewall. Which is what takes the majority of the time.
However, if you are seeing it say it expires but actually hasn't, then there might be a syncing issue between the web UI and the backend.
Or if it is expiring and then completely reverting what you tried to apply that is definitely a bug.

As far as the VLAN's needing an IP to pass traffic, can you double check this? I feel like if this was happening we would have heard of it long before now. What I think might be occurring, is if the first bug is correct. Then the VLAN's get reverted before the config is applied and thus eliminated so obviously any tagged traffic for the lost VLAN's won't pass.

What I suggest you try to test this is to create a small group of VLANs, small enough that it won't take long to apply them (maybe 3-4) And see if you can get them to pass traffic.
If that works than it is likely the symptom of a syncing issue with the config. Which I will investigate.

Oh one more thing. If you wish, you can disable the revert timer by going to Device->Config under the "Switch" panel in the top left corner the last option is "Revert Timer" if you set it to 0 it will not be enforced. Obviously doing so come's with a risk. But just be aware it's an option.

[bpnz]

Hi Stephen,

Many thanks for the reply.. my logic is it must be something I'm doing wrong as pretty much exactly what you said, there's so many people already using these switches and vlanning that something obvious would show up way sooner.. the vlans not passing traffic unless assigned an IP was an issue I used to have a couple of years ago, and to be fair I haven't tested it again since, so I will lab something up this week.. the more perplexing issue was it failing to apply the settings and the fact it was happening across multiple switches.

One other question, and may be related is I have a device connected to port 12 on the test bench.. and nothing plugged into port 11... and tagged traffic on port 2.. both ports 11 and 12 are untagged to form a bridge as such.. yet port 11 shows as moving a lot of data??? like gigabytes over the space of an hour or so.. there are no other vlans or untagged traffic on port 11 or 12... and port 2 has other tagged vlans and one untagged vlan (1) on it.. could this be part of the issue?? I'm wondering if having tagged and untagged on the same port isn't a good idea. If I change port 11 to tagged the same thing happens. Multicast is off.

Just to throw more confusion into the mix, if I make port 11 an untagged port, evenn thought there's nothing plugged into it, if I run a speedtest.net test, I get about 1/5 to 2/5 of the speed I get than if I disable the port.. I've also included a picture of all my vlans.. (I don't want to derail this thread into a training thread or anytihng, so if I should make this as a seperate topic elsewhere let me know)..

[Stephen]

One other question, and may be related is I have a device connected to port 12 on the test bench.. and nothing plugged into port 11... and tagged traffic on port 2.. both ports 11 and 12 are untagged to form a bridge as such.. yet port 11 shows as moving a lot of data??? like gigabytes over the space of an hour or so.. there are no other vlans or untagged traffic on port 11 or 12... and port 2 has other tagged vlans and one untagged vlan (1) on it.. could this be part of the issue?? I'm wondering if having tagged and untagged on the same port isn't a good idea. If I change port 11 to tagged the same thing happens. Multicast is off.

Just to throw more confusion into the mix, if I make port 11 an untagged port, evenn thought there's nothing plugged into it, if I run a speedtest.net test, I get about 1/5 to 2/5 of the speed I get than if I disable the port.. I've also included a picture of all my vlans.. (I don't want to derail this thread into a training thread or anytihng, so if I should make this as a seperate topic elsewhere let me know)..

This could potentially all be the result of this:

the more perplexing issue was it failing to apply the settings and the fact it was happening across multiple switches.

If when applying the config's, a revert was attempted by didn't work correctly. Then your switches actual configuration might be different then what is reported in the UI. This could lead to all sorts of confusion.

Probably the best way to troubleshoot this, is when you get a chance in the lab. Start with a switch that has been defaulted. Then apply a few vlan's (say 2-3) and see if they pass traffic, and all the stat's make sense.
If that works add a few more, repeat until say 6-8 in total. If that works.
Then clear them, and re-add them all at once.
My guess is that is when you add them all at once is when you'll start seeing issue's. If you can replicate that scenario, then it will give me something to hunt down.

[bpnz]

Hi Stephen, I took the time to try it tonight.. my setup was a Edge Router 12P with a ETH port setup with a handful of vlans plugged into a switch with a laptop plugging into that, very simple.. I verified the laptop was working fine plugging it directly into the ER and changing the tag on the lan port on the laptop.

Everything was fresh, meaning the switches I used had a full factory reset and were running the latest firmware (1.5.5 installed THEN the factory rest performed). The Edge Router was running the latest firmware with a very basic setup for testing, basically I did a factory reset, created some VLANs and added DHCP servers where needed.

So just adding two vlans on a WS-8-DC-12 switch.. no traffic would pass through the vlans.. the laptop could not get an IP address.. I tried repowering the switch, laptop and Edge Router.. nothing could get it to work.. I even did an identincal setup on a Edge Switch (that worked first pop)... The Netonix refused to work UNTIL I assigned an IP to the vlan on the switch.. then it worked perfectly.. VLAN traffic would only work on the VLANs I assigned IP addresses. As soon as I disabled those IP addresses it stopped.

I grabbed a WS-12-250A switch... latest firmware, factory reset.. and had exactly the same issue..

The two pictures below are from it working.. the picture of the vlans, I can plug the trunk from Edge Router into any of those four and it works.. I know the EdgeRouter and laptop are fine as I mentioned above the laptop plugged directly into the same ETH port on laptop with the tag added on the laptop worked perfectly, and it worked perfect with a Edge Switch. In both cases the laptop was able to get an IP via DHCP and browse the web.

The revert delays were set to 300 seconds and it didn't once get anywhere near hitting them.

You mention above if it didn't apply the changes properly the UI may be displaying the wrong config? but I'm assuming with at least a refresh of the webpage or a reboot of the laptop and switch then the UI and switch config would sync up??

[sakita]

I have seen this before. Oddly not on all switches. Everything is running 1.5.5.

One of our switches (a WS-12-250-AC) does this where VLANs will not pass through the switch until an IP address is added in the VLAN.

I just verified that one of the VLANs did not have an address and ping was broken. I added an address and now I can ping devices on that VLAN through the switch.

The save of the config with just adding that IP took a long time (300 second timer was down to low 200s - normally takes a couple of seconds to save).

Disabling the IP on the VLAN and hitting save again took around 100 seconds... and pings continued to work... and ping continued to work after rebooting the pass-through switch and the switch that the device is local to (I would suspect the MAC address is still in another switch or the router).

[Stephen]

Hmm, thanks for shedding some light on this.
It's starting to look more and more like there is a bug somewhere.

I haven't responded because I've been trying to replicate this in my lab and I haven't had any luck yet. I don't doubt it's happening - but I need to be able to see it to fix it. Is there any trends anyone has noticed on which switches this occur's on? As in, does it most often occur on say the WS-12-250-AC? Any other patterns will help me replicate and isolate this issue.

[bpnz]

Hi Stephan, No worries, I wasn't too worried about it as it's got an easy work around..

So I've had the two different models it's happened on.. I'm wondering if I do a fresh setup on one again... make sure it fails... then send you a copy of my config file???

(I also have had this happen a couple of years ago on different models as well but much older firmware too)

[Stephen]

Thanks bpnz that would be helpful. Feel free to send it to me in a PM. Also it could be helpful if you send me the mac address of the device that doesn't work. If you manage to find a switch that does work sending the mac address of that one also would be helpful.