MAC table not populating correctly with multiple vlans

[adunlap]

I've been chasing an issue for several weeks and narrowed the problem down to unicast flooding. The switch is forwarding traffic out to all CPEs on a VLAN because (from what I can tell) the MAC table is not populating correctly. Has anyone experienced this?
After switch software upgrades and replacements I finally had to pull the switch and replace it with a Cisco 2960-G to split my VLANs out.

Setup is layer 2 and pretty straight forward:
Trunk from our C.O. (central office) is feeding the tower (A) that had the issue. I'm also feeding a few other towers (B & C) on different VLANs.
I have about 130 customers (split between 6 RocketM2 and a few dozen on 6 Rocket-5AC) on tower A. All APs and CPEs are in bridge mode and on VLAN 18. Customers have their own routers and we use PPPoE.

All CPEs on tower A were receiving 1-3mbps and at times 5+ mbps; causing 100% airtime utilization on our APs. I would have a hard time logging into a CPE, sometimes taking minutes to load the WebUI.
I have a RoomAlert that monitors power running on port 21 on the same VLAN. Normally I see very little traffic (20kbps or less) on this unless we load the webUI to look at the status. As you can see below the switch is sending unicast traffic out of this port as well.

Originally the switch was running older firmware and I upgraded to the latest (1.4.9) to see if there was a bug. This did not help so we replaced the switch with a newer model. The new switch didn't fix the problem either.
-Upgraded firmware on WS-24-400A from 1.4.6 to 1.4.9. No fix
-Replaced with a WS-26-400-AC which had 1.4.7 pre-installed (did not upgrade to latest). No fix

I had no choice other than to install a different switch. The switch installed now is a Cisco 2960. Everything was moved to the 2960 except for the local PtMP APs (they remained on the Netonix). The Netonix is uplinked (port 24) to the 2960 and set as a trunk.

After this change the unicast flooding stopped. Below are several screenshots.

I also took screenshots of the MAC table. First screenshot is before the 2960 was installed and the second is after.
Notice, with all devices off of the Netonix other than the PtMP APs (on VLAN18) the MAC table is populated correctly.

I have some other Netonix switches that I'm also starting to noticing unicast flooding. I believe this issues has been going on for a while but slowly getting worse as this tower is more populated than others. And it's seems like after Christmas we seen this flooding escalate probably because of more traffic from users.

I have also attached a few more screenshots of different tabs in the Netonix to give a better understanding of the setup.
Thank you in advance for any help and/or suggestions.

[sirhc]

It would not matter which switch model you used as all of our switches (currently on the market) use the same switch core (VSC-7427) and use the same firmware so it would not matter if your using a WS-6-MINI, WS-24-400A, or WS-26-400-AC (Same switch core / same firmware)

I will ask Eric to look at this post but my guess is the issue is in your VLAN configuration?

[adunlap]

Thank you for the clarification that the switches use the same switch core and why I didn't see a change.

Thanks, I am curious if Eric has any ideas...

The majority of our towers have cisco switches (zero issues) and over the past year and a half we have been switching them out to Netonix to help clean up our boxes. I love that Netonix can poe power all of our radios and is the main reason we've went with these.

I forgot to add that the other end (at the C.O.) we have a cisco 2960 and is only allowing the vlans needed through the trunk.

2960G#sh run int gi0/3
Building configuration...
!
interface GigabitEthernet0/3
switchport trunk allowed vlan 14,15,18,19
switchport mode trunk
end

[Eric Stern]

My first thought is that you have trunking enabled on ports 4 5 and 26 and that might be incorrect. But I'm not a VLAN expert.

I'll be on vacation until the 30th, but I can look into it further when I get back.

[adunlap]

Eric,
Hope you are getting caught up from being on vacation...

Ports 4 & 5 are going to two other towers (B & C). Trunking is enabled and only the specific single VLAN was allowed through. Port 26 is connected to another Netonix at the top of the tower with two VLANs allowed through (switch management and access).

The Cisco 2960 that was put in place is pretty much configured the same port by port. Below I have pasted in the running config for the 2960.
Again, since this has been put in; the unicast flooding has stopped and the netonix is showing the full MAC table (since its only passing 1 VLAN).

Any assistance would be great since I am seeing this on some other towers with Netonix switches. If you need access to a live switch I will gladly allow access. If I can't get this resolved I will unfortunately be replacing all of them.

Code: Select all

 
TowerA-2960G-sw#sh run
Building configuration...

interface GigabitEthernet0/1
 description Uplink to C.O. via Mimosa
 switchport mode trunk
!
interface GigabitEthernet0/2
 description BACKUP Uplink to C.O. via airFiber (switchport at C.O. 0/10 shutdown)
 switchport mode trunk
!
interface GigabitEthernet0/3
!
interface GigabitEthernet0/4
 description Backhaul link to Tower C
 switchport trunk allowed vlan 19
 switchport mode trunk
!
interface GigabitEthernet0/5
 description Backhaul link to Tower B
 switchport trunk allowed vlan 15
 switchport mode trunk
!
interface GigabitEthernet0/6
!
interface GigabitEthernet0/7
 description PtMP Legacy AP
 switchport access vlan 18
 switchport mode access
 speed 100
!
interface GigabitEthernet0/8
!
interface GigabitEthernet0/9
!
interface GigabitEthernet0/10
!
interface GigabitEthernet0/11
 description PtP link to tower owner building 1
 switchport access vlan 14
 switchport mode access
!
interface GigabitEthernet0/12
 description PtP link to tower owner building 2
 switchport access vlan 18
 switchport mode access
!
interface GigabitEthernet0/13
!
interface GigabitEthernet0/14
!
interface GigabitEthernet0/15
!
interface GigabitEthernet0/16
!
interface GigabitEthernet0/17
!
interface GigabitEthernet0/18
!
interface GigabitEthernet0/19
!
interface GigabitEthernet0/20
 description Power & Temp monitor
 switchport access vlan 18
 switchport mode access
!
interface GigabitEthernet0/21
 description To Netonix in same cabinet for PtMP APs (WS-26-400-AC) <-WAS THE MAIN SWITCH
 switchport trunk allowed vlan 14,18
 switchport mode trunk
!
interface GigabitEthernet0/22
!
interface GigabitEthernet0/23
!
interface GigabitEthernet0/24
 description To Netonix on top of tower for PtMP R5-AC APs (WS-12-250AC) <-was on port 26 of WS-26-400-AC
 switchport trunk allowed vlan 14,18
 switchport mode trunk
!
interface Vlan1
 no ip address
 no ip route-cache
 shutdown
!
interface Vlan14
 ip address aaa.bbb.ccc.ddd 255.255.255.240
 no ip route-cache
!
ip default-gateway aaa.bbb.ccc.ddd
!
end

TowerA-2960G-sw#sh vlan brief
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi0/3, Gi0/6, Gi0/8, Gi0/9, Gi0/10, Gi0/13, Gi0/14, Gi0/15, Gi0/16, Gi0/17
 Gi0/18, Gi0/19, Gi0/22, Gi0/23
14 LocalTowerA-Mgmnt active Gi0/11
15 TowerB active
18 LocalTowerA-Access-PtMP active Gi0/7, Gi0/12, Gi0/20
19 TowerC active
1002 fddi-default act/unsup
1003 token-ring-default act/unsup
1004 fddinet-default act/unsup
1005 trnet-default act/unsup
TowerA-2960G-sw#
 

[sirhc]

One observation is your running v1.4.7

Please upgrade to current version which is v1.4.9

[adunlap]

Sorry, but I already tried v1.4.9 on the SW-24-400A as stated above...

"-Upgraded firmware on WS-24-400A from 1.4.6 to 1.4.9. No fix"
"-Replaced with a WS-26-400-AC which had 1.4.7 pre-installed (did not upgrade to latest). No fix"