Netonix Keeps Rebooting on 1.5.0

[mhoppes]

Our problem child switch just rebooted again this morning. I will upgrade it to 1.5.3 and see if there are any positive changes.

[mhoppes]

MAC Table -- I assume it's stored in memory. Have you tried overloading it to see if you can replicate a switch crash? I'm just thinking, if you have a very busy site running 25-30 VLANs trunking 1,500 customers through it -- maybe something is getting overloaded in the tables from time to time?

[sirhc]

MAC Table -- I assume it's stored in memory. Have you tried overloading it to see if you can replicate a switch crash? I'm just thinking, if you have a very busy site running 25-30 VLANs trunking 1,500 customers through it -- maybe something is getting overloaded in the tables from time to time?

The switch only stores the MAC address for devices that are in the same Flat non routed Layer 2 segment.

The switch can handle 4096 MAC entries. All it does when you exceed 4096 is drop the oldest not talked to address.

So meaning if a switch is at head end and behind the switch is a router then more customers those customers beyond a router are not in the same Layer 2 segment and the MAC of the router port is used to forward all packets headed in that direction to customers on the other side of the router.

If you have 4096 devices on a flat Layer 2 segment you have a poorly designed network and will have far worse issues. The more devices in a segment increase the potential for things like packet storms and other packet related issues bringing down and entire segment.

https://forum.netonix.com/viewtopic.php?f=6&t=2580&p=18057&hilit=+mac+addresses#p18057

[mhoppes]

Chris -- what you say is correct, and we don't have anywhere near 4,096 device .... BUT consider you could have one or two routers plugged into a switch with VLANs going various places and could easily have a large number of MACs....

OK - we will keep digging.

[sirhc]

Chris -- what you say is correct, and we don't have anywhere near 4,096 device .... BUT consider you could have one or two routers plugged into a switch with VLANs going various places and could easily have a large number of MACs....

OK - we will keep digging.

Does not matter if the packet is encapsulated in a VLAN, a VLAN will not preserve the Layer 2 source or destination MAC through a router which is Layer 3 (stating obvious). Layer 2 MAC source and destination is lost and reassigned as soon as a packet hits a router for the packets journey through the next Layer 2 segment.

So a packet can have many Layer 2 MAC source and destination addresses on it journey from the source to the destination.

So a packet that leaves my computer will have my computer source MAC address until it hits my UBNT radio Ethernet port (this is where we could discuss bridge groups for radios in bridge mode) which was its Layer 2 Destination address which is in router/NAT then it takes the MAC of the radio interface as it new source Layer 2 MAC address for its journey to the tower through the Netonix switch there and when it hits the router interface which was it destination Layer 2 MAC address. Once the packet leaves the router that same packet will assume the MAC of the router interface it leaves as its new source Layer 2 MAC address and go to the edge of that Layer 2 segment where the MAC on that packet will change again. This will happen over and over again each time that packet leaves one Layer 2 segment and enters another Layer 2 segment on its Layer 3 journey across the WWW to its final destination which is a Layer 3 IP address.

[mhoppes]

I understand that... I'm saying consider a core tower with a wheel and spoke approach...... many micro-sites or smaller tower sites all hanging off of this tower. The tower has one router with many interfaces/VLANs connecting to the switch. The switch has backhauls going out to each of the small sites.

The core site handles the routing of each of these towers. So while it is not a bridged network, the switch may be handling a large number of MAC addresses.

Anyway -- we've derailed the original subject.

[sirhc]

if your routing from tower to tower regardless of VLANs MAC address do NOT get passed through a Layer 3 router.

All packets headed out a router interface to say an AF24 bridge then to another router all MAC address on the second tower will never be seen on the switch in the first tower. Once a packet is "routed" MAC source destination is changed.

Meaning all packets headed to tower 2 if there is a router in each tower then tower 1 switch can not see MACs on tower 2, they see all packets headed for tower 2 as the single MAC of the router interface on tower 2.


So if you have this

SWITCH1--ROUTER1--AF24BRIDGE--ROUTER2---SWITCH2--2,000DEVICES

SWITCH1 will see all packets headed to all 2,000DEVICES on SWITCH2 as the same destination MAC address which is the MAC address of the interface on ROUTER1 and when the packets leave ROUTER2 all packets will have the same source MAC address of the ROUTER2 interface they leave headed to SWITCH2.

Below are 3 separate Layer 2 segments and as packets transition from 1 Layer 2 segment to another their source and destination MACs change. SWITCH1 below sees all packets headed to Layer 2 segment # 3 as the same destination MAC
|11111111111||22222222222222222222||33333333333333333333333333|
SWITCH1--ROUTER1--AF24BRIDGE--ROUTER2---SWITCH2--2,000DEVICES

Now you might say well if there are more than 4096 subnets beyond as even though there is only one MAC the Layer 3 IP address needs to be retained you have a point but this is where the MAC of the oldest inactive entry is dropped to make room.

However inactive MAC entries are dropped anyway as all entries have a TTL. So could you create an issue if 4096 destination streams are all active......well yes and no. Again the switch will flush and learn over and over and over again.

It does get complicated from here but I "suppose" if the moons of jupiter alined and such then in a perfect storm you could have a problem but the next question you need to ask yourself is do you have more than 4096 clients? Then you would need all of them in constant communications SOLID, and so on. Or you could be experiencing some sort of DOS attack from outside or from within sending packets to all your customers at the same time but then again this is a DOS attack. Or some bad device sending out packet storms and the list goes on but I am unaware of any switch or router that is bullet proof against attacks. So there are switches with larger MAC table capabilities but I would be more interested to find out what is causing this issue and stop it not throw hardware at it to mask something that is going wrong.

[Jeandre]

Hi,

We have the same issues, We have now updated to v1.5.4 but it seems like there is still Memory being used/lost day by day.

Please see graph attached when available memory reaches 0MB the switch reboots.

[img]C:\Users\User\Desktop\Netonix%20Mem%20usage.PNG[/img]

[Stephen]

We've been trying to track this one down for awhile. I have a few questions that will help us isolate it for your situation. Do you have any SFP module's inserted? Are you using Discovery at all? Do you use SNMP in anyway? Did you notice a difference in the rate of memory loss when upgrading to 1.5.4?

[mhoppes]

I have SFP modules. Discovery is off. I use SNMP.