Dropping ports on new WS, what is wrong with my setup?

[sirhc]

I would want to know what this unknown traffic is hitting the switch just before it locks up?

What is the traffic, where is it coming from.

Obviously the switch does not like whatever it is and eventually it breaks the switch.

So even if we figure out what type of traffic it is and somehow make our switch immune to it all that is doing is putting lipstick on it, do you really not want to find out what and where it is coming from?

[TheHox]

I've been using netonix's for a while and haven't had issues like this until now. Different situation than Adairs, but we are also getting the 8mbps at 16kpps taking down switches and flopping ports, etc.

I get it, you have thousands of switches deployed, and they are fine, we are not coming here saying your product is bad. We are just saying we are having issues (my fault, your fault, someone elses fault who cares) that we need help diagnosing as we have tried addressing them on our own internally and nothing is making any changes or sense here, so that is why we reach out to you.

[TheHox]

So far the only thing I can see that is similar, is that we are all using multiple switches. No idea what that has to do, but I had 2 switches in line and had issues, adair has 2 switches in line and has issues.

I moved my switches to my router and the issue stopped.

I added another switch down stream again, and the problem came back again.

'shrug'

I have that port disabled for now.

[Dave]

Has anyone noticed if the CPU usage goes up a lot on the status page when this happens??

[TheHox]

I don't think I have paid attention to that page when it happens, that is if I am even able to get to the switch.

I will try and look for that when it happens again.

[sirhc]

I don't think I have paid attention to that page when it happens, that is if I am even able to get to the switch.

I will try and look for that when it happens again.

You're going to try v1.4.0rc12 and let us know or wait a day or so for v1.4.0rc?? right?

[TheHox]

I'm playing with fire, I'm still running 1.3.9.

I think I've re arranged my network in that whatever was triggering the issues has gone away, we'll see.
Uptime 1 day 02:11:27


I want to see if it does it again with my current setup on 1.3.9, if it does, then I know it still persists and then I will test 1.4.0rc12.

[nickwhite]

Wow, so this thread just blew my mind. Glad to know I'm not crazy.

Chiming in to state that we've been seeing this same issue:
- ports/switches stop passing
- Interfaces (usually connected to AF24 or AF5) show a constant 8mbps or 15Mbps of traffic on RX
- Weird issues related with RSTP

I've been battling this off and on for at least 3 months. I thought it was an issue with our network, maybe a loop somewhere that squeaked through. But no... we know what loops look like and these aren't them. For a while I thought it was something with UBNT - primarily saw this on AF5 links. Our network has probably in the neighborhood of 50-60 Netonix switches. A handful of toughswitches still in the mix too. A total of around 90 POPs.

I'll post more details shortly... it's been a long weekend.

[nickwhite]

Has anyone noticed if the CPU usage goes up a lot on the status page when this happens??

From what I recall, no the CPU was not running high. I'll try and take note the next time I see this.

[adairw]

Some updates.
We replaced the netonix switch at our Morton site and it still locked up. In fact we had more lock ups than we've ever had. All switches running 1.4.0rc12
I went through and disabled all un needed services on all the switches in this chain and it still locked up.

The customer traffic for our sites normally comes in via a mikrotik router and vpls tunnel. Using VLANS to get out of the router and in to the switch. In desperation I reconfigured all the switches to just use VLAN's between each other and so far there hasn't been a crash in 34 hours. I'm gonna let this run for a few days and see what happens.
The mikrotik routers are still there, but they are only running the management traffic.