WS-12-250-AC - Locking up?

[sirhc]

So one thing that can make a switch appear locked up is a packet lock which is from Flow Control. There are many LONG discussions about this on this forum and on UBNT forum.

One thing you can try is disable all Flow Control on all ports and see if issue stops.

We know UBNT had or still has FLow Control issues with AF, AFX, and airMAX AC radios. Although I thought I heard the latest versions of firmware for some appear to fix this issue?

We also know that some Cambium epmp I "think" 2000 radios were also "suspected" to have a Flow Control issue.

AS I said when you have a switch that appear to be locked up use a console cable. If the console is working the switch is NOT locked up.

Then to find the offending port on the switch simply start unplugging cables 1 at a time, wait 5 seconds and plug it back in. WHen you find the offending port where the packets are all being held for once the Ethernet link drops the packets are flushed an traffic will resume. A better way is to use the console cable and simply disable Ethernet communications to each port for a couple seconds then enable the port again. This is disabling the Ethernet link not the POE so the radio will not reboot just the link drops and the packets held in buffers for that port are dropped by the switch allowing packets to flow again.

Remember switches used SHARED buffer memory for all ports so when the buffer is FULL this can cause all ports to refuse packets. This is not by our design this is simply the way most switch cores work. We do not write the code for the cores nor do other switch manufacturers that code is written by the core chip manufacturer we simply write the UI and such and configure the core with the desired options.

[jormerod]

Any further thoughts on this?

It is now getting to the point where we must replace the netonix switch with another manufacturer.

Again it has just crashed and I have to go and reboot it manually.

[sirhc]

Any further thoughts on this?

It is now getting to the point where we must replace the netonix switch with another manufacturer.

Again it has just crashed and I have to go and reboot it manually.

Did you do any of the things I asked you to do above/below and report back what you found?

So one thing that can make a switch appear locked up is a packet lock which is from Flow Control. There are many LONG discussions about this on this forum and on UBNT forum.

One thing you can try is disable all Flow Control on all ports and see if issue stops.

We know UBNT had or still has FLow Control issues with AF, AFX, and airMAX AC radios. Although I thought I heard the latest versions of firmware for some appear to fix this issue?

We also know that some Cambium epmp I "think" 2000 radios were also "suspected" to have a Flow Control issue.

As I said when you have a switch that appear to be locked up use a console cable. If the console is working the switch is NOT locked up.

Then to find the offending port on the switch simply start unplugging cables 1 at a time, wait 5 seconds and plug it back in. WHen you find the offending port where the packets are all being held for once the Ethernet link drops the packets are flushed an traffic will resume. A better way is to use the console cable and simply disable Ethernet communications to each port for a couple seconds then enable the port again. This is disabling the Ethernet link not the POE so the radio will not reboot just the link drops and the packets held in buffers for that port are dropped by the switch allowing packets to flow again.
Remember switches used SHARED buffer memory for all ports so when the buffer is FULL this can cause all ports to refuse packets. This is not by our design this is simply the way most switch cores work. We do not write the code for the cores nor do other switch manufacturers that code is written by the core chip manufacturer we simply write the UI and such and configure the core with the desired options.

[Julian]

Any further thoughts on this?

It is now getting to the point where we must replace the netonix switch with another manufacturer.

Again it has just crashed and I have to go and reboot it manually.

Sounds like you're having fun.. While you're there, plug in a console cable and make sure it's not a packet lock - if you get intelligence via rs232 and can reboot it via console, it actually is packet lock, and flow control needs to be turned off.

I have a thought, here, which is going to require some reconfiguring on your end, if it's not that.

There are two 24v power supplies in the 24-400s, one for each bank of ports. We rate them each at 110W, but they will push around 140W, give or take, at a non-constant draw. Your load is lower than the rated limit, but is entirely on bank 0, which may have some issue preventing it from pushing full capacity - I don't know, i'm thousands of miles away.

What I'd like you to do, though, is split your load between banks 0 and 1, roughly evenly, and see if your problem persists - If not, then we'll need to get your switch in for some troubleshooting.

[aaron@futuretk.com]

I'm having what sounds like a very similar problem with a WS-12-400-AC on 1.4.8 firmware. I've had three separate incidents in the last week where the switch has become inaccessible over the network. In two of the cases someone was able to power cycle it and it came back on it's own. The lights appeared normal on the switch prior to the reboots. In the third case it came back on it's own (about 30 min later) before we were able to get a tech to the location. The logs showed that the switch had just booted up in that third case.

This switch is configured with LAG and has two gig fiber uplinks to separate blades in a Cisco core switch. During the incident the Cisco shows no link on either fiber going to the switch. This is one of 5 Netonix switches all in the same rack, connected to the same UPS, and connected to the same two blades in the same Cisco core switch. Aside from the fiber uplinks there are only three devices connected, drawing 36, 38, and 14 watts of PoE power. Board voltages and temperatures appear normal.

Loop protection and spanning tree are both disabled, and I don't suspect it's a flow control issue since the switch rebooted on it's own in the most recent case.

This is a critical backhaul switch so I've replaced it with another new WS-12-400-AC and copied the config from this one. I have the pulled switch running on the bench with a continuous ping running to it in case I can catch it happening again.

[sirhc]

I am suspecting Flow Control

Why would you not hook up a console cable before rebooting?
Always make sure your techs have a serial cable and software that you pre tested to work.
This is why we spent all the extra money putting console ports on so your not throwing farts at a problem board in the dark.

If this happens again disable Flow Control on all ports and see if it happens again.

I have plenty of long winded posts explaining how Flow Control can packet lock a switch.

It would also help in these cases to provide more information such as screen grabs of the following TABs:
Status
Ports
VLANs
STP/RSTP
LAG
and any other TABs with significant changes.

Could also be the result of hard coding speed/duplex on ports which you should never do, always use AUTO else at your own peril.

Could also be a bad SFP module if your using one.

So many possibilities so little information.

[futur electronique]

i had a similar issue today, we had to remove power to the switch to get it back up and running, i couldn't access the Switch's GUI.

i did not try to console in when it was jammed but i will try if it ever happens to me again.

[sirhc]

i had a similar issue today, we had to remove power to the switch to get it back up and running, i couldn't access the Switch's GUI.

i did not try to console in when it was jammed but i will try if it ever happens to me again.

Next time it happens try console cable

If console works then the switch is not locked up but it may be packet locked from Flow Control.

Simply unplug one cable at a time for a few seconds then plug it back in. Work your way through the ports until you find the port that caused the packet lock. As soon as the port link drops all packets held in buffer memory will drop freeing up all the ports.

Flow Control is great on small Layer 2 segments if you set things up right but for LARGE flat networks I would be careful with using Flow Control.

Switches use a memory pool for packet buffering and if 1 port on the switch is receive Pause Frames from a device and there are a lot of packets destined to that port the switch memory will fill up fast then there is no memory buffers available anywhere and suddenly the switch starts refusing packets on all ports. This is why the AFX and airMAX AC Flow Control issue caused a lot of problems.

Also another BIG mistake people use is a single uplink port to the switch for backhauls and APs. I never share the same port in a router or uplink port in the switch for both APs and backhauls. Look here for my setup: viewtopic.php?f=30&t=452

You can also get a tour of my WISP tower setup in the 1.5 hour video on youtube: https://www.youtube.com/watch?v=8JvBEAD4MFM


Also make sure you are running v1.4.8

[futur electronique]

Next time it happens try console cable i will give it a try if it ever happens again for sure.

If console works then the switch is not locked up but it may be packet locked from Flow Control.

Simply unplug one cable at a time for a few seconds then plug it back in. Work your way through the ports until you find the port that caused the packet lock. As soon as the port link drops all packets held in buffer memory will drop freeing up all the ports. would this also indicate the packet lock condition? can i do this on Powered on Ports? i only use 24v and 48v, i dont suspect they are the ones locking up.

Flow Control is great on small Layer 2 segments if you set things up right but for LARGE flat networks I would be careful with using Flow Control.

Switches use a memory pool for packet buffering and if 1 port on the switch is receive Pause Frames from a device and there are a lot of packets destined to that port the switch memory will fill up fast then there is no memory buffers available anywhere and suddenly the switch starts refusing packets on all ports. This is why the AFX and airMAX AC Flow Control issue caused a lot of problems.

Also another BIG mistake people use is a single uplink port to the switch for backhauls and APs. I never share the same port in a router or uplink port in the switch for both APs and backhauls. Look here for my setup: http://forum.netonix.com/viewtopic.php?f=30&t=452 i am only using a single port to the switch.. i need to understand this better so i dont just create loops.. i will keep on reading and searching.

You can also get a tour of my WISP tower setup in the 1.5 hour video on youtube: https://www.youtube.com/watch?v=8JvBEAD4MFM im halfway trough this movie, i will probably have to listen to it a few times to catch everything but it is awesome thanks so much for taking the time to do this video.


Also make sure you are running v1.4.8 the switch that locked up is running 1.4.8, i have another one running 1.4.7 [/quote]

[sirhc]

would this also indicate the packet lock condition? can i do this on Powered on Ports? i only use 24v and 48v, i dont suspect they are the ones locking up.

We normally do not suggest you unplug and plug cables with POE ON but in this case we know the cable is NOT shorted and the device is using the proper POE just make sure to unplug and plug the connector in straight.

Most common cause of packet lock is excessive Pause Frames, this can occur on an AP that is trying to talk to a bad or limited connection and the client is getting a large stream faster than the AP can deliver. This causes switch buffers to fill up then suddenly there is no packet buffer memory left and any disruption on any port can cause that port to size up due to flow control. NOT A LOCK UP FLOW CONTROL IS BEHAVING CORRECTLY SADLY.

i am only using a single port to the switch.. i need to understand this better so i dont just create loops.. i will keep on reading and searching.
im halfway trough this movie, i will probably have to listen to it a few times to catch everything but it is awesome thanks so much for taking the time to do this video.

If all radios, backhauls and local APs all get to the router through a single port then turn Flow Control OFF on all port. If a pause frame is received on Port 4 going to an AP then a Pause Frame is sent out Port 1 to the router Pausing ALL traffic when the buffers fill up and if there is a large stream destined to a client on that AP then buffers can fill up fast. Watch that video several time I talk about separating backhaul links from local radios.

the switch that locked up is running 1.4.8, i have another one running 1.4.7

I do NOT think the switch locked up I really think you experienced a packet lock, as I said simply disable all Flow Control on ALL ports and it probably will not happen again. Flow Control is a good thing if implemented correctly on a routed network but on a large flat network unless you really know what your doing it is better to disable it all together.