Page 1 of 2

Problem with WS Generating Traffic and dying

Posted: Wed Feb 01, 2017 3:37 pm
by Mac_SPITwSPOTS
We have had 2 instances of this in the past week, the first instance I have no screen shots of but occurred on a WS-12-400-AC. The Second occurrence was on a WS-24-400A.
Both sites are set up switch first with a ubiquiti edgerouter and fed with AirFiber 24's. The WS-12-400-AC was operating on 1.4.6 and the WS-24-400A was on 1.4.2. We upgraded the WS-12-400-AC to 1.4.7rc4 however roughly 1.5Hrs later it occurred again on the switch at which time we replaced the physical switch with a new device configured from scratch to match the original configuration.


Symptoms: We lose communication to the switch and all devices connected to it aside from our router on site. We are unable to access the switch remotely. On site plugged directly into the switch with we are able to log into the switch. The screenshots + summary below are for the failure that occurred today on our WS-24-400A.

WS-12-400-AC
Switch was registering around 7-9Mbps constant TX on Port 1 which was powering the AF24 link off this site, there was minimal to no data being reported on every other port. Switch was operating on 1.4.6 initially and upgraded to 1.4.7rc4 after we got it back online. Rebooting the switch fixed it for roughly 1.5Hrs at which time the issue happened again. We replaced the switch at that time with a similar model running 1.4.7rc4 and configured from scratch by hand to avoid any problems with a broken config file.


WS-24-400A
Switch was registering 7-9Mbps constant TX on a single port at around 15Kpps. Disabling the port, as well as testing disabling ALL ports in the webUI had no affect and physically unplugging the cable leading into the port also had no affect. We tested disabling all VLAN's in the webUI and this had no affect either. A software reboot of the switch brought it back online. We have a spare configured by hand prepped for replacement in the event it happens again.

This IMGUR gallery shows screenshots of the switch while I was hardwired in and it was having problems, as well as after the software reboot at which time it was operating properly.

http://imgur.com/gallery/jxWhV

Re: Problem with WS Generating Traffic and dying

Posted: Wed Feb 01, 2017 3:56 pm
by Mac_SPITwSPOTS
data_charts.png
Broken_TX_Issue




operating properly.png
Proper operation of switch


Screen Shot 2017-02-01 at 10.53.16 AM.png
Config 1/2


Screen Shot 2017-02-01 at 10.53.29 AM.png
Config 2/2



ports.png
Ports

vlan.png
VLAN

LAG.png
LAG

STP.png
STP

overview_data_tx.png
Broken_TX_Issue

Re: Problem with WS Generating Traffic and dying

Posted: Wed Feb 01, 2017 5:11 pm
by tma
Please also have a look at my old post on the 8/15 issue which refers to even older posts. I will also add a link from my post to yours later.

The interesting thing about the 8/15 issue is (for me at least) that it can occur without involving an AF24 or other UBNT device that are usually taking the blame of generating pause frame storms. For me it occurred between two Netonix switches connected to really nothing else. Others, referred to in my post, reported similar absence of the typical UBNT suspects. A good summary was given in this post by TheHox, but that long thread then identified Airfibers as the source of all FC evil (which they are sometimes) and never came back to look at the initial descriptions.

If I read your post correctly, it happened (or at least continued) to send out 8 Mbps on a port that had no cable plugged in?

Re: Problem with WS Generating Traffic and dying

Posted: Wed Feb 01, 2017 5:16 pm
by Mac_SPITwSPOTS
Yes, even with the port either disabled in software, or with no physical cable plugged into it, it would continue to generate traffic.

Re: Problem with WS Generating Traffic and dying

Posted: Wed Feb 01, 2017 5:39 pm
by sirhc
What would be nice is when that traffic leaving the port is there check all the other ports to see if there is an input source to the switch coming from some other port.

Re: Problem with WS Generating Traffic and dying

Posted: Wed Feb 01, 2017 5:59 pm
by Mac_SPITwSPOTS
Updated with photos of the tabs requested. I don't have a screen shot of each individual port however I did check and there were no other ports showing traffic close the amount that was being generated on that port even if I aggregated those ports showing traffic together. When this issue happens again I will grab screen shots of each port being selected as well as an overview of the ports page.

Re: Problem with WS Generating Traffic and dying

Posted: Thu Feb 02, 2017 7:10 pm
by Mac_SPITwSPOTS
Switch had the same issue occur today. I have pulled screen shots of every port while the switch was not functioning properly and we have replaced it with a similar unit configured from scratch. Do you want me to dump a few posts of 10 Photos each into here re. the ports?

Re: Problem with WS Generating Traffic and dying

Posted: Thu Feb 02, 2017 7:27 pm
by sirhc
put all in 1 post up to 24 pics

Re: Problem with WS Generating Traffic and dying

Posted: Fri Feb 03, 2017 7:05 pm
by Mac_SPITwSPOTS
Port Overview for ports 1-24 while switch was not operating properly.

Re: Problem with WS Generating Traffic and dying

Posted: Fri Feb 03, 2017 7:23 pm
by tma
So the 8/15 stream occurred on port 17 (as before) with a router connected there. And you were connected on port 23 (guessing from the 400 kbps stream) ?