Problem with WS Generating Traffic and dying

DOWNLOAD THE LATEST FIRMWARE HERE
User avatar
Mac_SPITwSPOTS
Member
 
Posts: 19
Joined: Wed Aug 12, 2015 6:13 pm
Location: Homer, Alaska
Has thanked: 2 times
Been thanked: 2 times

Problem with WS Generating Traffic and dying

Wed Feb 01, 2017 3:37 pm

We have had 2 instances of this in the past week, the first instance I have no screen shots of but occurred on a WS-12-400-AC. The Second occurrence was on a WS-24-400A.
Both sites are set up switch first with a ubiquiti edgerouter and fed with AirFiber 24's. The WS-12-400-AC was operating on 1.4.6 and the WS-24-400A was on 1.4.2. We upgraded the WS-12-400-AC to 1.4.7rc4 however roughly 1.5Hrs later it occurred again on the switch at which time we replaced the physical switch with a new device configured from scratch to match the original configuration.


Symptoms: We lose communication to the switch and all devices connected to it aside from our router on site. We are unable to access the switch remotely. On site plugged directly into the switch with we are able to log into the switch. The screenshots + summary below are for the failure that occurred today on our WS-24-400A.

WS-12-400-AC
Switch was registering around 7-9Mbps constant TX on Port 1 which was powering the AF24 link off this site, there was minimal to no data being reported on every other port. Switch was operating on 1.4.6 initially and upgraded to 1.4.7rc4 after we got it back online. Rebooting the switch fixed it for roughly 1.5Hrs at which time the issue happened again. We replaced the switch at that time with a similar model running 1.4.7rc4 and configured from scratch by hand to avoid any problems with a broken config file.


WS-24-400A
Switch was registering 7-9Mbps constant TX on a single port at around 15Kpps. Disabling the port, as well as testing disabling ALL ports in the webUI had no affect and physically unplugging the cable leading into the port also had no affect. We tested disabling all VLAN's in the webUI and this had no affect either. A software reboot of the switch brought it back online. We have a spare configured by hand prepped for replacement in the event it happens again.

This IMGUR gallery shows screenshots of the switch while I was hardwired in and it was having problems, as well as after the software reboot at which time it was operating properly.

http://imgur.com/gallery/jxWhV
Operations Management is key.

User avatar
Mac_SPITwSPOTS
Member
 
Posts: 19
Joined: Wed Aug 12, 2015 6:13 pm
Location: Homer, Alaska
Has thanked: 2 times
Been thanked: 2 times

Re: Problem with WS Generating Traffic and dying

Wed Feb 01, 2017 3:56 pm

data_charts.png
Broken_TX_Issue




operating properly.png
Proper operation of switch


Screen Shot 2017-02-01 at 10.53.16 AM.png
Config 1/2


Screen Shot 2017-02-01 at 10.53.29 AM.png
Config 2/2



ports.png
Ports

vlan.png
VLAN

LAG.png
LAG

STP.png
STP

overview_data_tx.png
Broken_TX_Issue
Last edited by Mac_SPITwSPOTS on Wed Feb 01, 2017 5:55 pm, edited 4 times in total.
Operations Management is key.

User avatar
tma
Experienced Member
 
Posts: 122
Joined: Tue Mar 03, 2015 4:07 pm
Location: Oberursel, Germany
Has thanked: 15 times
Been thanked: 14 times

Re: Problem with WS Generating Traffic and dying

Wed Feb 01, 2017 5:11 pm

Please also have a look at my old post on the 8/15 issue which refers to even older posts. I will also add a link from my post to yours later.

The interesting thing about the 8/15 issue is (for me at least) that it can occur without involving an AF24 or other UBNT device that are usually taking the blame of generating pause frame storms. For me it occurred between two Netonix switches connected to really nothing else. Others, referred to in my post, reported similar absence of the typical UBNT suspects. A good summary was given in this post by TheHox, but that long thread then identified Airfibers as the source of all FC evil (which they are sometimes) and never came back to look at the initial descriptions.

If I read your post correctly, it happened (or at least continued) to send out 8 Mbps on a port that had no cable plugged in?
--
Thomas Giger

User avatar
Mac_SPITwSPOTS
Member
 
Posts: 19
Joined: Wed Aug 12, 2015 6:13 pm
Location: Homer, Alaska
Has thanked: 2 times
Been thanked: 2 times

Re: Problem with WS Generating Traffic and dying

Wed Feb 01, 2017 5:16 pm

Yes, even with the port either disabled in software, or with no physical cable plugged into it, it would continue to generate traffic.
Operations Management is key.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Problem with WS Generating Traffic and dying

Wed Feb 01, 2017 5:39 pm

What would be nice is when that traffic leaving the port is there check all the other ports to see if there is an input source to the switch coming from some other port.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
Mac_SPITwSPOTS
Member
 
Posts: 19
Joined: Wed Aug 12, 2015 6:13 pm
Location: Homer, Alaska
Has thanked: 2 times
Been thanked: 2 times

Re: Problem with WS Generating Traffic and dying

Wed Feb 01, 2017 5:59 pm

Updated with photos of the tabs requested. I don't have a screen shot of each individual port however I did check and there were no other ports showing traffic close the amount that was being generated on that port even if I aggregated those ports showing traffic together. When this issue happens again I will grab screen shots of each port being selected as well as an overview of the ports page.
Operations Management is key.

User avatar
Mac_SPITwSPOTS
Member
 
Posts: 19
Joined: Wed Aug 12, 2015 6:13 pm
Location: Homer, Alaska
Has thanked: 2 times
Been thanked: 2 times

Re: Problem with WS Generating Traffic and dying

Thu Feb 02, 2017 7:10 pm

Switch had the same issue occur today. I have pulled screen shots of every port while the switch was not functioning properly and we have replaced it with a similar unit configured from scratch. Do you want me to dump a few posts of 10 Photos each into here re. the ports?
Operations Management is key.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Problem with WS Generating Traffic and dying

Thu Feb 02, 2017 7:27 pm

put all in 1 post up to 24 pics
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
Mac_SPITwSPOTS
Member
 
Posts: 19
Joined: Wed Aug 12, 2015 6:13 pm
Location: Homer, Alaska
Has thanked: 2 times
Been thanked: 2 times

Re: Problem with WS Generating Traffic and dying

Fri Feb 03, 2017 7:05 pm

Port Overview for ports 1-24 while switch was not operating properly.
Attachments
Port 1.png
Port 1
Port 2.png
Port 2
Port 3.png
Port 3
Port 4.png
Port 4
Port 5.png
Port 5
Port 6.png
Port 6
Port 7.png
Port 7
Port 8.png
Port 8
Port 9.png
Port 9
Port 10.png
Port 10
Port 11.png
Port 11
Port 12.png
Port 12
Port 13.png
Port 13
Port 14.png
Port 14
Port 15.png
Port 15
Port 16.png
Port 16
Port 17.png
Port 17
Port 18.png
Port 18
Port 19.png
Port 19
Port 20.png
Port 20
Port 21.png
Port 21
Port 22.png
Port 22
Port 23.png
Port 23
Port 24.png
Port 24
Operations Management is key.

User avatar
tma
Experienced Member
 
Posts: 122
Joined: Tue Mar 03, 2015 4:07 pm
Location: Oberursel, Germany
Has thanked: 15 times
Been thanked: 14 times

Re: Problem with WS Generating Traffic and dying

Fri Feb 03, 2017 7:23 pm

So the 8/15 stream occurred on port 17 (as before) with a router connected there. And you were connected on port 23 (guessing from the 400 kbps stream) ?
--
Thomas Giger

Next
Return to Hardware and software issues

Who is online

Users browsing this forum: No registered users and 39 guests