Dropping ports on new WS, what is wrong with my setup?

DOWNLOAD THE LATEST FIRMWARE HERE
User avatar
tma
Experienced Member
 
Posts: 122
Joined: Tue Mar 03, 2015 4:07 pm
Location: Oberursel, Germany
Has thanked: 15 times
Been thanked: 14 times

Re: Dropping ports on new WS, what is wrong with my setup?

Sat May 14, 2016 7:03 pm

tma wrote:... I'd say the link that took all the traffic and produced many pause frames was doing similar to the mis-aimed link I had after deployment the other day -> lots of pause frames from the AF5X. I'm still trying to find a link in my network where I could try that without harming my customers too much.


Here's a video that shows: If an AF5X runs at 1/4 QPSK SISO, it will send a 600 Mbps / 1.25 Mpps FC pause frame stream into the Nextonix switch. Actually I did it twice (once as rehearsal before recording the video), and it happened to a few days ago. So this is reproducible:

http://www.tgnet.de/downloads/screenshots/600-mbps-fc-storm-from-af5x.wmvYou can read the full screenplay at the UBNT forum, titled 640 Mbps FC storm while in 1/4x SISO
I'd like to emphasize, though, that this test shows two things:

(1) Yes an AFX5X (and presumably other AF models as well) can produce a 600 Mbps flood of FC frames under certain conditions. I'm not sure (and don't want to try it) whether this only happens in 1/4x SISO modulation or whenever load is substantially higher than capacity.

(2) A Netonix switch will survive such storms without becoming inaccessible. In fact, I was connected to the CLI of the AF5X on its DATA port (using inband management) or else I wouldn't have been able to restore normal TX power.

So what does that do to the theory it will become inaccessible from only 16 Mbps of FC frames?
--
Thomas Giger

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Dropping ports on new WS, what is wrong with my setup?

Sat May 14, 2016 7:24 pm

WOW - AWESOME video Thomas!

That is a CRAP LOAD of Pause Frames!

Thank you for putting all the time and effort into that!
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Dropping ports on new WS, what is wrong with my setup?

Mon May 16, 2016 5:23 pm

So this issue is resolved, or it will be soon when ubnt release a new AF firmware. For now if you have any problems described in this thread simply disable Flow Control on the airFIBER radio or the switch port facing the airFIBER radio.

http://community.ubnt.com/t5/airFiber-B ... true#M3769

We are also going to put back in our Pause Frame Storm Protection Routine on our next firmware release which you can disable if you do not want to use it as a fall back. Also we too are looking into making sure we are properly implementing water marks which could have helped prevent this from being as bad of situation as it was for some people.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mayheart
Experienced Member
 
Posts: 166
Joined: Thu Jan 15, 2015 1:42 pm
Location: Canada
Has thanked: 43 times
Been thanked: 40 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 1:33 am

Has anyone had any success with enabling flow-control on the AC gear?

This is my port after 5 minutes. There's only 4 clients behind this AP and only 2mbps of traffic. I'm seeing this on other towers too with AC gear.

Running rc25 here, 1.3.9 on the other site. 8.0-b12 for the AC APs.

Rx Packets 49512
Rx Octets 6396711
Rx Unicast 45523
Rx Multicast 3857
Rx Broadcast 132
Rx Pause 19156

User avatar
mayheart
Experienced Member
 
Posts: 166
Joined: Thu Jan 15, 2015 1:42 pm
Location: Canada
Has thanked: 43 times
Been thanked: 40 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 12:33 pm

I created a post about it on the Ubiquiti forums. http://community.ubnt.com/t5/airMAX-AC- ... -p/1568041

The odd thing is having flow-control enabled KILLS the upload speed on the CPE end which shouldn't be happening.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 1:11 pm

mayheart wrote:I created a post about it on the Ubiquiti forums. http://community.ubnt.com/t5/airMAX-AC- ... -p/1568041

The odd thing is having flow-control enabled KILLS the upload speed on the CPE end which shouldn't be happening.


I only have one type of airMAX AC links in service and they are PowerBeam 500mm AC PTP links with RF Armor of course to high end customers. I do have flow control enabled and I am not having any issues but let me stress that these are relatively short links so the modulation is VERY STABLE as the signal is EXCELLENT. Note in the second picture no Rx or Tx pause frames are present but Flow Control is Active. But this is a PTP link and very stable modulation and GREAT SNR. SEE EXAMPLE PICTURE BELOW

One other person (Yahel) already reported that he "may" have seen this same weird behavior with airMAX AC equipment but was not sure.

I am not sure how the AC gear works compared to the airMAX M gear yet. The airMAX M gear did not transmit when there was no data going through the link and it only reported air rates which was not capacity but the AC gear seems to act similar to AF gear in that it reports capacity which could mean that they are transmitting all the time even with no payload. In that event they "could" suffer the same affliction as the AF radios when modulation changes or interference drops capacity?

This is only a guess?

CLICK IMAGE TO VIEW FULL SIZE
AC Link.jpg

AC Link Switch.jpg
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mayheart
Experienced Member
 
Posts: 166
Joined: Thu Jan 15, 2015 1:42 pm
Location: Canada
Has thanked: 43 times
Been thanked: 40 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 2:27 pm

Here's a screen shot of one AC unit.

Image

This is 42 seconds later...

Image

That's 15,448 pause frames within 42 seconds. Kinda excessive. FYI everyone on this tower is at 256QAM.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 3:44 pm

I am going to "guess" that maybe the airMAX AC units might suffer from the same Flow Control problem as the AF radios?

You're going to have to disable Flow Control I guess for those ports facing your airMAX AC units.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
WisTech
Associate
Associate
 
Posts: 213
Joined: Mon Aug 04, 2014 3:57 pm
Has thanked: 8 times
Been thanked: 64 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 5:39 pm

*sigh* Not again. Freaking ubiquiti. This AC line is plagued with so much nonsense.

User avatar
Rogue Broadband
Member
 
Posts: 19
Joined: Fri Mar 11, 2016 12:24 pm
Location: Oregon
Has thanked: 3 times
Been thanked: 17 times

Re: Dropping ports on new WS, what is wrong with my setup?

Wed May 18, 2016 6:18 pm

I can confirm this is happening on my tower with all AC gear. I turned Flow Control off and now I am seeing 0 Tx/Rx pause frames on all 5 of my Rocket AC PTMP units.
Attachments
pause_frames.JPG
Screen before reset port stats showing pause frames
FC_off.JPG
Flow Control Off
Pause_0.JPG
Pause Fames

PreviousNext
Return to Hardware and software issues

Who is online

Users browsing this forum: No registered users and 61 guests