where did my bandwidth go?

[cyberbroadband]

Howdy ho!

Very difficult to come up with a proper subject line here. Lots to share. Lots of individual facts to take into consideration leaving me wondering where the right place to go for answers is. I came here. yay me! first post too, I think!

Been using Netonix since about April. Very happy. They do great things and really clean up our tower cabinet...

Biggest regret? I purchased a 24 port without measuring our cabinet and wound up having to "temporarily balance" the 24 port unit on top of some cardboard light bulb packaging until i got two 12 port units. It is all better now ! :)

Ok - so my problem is this. It contains a lot of detail so read the whole post before replying please.

There was a fiber cut about a month ago. Our upstream apparently installed another customer and did not reassemble the splice case correctly disrupting our service (their service techs words). I am not stating this as an overall cause - just stating facts.

Sometime after this we got some random speed complaints on our network. Nothing major, just a few to look into. We're mostly a 900 mhz canopy network anyway (although we are looking at some newer gear) - but recently we have added some 2.4 ubnt and EPMP as well - so there are some customers with 10 and 15 meg packages.

Using a mikrotik btest server we determined we were not getting fully capacity incoming but were getting full capacity outcoming. At times it was about a 2:1 radio. If we got 900 meg sending data out we got 400 meg incoming. If we went further away (across the country for example) if we got 600 sending data out we only got 250 data coming in.

At peak usage we are only using about 250 meg. We mainly have the gig circuit in preparation for a fiber deployment

Upon some peer suggestions, we upgraded our mimosa backhaul radios (we have 6 - both b5 and b5c) from 1.4.1 to 1.4.4 . Immediately the GUI of the mimosas showed almost 2x the speed, showing it could do 300+ physical traffic. Indeed using a mikrotik bandwidth test I could send that much UDP. I couldn't send that much TCP....if i was lucky i could send 150 TCP. Running the same test with UDP I could get 305 to 320 meg. I know UDP does not require an acknowledgement packet - but it still amazes me the results are that different. (back to the fiber tests, I could UDP better speeds than I could TCP - but still results were similar)

On additional peer suggestion, I also upgraded our netonix switches (two of them, 12 port) to 1.4.6. They were on a release candidate that was out prior to my installing the switches (which was late summer).

Another peer redirected me to the flow control discussion happening here. I read through some of that. The master side of my mimosas are coming off a mikrotik cloud router switch. flow control is not enabled. my slave side (and 2nd hop master) is a netonix. Flow control is now on by default (I think I read in a firmware update here at Netonix?) The mimosas do NOT show the FLOW CONTROL icon on the status screen like my EPMP radios do. (and my UBNT radios for that matter). Does not matter if flow control is on or off in the radio - the mimoas do not show the flow control (two arrows left/right) icon for beeing enabled.

I am still able to send the above mentioned speedtests out our network - from our core router across to the remote side - so I don't think it is a speed issue there.

Is it flow control somehow on our epmp radios? or ubnt radios? I graph quite a bit using MRTG (and we have some internal graphing with powercode) and it looks like the ubnt radios are performing just fine.

Should I be expecting an overall reduction of 1/4th the bandwidth across our network due to our fiber feed being slower? ( trust me I have a ticket open with them).
We can still pull 400-500 meg from inside our office and our customers before any of this happened were only using 250-300 meg.

When enabling flow control, I know look under stats -> pause frames, but is it like a vlan where you need to enable it on every switch it touches? including mikrotik? does it play well with mikrotik? I have tweaked and tweaked but haven't really found a configuration that gets us back where we were before this began.

Even worse, I don't know what really broke it....

We have over 600 customers. I'd say we've heard from 50 of them. Especially the ones with higher packages.

Thanks in advance.

Jay Fuller
Cyber Broadband Inc
Cullman AL

[cyberbroadband]

quick update. Plugged a laptop into our cloud router switch colocated with our fiber (just before our first mimosa hop) - I get 450 down and 900 up there using an internet speed test.

[mike99]

Hard to tell with so few but I would suspect something wrong with the mimosa. Not sure how Mikrotik handle UDP bandwidth test but with UDP, sending side can send more than actually received on the other side so only the receiving side bandwidth should be look at.

Also, you must know what your doing with flow control and not enable it blindly else it could make things a lot worst than without it.

[cyberbroadband]

from :

viewtopic.php?f=6&t=2170

...mimosa flow control does not work ?

also trouble with epmp 2000? but we run epmp 2.4 , which i think is 1000

....

ugh

so why did a firmware update enable flow control on everything?
i think i was told that - - i need to verify it.
i am wondering if i should turn all the flow control off

[cyberbroadband]

ok, since i'm having a good conversation with myself here, i've just finished doing a ton of reading.

About mimosa - thank you google - they've been asked on their forum about flow control. They insist it works. I have enabled it on our network (after reading the 10 page flow control discussion, haven't had time for the movie yet....maybe tomorrow).

i still do not know what caused all of our issues earlier - - it could be a water issue, a port flapping issue (we moved our mimosa to another port in our mikrotik cloud router switch, it seeems to have stopped flapping) - did i mention flapping in my initial post?

speeds were more normal this evening - - i JUST NOW enabled flow control on everything again - - but i did do a speedtest across our mimosa network and it hit 300 meg almost immediately whereas before the speedtest would take a while to get there or work its way there.

will also add no pause frames appeared on my one netonix switch (most of the links are mikrotiks being used as switches - there are two links before we get to another router)

I look forward to experimenting more - but i have to wait for another "prime time". Darn....

[sirhc]

MIMOSA is full of crap, their Flow Control does NOT work or even negotiate, I have stated this many times.

HERE ARE A FEW:
viewtopic.php?f=17&t=1654&p=12766&hilit=+mimosa+flow+control#p12766
viewtopic.php?f=6&t=2183&p=15656&hilit=+mimosa+flow+control#p15656
viewtopic.php?f=6&t=2170&p=15543&hilit=+mimosa+flow+control#p15543
viewtopic.php?f=5&t=1710&p=12826&hilit=+mimosa+flow+control#p12826

One person reported an issue with epmp 2000 Flow Control but has never been verified and as I do not use epmp gear I can not test it at my WISP.
viewtopic.php?f=17&t=2158&p=15555&hilit=+epmp+flow+control#p15555

I am not ignoring your post, there just is not much I can add yet as you need to do more testing and narrow it down.

Plus I am working with pneumonia, feel like crap.

[cyberbroadband]

I am sorry to hear about the pneumonia - I've been fighting a cold all week myself. Things have looked much better since enabling flow control. However, I can't understand what changed "suddenly" to require this. I guess our issue could have been resolved simply by moving the problematic backhaul radio to another port and not enabled flow control but since we performed both steps at the same time I guess we won't ever know...

Nevertheless, appreciate all the advice and information here. I'll keep reading. :)