Loop failure rebooted switch, through issues 2nd reboot

[TheHox]

Short version: I accidentally created a loop, a netonix locked up and rebooted itself, and slowed TCP traffic down until I manually rebooted the switch again. UDP traffic was not effected which made it hard to track down.



Long version:

So I was firing up a new backhaul link last week, and I power those via the Netonix as a midspan, and VLAN them back. The port on the mikrotik for some reason had a bridge created with another port (something I must have been messing with a long time ago), so when I plugged everything in, I saw the mikrotik spitting out loop detected errors, I was watching it go crazy wondering what is going on.. Next thing I know, the Netonix was rebooting.. I disabled the interface on the mikrotik and finally saw the bridge I had created a long time ago..

I received a few phone calls that day about the internet dropping out, but it shortly came back after the netonix booted back up..
I removed the bridge, fired up my new backhaul and everything was fine, I left the tower site.



The next day I got 3 calls from customers complaining about slow speeds. They all usually get 10-20Mbps but were getting 2Mbps or less, and lots of buffering. I saw no issues, this tower has fiber at it, I did speed tests from their CPE to the sector, 70-80Mbps, I did speed test to other sectors on the tower to be sure traffic passed through the switch, and the UBNT speed tests in the CPE showed again 70-80Mbps. I was stumped..

I told the customer everything looks great on our end, so he tested right off our POE adapter, and said same problem, 1.5Mbps.. He asked for us to come out..

So Friday, I go to his house thinking the guy is nuts, I plug into the POE adapter inside his house, and sure enough, I get 2Mbps.. What the hell..

I ended up going on the roof and testing a new wire, new IP, new CPE, nothing helped. I then had a hunch. Did the switch that took a crap and rebooted itself from that loop still have issues? I triggered a reboot from his roof, did a speed test, POOF. 20Mbps down.

Not sure what that was about, but, something to remember next time I get some weird speed issues.

[sirhc]

Not sure what to say here, weird.

Also not sure what switch firmware version you are running but I strongly suggest v1.4.3rc7 - READ RELEASE NOTES

Next why do you not have RSTP enabled which would have prevented a loop on the switch.

And/Or you can run Loop Protection which will prevent loops from anywhere on your layer 2 segment from causing the switch grief.

[TheHox]

Odd,
So that switch was running 1.4.2, I just had another odd customer 3 hops down saying hes only getting 1mbps. We got 133Mbps when we installed him on Wednesday.

I tested the tik at his tower to my core and got 160Mbps TCP using the btest took in the mikrotik. If you've used the mikrotik btest, you know it slowly ramps up speed. When I did the test, it instantly got about 160Mbps, no ramp up, odd, but whatever.

I decided to reboot the netonix, now I do a btest, and it ramps up as usualy, and i got over 200Mbps...
This switch is running 1.3.9

Yea yea I know you are going to tell me to update, but, too many PM's going out with URGENT BUGS, that I'll never update right away, I say back a few using the stable ones..

[sirhc]

We know there are several bugs found in v1.4.2 that could be causing your issue.

Yes if you find a bug in the latest version please try the latest RC version and if the bug still exist arrange a call and Team View demonstration with Eric and we will fox it ASAP.

The reason is because there is no way to fix a bug in a closed firmware and v1.4.2 is closed and no longer being developed whereas v1.4.3rcX is being actively developed and if a customer has a bug in the code being developed and wants it fixed right away we will work with them.

If you read the release notes you will see that there really is no significant NEW features added to v1.4.3rcX just bug fixed that have been found in v1.4.2.

I am running v1.4.3rc7 at my WISP and yes the bugs found in v1.4.2 did affect me so I needed to upgrade to get the fixes.