Loop failure rebooted switch, through issues 2nd reboot
Posted: Sat Aug 27, 2016 8:17 pm
Short version: I accidentally created a loop, a netonix locked up and rebooted itself, and slowed TCP traffic down until I manually rebooted the switch again. UDP traffic was not effected which made it hard to track down.
Long version:
So I was firing up a new backhaul link last week, and I power those via the Netonix as a midspan, and VLAN them back. The port on the mikrotik for some reason had a bridge created with another port (something I must have been messing with a long time ago), so when I plugged everything in, I saw the mikrotik spitting out loop detected errors, I was watching it go crazy wondering what is going on.. Next thing I know, the Netonix was rebooting.. I disabled the interface on the mikrotik and finally saw the bridge I had created a long time ago..
I received a few phone calls that day about the internet dropping out, but it shortly came back after the netonix booted back up..
I removed the bridge, fired up my new backhaul and everything was fine, I left the tower site.
The next day I got 3 calls from customers complaining about slow speeds. They all usually get 10-20Mbps but were getting 2Mbps or less, and lots of buffering. I saw no issues, this tower has fiber at it, I did speed tests from their CPE to the sector, 70-80Mbps, I did speed test to other sectors on the tower to be sure traffic passed through the switch, and the UBNT speed tests in the CPE showed again 70-80Mbps. I was stumped..
I told the customer everything looks great on our end, so he tested right off our POE adapter, and said same problem, 1.5Mbps.. He asked for us to come out..
So Friday, I go to his house thinking the guy is nuts, I plug into the POE adapter inside his house, and sure enough, I get 2Mbps.. What the hell..
I ended up going on the roof and testing a new wire, new IP, new CPE, nothing helped. I then had a hunch. Did the switch that took a crap and rebooted itself from that loop still have issues? I triggered a reboot from his roof, did a speed test, POOF. 20Mbps down.
Not sure what that was about, but, something to remember next time I get some weird speed issues.
Long version:
So I was firing up a new backhaul link last week, and I power those via the Netonix as a midspan, and VLAN them back. The port on the mikrotik for some reason had a bridge created with another port (something I must have been messing with a long time ago), so when I plugged everything in, I saw the mikrotik spitting out loop detected errors, I was watching it go crazy wondering what is going on.. Next thing I know, the Netonix was rebooting.. I disabled the interface on the mikrotik and finally saw the bridge I had created a long time ago..
I received a few phone calls that day about the internet dropping out, but it shortly came back after the netonix booted back up..
I removed the bridge, fired up my new backhaul and everything was fine, I left the tower site.
The next day I got 3 calls from customers complaining about slow speeds. They all usually get 10-20Mbps but were getting 2Mbps or less, and lots of buffering. I saw no issues, this tower has fiber at it, I did speed tests from their CPE to the sector, 70-80Mbps, I did speed test to other sectors on the tower to be sure traffic passed through the switch, and the UBNT speed tests in the CPE showed again 70-80Mbps. I was stumped..
I told the customer everything looks great on our end, so he tested right off our POE adapter, and said same problem, 1.5Mbps.. He asked for us to come out..
So Friday, I go to his house thinking the guy is nuts, I plug into the POE adapter inside his house, and sure enough, I get 2Mbps.. What the hell..
I ended up going on the roof and testing a new wire, new IP, new CPE, nothing helped. I then had a hunch. Did the switch that took a crap and rebooted itself from that loop still have issues? I triggered a reboot from his roof, did a speed test, POOF. 20Mbps down.
Not sure what that was about, but, something to remember next time I get some weird speed issues.