Random port drops

[dmcken]

Good Day,

Every so often we are getting a group of ports dropping, logs are telling me the following:

Jun 11 00:15:57 kernel: link state changed to 'down' on port 5
Jun 11 00:15:57 kernel: link state changed to 'down' on port 6
Jun 11 00:15:57 kernel: link state changed to 'down' on port 8
Jun 11 00:15:57 kernel: link state changed to 'down' on port 7
Jun 11 00:15:57 kernel: link state changed to 'down' on port 10
Jun 11 00:15:57 kernel: link state changed to 'down' on port 3
Jun 11 00:15:57 kernel: link state changed to 'down' on port 9
Jun 11 00:15:58 kernel: link state changed to 'up' on port 6
Jun 11 00:15:58 kernel: link state changed to 'up' on port 5
Jun 11 00:15:58 kernel: link state changed to 'up' on port 8
Jun 11 00:15:58 kernel: link state changed to 'up' on port 10
Jun 11 00:15:58 kernel: link state changed to 'up' on port 9
Jun 11 00:15:59 kernel: link state changed to 'up' on port 3
Jun 11 00:15:59 kernel: link state changed to 'up' on port 7

Repeating a few times over the past couple hours (seems kinda random atm but the time difference between the last outage and now is about 1 hour 45 minutes)

I'm attaching an overview of my particular switch. The devices in ports 3-6 have been there since last week Friday with no issues.

Is there any way to intelligently debug which device may be causing the issue (I believe it may be some sort of cabling issue at this point). Or if anyone can point out something to check to see where the problem might lie.

[adairw]

What's plugged in to the ports? Ubiquiti radios or something else?
Cable tests good? What was powering this stuff before the netonix?

[sirhc]

Yea, a little more info would help, why are you blocking the port descriptions which may help identify the device in those port?

What firmware version are you running?

I would examine the cabling and ends, like Adair said to cable diagnostics test from switch UI.

Tell us what is in each port and the cable diagnostic results.

Check the cable to make sure it passes the wiggle test.

Who knows maybe the connector at the radio is not good as well. Always do the wiggle test at the radio when installing them.

Did you check the devices to see if they are rebooting?

Provide as much information as possible.

[dmcken]

Everything is the standard UBNT Rocket M5.

Ports 3 - 10 are our Sectors (I think 45 degree KP antennas).

Port 13 is a M2 running on a licensed frequency and 14 and 15 are rockets on dishes (backhauls to other sites).

Ports 17-22 are to a cluster of 6 Canopy APs (not powered by the netonix).

Port 24 is the uplink to our mikrotik router at the site.

It is running the latest stable firmware (1.2.2), as a side note is there any way to see what version it is running? As we look to roll out more of them we will need that to see which ones need upgrading.

They were previously powered by digital logger 24volt for the better part of 2 years, cable is shielded UBNT tough cable with shielded ends.

I assume you mean with wiggle test that if we move the cable the port drops, none of the cables have that problem. We re-terminated a few just because we thought the termination looked funny (mainly the jack was biting onto the wires instead of the outer insulation).

All the cables leading to UBNT devices are coming out with a cable diagnostics similar to what I have attached (the only difference is the cable length, which varies between 47 and 54 meters).

I have seen the issue happen twice, I don't hear anything from the switch, just see the ports ethernet link drop and go red (the first state while the device is booting, until it drops the link and comes up at 100-F). Eventually the device boots up and operates normally, logging into to the device and checking the uptime confirms the device has rebooted.

I'm willing to believe this is a cable issue. We have swapped the UBNT devices in ports 9 and 10 with two unpowered canopy devices in port 20 and 21. This is due to the fact that we had 4 devices successfully powered up on it since friday and only moved the remainder of the kit over on Wednesday once we got an issue with site cooling sorted out. Going through each cable and testing it will take some time so I was hoping to find some way of identifying which one of the cables is most likely to be the one causing us trouble. Right now only ports 3 to 10 are dropping and 3 to 6 have been there for almost a full week before the issue started so we are trying to isolate the 4 new UBNT devices to see if we can identify which one might be the culprit (hopefully this way will limit ti to 2 cables that need to be checked).

[sirhc]

Well you can click on the Device/Status Tab and see the firmware version

Your cable diagnostic there shows abnormal termination on pair 3 and 4 and it should report Short and Short NOT TO BE CONFUSED WITH CROSS SHORT.

Pair 3 and 4 carry the DC power on standard airMAX 10/100 gear and it should report
Pair 1 OK
Pair 2 OK
Pair 3 Short (since the Blue and Blue/White are bonded)
Pair 4 Short (since the Brown and Brown/White are bonded)

Since you are showing abnormal termination I would "guess" one of the wires in each pair is not all the way up in the connector or the wire is damaged. If this is the case then the radio is only getting power on 1 wire for positive and 1 wire for negative which would cause the device to not be getting enough current so when the AP starts transmitting heavily it gets staved for power and reboots.

Also pair lengths should all be the same. When pair lengths are not reporting the same length it usually means the cable was stretch or kinked during installation and it is now JUNK! Ethernet surge protector will also mess the lengths up. Also always check the wire before you power up and after as some equipment mess with pair lengths until you power them up.

Another possibility is your using an Ethernet Surge protector WHICH I HATE and it is bad.

I would like to take this opportunity to stand on my soap box - not directed at poster just PREACHING TO THE MASSES!!!
People you need Multiple ground rods far apart for your tower and you need at least 2 ground rods for your electrical service AND THEN THE MOST IMPORTANT THING IS TO RUN A GROUND WIRE CONNECTING YOUR TOWER GROUND RODS TO YOUR ELECTRICAL SERVICE GROUND RODS SO THERE IS NO GROUND POTENTIAL DIFFERENCE BETWEEN YOUR EQUIPMENT ON THE TOWER AND THE EQUIPMENT IN THE BOX OR YOU WILL FRY ETHERNET PORTS!!!

[dmcken]

Doing the port swap seems to have stopped the random reboots:

9 <-> 19
10 <-> 20

The devices originally in 19 and 20 are not powered by the netonix (Canopy PTMP 100s, plugged into a last mile gear CTM).

This caused us to look at the other netonix device we have on our network and we are getting 3 oks and 1 short (screenshot attached). This site was brought up with a nextonix from day one (about a month or two ago) and we haven't seen any issues with it so far so wanted to see how it compared (all cables were tested with a fluke before the site went live).

In office testing with a unit whose deployment has been put on hold is showing abnormal terminations on ports 3 & 4 if we use a shielded jack into the netonix. Use a normal jack into the nextonix (continue with shielded into rocket) and it changes to just short on pairs 3 & 4.

We have a trip planned to the site with the reboots with the fluke on Monday to see which of the cables need to be re-run / re-terminated.

[dmcken]

Just realized I forgot to mention the site in the screenshot has a cluster of 7 UBNT Rockets (should be 8) plugged into ports 1-7 and the trunk to the site's router is on port 24.