Long Pings and Packet Loss

[BaldwinCo]

Background: running 1.4.3rc7. The symptom is that pings from, to and across the switch have occasional 100ms+ times and packet loss (e.g. 3 failures in 50 pings) when typical time would be less than 4 ms. The problem seems to be independent of enabling/disabling flow control, VLANS and individual ports. All cable diagnostics are good (can’t test some radios). The problem is the same for 1.4.2. The amount of packet loss and the length of the delay seems to be directly proportional to total traffic.

May be related: Three days ago, port 9 started tripping loop protection. This was during and after a storm. It mostly went away the next day. We replaced the Rocket on Port 9 and moved the connection to port 10. That problem has not shown up since the change. Ping tests before this event were fine - no delays and 0% packet loss.

Testing: In addition to mentioned above, I disabled ports while continuing the ping tests. The failures lessened with fewer ports enabled but never disappeared.

Ports 1, 3 and 5 are AF5X. Ports 13, 15 are Rocket AC. Ports 7, 8, and 11 are Rocket Ti. Port 10 is a Rocket M.

The switch has been in use about 3 months.

[sirhc]

Now you had said in the email to me that this started after a storm?

Was this current configuration working without issue before storm or was changes made after?

Can you post up Port Stats on any ports with Errors.

Also maybe a network diagram so I know what is on what port and going where.

[BaldwinCo]

The original post describes what I changed after the storm. Everything was working without issue before storm.

[BaldwinCo]

Port Details for Port 3 and Port 1 attached. Port 1 is the one with pause frames. In terms of network diagram, port 1 is backhaul, the rest are AP's to customers. On the other end of the backhaul is another Netonix ws-24.

[sirhc]

You do know to diagnose this you're going to have to disturb your network!

Normally I do this on my network after midnight.

You're getting a lot of Drops on those ports?

I would start at the front end with my laptop and work my way out looking for where the issue begins, or you could start here and work backwards.

Unplug the radio (your backhaul) on port 1 and plug into your laptop using a POE brick and do speed tests and watch a constant ping.
Is you have no issues then you know your issue is HERE, at this location.

Also if ports 1,3,5 are AFX radios you should check and see if they are the newer models that can take 48VH

Read this post to find out: viewtopic.php?f=6&t=1215#p9040

If these AF5X radios will take 48VH you should use 48VH but if you have no 48VH port available then use 48V over 24V

So this is one big flat Layer 2 segment?

Port 3 and 5 are also AF5X radios that go where?

[BaldwinCo]

Thanks for the related advice on AF5X radios. Note that two of the AF5X's use GPS Synch.

I did disrupt network for about two hours last night doing testing. With all ports off but the backhaul, the problem persists on that port. But I see the same problem if I do a ping across the switch from Rocket to Rocket. If I ping from the backhaul out to 8.8.8., no problem.

It is flat, note that I have an ER-PRO plugged into the switch and will be moving the whole system to a routed solution - as soon as I fix this problem.

[sirhc]

Well if you started seeing this issues after a storm and you say you are seeing issues pinging across the switch and you had to replace a radio that was on port 9 and move to another port maybe I would try swapping out the switch and see what that does?

Then I would take this switch back to your office and default it and test each port 1 by 1 using 2 laptops to copy large amounts of data across the ports from laptop to laptop and make sure each port is OK.

You do have a spare switch right? Or break out a good NON POE and your POE bricks.

Also you know I am going to take this opportunity to tell you and all the viewers our there to read up on our grounding advice!!!


Here are some good posts on grounding:
viewtopic.php?f=30&t=1816
viewtopic.php?f=30&t=188
viewtopic.php?f=30&t=1429
viewtopic.php?f=17&t=1786&start=30#p13447
https://community.ubnt.com/t5/airFiber/ ... rue#M31070
But sometimes even with the best grounding you will still take damage but I have no idea what your grounding is at this site.

Just so you know a poor ground system can cause Ethernet devices tons f grief and cause errors and packet loss which is talked about in the posts above.

[BaldwinCo]

I have read your grounding advice and changed my practices accordingly. Note that this location has a ground web of about 10,000 sq ft and I am tied into that web at the antennas and at the switch. It is an old ATT Microwave tower and shelter.

Yep, keep a stock of redundant equipment on hand and will replace. Thanks for looking into this problem.

[sirhc]

Thanks for looking into this problem.

OH I do not think we are done, this is just the next step, could be the answer could not be.

[BaldwinCo]

Chris,

With a brand new out of the box WS-24, loaded firmware and duplicate configuration. Connected a AFX and a Rocket M on the bench and did the same ping test across the switch - from AF5X to Rocket. And it failed. So, I have replicated the problem on the bench. No storm involved. Any ideas?

Thank You