Periodic high latency....

[jjonsson]

I have just upgraded all switches to v1.4.4 firmware. I don't know if it's a firmware issue. Could also be that the reboot has changed something, so that something is not configured correct.

Result is poor network performance. Looks to me like a loop ? (Loop protections is enabled)


Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=421ms TTL=64
Reply from 10.0.1.93: bytes=32 time=300ms TTL=64
Reply from 10.0.1.93: bytes=32 time=2ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=243ms TTL=64
Reply from 10.0.1.93: bytes=32 time=1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Sometimes is like 1200 ms.....


This is ongoing......

Completely switched network.

Any help is appreciated....

[sirhc]

If you're going to upgrade why not go for v1.4.5rc2 which has nothing new just bug fixes for bugs reported in v1.4.4?

How can we help?

You need to provide a LOT more information.

What version did you upgrade from?

Are you sure this intermittent high ping did not exist prior?

You're going to have to start at the head end and work your way through your network looking for where the latency starts.

You can try disabling features in the Device/COnfiguration Tab to see if any feature is causing it.

Features like:
Discovery
Discovery Tab
SMTP
SNMP
FTP Backup

[jjonsson]

If you're going to upgrade why not go for v1.4.5rc2 which has nothing new just bug fixes for bugs reported in v1.4.4?

Ok, I can upgrade to that version....

How can we help?
You need to provide a LOT more information.

Yes, could you maybe point me in the right direction. I have checked that no loops should be present.

What version did you upgrade from?

v1.3.9

Are you sure this intermittent high ping did not exist prior?

No, I actually think is was there before upgrading. Maybe because of configuration error somewhere...

You're going to have to start at the head end and work your way through your network looking for where the latency starts.

Since it's a complete flat switched network it is everywhere, also at head end and far end...

You can try disabling features in the Device/COnfiguration Tab to see if any feature is causing it.

Features like:
Discovery
Discovery Tab
SMTP
SNMP
FTP Backup

OK, will give that a try....

I was started converting this network to fully routed. But had to rollback as I thought that was what causing the latency spikes...

[sirhc]

No if it it existed with 1.3.9 it is not any of the services I mentioned.

What makes you think this is a switch issue?

You have a large flat network, chances are this is some intermittent traffic.

[jjonsson]

What makes you think this is a switch issue?

You have a large flat network, chances are this is some intermittent traffic.

Because work on the network the last several times has all been around the switches.

Can you explain what you mean by "intermittent traffic" ?

[sirhc]

Because work on the network the last several times has all been around the switches.

Well that could be a coincidence. I have no idea how your network is setup but you want me to tell you what your problem is?

Can you explain what you mean by "intermittent traffic" ?

What you show below is intermittent congestion which could be caused by MANY things.

Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=421ms TTL=64
Reply from 10.0.1.93: bytes=32 time=300ms TTL=64
Reply from 10.0.1.93: bytes=32 time=2ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time=243ms TTL=64
Reply from 10.0.1.93: bytes=32 time=1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Reply from 10.0.1.93: bytes=32 time<1ms TTL=64
Sometimes is like 1200 ms.....

Personally I think you have some sort of traffic occurring such as broadcast storm or maybe BitTorrent traffic.

I would try starting at the front end and shut down further out links and see if the latency spikes stop and if so move further out shutting the next segments down and then when I isolate the segment I think is the problem I would try shutting down AP's one at a time to see if I could find the source.

Disable segments and APs by simply disabling the Ethernet port not turning the POE off that way it is quicker to restore normal traffic.

Another thing you can try is disabling Flow Control on back haul radios. So far radios with KNOWN Flow Control bugs are airFIBER and epmp 2000.

I do not think this is a switch issue.

[jjonsson]

Reboot of all switches and the error is gone....

If this was a loop prevented by the loop protection, can I see that in the log file of the switch ? If yes, what to look for ?

[jjonsson]

Hmm....

No problem is back :-(

Could anyone please suggest something here ?

[jjonsson]

@sirhc....

Disabling FC on AF24 link seems to fix the problem....

[jjonsson]

Further investigation reveals that firmware 1.4.4 and FC enabled on port on switch and in device is no go suddenly......

I will try downgrade to 1.3.9 again.

Looks like UBNT also have an issue with FC. Most devices show huge amount of pause frames on switch ports.