airFiber24 Ethernet Errors

[jedler]

We recently ran into a situation with a WISP Switch 250B, we had Port 1 setup as 48VH to power our airFiber and recently started getting errors on the port. We use Wireless Beehive GiG-E grounding, we bypassed that and still saw the same issue. We then tried Port 2 on the switch with the same issue. In turn we replaced the airFiber24 unit and cabling, and we are still running into the same issue. The switch is running 1.2.3 Firmware, and as best practice when plugging the unit in to power it up, we plugged the cable in, logged into the switch and powered the port on. We also tried to downgrade firmware and at that time upgrade to a later version, no change. We are now using the same cable and unit, just no power running to a Mikrotik CCR1016 with no errors reported. Has anyone else ran into this issue?

[sirhc]

What type of Errors, CRC?

The fact that you tried a different port and the errors went with it tells us it is not the port in the switch. However it is "possible" I suppose that the Ethernet Transformer is bad and since Port 1 and Port 2 use the same dual Channel Ethernet Transformer it is a "possibility".

The firmware versions will have no effect on port errors so long as not older than v1.1.8 . The only reason older firmware than v1.1.8 has "some possible" effect on this was we were not properly shutting the PHY down prior to soft rebooting the switch which could cause some packet corruption as the PHY could possibly be active during the port initialization process but that was corrected in v1.1.8. But other that that the firmware version has no effect on port errors as port communications is handled internally by the switch core code.

So the AF24 comes in on port 1, how does the AF traffic get to the router from there, are you going through an SFP port to the router?

Can you post up your VLAN and Ports Tabs and explain what type of Errors we are dealing with.

I know you guys have lots of WISP Switches so is this issue happening on any other WISP Switch at other sites?

I do get some small amount of Rx CRC errors on a couple of my heavily loaded AF24 link ports during primer time at night when traffic exceed 500 Mbps of the airFIBER link. The fact that it is a Rx CRC Error we are assuming the fault lies with the AF?

Now I use the WISP Switch as a MID-SPAN injector for my AF24 back haul links where I create a VLAN with just Port 1 and Port 25 (SFP) using "U" on both ports and "E" on all others. On one tower I had Rx Errors on Port 1 but when I replaced the SFP module in Port 25 the Errors stopped on Port 1? - WEIRD but thought I would share as it may possibly be relative to your issue depending on your setup?

[jedler]

Appreciate the reply,
I also failed to mention in the first post that we also tried a brand new switch and reproduced the same issues. We originally had a 250B installed and went to a 250A with the same setup, they were Rx CRC errors that will stack significantly per couple seconds. The AF24 came into Port 1 (being powered by the switch), then Port 2 goes to Ether1 of our Mikrotik CCR1016 via ethernet. We were using an Untag VLAN on Port 1 and Port 2 with all other VLAN Excluded. We do not have the Netonix powering any other airFibers in the field due to the issues with this one right now, otherwise the switches are awesome.
Thanks!

[lligetfa]

Did you check all your grounding and bonding? I've seen ground loops generate CRC errors.

[sirhc]

I know this is not what you want to hear but it has to be something going on with your wiring, grounding, or something.

Since you swapped out the port and the switch and the problem persists it can not be the switch.

Since you swapped out the AF24 it can not be that but I have had THREE out of twentyfour AF24 units with bad Ethernet ports that I had to either RMA or even chuck as they were out of warranty.

Is your tower grounding system bonded to your Electric service ground rods with a HEAVY #2 ground wire?

As Les said ground potential differences can cause issues with Ethernet communications. <== DO NOT DISCOUNT THIS.
So many people ignore ground potential differences and say ground is ground and this is simply NOT true.

Another thing you can do is SSH into the AF and check the Errors on that side, the linux command on the AF24 radio to view Errors is "emacstat"
I know this is a 24 port but it is the "exact" same design as the 12 port.
As you can see below is a switch that has been up for 50 days and only had 2 CRC errors on my AF24 link. - I JUST TOOK THIS SCREEN SHOT

CLICK IMAGE BELOW TO VIEW FULL SIZE

AF Errors.jpg (338.77 KiB) Viewed 39841 times

[sirhc]

Another thing we have found is we were using a 4 port EHWIC switch card in our Cisco 2951 router and VLANing the EHWIC ports off to make them routable ports in the router and we found that when we did this and fed the AF24 with one of these ports that the switch port feeding the AF24 would get CRC errors during heavy load. This might be because the router was mangling packets with all the overhead or because the EHWIC had to handle each packet 4 times when passing packets in one AF24 link and out the next to the next tower.

WHen we replaced the EHWIC with single ROUTABLE port EHWICs to feed the WISP Switch port that was bonded to the port that powered the AF24 that CRC errors were MUCH MUCH less on the port feeding the AF24 almost NON Existent as seen in screen grab above.

I know you are going to say I am crazy but SERIOUSLY this example below is what we found. Do not ask me why, I did not care it just stopped the errors.

Trust me I just spent 45 minutes in MS Paint to show you this so if I did not think it "might" so how relate to your issues I would not bother but it could but I would start with checking the wiring and grounding then look at this.

CLICK IMAGE BELOW TO VIEW FULL SIZE

AF Errors2.jpg (163.2 KiB) Viewed 39838 times

[sirhc]

So I guess what I am saying in the post above is try another port in the router, also check/replace the cable between the router and the switch.

You might be surprised, it made no sense (above example) but it is the truth.

Another thing to try is make sure you have Flow Control on all the ports on the switch and make sure the router has Flow Control turned on and that the airFIBER also has FlowControl turned on.

[jedler]

Apologies for the delayed response. Appreciate all of your input, the next one we deploy powering an airFiber we will take all of your input in mind.
Thanks Again!

[sirhc]

So we have an AF24 link that looks like this

[VRTOWER]CISCO2951-----WS24400A-----AF24 ~~~~ 4 miles ~~~AF24-----WS24400A-----CISCO2951[BBTOWERS]

Now the WS-24-400A on the BB Tower was getting TONS of errors on the port connected to the AF24 and LOW throughput with packet loss so this is what we tried and what we found to be the problem.

1) Replaced wiring between WS-24-400A and AF24 on BB Tower= ERRORS PERSISTED
2) Replaces AF24 on BB Tower = ERRORS PERSISTED
3) Replaced WS-24-400A at BB Tower = ERRORS PERSISTED
4) Replaced EHWIC in Cisco 2951 at BB Tower going to AF24 = ERRORS PERSISTED
5) Replaced EHWIC in Cisco 2951 at VR Tower going to AF24 = ERRORS PERSISTED
6) Replaced WS-24-400A at VR Tower = ERRORS PERSISTED
7) Replaced wiring between WS-24-400A and AF24 on VR Tower= ERRORS PERSISTED
8) Replaced AF24 on VR Tower = ERRORS GONE THROUGHPUT RESTORED

This just goes to show you that Ethernet Errors can come from other ports and other devices but manifest themselves anywhere. My guess is the packets were getting mangled in the wireless transmission or between the ethernet port of the AF and the FPGA radio and showing up on the switch Ethernet port.

We have also seen Errors how up on a port connected to an AF but the problem was from the SFP uplink port from the router, replaced the SFP module errors went away.

We have also see errors created on the AF24 Ethernet port and they show up on the switch or on the switch on the other side of the link.

To check the stats on the AF24 radios you need to telnet into them and run this command on the CLI "emacstat"