Issue with epmp 2000

[jshoosh]

This tower was working fine then we replaced (2) epmp 1000 radios with (2) epmp 2000 radios and every night the switch becomes inaccessible.

A reboot of the switch and tower brings things back up.

[sirhc]

Jeff, I just talked to you on the phone.

I was Team Viewed in and the switch was performing as expected?

We sat there and watched the switch for 20 minutes and all was fine?

So what has changed now?

Now I did discuss your grounding was in need of correction has this been done?

[sirhc]

OK so I called Jeff.

What he has is a Triplite Inverter Charger 1200 watt UPS with external batteries.

He had it plugged into the receptacle that pulls AC power from an adjoining warehouse which has it's electrical service grounds a great distance away.

I had suggested that Jeff isolate the ground in his box to only use the tower ground system as described in this post: viewtopic.php?f=17&t=1786&start=30#p13447

They were in the process of doing this when this occurred.

They unplugged the Inverter Charger from the AC socket and this occurred.

My guess is the switch lost power and as there was a ground issue as they are in the process of separating the grounding and the Triplite probably did not like lose access to earth ground when he unplugged it.

I instructed him to finish his grounding changes and then I would reboot the switch and see what happens after that.

[jshoosh]

OK so I called Jeff.

What he has is a Triplite Inverter Charger 1200 watt UPS with external batteries.

He had it plugged into the receptacle that pulls AC power from an adjoining warehouse which has it's electrical service grounds a great distance away.

I had suggested that Jeff isolate the ground in his box to only use the tower ground system as described in this post: viewtopic.php?f=17&t=1786&start=30#p13447

They were in the process of doing this when this occurred.

They unplugged the Inverter Charger from the AC socket and this occurred.

My guess is the switch lost power and as there was a ground issue as they are in the process of separating the grounding and the Triplite probably did not like lose access to earth ground when he unplugged it.

I instructed him to finish his grounding changes and then I would reboot the switch and see what happens after that.

Chris,
We did isolate the ground as described, so far for past 3 hours has been working fine. We also bypass the surge suppressor per your suggestion. Since everything went crazy 2 days ago, after we installed 2 epmp2k, on ports 3 & 4, I am wondering what could have caused this phenomenon!

[sirhc]

We have narrowed this down to the (2) new epmp 2000 radios.

It happened again today and I was able to recover the switch for Jeff by bouncing the port to one of the epmp 2000 radios and the switch woke up.

Looks very similar but not the same as the Pause Frame Storm issue with AFX radios but slightly different????

I disabled Flow Control on the epmp 2000 ports and we will see if this eliminates the problem but then we need to find out why and what to do.

The normal Pause Frame Storm protection did not catch this as the Pause Frames were in the opposite direction which is weird??? Well let me explain, the Pause Frame Storm Protection looks for a ton of unsolicited Rx Pause Frames on a port then disables Flow Control on that port. But in this case I could not find a ton of Rx Pause Frames on any ports but did see tons of Tx Pause Frames on all the ports with Flow Control "except" the port that bouncing resolved the issue which was port 4 which was an epmp 2000 radio. It did have some Rx Pause Frames but not nearly as many Tx Pause Frames as seen on the other Flow Control ports?

This tower had been working fine for a long time and only started this behavior after he swapped out 2 epmp 1000 radios for 2 epmp 2000 radios.

For now I would suggest anyone playing with epmp 2000 radios to look for this and if they experience it to disable flow control on the port facing the epmp 2000 radios.

Not saying this is a Cambium issues or what exactly it is, I am confused at the moment so I am just alerting people to something I am looking at and how to avoid it

[wNieds]

Has issue been set in stone? We have had multiple instances of outages with ePMP 2000's and Netonix switches. With gear properly grounde dat the bottom of the tower, we are at a loss with the issue. Should we just shut off flow control?

[sirhc]

There has been a couple reports of Flow Control issues with the ePMP 2000. Now there was also a MT router at that locations as well.

Yes I would try disabling Flow Control on ports facing ePMP 2000 radio and see what happens.

Also make sure your firmware is up to date, we are on v1.4.7rc10, not sure what version the Cambium is.

You need to narrow it down.
When you have an issue always upgrade all firmware
You can disable all Flow Control

Now if problem goes away slowly enable FC on ports over time examining the ports stats and narrow it down.

Sometimes in IT you need to be a detective and figure it out through trial and error.

[gilljr]

I just found this post and I was having issues with my epmp2000 radios. I looked and the sites that randomly have issues all have Flow Control enabled on the switch ports. I have now disabled flow control on those ports and hope it solves the issue. I was thrown off because I did not distiguish between epmp1000 and epmp2000.

Switch: WS-12-250-DC (version 1.5.0)
Power: NewMar DIN-UPS 48-10

[mike99]

Cambium radios handle way better their queue than ubnt. You don't need flow control with those radios. Flow control should be a last resort option for radio with scrappy scheduler.