Total failure of WS-8-150-DC on a tower

[cwachs]

I have a brand new WS-8-150-DC. It is running 1.4.7rc11. We configured it to be fed by fiber and in turn feed/POE three Cambium ePMP APs a the top of an FM tower.

We tested everything at the bottom of the tower (125' fiber, 125' DC cable) and powered an ePMP AP at the bottom of the tower to check everything. It worked just fine for the 10 minutes or so we tested.

Installed it at the top of the tower, ran the very same fiber and power cable to it. Grounded the switch and enclosure to a good ground bar at the top of the tower (dedicated ground wire to the base and connected to power ground as well). We powered it up. All worked great. Both fiber ports (7 & 8) worked fine - everything looked good.

I then turned on 48v on a port to power up the very same ePMP AP we tested at the bottom using the very same Ethernet cable (which tested good in the cable test). It powered up the AP and we could log into it. So far, so good. About 2 minutes later, we lost access to the AP. I tried to log into the DC switch. It would not respond. According to the Netonix at the bottom of the tower, we still had a good 1G link to the DC switch over the fiber. Re-powered the DC Netonix - no change. Can't ping it and it does not show up in the MAC list for either fiber interface on the bottom Netonix. Climbed back up the tower and the ePMP AP is still showing power.

Long story short (sort of), we pull the new DC switch down off the tower to bench test it. Can not get it to come alive (other than green power light). We only get an Ethernet link on two of the 6 ports but still can't ping either factory IP or our IP.

I try to "soft" reset it by holding in the reset button for 20 seconds while it is powered up. Nothing happens. I then do a hard reset by holding in the reset while I power it up and wait 20 seconds. This brings the switch back to life in default settings.

Additionally, we discovered that our initial 24v power supply at the bottom of the tower had died over night while powering the switch in the bad state. It is a 24v 13.34 Amp power supply - more than enough to power the switch and 3 APs. We have connected to a second identical power supply for the bench tests. While it was on top of the tower and working, we did verify 24.1 volts of input power so no line loss.

I'm a little nervous to put this back up on the tower until I have some idea of what went wrong and if we can prevent it from happening again. Thoughts? I was very surprised a hard reset brought it back since I figured I was dealing with a hardware problem that I somehow must have caused.

[Julian]

Let's see if I get this right:

Power supply at bottom of tower (24v 330w, seems okay) -> ws-8-150-DC at top of tower -> ePMP. What kind of cable do you have installed?

24.1 @ posts on 8-150-DC, seems good

Did you plug in a console cable while trying to get it to boot in a failed state? I'd be willing to bet it was outputting -something-, would also give my left foot to see what that something was.

-sounds- like corrupted FW, and the fact that a hard reset solved it seems to support that. The bootloader's the piece of software that does the heavy lifting when you keep the button depressed during poweron. While up and booted (even if in a locked-up state) that's a kernel thing, which is on a different partition in flash. If you'd like to know more, there's a sticky post somewhere around here titled 'not all factory defaults are created equally' in which Chris explains in more detail.

Since we've done the modification that hardens the switch to ground current, there have been a few more scattered cases of corrupted firmware, so I'm starting to wonder if the two are linked somehow.. Weird stuff happens when what's supposed to be 0v starts being 3 or more volts, but lacking a crystal ball and/or a telescope powerful enough to see your setup.. I got nothing.

If you'd like me to take a look, just RMA it, and I can run it back through testing for you.

[sirhc]

I would have no idea unless I had the unit on my bench to test it.

I also have no idea what your configurations were, maybe if I could see all your config TABs.

I do know if the DC connection is sparked (connect / disconnect fast ON AND OFF) it can corrupt the flash but not saying that happened.

The fact that you recover it via default would indicate the switch is running but you can not access it.

THis is where a console cable comes in handy and should be a MUST in everyone's tool bag.

What gauge wire are you running up the tower.

Measuring the voltage without load is not accurate and there would be no or little drop unless load is applied.

[cwachs]

Cable running up the tower for power is 12 gauge stranded cable. Ethernet cable from the switch to the radio is Ubiquiti Carrier double shielded. Ethernet is about 3 feet long. 12 gauge power is 125' long.

True on the load/voltage. I get that. However, at the bottom of the tower, we powered the ePMP AP off the DC switch with 125' of the 12 gauge power cable in between - the exact scenario at the top. Power supply was showing ~80% efficiency with the AP powered.

Luckily, you don't have to loose your left foot since I did not have a console cable plugged in. Time to make one.

You recommend sending it back for your testing or re-climb the tower???

[cwachs]

Here are screen shots of relevant tabs. The last Device -> Status shot is with the switch powering an ePMP AP on port 3 - the same scenario I had at the top of the tower. The only difference in the bench test is there is no fiber feeding the switch.

Here is the same Status -> Power with 100' of 12-2 gauge power cable in place on the bench - still powering one ePMP AP:

[sirhc]

So you have the management VLAN with all 'T''s so when you brought it down off the tower you could not talk to it by simply plugging into the switch and access the UI as it would only respond to packets with VLAN ID 5.

I mean you could have gone into your laptop and setup a VLAN on the interface but if you had not done this then you could not log into it which makes sense that a factory default allowed to then see it.

I am not sure your issue was not a network configuration issue?

[cwachs]

Myt laptop is set up for tagged VLAN5. That is how we communicate with all switches and radios on the network. Everything is tagged 5 on the mgt interface. So, I know the laptop is setup correctly. However, I did turn off VLAN5 tagging and tried to access it that way (as well as 192.168.1.20 untagged).

It was not a VLAN issue that prevented me from getting at it.

It's been running on the bench for 2 hours powering one ePMP AP with 100' of power cable feeding the DC switch. So far, it's stable. But I hate to climb back up the windy and cold 100' tower, drop the mounting screws three times and try to install it without knowing why it failed...

[sirhc]

I have no idea either.

Do you have another one to put up there and put this one somewhere else?

I have at least 2 of everything as spares, just in case. And with AP radios I have enough for 2 towers.

[cwachs]

No, this is the only DC switch at the moment. We finally gave up battling FM interference on the tower and are ready to migrate 3 APs to this switch to get away from the FM noise on the Ethernet.

I can delay this project for a bit and send you the switch if you think that would be at all helpful. I find it very odd I have a DC power supply that went bad during this (caused by it??) and the switch seems to be totally fine after a hard reset...

[sirhc]

No, this is the only DC switch at the moment. We finally gave up battling FM interference on the tower and are ready to migrate 3 APs to this switch to get away from the FM noise on the Ethernet.

I can delay this project for a bit and send you the switch if you think that would be at all helpful. I find it very odd I have a DC power supply that went bad during this (caused by it??) and the switch seems to be totally fine after a hard reset...

If it is bench testing fine then I would say try it.