Copper Port Resets on WS-12-250A, 48VH

[qlex]

Hi, this past weekend we had one of our WS-12-250A units go through an odd failure and I believe it to be a hardware issue possibly stemming from too much power draw. The configuration had Port 12, non-powered, acting as the uplink to the core network while ports 2, 3, and 4 were all powering Dragonwave radios. Firmware was v1.3.3.

Ports 2, 3, 4 were set to 48VH, with ports 2 and 3 drawing 45-49w each. The radio on Port 4 had its Transmitter off, and thus was only drawing 28-31w. The average draw total looked to be about 135-140w. It had been running this way for several weeks, but the failures started suddenly and got worse over the course of about 8 hours.

What alerted us to this was that we noticed the ports facing the radios were dropping. The WS was not rebooting, and the radios were not losing power, either. It was just the data link that was being lost (i.e. ping drops, and port drop -- radios reported they were losing connectivity on their copper ports facing the Netonix). It was happening to all three radios at the same time. Over the course of about eight hours, the issue got progressively worse, to where the port flaps were happening about every hour. I applied v1.3.8 but this did not help (or make things any worse).

Now, across our relatively modest network, no other Netonix we have draws this much power, but our ~140w draw here is still well under 250w or even the "80% rule" (200w). But maybe we were drawing too much power on this switch by using three of the 48VH ports? As for a fix, we Restored the config to a new WS-12-250A (running v1.3.8) and I turned the power off to the Dragonwave that had its Transmitter off, too. Average draw about 105w now, no issues, but we didn't have any issues on the previous switch either for several weeks and that was with an additional 35w load.

Would appreciate any feedback on this; thank you.

[sirhc]

First PLEASE upgrade firmware. We do no support older firmware as it becomes a nightmare as there are so many changes.

What Temperature is the switch environment?

Under Device/Configuration what is the Board Rev?

If is older than RevD maybe you could switch/swap this unit with another unit that is RevD.

The biggest change between RevC and RevD was higher rated PolyFuses.

[qlex]

Thanks Chris. I did switch to v1.3.8, but only after the problem began as a shot in the dark to see if would help. Switch environment is comfortable, room temperature or better, it's in a 'finished' room with air conditioning (several other racks of equipment there). I'll check the Board Revision, might be Friday before I can get that answered, though. I can confirm that the new WS-12-250A that we put in this weekend is Revision D, as is the WS-24-400A we just installed, too.

Will report back when I have the old unit's Revision.
Thanks.

[qlex]

Was actually able to just confirm the old WS-12-250A is a Revision D board.
Thanks.

[sirhc]

Let me know how the new switch works in that location.

If this occurs next time go look at the switch log.

THe only other thing I can think of is Loop Protection which can give false loops and disable Ethernet ports for 90 seconds.

It will be in the log and you can disable Loop Protection on the Device/Configuration Tab, it is a check box

[qlex]

Actually one (very) important detail I forgot to mention -- I could verify that the timing and reason for the ports flapping was due to the Netonix OS or management interface restarting itself. Similar to when you do a firmware update, pings to the Netonix and the radios its powering drop out for 8-10 pings, but power to the radios does not drop (which is great, by the way). This continued after moving to v1.3.8.

Here are some logs that I pulled that night, after its first reboot (note uptime of 3h37m, before long it was doing this every 30-45 minutes):

Uptime 03:37:20

Dec 31 19:00:19 kernel: link state changed to 'down' on port 2
Dec 31 19:00:19 kernel: link state changed to 'down' on port 3
Dec 31 19:00:19 kernel: link state changed to 'down' on port 4
Dec 31 19:00:19 kernel: link state changed to 'down' on port 12
Dec 31 19:00:20 kernel: eth0: no IPv6 routers present
Dec 31 19:00:20 kernel: eth0.1511: no IPv6 routers present
Dec 31 19:00:22 kernel: link state changed to 'up' on port 3
Dec 31 19:00:22 kernel: link state changed to 'up' on port 2
Dec 31 19:00:23 kernel: link state changed to 'up' on port 4
Dec 31 19:00:23 pinger: Ignoring port bounces because date is wrong
Dec 31 19:00:23 kernel: link state changed to 'up' on port 12

Is there an RMA process I should go through or anymore testing I can do at the bench now that this is out of production?

No rush on a reply, and thank you for taking the time to answer these questions directly. Also, have a great Thanksgiving! :)

[qlex]

Update on this as well -- the WS-24-400A with just the one Dragonwave radio removed from the 250 (see above), powered at 48VH, is having similar issues to the original WS-12-250A which is quite interesting. This makes me think that the issues I described above are somehow being caused by this one Dragonwave radio or something about its installation (damaged connector perhaps?). So the root problem lies with it, it's causing the Netonix to drop port and reset its OS and/or reboot altogether.

I'm still gathering more info and we hope to climb to the radio late this week or for sure first of next to do more troubleshooting.