We had a WS-12-250-DC last night that stopped passing all traffic. It continued to power radios that were connected to it, but the switch was not reachable via it's management interface, nor were any any radios reachable that were directly connected to it. The only way to fix this was to reboot the switch.
The switch is running 1.3.9 final. I'm not finding any logs from before the reboot. /var/log/messages only contains logs AFTER the reboot. How can I determine what caused this? This resulted in a tower climb in snowy weather.
Thanks.
WS-12-250-DC Stopped passing traffic
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC Stopped passing traffic
Why would it result in a tower climb to re-boot the switch, even if you mounted it up on a tower the power source I would assume is on the ground?
Of course the log is cleared when you reboot the device as does most any device log.
The time to review the log was before you reboot it via a serial console cable if need be.
Maybe setup a syslog server so the log is captured to a server that you can review but if the issue was from something sudden there may not be an entry as to the last thing that happened but maybe there would be some things in the log to hint to what is about to happen?
Since there is next to no information as to your setup all I could do is read the tea leaf patterns in the bottom of a salad shooter to determine the cause?
Maybe post up all your configuration tabs and describe the installation in detail. I would also setup a syslog server as you never know, it could point to something? Detail out what happened and what you did to try and access the switch before you rebooted it.
Of course the log is cleared when you reboot the device as does most any device log.
The time to review the log was before you reboot it via a serial console cable if need be.
Maybe setup a syslog server so the log is captured to a server that you can review but if the issue was from something sudden there may not be an entry as to the last thing that happened but maybe there would be some things in the log to hint to what is about to happen?
Since there is next to no information as to your setup all I could do is read the tea leaf patterns in the bottom of a salad shooter to determine the cause?
Maybe post up all your configuration tabs and describe the installation in detail. I would also setup a syslog server as you never know, it could point to something? Detail out what happened and what you did to try and access the switch before you rebooted it.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
jbaird - Member
- Posts: 43
- Joined: Fri Oct 09, 2015 3:47 pm
- Location: Danville, KY
- Has thanked: 0 time
- Been thanked: 2 times
Re: WS-12-250-DC Stopped passing traffic
Because the switch is on a grain leg, and we get AC power from the top of the leg.
I'll setup a syslog server, and have someone with a laptop and serial cable if it happens again. I would expect to see historical logs in /var/log. Why do they have to be deleted on reboot? If a daemon on the switch crashed, could there be a coredump somewhere that we could look at?
I tried to access the switch via it's management interface. This is the only way to access the switch remotely. I also tried to access radios that were directly connected to it. The guy who climbed the tower did so in rain and snow, so all he did was connect to an interface and try to access the management interface. He had an ethernet link, but could not reach the switch so he rebooted it. Has an issue like this been reported previously?
There is nothing complex about this config. A few radios, a few VLANs.
I'll setup a syslog server, and have someone with a laptop and serial cable if it happens again. I would expect to see historical logs in /var/log. Why do they have to be deleted on reboot? If a daemon on the switch crashed, could there be a coredump somewhere that we could look at?
I tried to access the switch via it's management interface. This is the only way to access the switch remotely. I also tried to access radios that were directly connected to it. The guy who climbed the tower did so in rain and snow, so all he did was connect to an interface and try to access the management interface. He had an ethernet link, but could not reach the switch so he rebooted it. Has an issue like this been reported previously?
There is nothing complex about this config. A few radios, a few VLANs.
- Attachments
-
-
-
-
-
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC Stopped passing traffic
Please add the Power and STP Tab to your post.
Is there batteries involved or simply a DC power supply?
Is there batteries involved or simply a DC power supply?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC Stopped passing traffic
If there are no batteries it could have been a power blip where the power is cut to the switch then restored real fast which can cause issues as the board is in the process of de-powering but not all circuits are fully discharged then power is restored which does not force a reboot but instead locks it up.
Here is a post where I had a similar issue last year, in fact twice at this same tower as it has faulty electric and my guess is the power flickered on and off very fast and the Tripp Lite Inverter charger was not fast enough switching to battery and back again fast enough.
viewtopic.php?f=17&t=733&p=5427&hilit=bareville#p5427
But if you do not have batteries why not simply use an AC switch?
Here is a post where I had a similar issue last year, in fact twice at this same tower as it has faulty electric and my guess is the power flickered on and off very fast and the Tripp Lite Inverter charger was not fast enough switching to battery and back again fast enough.
viewtopic.php?f=17&t=733&p=5427&hilit=bareville#p5427
But if you do not have batteries why not simply use an AC switch?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
jbaird - Member
- Posts: 43
- Joined: Fri Oct 09, 2015 3:47 pm
- Location: Danville, KY
- Has thanked: 0 time
- Been thanked: 2 times
Re: WS-12-250-DC Stopped passing traffic
Yes, I have batteries (Traco TSP PSU, Traco BCM battery charge controller). This is a 48V site, and it never lost AC from what we can tell, but I guess it could have been a very quick blip.
- Attachments
-
-
-
sirhc - Employee
- Posts: 7416
- Joined: Tue Apr 08, 2014 3:48 pm
- Location: Lancaster, PA
- Has thanked: 1608 times
- Been thanked: 1325 times
Re: WS-12-250-DC Stopped passing traffic
Your config looks fine.
It could have been a power blip but the batteries should have buffered that so lets move that down the list to not very probable.
It could also be a rare event that you do not see again?
I would setup syslog server and also if it happens again and weather permits try and console into switch rebooting.
Maybe try using RSTP instead of Loop Protection if it happens again as Loop Protection can be triggered by a any type of loop or leaky VLANs even at a customer location.
Could also be a bad unit which if it continues to happen and you can not figure it out you swap the unit out and see of it stops.
It could have been a power blip but the batteries should have buffered that so lets move that down the list to not very probable.
It could also be a rare event that you do not see again?
I would setup syslog server and also if it happens again and weather permits try and console into switch rebooting.
Maybe try using RSTP instead of Loop Protection if it happens again as Loop Protection can be triggered by a any type of loop or leaky VLANs even at a customer location.
Could also be a bad unit which if it continues to happen and you can not figure it out you swap the unit out and see of it stops.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.
-
jbaird - Member
- Posts: 43
- Joined: Fri Oct 09, 2015 3:47 pm
- Location: Danville, KY
- Has thanked: 0 time
- Been thanked: 2 times
Re: WS-12-250-DC Stopped passing traffic
Ok - syslog configured. If it happens again, we'll try to get someone up there with a serial console.
8 posts
Page 1 of 1
Who is online
Users browsing this forum: No registered users and 32 guests