Dropping ports on new WS, what is wrong with my setup?

[sirhc]

And it just happened again. That was on 1.3.8, rolling it back to 1.4.0rc12.

Any thoughts? Anything I can do? This is happening pretty frequently, so if you guys want access to troubleshoot this, I can provide it.

It did look like the CPU was running at 98% the last two times this was happening. I didn't catch what process was taxing it though. I will try and catch it the next time.

Well try and eliminate things by turning them off on the device/configuration tab

Turn off and or stop using all un-essential services such as SNMP, Discovery (all), IGMP

Some people have reported issues trying to poll the switch to fast with SNMP which Eric made a post about that recently.

[nickwhite]

Well try and eliminate things by turning them off on the device/configuration tab

Turn off and or stop using all un-essential services such as SNMP, Discovery (all), IGMP

Some people have reported issues trying to poll the switch to fast with SNMP which Eric made a post about that recently.

Thanks, went through all that already. The only things enabled are SSH, HTTPS, Remote Syslog, SMTP config, Loop Protection, NTP.

[Eric Stern]

Try turning off storm control (in a previous post you mentioned it was on).

[adairw]

We just crashed again... 1.4.0rc12 on all switches, only services that were enabled were syslog (on a few) NTP and SMTP, which are now off.
All discovery, igmp, lag and stp were an are off.. I guess tomorrow we will go throw some more hardware at it..
why sometimes it runs longers than others is beyond me.. some nights it will lock up four times others it wont at all..

its like some certain amount of traffic triggers something. is there not any deeper logging we could turn on or watch?

[Dave]

did you have storm control on?

[adairw]

No sir. No services turned on except, ssh, ssl

[sirhc]

We just crashed again... 1.4.0rc12 on all switches, only services that were enabled were syslog (on a few) NTP and SMTP, which are now off.
All discovery, igmp, lag and stp were an are off.. I guess tomorrow we will go throw some more hardware at it..
why sometimes it runs longers than others is beyond me.. some nights it will lock up four times others it wont at all..

its like some certain amount of traffic triggers something. is there not any deeper logging we could turn on or watch?

All your switches are locking up or are you saying you just have 1.4.0rc12 on all of them and it is just this one problem switch/location still giving you grief?

[adairw]

If you refer to the diagram I posted showing the switch layout the only two switches that lock up are at Dumas and Morton.

We are trying to figure out how to setup remote access to these switches since they lock up and a netbooter has to power cycle them to keep them running.

Dumas and Morton switches are basically rebooting at the same time since the netbooters are using the same settings. I think I will adjust Morton for a longer delay time so I can see if just Dumas rebooting on its own will bring it back up. I might regret this as there is already about 10 minutes of downtime ever time this happens..

All switches in this chain are running 1.4.0rc12

[sirhc]

Yea, this is the part that stumps me:

1) You have many switches that have no issues at all and work as desired all over your network.

2) All of your towers are setup essentially the same way so why this chain (2 towers) has issues?

3) I "think" you swapped out the switches causing the issues to insure it is not just a bad switch (it does happens)

Will be interesting to see what your testing with rebooting 1 switch as maybe it is just 1 bad switch and it is only one of the two causing problems for both but as I said in 3 above I "think" you tried swapping out the units having the problems?

Which if all your towers are setup the same and it is not a bad switch then something is causing the traffic that is causing the switches to lock up, but what is it?

Is it a bad radio, a bad router, a mis-behaving customer, or the the moons of Jupiter aligned wrong?

I really really really wish we knew what it was and if it is us we could fix it for you Adair.

There will be a new v1.4.0rcX released "maybe" tomorrow but since we do not know what the problem is it is just a poke in the dark that it will help.

[nickwhite]

So no lockups on our end overnight. Last one was yesterday around 5pm or so. EDIT: Apparently I missed it while reviewing, but there was a lockup around 2:40AM. We threw a DLI Web Power Switch on to power cycle if it locks.

I was going through and realized I had this switch being monitored in LibreNMS. Here are some interesting graphs:


Memory over last several months:





Memory last 48 hours (I shut off SNMP yesterday, so it didn't catch the last 2 lockups):



CPU the last 48 hours: