High CPU usage

[colinhowlin]

Getting high CPU usage on our DC switches.
Were all running 1.4.5RC2 so upgraded to 1.4.5 final but still seeing CPU at 90%+.
Running top gives multiple dropbear processes - ssh access is quite slow.
Only seems to be affecting our DC switches.
Have AC switches with similar configurations and CPU usage around 50-60%.

Any more info needed let me know.

[sirhc]

Can you debug the situation with Standard debugging practice is to determine what service (Daemon) is using the CPU?

So disable all nonessential service on the Device/Configuration Tab such as:
SMTP - Email alerts
SNMP
TFTP backup
Discovery protocols - UDP, CDP, LLDP
Discovery Tab
SSH - Use web UI during this time

After disabling all that and anything else you can you may need to reboot the switch then if the CPU load is normal then enable 1 at a time until you find the cause.

[colinhowlin]

It seems to be SNMP causing the high CPU.
Disabled all suggested services on one switch and CPU dropped off to around 50%.
Re-enabled one by one and no dramatic increase in CPU so could not determine.

On another switch I disabled one by one and monitored CPU usage.
Seems to be SSH causing the issue which I had suspected due to the multiple Dropbear daemons in top output.

We can live without SSH for the most part so I've disabled this on all our DC switches for now.
However, does the monitoring server use SSH for scheduled upgrades?

[sirhc]

yes SSH is used for manager

My guess is someone is attempting to hack your switch via SSH as repeated SSH login attempts will cause CPU to spike.

Implement the Access control list so only your management IPs can talk to the switch but if your switch is on a NATed IP address this would have to be done on the router providing the NAT

Hopefully your switch is not accessible from the WWW???

Also I have seen people set their SNMP server to query the switch every second which is also going to cause this, do not query the switch with SNMP more than every 10 seconds or preferably every minute.

The CPU in the switch that runs the UI/CLI and Daemons like SNMP is not a powerhouse but a simple MIPS 24K CPU running at 416MHz

[colinhowlin]

Switches are not accessible outside our core network so hacking attempts are not a possibility.
All on private IP's, NATted, and with ACL's on all our core equipment.

We have left SNMP running on the switches and they are polled every 15 seconds.
Definitely seems to be SSH causing this issue.

After re-enabling SSH, CPU is staying stable around 50%.
I'll leave the SSH service running and monitor CPU usage and restart service is usage spikes to confirm cause.

[sirhc]

If your switches are accessible within your network to any customer a device somewhere on your net may have a virus/worm that is looking for other devices to infect and found the switch in a scan. This is how self replication works.

Personally we block access to ALL our WISP infrastructure equipment so only our management IPs can access them for SSH, SNMP, HTTPS...everything

Should should be able to determine where the SSH login attempts are coming from either from the switch itself at Linux level or the router in front of the switch.

I would track it down.

If you do not put your switch on a NATed address but rather a routed invalid address you can use the switches Access Control List but not if the switch is accessed via a NATed address, in that case you need to define an Access List on the router handling the NAT.

[colinhowlin]

Switches are not accessible to customer devices.

I'm not convinced these are SSH login attempts.
However, I will trace it on the switches.
I would normally do this by checking the auth.log file in /var/log but this doesn't seem to be there.
find / auth.log doesn't return any results.

Where are SSH login attempts logged?

[Eric Stern]

Incorrect SSH logins (ie invalid username or password) are logged in the log (/var/log/messages), valid logins are not logged.

[colinhowlin]

As expected, no incorrect SSH logins logged to /var/log/messages.
This is not being caused by attempted logins by bots/other.

I've just noticed the SSH service is not running on the switches after being re-enabled.
I will have to schedule a reboot for later tonight to enable again.
I've set up a monitor through SNMP to report when CPU usage hits 85%+ so I'll know as soon as it does and can take a closer look at the switches.

Again, this is only happening on our DC switches.
All WS-12-250-DC, board REV F.
All our AC switches with similar configurations are not having this issue.

[sirhc]

All our switches use the EXACT SAME chip and firmware.

The DC switches do a little more in stats collecting and the WS-12-250-DC talks to the DC power supply board but you said the COU dropped when disabling SSH?