WS3 that almost freezes with much broadcast traffic

[giannici]

We have a WS3 that quite works, with some minor strangeness. E.g. sometime it gives error when we try to save config.

But the greatest problems arise when it is connected to a LAN with a lot of broadcast traffic (a few tens of broadcast packets per second): it almost freezes, everything is very slow, and CPU is almost 100%:




Strange things happen to the bandwidth graphs (300 Gbps????):

[img]Screenshot_2022-03-01_17-36-28-Status0.png[/img]



And STP is active but without roles:




We really hope all these problem are solved in next firmware release...
Thanks.

[sirhc]

Yes we are working on a new WS3 firmware.

However if you are getting the error when saving a config I made a post on this and how to correct that error.

Please read this post for details on how to fix that error and what we are working on.
viewtopic.php?f=17&t=7315&p=36418#p36418

Keep in mind that the CPU has nothing to do with packet handling as the switching is handled by the switch core and has nothing to do with the CPU reported in the UI.

The CPU is small CPU that is only responsible for driving the UI/CLI and configuring the switch CORE which is rated at line speed of all ports.

So unless your switch IP address is on a subnet with thousands of broadcast per second the switch CPU would not see them.

Or if your switch is on a VALID routable IP it could be under attack from anywhere in the world which in that case you should enable the Access Control list in the switch to limit the IP addresses that it will even listen too else it could be under attack from anywhere. Or if it is even on an invalid IP address but your customer valid IP addresses can get to it then it still can be under attack. I am not sure of your network architecture, is it a large flat network which is dangerous in my opinion or is it segment with VLANs or segmented via routing. My infrastrure is all sitting on INVALID non routable IPs that can only be accessed within my network and then I prevent my users from within my network from getting to them and devices such as my Cisco routers at each tower which have to have valid IPs I use an access control list to prevent the outside world from getting to them and also my users valid IP address from accessing them.

But anyway I would default your switch to clear the corrupted config and manually set it back up. Then the only way your switch CPU is getting effected from broadcasts storms is it is setting at an IP that either the web or your customers can get too?????

Again the CPU in the switch has nothing to do with switching load but rather only the linux running on a small embedded process that is used to run the UI/CLI and configure the switch core. If any packet storm is simply being switched through the switch and not on the same subnet or segment as the switch IP address which is for configuring the unit then this would have no effect on the CPU.

I want to stress that our switches are not software switched but rather switch cores and the CPU has no bearing on the switch core packet handling.

[giannici]

OK, but our main problem are all these strangeness when a lot of broascast traffic is present.
Have you noticed this problem?

[giannici]

So unless your switch IP address is on a subnet with thousands of broadcast per second the switch CPU would not see them.

No, they are broadcast packets, so CPU is seeing them.

Anyway, the switch IS WORKING, the traffic flows correctly.

The problem is that the GUI is almost unusable.
And please note that many other WS (no WS3) switches are on the same network, and they have NO problem at all!
So this is a bug specific of the WS3.

[sirhc]

OK, but our main problem are all these strangeness when a lot of broadcast traffic is present.
Have you noticed this problem?

Again if those broadcast storms can get to the switch IP address which is a small cpu used to run the UI/CLI to configure the core then it could overwhelm that small CPU and cause problems.

broadcast storms going through the switch core and not on the same subnet as the switch UI/CLI then the CPU would not see them it is just traffic going through the core.

If the small embedded CPU which is reported in the UI is being maxed out by broadcasts that is because those broadcasts are on the same subnet as the switch UI/CLI ip address and the switch core is sending those packets to the IP address assigned to the switch for you to access it to configure it.

For instance the switch core is rated to pass line speed on all ports at the same time and if that traffic is not seen by the small CPU in the switch used to run the UI/CLI it would not affect that CPU load. Only if traffic going through the switch core should be forwarded to the IP address assigned to the switch for access to the UI/CLI would thoise packets be forwarded to that IP address and then affect the CPU load you see in the UI. The CPU that runs the UI/CLI is a very small CPU and we report that load to make sure the linux operating system that runs the CLI and UI is not messing up but again that CPU has nothing to do with how much data the switch core is passing unless those packets should go to that small CPU because the IP assigned to to the switch should see those broadcasts.

Basically the switch core looks at all packets going through the core in the management VLAN and is any of those packets are in the same subnet as the switch IP address forwards those packet to that small CPU which is a separate embedded CPU whose sole purpose to to tun a little linux operating system to run the UI/CLI and then when you hit save the config configures the switch core.

[sirhc]

So unless your switch IP address is on a subnet with thousands of broadcast per second the switch CPU would not see them.

No, they are broadcast packets, so CPU is seeing them.

Anyway, the switch IS WORKING, the traffic flows correctly.

The problem is that the GUI is almost unusable.
And please note that many other WS (no WS3) switches are on the same network, and they have NO problem at all!
So this is a bug specific of the WS3.

Again you reported you get an error when saving config changes.
I already answered how to fix that problem, you have to default the switch and set it back up manually as the JASON config file is corrupted.

After you do that and you can then save changes without getting an error we can go from there.

So please DEFAULT the WS3, then manually set it back up and confirm you can save changes without getting that error. A corrupted config can cause a lot of problems. Please do that first and report back what you see after you fix that issue.

[giannici]

So please DEFAULT the WS3, then manually set it back up and confirm you can save changes without getting that error. A corrupted config can cause a lot of problems. Please do that first and report back what you see after you fix that issue.

OK, we have RESET the WS3, so we are starting from a default config, and only set an IP address. We connect from a PC to one of the 10G ports. No problem. As soon we connect the second 10G port to the LAN with a lot of broadcast/multicast packets the WS3's CPU goes to 98% and the GUI almost freezes (but traffic is still correctly flowing).

I want to reiterate that in the same LAN there are a lot of WSs that have NO problem at all. Only WS3s have this problem.

As a side note, much of the broadcast traffic in the LAN are gratuitous ARPs generated from WS switches every 10 seconds! This problem was reported more than an year ago (viewtopic.php?f=17&t=6874) but never resolved.

Thanks.