Issues with Netonix

[colinhowlin]

On one of our sites, due to space restrictions, we have 2 WS-12-250A's.
Set up is Backhaul -> Switch1 -> Switch2.
We have a Mikrotik CCR attached to Switch1 as well.

On Saturday, we lost access to Switch2.
Switch1 had the following entry in the log which was just before we lost Switch2:

Code: Select all

 Jan 11 06:14:24 switch[750]: Excessive flow control pause frames received on port 14, disabling flow control

I swapped out Switch2 with a new switch and disabled flow control in case that was the root of the problem.
Just powered it up back in the office and can access via CLI but not WebUI.
I've attached the log but the following entry stands out:

Code: Select all

 Dec 31 19:09:23 kernel: vcoreiii_fdma_inj:4360 Assert failed: props->switch_frm || props->port_mask != 0
Dec 31 19:09:23 kernel: *** Failure:/home/dev/vtss_linuxbsp_4_48a/build_dir/linux-vcoreiii_lu26/kmod-vtss_switch/linux_support/kernel/vtss_switch-

eth.c:212: vtss_fdma_inj(NULL, l, DMACH_TX, raw_frm_sz_bytes, &netinj_opts)

Any idea what might have caused this issue?
Both switches running 1.4.1 with no issues prior to this.
I see 1.4.2 has gone final so I'll be scheduling an upgrade to both switches shortly.

thanks
Colin

[sirhc]

Well yes, upgrade to v1.4.2 as there is a BIG difference in how v1.4.1 and v1.4.2 determines if a Pause Frame Storm is occurring.

v1.4.1 only looks if there are 10K Pause Frames Per Second for 5 consecutive seconds on s specific port

v1.4.2 looks if there are 10K more Pause Frames Per Second than Tx Packets for 5 consecutive seconds on a specific port

You can disable Pause Frame Storm Control on the Device/Configuration Tab but I would not unless absolutely necessary


Beyond that you provided to useful information as to what is going on.

Would need to have a port map as to what is connected to each port of each switch to start with.

As far as WEB UI CLI I have no idea of your configuration and such.

I would factory default the switch and then upgrade it to v1.4.2

[colinhowlin]

Factory defaulted switch and upgraded to 1.4.2 and it is now fully accessible, either by CLI or WebUI.
Replacement switch we put in has not given any issues.
I have left flow control disabled however.

[sirhc]

Factory defaulted switch and upgraded to 1.4.2 and it is now fully accessible, either by CLI or WebUI.
Replacement switch we put in has not given any issues.
I have left flow control disabled however.

Glad you recovered the switch!

When you say disabled Flow Control did you disable it on the port or the AF radio, either works just curious?

[colinhowlin]

We have a Ceragon feeding this site.
The issue seemed to be between the 2 Netonix - I disabled flow control on both switches.
It was already disabled on the Ceragon and Mikrotik CCR.

Colin

[sirhc]

We have a Ceragon feeding this site.
The issue seemed to be between the 2 Netonix - I disabled flow control on both switches.
It was already disabled on the Ceragon and Mikrotik CCR.

Colin

I have no Idea, I have many WISP Switches connected together some even with LAGs and Flow Control is active on all of them?

Now maybe if you have an AF radio on one switch that generates a Pause Frame Storm to the switch it can eventually cause that switch to send out tons of Tx Pause Frames on all other ports when the buffers become full.

Here are 3 screen grabs of 3 switches with Flow Control going between WISP Switches

Picture 1 and 2 are 2 switches in my NOX that are tied together
Picture 3 is a tower switch that feeds a FIBER LACP LAG to as WS-12-250-AC for a high end customer
Picture 4 is the WS-12-250-AC at high end customer

CLICK IMAGE TO VIEW FULL SIZE