v1.5.0rc1 and rc2 Bug reports and Comments - CLOSED

[sirhc]

What happened to dropping rc from the version?

After v1.4.8 if finalized we will stop the naming scheme we use and simply uses dots so in the future it will v1.4.8.1 and so on

Due to a stability issue affecting the WS-25-500-DC rebooting we had no choice, we had to get a version out ASAP and as it was not ready to be FINAL we had no choice, how would you have handled it?

[sirhc]

v1.5.0rc2 was released with more improvements to prevent reboots of WS-26-500-DC

Anyone with a WS-26-500-DC should install this version.

[IntL-Daniel]

Any update on this QOS issue introduced in 1.4.9 http://forum.netonix.com/viewtopic.php? ... =20#p22394 ? We still stay on 1.4.8 to keep our custommers working. Thanks.

There were no changes that would affect QOS between 1.4.8 and 1.4.9. Any difference between them you are seeing is strictly coincidental.

Unfortunately the issue somehow persist and one more! Today early morning I gave another try to put 1.5.0rc2 to all of our 5 Netonix switches (WS-12-250-AC or WS-10-250-AC) instead of 1.4.8.

The result:
1/ The QOS issue (described in message reffered above) still persist, BUT maybe it is only releated to wrong re-appliing QOS rules during upgrade(?). Upgrade to 1.5.0rc2 caused QOS issue again...BUT when I disabled (Save) and enabled (Save) all QOS rules on all switches, the isssue seems like disappeared (tested another 2 hours later).

2/ Unfortunalelly a few hours later when full traffic in our network started we started to lose the whole network like a "domino"..at the begining there was a "loop protection warning" on one switch and disabled port with one of our backhaul wireless link. The log said that disabled port for 180s was immediatelly disabled again after it was re-enabled. Reboot of that switch did not help and disable/enable the port as well (the same as bounce power to that port). When I disabled loop protection on that one switch then domino effect started and loop protection appeared on another four Netonix switches. Each switch downgraded to 1.4.8 one by one re-establish previous working network..no more loop protection warnings.

No VLAN used, STP disabled, never happen before. I bet it could be releted to Watchdog changes in 1.5.0rcX release, while we have enabled watchdog to the first loop protection affected port with Ubiquti NanoBridge M5 with the IP of that unit. I had no chance to do more investigation..I am back on 1.4.8 again without an issue.

Edit: I can see another possible root cause of this in changelog (BTW we do not use SFP ports):
- Fixed issues with Loop Protection on SFP ports - RC1

[sirhc]

I bet it could be releted to Watchdog changes in 1.5.0rcX release, while we have enabled watchdog

I will ask Eric to look at your claims on QOS further this coming week but the changes we did in regards to "watchdog" was to "watchdog service" that is internal to the switch core and has nothing to do the watchdog feature located on the Tools/Watchdog and you are not able to configure it.

The watchdog service we are working with has nothing to do with the user configurable watchdog feature on the Tools/Watchdog TAB, QOS, Loop Protection, or any other service of the switch it is an internal watchdog that only monitors communication between the switch core and the Linux running the UI/CLI.

Edit: I can see another possible root cause of this in changelog (BTW we do not use SFP ports):- Fixed issues with Loop Protection on SFP ports - RC1

What I can tell you is Loop Protection was fixed in v1.5.0rc1 for SFP ports, it was broken and not working.
Not everyone can use Loop Protection with their network configuration and the equipment they use especially large flat networks.
I would suggest you use STP instead of Loop Protection.
If your positive this is a FALSE Loop detection then DIsable Loop Protection. If it is a False Loop Protection report then nothing bad will happen.

When I disabled loop protection on that one switch then domino effect started and loop protection appeared on another four Netonix switches. Each switch downgraded to 1.4.8 one by one re-establish previous working network..no more loop protection warnings.
The fact that you

On Monday or Tuesday I will LAB v1.4.8 Loop Protection to confirm it is broken on SFP ports.
As I said I would suggest you use STP and DIsable Loop Protection especially on a large flat network.

If I was you I would also take the time to investigate all your switches while on v1.4.8 close inspecting the stats, traffic, and packets on each port looking for something amiss.

[IntL-Daniel]

Thanks for the review of QOS rules tab in 1.4.9 and above during upgrade... About (probably) false loop protection alarm, I have disabled that option on all of our switches now and enabled RSTP instead (even though there is no possible loop in our scenario). But this still not explain different behaviour in 1.5.0rc2 and 1.4.8 - we had never false loop protection on any previous firmwares. Finally I will stay on 1.4.8 until some reply about QOS review. Thanks.

[sirhc]

Thanks for the review of QOS rules tab in 1.4.9 and above during upgrade... About (probably) false loop protection alarm, I have disabled that option on all of our switches now and enabled RSTP instead (even though there is no possible loop in our scenario). But this still not explain different behaviour in 1.5.0rc2 and 1.4.8 - we had never false loop protection on any previous firmwares. Finally I will stay on 1.4.8 until some reply about QOS review. Thanks.

Well as I said Loop Protection was not working on SFP ports in v1.4.9, not sure if it was broken in v1.4.8 but I suspect it was broken on SFP ports for awhile.

So when you upgraded to v1.5.0rc2 Loop Protection was now working on SFP ports and your network configuration/topology somehow allowed the broadcast type packet for Loop Protection to bounce around and get picked up.

If you allow your customers to have layer 2 access to your network this issues could be any of your customers internal network.

At my WISP we do not allow customers Layer 2 access to our network for this very reason.

ANother cause for false Loop Protection events is misconfigured VLANs, and or leakie VLANs which is most often a misconfigured VLAN or the improper use of VLAN Trunking type option or allowed VLAN ID lists. These allowed VLAN ID list use does not cause an issue but since Loop Protection sends out a packet designed to bounce around a misconfigured or leaky VLAN will trigger a False Loop Protection.

I have seen over the years people that run a large flat network often have a misconfigured VLAN setting somewhere that really is not causing an issue as it is relying on the Layer 2 switching and the way they have the VLANs setup is with Trunking or allow all VLAN IDs to pass basically making the VLAN a placebo but it works so they never realize they really are not segregated like they think they are.

[IntL-Daniel]

If you allow your customers to have layer 2 access to your network this issues could be any of your customers internal network.

At my WISP we do not allow customers Layer 2 access to our network for this very reason.

BTW how you do that?

[sirhc]

So if your one of my residential customers the CPE is in Router/NAT mode. This prevents them from having Layer 2 access.

If your a commercial customer your CPE is in router mode and your default gateway is on the CPE and the customer is prevented from Layer 2 access to my network. This requires the commercial customer to be subnetted and a route put into the Tower router, in my case a Cisco 2951.

My CPE radios are never in Bridge Mode because I do not trust customers to have Layer 2 access to my network.

Each of my AP's on a tower is segmented so even if a problem occurs on the AP segment which each AP is its own Layer 2 segment it can not affect another AP on that tower and I use OSPF with BFD between all towers.

viewtopic.php?f=30&t=452#p2961

[mgthump]

is multicast IGMP working within a vlan? I know its marked as developing, but I'm out of time, unfortunately. I need to start changing out switches this week if these features are still pending.

If you suspect they are ready, I'll try it on a couple switches. in 1.4.8 if I turn on igmp snooping our streams die in vlans. they carry as expected in Juniper cisco and netgear switches.... don't hang me for mentioning Netgear with Netonix Juniper and Cisco lol




EDIT: just gave it a test on 1.5RC2. IGMP enabled breaks the multicast, disable and it flows as expected :(

[Eric Stern]

is multicast IGMP working within a vlan? I know its marked as developing, but I'm out of time, unfortunately. I need to start changing out switches this week if these features are still pending.

If you suspect they are ready, I'll try it on a couple switches. in 1.4.8 if I turn on igmp snooping our streams die in vlans. they carry as expected in Juniper cisco and netgear switches.... don't hang me for mentioning Netgear with Netonix Juniper and Cisco lol

Nothing has changed in regards to IGMP since 1.4.8 and I have no confirmed reports of any issues with it.