v1.5.5rcX Bug Reports and Comments

[Stephen]

**** IMPORTANT NOTICE ****
Version 1.5.5rc2 was a collective effort where all engineer's where recalled to Netonix headquarter's to tackle what was considered the most important issue's we HIGHLY recommend installing this release ASAP and reporting your result's because we are working on a mean's of improving production and need to finalize version 1.5.5, if there where any issue's you where waiting on in this release - please note that we are still aware of them and normal development will continue after this release.

FIXED/CHANGED
- WS-12-400-AC would not display port 5 and port 6 monitored power correctly for VH (This bug has been present since before 1.4.9) - rc1
- Added Firmware Version to page footer - rc2
- DCDC Bootloader Version and DCDC Board Rev display correctly if i2c link to Power Supply is broken on boot up - rc2
- vtss_appl is now SIGNIFICANTLY more stable than it was. This was probably the root of many other issue's experienced by users. - rc2
- SFP module interaction has improved SIGNIFICANTLY, rebooting, and general operation should no longer cause failure's. It's possible more module's are compatible as well (this requires further testing) - rc2
- Boot up now performs a gratuitous ARP broadcast, which prevents the scenario where upon a warm or cold boot (aka firmware upgrade, etc) where loss of connectivity
to the switch's management site or ssh access occurs even if sometimes otherwise the switch is operating. - rc2
- Fixed issue, upon boot or config changes, PoE wattage values would disappear on smart DC switches if i2c to the Power Supply failed. - rc2

ENHANCEMENTS

KNOWN ISSUES
- WEB UI issues when not at 100% Zoom on browser especially on VLAN TAB
- Some language templates need help - please private message Eric Stern to help

R1 Released 8/27/2019
R2 Released 9/20/2019

[sirhc]

Software engineers were flown here so we could collectively attack several long standing issues that have festered in some cases years but could never seem to get resolved. We spent all this past week collectively attacking these issues and testing as best we could and think we have finally put many of these issues behind us.

Also we need to finalize v1.5.5 SOON so we can provide this to production so it can come pre flashed on new boards.

Also we worked on a new custom written centralized testing and serialization software suite that can be accessed by multiple locations to soon ramp up production by bringing on another production facility/location to better handled increased demand and facilitate finally bringing the new WS3 line to market. This software will also eventually tie into an online automated and tracking of RMAs greatly simplifying and speeding up this process.

Starting in early 2020 we should be able to start morphing into something bigger and better.

Also a lot of internally developed technology that is in the WS3 line will be eventually back ported into existing WS product lines.

Strongly suggest using v1.5.5rc2 as we need feedback on these fixes.

This was a LONG week especially for Eric and Stephen, we left our facility last night [Friday] around 1:30 AM and they headed back to their hotel without our planned last night meal and drinks.

They are currently now on the road headed to the PHL airport to go home, thanks guys for the massive efforts this past week.

[sirhc]

NOTICE: When upgrading my WISP towers I noticed that a couple OLDER v1.4.8 and v1.4.9 as I had been trying to run all versions before for reference those older firmware switches incorrectly detected COLD boot which is a power cycle reboot and for whatever reason appeared to fail to link up initially with my AF24 backhauls. The first tower was close so I quickly rebooted it within minutes and all was fine.

Another tower was further away and until I got there either 1 or 2 things occurred:
1) The AF24 watchdog rebooted the AF24 then the links were automatically re-established and the tower came up on its own just took 6+/- minutes.

2) During COLD reboots or FALSE detected COLD reboots the switch turns OFF POE which would have caused the AF24 to reboot and we all know they boot slow. Then after reboot they had to establish links then OSPF had to negotiate and converge which would take about the same down time about 6+/- minutes.

So when Stephen gets home and has time Monday he will investigate this. I think this has to do with the way we detect the type of boot WARM or COLD as it does things differently and the fact these were OLDER firmwares handing the upgrade.

Now keep in mind when upgrading and then rebooting the switch is using the OLD firmware to perform the upgrade and reboot so this issue may be related to the OLDER firmware (older than v1.5.0 as I "think" that is when we implemented the WARM COLD reboot detection). I plan to upgrade towers repeatedly late at night this weekend and see if already a newer firmware v1.5.0 or newer if it never does this or very rarely does as I also "think" there was a very low percentage chance that a FALSE cold reboot could be detected based on weird factor but this was a rare event, again think I will have Stephen look at why it can even ever happen. But now that I think about it when upgrading from older firmware that did not have COLD WARM reboot code it would always think it was a COLD reboot as the way it works is the firmware handling the reboot writes a maker flag which on reboot the firmware sees then deleted but then knows it was a WARM reboot. Again this was from so long ago and Stephen will have to review all the change notes to confirm but I think I am correct on how I recall it works.

But anyway once upgraded to v1.5.5rc2 this firmware has several issues FINALLY fixed that we had been chasing for years such as doing something insignificant like turning POE ON or OFF on a port not even in use causing vtss to reload causing RSTP to cycle the ports. Also as said better handling of SFP modules, and finally rebooting the switch causing your laptop to lose access unless you do an ARP CLEAR "arp -d *" which forces a broadcast then the laptop can ping it else you would have to wait until the switch sees a broadcast or the arp table times out on your laptop and issues an arp broadcast. By using what is called an gratuitous arp broadcast this is resolved and once we figured this out we were able to discover larger companies like Cisco, Netgear, and so on also struggled with this until they also discovered it and implemented what we just did to correct this issue.

Anyway v1.5.5rc2 once installed I think will fix a lot of issues people have experienced for a long time and we will investigate this false COLD reboot but initial thoughts this is when upgrading from OLDER firmware to v1.5.5rc2 as again the process is handled by the existing firmware already installed on the switch.

And just so you know I like v1.5.5rc2 so much that I just finished upgrading ALL my towers to this version as these bugs which we knew were there for a long time and could not seem to fix are now fixed or at least we are 99.999% sure they are.

[sirhc]

Oh, and one more thing I just remember we forgot to do as it was a LONG week was to look and see if there was a newer version of the HTTP and SSH packages we use with new security patches so Stephen will check Monday and will put them in the next release but our goal is to finish off v1.5.5 soon.

So far all our security issues have been related to these packages we use to run our HTTP and SSH as we do not write them and we have to constantly check for newer releases as vulnerabilities are discovered and patched by those package creators. Security is a never ending battle, sad that people exist that chose to exploit other people and cause harm to others but that is just the way it is.

[md1man42]

I upgraded 2 WS-12-250-DC hoping to fix a couple issues Based on these 2 changes
- Boot up now performs a gratuitous ARP broadcast, which prevents the scenario where upon a warm or cold boot (aka firmware upgrade, etc) where loss of connectivity
to the switch's management site or ssh access occurs even if sometimes otherwise the switch is operating. - rc2
- Fixed issue, upon boot or config changes, PoE wattage values would disappear on smart DC switches if i2c to the Power Supply failed. - rc2

1. I would occasionally lose access to the switch requiring a truck roll to plug a laptop into an open port on the switch. Once the laptop was plugged I would be able to remote into the switch again. After the upgrade the switch came back and no intervention was needed by the tech on site so that looks good. However The Total Throughput numbers and tx/rx data no longer update unless I refresh the screen. I have tried this in Chrome and Edge

2. On the 2nd switch I have port with a Cambium PTP550 plugged in and powered by the Netonix. The port shows 0 Watts power draw. Upgrading to 1.5.5rc2 did not resolve this issue. On this switch I do not have the Total Throughput number and tx/rx issue that I have on the first switch. Both switches have the same boards
Board Rev F
Power Supply Firmware Version 70
Power Supply Board Rev B

[Stephen]

md1man42,

Glad to hear that it helped a bit.

As far as the issue's for 1 & 2, they sound like related problems to me. Can you check that your browser's have JavaScript enabled and that you don't have any plugin's that might be interfering? Such as ScriptSafe or NoScript. Or if that doesn't help, while looking at the switches page, try going into the developer view on your browser (on chrome, just hit CTRL + SHIFT + i) and click on the "console", take a screenshot of what you see there and post it here. It is most likely a javascript issue.

[md1man42]

Issue 1 appears to have resolved itself so at this point it looks good.

Issue 2

Attached the dev console and JSON screenshot. This wattage display issue only is on one port which is port 2. This does not occur on every port, and it does not follow the device when it is plugged into another port.

[Stephen]

Ahh, it's working on my WS-12-250-DC on that port, so it must be a bad port sensor. My suggestion is to RMA the unit.

[Ludvik]

I think is something wrong with IPv6. I have this disabled in configuration, but switch has IPv6 global address from SLAAC!

Internal services (https, ssh, snmp) listen on IPv6.
I try configure Access control, but ipv6 netfilter rules are still empty ... And saving configuration takes about 150 seconds (without ipv6 access control cca 10 sec).

Second problem is IP address on VLAN. If I configure it, i don't see address in "ip addr" listing. It appear after reboot, or if I change something else.
And disabling this is same problem. In configuration it is disabled, but ip addr still show ip address.

And I have question: If I disable spanning-tree - is BPDU forwarding enabled, or disabled?

[Stephen]

After some testing, it appears that IPv6 does require a little bit of debugging when the config disables it. VLAN IP's do seem to have an issue. Do you mind letting me know what IP you assigned to your VLAN's as well as the IP your switch was at? This will help while I work on this.
I will try and add that in before the final release.

BDPU packets do not cross switches whether STP is disabled or not.