Netonix Manager / Control Software

[sakita]

Recently my Netonix Manager is not acting properly:

- Many switches are not being monitored (red ball). Using the edit pencil to turn off polling, save, and then turn back on doesn't fix it. Only deleting and re-adding gets them polling again.

- The manager cannot upgrade firmware. It starts to do the upgrade, fails, and then adds this entry to the log (NAME and IP ADDRESS are actual name and IP of switch):

Device 'NAME' (IP ADDRESS) error during upgrade: Error unpacking firmware file /tmp/upgrade.bin

Manager v1.0.17
Ubuntu 18.04.5 LTS
node.js v8.17.0
133 switches in manager

[Stephen]

If this recently started happening, it suggests that something has recently changed. So a couple of questions to help find the problem.

There are 133 switches, did you recently get to this number?
Are there any other process's or activities that run on the server that have been added recently?
Was the server updated/upgraded recently?

The manager has to maintain several threads and proc's at runtime, please run something like top to get an idea of it's cpu/mem footprint on the server.

Also, try restarting the service, sometimes it helps.

[david.sovereen@mercury.net]

I am having a similar problem as above on a brand new Netonix Manager install. Did not know Netonix Manager existed until a couple days ago. I entered all my switches in (about 30, don't see count anywhere in Netonix Manager).

Windows 10 x64
Netonix Manager 1.0.17
NodeJS 8.17.0 x64

I almost scheduled all switches to get an upgrade, but decided to do one as a test first. I scheduled the single firmware update yesterday to be performed overnight. When I looked this morning, the unit was not updated and the log says:

error during upgrade: Error unpacking firmware file /tmp/upgrade.bin

Any ideas?

Also, sorting does not work correctly on some fields. For example, which I sort by name or address (which are the same for us), I get:

sw-1.aglmwi-1.mercury.net
sw-1.cascwi-1.mercury.net
sw-1.twrvwi-4.mercury.net
sw-1.citnwi-0.mercury.net
sw-1.clevwi-1.mercury.net

sw-1.twrvwi-4.mercury.net should be at the bottom of that particular list. But there are other switches further down that should be up in this list, a sw-1.clevwi-0.mercury.net, for example. Makes it impossible to find what you're looking without using search.

And since name and address are the same for us, it would be nice to remove one completely from the display so that there isn't odd text wrapping.

Thanks,

Dave

[david.sovereen@mercury.net]

I think the problem with error during upgrade "Error unpacking firmware file /tmp/upgrade.bin" has to do with using the "Download a firmware from Netonix" option. I used that option to get the firmware originally and when I did, I got the /tmp/upgrade.bin error. I decided to manually upgrade a switch and to upload the file fo Netonix Manager from my computer in hopes of figuring out the problem. Uploading the file, the upgrade worked.

Now what I don't know is if the 1.5.6 that is now in Netonix Manager is the one I uploaded or the one it Downloaded from Netonix. I just tested by upgrading another switch and telling Netonix Manager to use the 1.5.6 in the local repository. The upgrade worked without error.

I can't be certain, but it would seem that the "Error unpacking firmware file /tmp/upgrade.bin" has to do with using the "Download a firmware from Netonix" option.

Last thing: I have one switch that I cannot get to via SSH. Initially, I had some switches that Netonix Manager could not get to via SSH, but that was caused by Access Control rules and I resolved those. I have one remaining switch that not I, nor Netonix Manager, can get to via SSH, but I can access via HTTPS. All Access Control rules have been removed. I turned off the tarpit. I tried turning off SSH in Device -> Configuration, Applying, then turning back on, but it didn't help. I tried switching the port SSH listened on. That didn't work. I can access the console at Device -> Console. Is there something I can do in there to restart the SSH service or see if firewall rules might be mixed up to try and resolve this without rebooting it?

Thanks,

Dave

[Stephen]

Hey david, I can't be certain as of yet, but it's very possible some of the issue's you're experiencing may be related to:

viewtopic.php?f=17&t=6741

and

viewtopic.php?f=17&t=6739

I have multiple priorities right now but for the WS series and the manager my priority is determining the best course of action to handle the recent deprecations.

[sakita]

Same results as David had above...

I went in and deleted 1.5.6 from the manager's firmwares tab. Then I downloaded a copy from Netonix site. Next I did an upgrade and uploaded it from my PC and it worked. After that I was able to update another switch using the one that is now in the manager.

So, it seems like there was something about the firmware copy that the manager had and replacing it with a fresh one fixed it?

What I haven't tried is deleting the firmware from the manager and then doing upgrade asking it to get it from Netonix.

[OacysShop]

I'm having the same issue as below but we are on Windows and I am trying to find what crashed. We have a WS3-14-600-AC switch that had issues over the weekend and when we try to look at logging on the manager, we get that same message as below but not sure where to find the crash log.

My manager always says "Crash detected. Click here to send the crash log to Netonix." Clicking the link doesn't seem to help, and rebooting the server does not help.

What am I missing?

Also, not sure if the issue where logging on the switch itself spamming the following has been addressed:

Apr 20 05:32:20 dropbear[13879]: syslogin_perform_logout: logout(pts/1) returned an error: No such file or directory

[david.sovereen@mercury.net]

Last evening we had 30+ Netonix switches reboot within seconds of each other. When we logged into the Netonix Manager (running 1.0.17) it had a crash message across the top and we sent in the reporting. Have to believe the manager was involved in this reboot issue because the switches were spread across a pretty large network. Any thoughts what might cause something like this and how we can help debug / prevent in the future.

Happy to provide any information that might assist in getting to the bottom of this.

[Stephen]

Hello david, I've been monitoring as a couple of crash reports reports have come in over the last few weeks. So far they are all apparently the same error, but it is somewhat generic. Could you provide me with the csv for the device's list? You can get it by clicking the DownloadCSV button which is circled in red in the picture below.

DownloadCSV_Button.PNG (3.16 KiB) Viewed 25182 times

It will download a (dot)CSV file that can be viewed in Excel.

I think that maybe helpful in identifying the issue in this case.

[david.sovereen@mercury.net]

Where do you want me to send it? We thought that Netonix Manager might be causing the mass simultaneous reboots, but it looks like the mass-simultaneous reboots were probably causing Netonix Manager to crash. We just recently upgraded to 1.5.9rc1 in hopes of taking advantage of MST, but like previous attempts, were met with instability. We mass-downgraded to 1.5.2 and the reboots have gone away and network stability is back. We've reported this before, but have nothing to give you because the switches just reboot and any log of why they rebooted is gone.