Switch reboots - High mem usage

DOWNLOAD THE LATEST FIRMWARE HERE
User avatar
mhoppes
Associate
Associate
 
Posts: 664
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 125 times

Re: Switch reboots - High mem usage

Wed Feb 13, 2019 6:59 pm

I don’t mean to be the bearer of bad news, but I logged into a switch running 1.4.7 this morning. This switch had an up time of two years. Upon logging in the entire switch rebooted about 15 seconds later.

This switch did have discovery enabled, and I have since disabled it

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Switch reboots - High mem usage

Wed Feb 13, 2019 11:43 pm

mhoppes wrote:I don’t mean to be the bearer of bad news, but I logged into a switch running 1.4.7 this morning. This switch had an up time of two years. Upon logging in the entire switch rebooted about 15 seconds later.

This switch did have discovery enabled, and I have since disabled it


I don't take that as bad news.

1) It was up 2 years - pretty damn good, my Cisco routers don't like being up that long
2) I said there was no significant changes that would make this issue more prevalent in v1.5.x than v1.4.X

Although v1.4.7 was STRONGLY SUGGESTED TO STOP USING A LONG TIME AGO

Keep your units updated.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

wmackay
Member
 
Posts: 6
Joined: Sun Feb 05, 2017 4:42 pm
Has thanked: 0 time
Been thanked: 1 time

Re: Switch reboots - High mem usage

Wed Feb 27, 2019 6:15 pm

I disabled the four discovery options on a switch that was exhibiting the memory leak. This fixed the leak - memory use immediately stopped increasing, and remained stable at the level it was at when I disabled discovery. With these settings disabled, firmware 1.5.1 is stable for me on multiple switches.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Switch reboots - High mem usage

Wed Feb 27, 2019 9:28 pm

wmackay wrote:I disabled the four discovery options on a switch that was exhibiting the memory leak. This fixed the leak - memory use immediately stopped increasing, and remained stable at the level it was at when I disabled discovery. With these settings disabled, firmware 1.5.1 is stable for me on multiple switches.


Yea we are working on several bugs in v1.5.1 and currently in v1.5.2rcX

1) The the discovery TAB being enabled causing memory leak under "some" conditions. also affects previous versions.

2) A bug with STP that can affect the switch in a BAD way with LAGs, this was introduced in v1.5.1 while fixing a MSTP bug that was fixed. Goes to show you fix one thing you can sometimes break another thing.

We hope to have a v1.5.2rcX soon to fix both the above issues.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mhoppes
Associate
Associate
 
Posts: 664
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 125 times

Re: Switch reboots - High mem usage

Thu Feb 28, 2019 8:43 am

Can you elaborate on the STP and LAGs and bad ways?

Wondering if this was potentially my issue or related to it? I plan to re-enable things when I get back from traveling a bit and see if I can get more log information if you still need it.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Switch reboots - High mem usage

Thu Feb 28, 2019 9:21 am

mhoppes wrote:Can you elaborate on the STP and LAGs and bad ways?

Wondering if this was potentially my issue or related to it? I plan to re-enable things when I get back from traveling a bit and see if I can get more log information if you still need it.


STP/RSTP/MSTP are all tied together behind the scene. A recent fix (or so we thought) that happened between v1.5.0 and v1.5.1 for an issue reported with MSTP is causing an issue with STP/RSTP with LAGs. Basically when an insignificant change occurs, like changing a Port Description, when it restarts/reloads vtss it causes STP/RSTP to change ports on a LAG to get stock in blocking mode.

If I knew more it would already be fixed. I do not think it is your issue as you reported your switch rebooted due to memory which is probably more related to the Discovery TAB?
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mhoppes
Associate
Associate
 
Posts: 664
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 125 times

Re: Switch reboots - High mem usage

Thu Feb 28, 2019 9:27 am

Gotcha.... except I didn't disable discovery... only STP. I'll keep oking it until I can figure out what happened -- it's stable at this point in time.

User avatar
sirhc
Employee
Employee
 
Posts: 7416
Joined: Tue Apr 08, 2014 3:48 pm
Location: Lancaster, PA
Has thanked: 1608 times
Been thanked: 1325 times

Re: Switch reboots - High mem usage

Thu Feb 28, 2019 9:40 am

mhoppes wrote:Gotcha.... except I didn't disable discovery... only STP. I'll keep oking it until I can figure out what happened -- it's stable at this point in time.


Well we know STP has an issue right now, so far we only have proof it affects LAGs and the symptom is not memory growth which is what you originally reported?

This STP issue first discovered in v1.5.1 or newer about a week ago, working on this issue right now.
Support is handled on the Forums not in Emails and PMs.
Before you ask a question use the Search function to see it has been answered before.
To do an Advanced Search click the magnifying glass in the Search Box.
To upload pictures click the Upload attachment link below the BLUE SUBMIT BUTTON.

User avatar
mhoppes
Associate
Associate
 
Posts: 664
Joined: Thu Apr 10, 2014 9:14 pm
Location: Pennsylvania
Has thanked: 10 times
Been thanked: 125 times

Re: Switch reboots - High mem usage

Thu Feb 28, 2019 9:43 am

My issue, though, was extremely fast growing memory -- not slow growth like others have seen. My switch would be fine, then suddenly memory would shoot up and within 30 minutes would go into a reboot.

Again - haven't seen it since disabling STP, but I haven't re-enabled yet. I will and have invasive logging on so I can hopefully see what's happening.

User avatar
Stephen
Employee
Employee
 
Posts: 1033
Joined: Sun Dec 24, 2017 8:56 pm
Has thanked: 85 times
Been thanked: 181 times

Re: Switch reboots - High mem usage

Fri Mar 22, 2019 3:00 pm

Try out wispswitch-1.5.2rc6 when you can.

Both of these issue's should be resolved there.
Some of the details of the memory increase you mention here seemed to differ from other reports so there might be something else happening but for the moment both the STP over LAG issue and the discovery memory leak seems to have been resolved.

PreviousNext
Return to Hardware and software issues

Who is online

Users browsing this forum: No registered users and 76 guests