Dropping ports on new WS, what is wrong with my setup?

[nickwhite]

Turning flow control off now.

FYI, I did catch the last crash and was able to run ps and top. The CPU was NOT being taxed.

Code: Select all

top - 11:06:24 up 3:11, 0 users, load average: 0.73, 0.66, 0.44
Tasks: 62 total, 3 running, 59 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.2%us, 63.3%sy, 0.0%ni, 24.5%id, 0.0%wa, 1.4%hi, 0.7%si, 0.0%st
Mem: 126348k total, 53104k used, 73244k free, 4120k buffers
Swap: 0k total, 0k used, 0k free, 15116k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 831 admin RT 0 2964 1064 552 S 13.1 0.8 6:06.25 i2c thread
1753 admin 20 0 2120 640 452 R 8.2 0.5 0:01.29 top
1207 admin 20 0 15184 8872 1680 S 6.6 7.0 8:59.65 pinger
 819 admin 20 0 2964 1064 552 R 4.9 0.8 15:16.75 switch
 797 admin 20 0 6804 1832 1052 S 3.3 1.4 0:00.94 lighttpd
 795 admin 20 0 6860 1880 1052 S 3.3 1.5 0:01.26 lighttpd
1784 admin 20 0 2680 456 372 S 3.3 0.4 0:00.02 sh
 832 admin 20 0 2964 1064 552 S 1.6 0.8 0:11.40 poe thread
1785 admin 20 0 2676 452 364 S 1.6 0.4 0:00.01 ping
 796 admin 20 0 6888 1892 1052 S 1.6 1.5 0:01.43 lighttpd
 2 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd
 3 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
 4 admin RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
 47 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 kswapd0
 281 admin 20 0 2688 300 204 S 0.0 0.2 0:00.00 init
 290 admin 20 0 2700 416 324 S 0.0 0.3 0:00.09 syslogd
 5 admin 15 -5 0 0 0 S 0.0 0.0 0:24.83 events/0
 1 admin 20 0 2688 488 392 S 0.0 0.4 0:01.12 init
 6 admin 15 -5 0 0 0 S 0.0 0.0 0:00.03 khelper
 502 admin 20 0 2684 392 300 S 0.0 0.3 0:00.99 watchdog
 292 admin 20 0 2676 384 300 S 0.0 0.3 0:00.03 klogd
 293 admin 20 0 2700 416 324 S 0.0 0.3 0:00.00 syslogd
 763 admin 20 0 19236 13m 952 S 0.0 10.6 0:26.21 RX thread
 48 admin 15 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
 45 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
 46 admin 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
 781 admin 20 0 19236 13m 952 S 0.0 10.6 1:33.15 loop_protection
 783 admin 20 0 6272 748 500 S 0.0 0.6 0:00.03 lighttpd
 784 admin 20 0 39144 1912 1340 S 0.0 1.5 0:00.73 php-fcgi
 785 admin 20 0 19236 13m 952 S 0.0 10.6 0:00.00 vtss_appl
 786 admin 20 0 19236 13m 952 S 0.0 10.6 0:19.62 vtss_appl
 788 admin 20 0 39144 1916 1340 S 0.0 1.5 0:00.76 php-fcgi
 790 admin 20 0 39144 1912 1340 S 0.0 1.5 0:00.68 php-fcgi
 792 admin 20 0 1452 248 196 S 0.0 0.2 0:00.03 stats.fcgi
 793 admin 20 0 1452 248 196 S 0.0 0.2 0:00.00 stats.fcgi
 794 admin 20 0 1516 368 248 S 0.0 0.3 0:00.05 stats.fcgi
 764 admin 20 0 19236 13m 952 S 0.0 10.6 0:00.87 event thread
 778 admin 20 0 19236 13m 952 S 0.0 10.6 0:22.12 mstp_thread
 780 admin 20 0 19236 13m 952 S 0.0 10.6 0:00.03 lacp_thread
 799 admin 20 0 1936 312 236 S 0.0 0.2 0:00.00 dropbear
 804 admin 20 0 19236 13m 952 R 0.0 10.6 0:29.73 port thread
 805 admin 20 0 19236 13m 952 S 0.0 10.6 1:47.02 status_thread
 812 nobody 20 0 2280 668 576 S 0.0 0.5 0:01.27 shellinaboxd
admin@P#5:/www# ps -aux
 PID USER VSZ STAT COMMAND
 1 admin 2688 S init
 2 admin 0 SW< [kthreadd]
 3 admin 0 SW< [ksoftirqd/0]
 4 admin 0 SW< [watchdog/0]
 5 admin 0 SW< [events/0]
 6 admin 0 SW< [khelper]
 21 admin 0 SW< [kblockd/0]
 45 admin 0 SW [pdflush]
 46 admin 0 SW [pdflush]
 47 admin 0 SW< [kswapd0]
 48 admin 0 SW< [aio/0]
 58 admin 0 SW< [mtdblockd]
 61 admin 0 SW< [spi-vcoreiii.0]
 269 admin 0 SWN [jffs2_gcd_mtd6]
 281 admin 2688 S init
 290 admin 2700 S syslogd -S -s128 -L -R [162.210.111.7]:514
 292 admin 2676 S klogd
 293 admin 2700 S syslogd -S -s128 -L -R [162.210.111.7]:514
 306 admin 1132 S /sbin/hotplug2 --override --persistent --max-childre
 502 admin 2684 S watchdog -t 500ms /dev/watchdog
 759 admin 19236 S vtss_appl -l -m -s -a -i -p
 762 admin 19236 S vtss_appl -l -m -s -a -i -p
 763 admin 19236 S {RX thread} vtss_appl -l -m -s -a -i -p
 764 admin 19236 S {event thread} vtss_appl -l -m -s -a -i -p
 778 admin 19236 S {mstp_thread} vtss_appl -l -m -s -a -i -p
 780 admin 19236 S {lacp_thread} vtss_appl -l -m -s -a -i -p
 781 admin 19236 S {loop_protection} vtss_appl -l -m -s -a -i -p
 783 admin 6272 S lighttpd -f /etc/lighttpd.conf
 784 admin 39144 S /usr/bin/php-fcgi
 785 admin 19236 S vtss_appl -l -m -s -a -i -p
 786 admin 19236 S vtss_appl -l -m -s -a -i -p
 788 admin 39144 S /usr/bin/php-fcgi
 790 admin 39144 S /usr/bin/php-fcgi
 792 admin 1452 S /www/stats.fcgi
 793 admin 1452 S /www/stats.fcgi
 794 admin 1516 S /www/stats.fcgi
 795 admin 6928 S lighttpd -f /etc/lighttpd.conf
 796 admin 6892 S lighttpd -f /etc/lighttpd.conf
 797 admin 6804 S lighttpd -f /etc/lighttpd.conf
 799 admin 1936 S /usr/sbin/dropbear -p 22 -z 5 -T 60
 804 admin 19236 S {port thread} vtss_appl -l -m -s -a -i -p
 805 admin 19236 S {status_thread} vtss_appl -l -m -s -a -i -p
 812 nobody 2288 S shellinaboxd -s /shell:admin:root:/root:-/bin/sh --l
 814 admin 39144 S /usr/bin/php-fcgi
 816 admin 39284 S /usr/bin/php-fcgi
 817 admin 39144 S /usr/bin/php-fcgi
 819 admin 2964 S switch -g
 823 nobody 2268 S shellinaboxd -s /shell:admin:root:/root:-/bin/sh --l
 824 admin 2692 S sh /usr/bin/monitor
 826 admin 2964 S switch -g
 827 admin 2964 S {discovery threa} switch -g
 831 admin 2964 S {i2c thread} switch -g
 832 admin 2964 S {poe thread} switch -g
 1207 admin 15184 R {pinger} /usr/bin/php-cli /usr/bin/pinger
 1658 admin 2680 S -/bin/sh
 1668 admin 2756 S switch -s
 1743 admin 2684 S sh -c /bin/ash
 1744 admin 2684 S /bin/ash
 1783 admin 2676 S sleep 10
 1788 admin 2680 R ps -aux
admin@P#5:/www#


[nickwhite]

Eric is investigating a possible bad interaction between Flow Control and RSTP.
Will keep you posted.

Initially I only had RTSP enabled on the #22 Netonix switch. #91 is a toughswitch - which also had RTSP enabled. None of the others have had RTSP enabled.



I went ahead and turned off Flow Control on all ports.

[nickwhite]

After about 40 minutes of Flow Control disabled, I had another switch on the opposite end of the network do the same thing... this one gave me similar issues once about 3 weeks ago, but that's it. I went ahead and disabled Flow Control on that one as well.

[adairw]

Nick, are these all WS-12-250 AC's?

[nickwhite]

Nick, are these all WS-12-250 AC's?

No. The latest one is a WS-24-400A

[sirhc]

Nick, are these all WS-12-250 AC's?

No. The latest one is a WS-24-400A

The model of the switch will make no difference.

All our switches use the exact same switch core chip and all of course use the exact same firmware.

WHen it comes to switching issues all are the same.

The only difference between our models and board Revs has to do with POE options and such as ALL switch core functions are the same.
You could literally take the switch core from the WS-6-MINI and install it in the WS-24-400A and if we were willing to violate our agreement with Vitesse it would work.

*SAME CHIP IN REALITY*

[adairw]

Chris, I understand. I'm just thinking out loud. Most of the other switches in the network are ws-12-250B's. This new leg is all 250 ac's. Just trying to find some correlation

[sirhc]

Chris, I understand. I'm just thinking out loud. Most of the other switches in the network are ws-12-250B's. This new leg is all 250 ac's. Just trying to find some correlation

LOL, I hear you brother, I appreciate the grasping at straws method, we have been doing that for days internally.

You should see our internal Skype conversations right now on this issue.

We are not even sure this is us, could be another device interacting with us poorly.

Eric is looking at Flow Control right now and its interaction with RSTP possibly?

Also noticing that most of the situations involve an airFIBER radio but yet I have tons of AF and AFX radios and I have FC enabled and RSTP enabled and I see no issues at all but I do not rely on RSTP failover.

What we would really like to know is what that mystery traffic is but port mirror will not work as certain types of packets such as BPDU and pause frames are not mirrored. And who is generating these packets?

[sirhc]

I will be pushing out v1.4.0rc16 possibly shortly but this does NOT have a fix for this issue but if you are running rc12 or have a smart DC switch you should upgrade as rc12 has a memory leak and rc16 fixes some I2C issues.

Dave tested rc16 for the past several days and right now I am rolling rc16 out through my towers and if I have no issues I will push it. I will also do some sanity checks on rc16 as well.

[mayheart]

I have a similar problem with Cisco and Airfiber units.

Having flow-control and RSTP will cause the interface to randomly lockup, shutting it down and bringing it back up is the only way to get it to work again. The interface will report 0 TX packets and lots of RX packets. Disabling flow-control & spanning-tree fixed the problem but that's not a good solution.

Perhaps the Airfiber units are sending some funky traffic when a certain situation occurs?

It's a shot in the dark but the situation looks similar. I just use Netonix for access points & PoE switches up towers when height is an issue.