WS-12-250DC lockup

[beambarossa]

Evening,

One of our WS-12-250 switches decided to semi stop responding today. Basically all POE ports were still powered and flowing traffic from looking at the switch physically but we were unable to login to the switch via web, ping and ssh was fine. The switch seemed to be passing very little traffic. Uptime was 60 odd days without issue. Current F/W is 1.4.7 - is there known issue on this?

Here is the dmesg hopefully it helps. To recover the switch we had to physically power cycle.

Thanks,
Sam

Jul 23 16:37:17 Port: link state changed to 'down' on port 10Jul 23 16:37:17 STP: set port 10 to discardingJul 23 16:37:17 Port: link state changed to 'down' on port 11Jul 23 16:37:17 STP: set port 11 to discardingJul 23 16:37:17 Port: link state changed to 'down' on port 1Jul 23 16:37:17 STP: set port 1 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 2Jul 23 16:37:18 STP: set port 2 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 3Jul 23 16:37:18 STP: set port 3 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 4Jul 23 16:37:18 STP: set port 4 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 5Jul 23 16:37:18 STP: set port 5 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 6Jul 23 16:37:18 STP: set port 6 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 7Jul 23 16:37:18 STP: set port 7 to discardingJul 23 16:37:18 Port: link state changed to 'down' on port 8Jul 23 16:37:18 STP: set port 8 to discardingJul 23 16:37:19 Port: link state changed to 'down' on port 9Jul 23 16:37:19 STP: set port 9 to discardingJul 23 16:37:20 Port: link state changed to 'up' (1G) on port 10Jul 23 16:37:20 STP: set port 10 to discardingJul 23 16:37:20 Port: link state changed to 'up' (100M-F) on port 8Jul 23 16:37:20 STP: set port 8 to discardingJul 23 16:37:20 Port: link state changed to 'up' (100M-F) on port 2Jul 23 16:37:20 STP: set port 2 to discardingJul 23 16:37:21 switch[3621]: !unexpected link change on port 8 (RRR-NWA1) from 1G to 100M-FJul 23 16:37:21 Port: link state changed to 'up' (100M-F) on port 3Jul 23 16:37:21 STP: set port 3 to discardingJul 23 16:37:21 Port: link state changed to 'up' (100M-F) on port 4- /var/log/messages 895/895 100% [1]+ Stopped (signal) topadmin@RRR-SW02:/www# dmesgLinux version 2.6.26.8 (dev@netonixdev) (gcc version 4.1.2) #2 Sun Sep 4 20:55:16 EDT 2016Invalid kernel arglist - use RedBoot "exec" command to boot kernel.Using predefined kernel options.CPU revision is: 02019654 (MIPS 24K)Determined physical RAM map:User-defined physical RAM map: memory: 08000000 @ 00000000 (usable)Entering add_active_range(0, 0, 32768) 0 entries of 256 usedZone PFN ranges: Normal 0 -> 32768Movable zone start PFN for each nodeearly_node_map[1] active PFN ranges 0: 0 -> 32768On node 0 totalpages: 32768 Normal zone: 256 pages used for memmap Normal zone: 0 pages reserved Normal zone: 32512 pages, LIFO batch:7 Movable zone: 0 pages used for memmapBuilt 1 zonelists in Zone order, mobility grouping on. Total pages: 32512Kernel command line: console=ttyS0,115200 init=/etc/preinit mem=128MPrimary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytesWriting ErrCtl register=8000000aReadback ErrCtl register=8000000aCache parity protection enabledPID hash table entries: 512 (order: 9, 2048 bytes)Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)Memory: 126052k/131072k available (1917k kernel code, 4840k reserved, 417k data, 116k init, 0k highmem)SLUB: Genslabs=6, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1Calibrating delay loop... 277.70 BogoMIPS (lpj=1388544)Mount-cache hash table entries: 512net_namespace: 640 bytesNET: Registered protocol family 16NET: Registered protocol family 2Switched to high resolution mode on CPU 0IP route cache hash table entries: 1024 (order: 0, 4096 bytes)TCP established hash table entries: 4096 (order: 3, 32768 bytes)TCP bind hash table entries: 4096 (order: 2, 16384 bytes)TCP: Hash tables configured (established 4096 bind 4096)TCP reno registeredNET: Registered protocol family 1squashfs: version 3.0 (2006/03/15) Phillip LougherRegistering mini_fo version $Id$JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.msgmni has been set to 246io scheduler noop registeredio scheduler deadline registered (default)Serial: 8250/16550 driver $Revision: 1.90 $ 1 ports, IRQ sharing disabledserial8250.0: ttyS0 at MMIO 0x70100000 (irq = 14) is a 16550Aconsole [ttyS0] enabledNo NAND device found!!!m25p80 spi0.0: m25p128 (16384 Kbytes)Mapped SPI read area @0x40000000 len 16777216Creating 10 MTD partitions on "m25p80":0x00000000-0x00040000 : "RedBoot"0x00040000-0x00080000 : "config"0x00080000-0x001c0000 : "linux_recovery"0x001c0000-0x003c0000 : "rootfs_recovery"0x003c0000-0x00500000 : "linux"0x00500000-0x00b00000 : "rootfs"mtd: partition "rootfs" set to be root filesystem0x00b00000-0x00f40000 : "rootfs_data"0x00f40000-0x00f80000 : "FIS directory"0x00f80000-0x00f81000 : "RedBoot config"0x00fc0000-0x01000000 : "Redundant FIS"ip_tables: (C) 2000-2006 Netfilter Core TeamTCP cubic registeredNET: Registered protocol family 10lo: Disabled Privacy Extensionsip6_tables: (C) 2000-2006 Netfilter Core TeamNET: Registered protocol family 17802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>All bugs added by David S. Miller <davem@redhat.com>VFS: Mounted root (squashfs filesystem) readonly.Freeing unused kernel memory: 116k freedPlease be patient, while OpenWrt loads ...Algorithmics/MIPS FPU Emulator v1.5mini_fo: using base directory: /mini_fo: using storage directory: /jffsvtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.switch: 'Luton26' board detectedvtss_port: Loaded port module on board Luton26, type 5nf_conntrack version 0.5.0 (2048 buckets, 8192 max)i2c /dev entries driveri2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19eth0: no IPv6 routers presentvcoreiii_fdma_inj:4360 Assert failed: props->switch_frm || props->port_mask != 0*** Failure:/home/dev/vtss_linuxbsp_4_48a/build_dir/linux-vcoreiii_lu26/kmod-vtss_switch/linux_support/kernel/vtss_switch-eth.c:212: vtss_fdma_inj(NULL, l, DMACH_TX, raw_frm_sz_bytes, &netinj_opts)admin@RRR-SW02:/www# dmesgLinux version 2.6.26.8 (dev@netonixdev) (gcc version 4.1.2) #2 Sun Sep 4 20:55:16 EDT 2016Invalid kernel arglist - use RedBoot "exec" command to boot kernel.Using predefined kernel options.CPU revision is: 02019654 (MIPS 24K)Determined physical RAM map:User-defined physical RAM map: memory: 08000000 @ 00000000 (usable)Entering add_active_range(0, 0, 32768) 0 entries of 256 usedZone PFN ranges: Normal 0 -> 32768Movable zone start PFN for each nodeearly_node_map[1] active PFN ranges 0: 0 -> 32768On node 0 totalpages: 32768 Normal zone: 256 pages used for memmap Normal zone: 0 pages reserved Normal zone: 32512 pages, LIFO batch:7 Movable zone: 0 pages used for memmapBuilt 1 zonelists in Zone order, mobility grouping on. Total pages: 32512Kernel command line: console=ttyS0,115200 init=/etc/preinit mem=128MPrimary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytesWriting ErrCtl register=8000000aReadback ErrCtl register=8000000aCache parity protection enabledPID hash table entries: 512 (order: 9, 2048 bytes)Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)Memory: 126052k/131072k available (1917k kernel code, 4840k reserved, 417k data, 116k init, 0k highmem)SLUB: Genslabs=6, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1Calibrating delay loop... 277.70 BogoMIPS (lpj=1388544)Mount-cache hash table entries: 512net_namespace: 640 bytesNET: Registered protocol family 16NET: Registered protocol family 2Switched to high resolution mode on CPU 0IP route cache hash table entries: 1024 (order: 0, 4096 bytes)TCP established hash table entries: 4096 (order: 3, 32768 bytes)TCP bind hash table entries: 4096 (order: 2, 16384 bytes)TCP: Hash tables configured (established 4096 bind 4096)TCP reno registeredNET: Registered protocol family 1squashfs: version 3.0 (2006/03/15) Phillip LougherRegistering mini_fo version $Id$JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.msgmni has been set to 246io scheduler noop registeredio scheduler deadline registered (default)Serial: 8250/16550 driver $Revision: 1.90 $ 1 ports, IRQ sharing disabledserial8250.0: ttyS0 at MMIO 0x70100000 (irq = 14) is a 16550Aconsole [ttyS0] enabledNo NAND device found!!!m25p80 spi0.0: m25p128 (16384 Kbytes)Mapped SPI read area @0x40000000 len 16777216Creating 10 MTD partitions on "m25p80":0x00000000-0x00040000 : "RedBoot"0x00040000-0x00080000 : "config"0x00080000-0x001c0000 : "linux_recovery"0x001c0000-0x003c0000 : "rootfs_recovery"0x003c0000-0x00500000 : "linux"0x00500000-0x00b00000 : "rootfs"mtd: partition "rootfs" set to be root filesystem0x00b00000-0x00f40000 : "rootfs_data"0x00f40000-0x00f80000 : "FIS directory"0x00f80000-0x00f81000 : "RedBoot config"0x00fc0000-0x01000000 : "Redundant FIS"ip_tables: (C) 2000-2006 Netfilter Core TeamTCP cubic registeredNET: Registered protocol family 10lo: Disabled Privacy Extensionsip6_tables: (C) 2000-2006 Netfilter Core TeamNET: Registered protocol family 17802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>All bugs added by David S. Miller <davem@redhat.com>VFS: Mounted root (squashfs filesystem) readonly.Freeing unused kernel memory: 116k freedPlease be patient, while OpenWrt loads ...Algorithmics/MIPS FPU Emulator v1.5mini_fo: using base directory: /mini_fo: using storage directory: /jffsvtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.switch: 'Luton26' board detectedvtss_port: Loaded port module on board Luton26, type 5nf_conntrack version 0.5.0 (2048 buckets, 8192 max)i2c /dev entries driveri2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19eth0: no IPv6 routers presentvcoreiii_fdma_inj:4360 Assert failed: props->switch_frm || props->port_mask != 0*** Failure:/home/dev/vtss_linuxbsp_4_48a/build_dir/linux-vcoreiii_lu26/kmod-vtss_switch/linux_support/kernel/vtss_switch-eth.c:212: vtss_fdma_inj(NULL, l, DMACH_TX, raw_frm_sz_bytes, &netinj_opts)

[sirhc]

Your running very old firmware please upgrade.

Many things have been fixed since v1.4.7

[beambarossa]

Thanks - this just happened this evening again. I have upgraded to 1.5.0 now.

Very interesting issue - this time one of our AP's was still flowing traffic but the other 9 were not. Switch would ping and web / ssh at first then after about 10 minutes only ping worked as ssh / web stopped responding.

Were there any specific fixes for this sort of behaviour or should i be looking for an unusual device thats plugged into it?

[beambarossa]

Shortly (1hr) after upgrading to 1.5.0 and disabling STP the switch disconnected all ports and reconnected again. This time they came back and we were able to keep logged into the switch.

Jul 24 21:43:54 Port: link state changed to 'down' on port 10

Jul 24 21:43:54 Port: link state changed to 'down' on port 11

Jul 24 21:43:54 Port: link state changed to 'down' on port 1

Jul 24 21:43:54 Port: link state changed to 'down' on port 3

Jul 24 21:43:54 Port: link state changed to 'down' on port 5

Jul 24 21:43:54 Port: link state changed to 'down' on port 6

Jul 24 21:43:54 Port: link state changed to 'down' on port 7

Jul 24 21:43:54 Port: link state changed to 'down' on port 8

Jul 24 21:43:54 Port: link state changed to 'down' on port 9

Jul 24 21:43:57 Port: link state changed to 'up' (1G) on port 10

Jul 24 21:43:57 Port: link state changed to 'up' (100M-F) on port 1

Jul 24 21:43:57 Port: link state changed to 'up' (100M-F) on port 3

Jul 24 21:43:57 Port: link state changed to 'up' (100M-F) on port 6

Jul 24 21:43:57 Port: link state changed to 'up' (100M-F) on port 8

Jul 24 21:43:57 Port: link state changed to 'up' (100M-F) on port 11

Jul 24 21:43:57 Port: link state changed to 'up' (100M-F) on port 5

Jul 24 21:43:58 Port: link state changed to 'up' (100M-F) on port 7

Jul 24 21:44:09 Port: link state changed to 'down' on port 10

Jul 24 21:44:12 Port: link state changed to 'up' (1G) on port 10

Jul 24 21:44:15 Port: link state changed to 'down' on port 10

Jul 24 21:44:17 Port: link state changed to 'down' on port 1

Jul 24 21:44:18 Port: link state changed to 'down' on port 11

Jul 24 21:44:18 Port: link state changed to 'down' on port 6

Jul 24 21:44:18 Port: link state changed to 'down' on port 7

Jul 24 21:44:18 Port: link state changed to 'down' on port 3

Jul 24 21:44:18 Port: link state changed to 'down' on port 5

Jul 24 21:44:18 Port: link state changed to 'down' on port 8

Jul 24 21:44:18 Port: link state changed to 'up' (1G) on port 10

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 1

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 6

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 3

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 7

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 11

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 8

Jul 24 21:44:20 Port: link state changed to 'up' (1G) on port 5

Jul 24 21:46:18 Port: link state changed to 'up' (1G) on port 9

Jul 24 21:46:38 Port: link state changed to 'down' on port 9

Jul 24 21:46:40 Port: link state changed to 'up' (1G) on port 9

Jul 24 21:56:56 Port: link state changed to 'down' on port 9

Jul 24 21:56:56 Port: link state changed to 'down' on port 10

Jul 24 21:56:56 Port: link state changed to 'down' on port 1

Jul 24 21:56:56 Port: link state changed to 'down' on port 2

Jul 24 21:56:56 Port: link state changed to 'down' on port 3

Jul 24 21:56:56 Port: link state changed to 'down' on port 4

Jul 24 21:56:56 Port: link state changed to 'down' on port 5

Jul 24 21:56:56 Port: link state changed to 'down' on port 6

Jul 24 21:56:56 Port: link state changed to 'down' on port 7

Jul 24 21:56:59 Port: link state changed to 'up' (100M-F) on port 4

Jul 24 21:56:59 Port: link state changed to 'up' (100M-F) on port 1

Jul 24 21:56:59 Port: link state changed to 'up' (100M-F) on port 7

Jul 24 21:56:59 Port: link state changed to 'up' (100M-F) on port 5

Jul 24 21:56:59 Port: link state changed to 'up' (100M-F) on port 3


There are two 12-250-DC switches at this particular site. The only change is we installed a new AF5xHD link a few months ago.

Any thing else to check?

[sirhc]

Post up the Stats and Device/Status TABs

If your port descriptions do not clarify what type POE device is on each port make a list of what each port is powering

[beambarossa]

Here are pictures.

Ports as follows:

1. Ubiquiti Prism Gen 2
2. Ubiquiti Prism Gen 2
3. Ubiquiti Prism Gen 2
4. Ubiquiti Prism Gen 2
5. Ubiquiti Prism Gen 2
6. Ubiquiti Prism Gen 2
7. Ubiquiti Prism Gen 2
8. Ubiquiti Prism Gen 2
9. Ubiquiti Airfibre 5X
10. Ubiquiti Airfibre 5XHD
11. Ubiquiti Prism Gen 2
12. Tycon TPDIN
13. Mikrotik CCR1009
14. Spare


Thanks!

[sirhc]

OK, your units MFG date: 09/30/2016

Your unit only has 108+/- watts of 24V
Newer units have 160+/- watts of 24V

Read this post: viewtopic.php?f=17&t=3934#p24989

My guess is since you have 9 airPRISMS at some point they are all in transmit mode at the same time then you exceed 106 watts and voltage drops.

This could cause many issues, first would be that radios randomly reboot, or if voltage drops enough then this could cause the unit to do worse things.

Solutions:
Swap unit with a little newer unit and you can do the upgrade yourself as described in post above or RMA it and have us apply that MOD and all other newer hardware MODs.

Now keep in mind I am "guessing" what the cause is.

When we started this the most common 24V device was an airMAX n radio with an average demand of 6 watts (2 min to 12 max watts) now the airPRISIM with the dual radios is considerably higher.

So if you assume average of 6 watts per port x 12 ports that is 72 watts + fan 1 watt + 6 watts max for switch itself is 79 watts so we had a cushion of 27 watts. Plus we assumed there would be 1 to 4 ports that were 48V for backhauls which is not limited as it draws directly from power supply.

[beambarossa]

This would make sense as prior to this one of the airfibers was running on 24v instead of 48v. At night time i have seen these prisms pull up to 12w at times under load.

Is there a way to exactly tell which ones are limited as we have a lot of loaded switches out there. Is there a limitation on the 48v side also?

Sam

[beambarossa]

We also run the IDC and 8-150 version - is there a 24v limit on these?

[sirhc]

Is there a way to exactly tell which ones are limited as we have a lot of loaded switches out there.

Not really easy to define for you which ones are and are not but basically if it was manufactured in the last 18 months it has the 160 watt limit verses the 106 watts. But as the post explains you can modify an older board to also produce 160 watts if you are any good with a soldering iron.

Is there a limitation on the 48v side also?

Yes, the "main" power supply as we simply pass the main power supply 48V (which is really 50V) straight through the switch whereas the switch board must convert 48V to 24V.

We also run the IDC and 8-150 version - is there a 24v limit on these?

The 24V power supply in the WS-8 is not as big as the one in the WS-12 but should not be a problem as there are fewer ports. Also use 48V when you can such as powering AFX radios, do not power AFX 2/3/5 radios with 24V.

Technically the 24V power supply in the WS-6 and WS-8 is rated to 90 watts MAX but if you push the WS-6 that hard it would generate too much heat as it does not have a fan or sufficient PCB board area to dissipate the heat.

Once again we always assume people will use several ports to power 48V devices.