Console reports this on boot, what next?

[dougncbp]

Highlighted one part that caught my attention, but otherwise I can't see any problem. Switch won't finish booting, and I can't interrupt the process with the keyboard to initiate a recovery. Never get a Redboot prompt either... Hangs forever after the last line.

This is a WS-12-250B that recently was flashed to 1.4.6 firmware, and died after MTU Settings were changed on every port. It was also preventing IPv6 from working, and a temporary WS-10-250-AC is working in its place, and handling IPv6 just fine.

------------------------------------------------------------------------------------

RedBoot(tm) bootstrap and debug environment [ROMRAM]
Non-certified release, version 1_18-Vitesse - built 05:56:33, Jan 31 2014

Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
Free Software Foundation, Inc.
RedBoot is free software, covered by the eCos license, derived from the
GNU General Public License. You are welcome to change it and/or distribute
copies of it under certain conditions. Under the license terms, RedBoot's
source code and full license terms must have been made available to you.
Redboot comes with ABSOLUTELY NO WARRANTY.

Platform: VCore-III (MIPS32 24KEc) LUTON26
RAM: 0x80000000-0x88000000 [0x80021b68-0x87fafffc available]
FLASH: 0x40000000-0x40ffffff, 64 x 0x40000 blocks
== Executing boot script in 1.000 seconds - enter ^C to abort
RedBoot> fis load -d linux
Image loaded from 0x80100000-0x8036422c
RedBoot> go
Linux version 2.6.26.8 (dev@netonixdev) (gcc version 4.1.2) #2 Sun Sep 4 20:55:16 EDT 2016
Invalid kernel arglist - use RedBoot "exec" command to boot kernel.
Using predefined kernel options.
CPU revision is: 02019654 (MIPS 24K)
Determined physical RAM map:
User-defined physical RAM map:
memory: 08000000 @ 00000000 (usable)
Zone PFN ranges:
Normal 0 -> 32768
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0 -> 32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
Kernel command line: console=ttyS0,115200 init=/etc/preinit mem=128M
Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
Writing ErrCtl register=80000000
Readback ErrCtl register=80000000
Cache parity protection enabled
PID hash table entries: 512 (order: 9, 2048 bytes)
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 126052k/131072k available (1917k kernel code, 4840k reserved, 417k data, 116k init, 0k highmem)
SLUB: Genslabs=6, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
net_namespace: 640 bytes
NET: Registered protocol family 16
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
NET: Registered protocol family 1
squashfs: version 3.0 (2006/03/15) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
msgmni has been set to 246
io scheduler noop registered
io scheduler deadline registered (default)
Serial: 8250/16550 driver $Revision: 1.90 $ 1 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0x70100000 (irq = 14) is a 16550A
console [ttyS0] enabled
No NAND device found!!!
m25p80 spi0.0: m25p128 (16384 Kbytes)
Mapped SPI read area @0x40000000 len 16777216
Creating 10 MTD partitions on "m25p80":
0x00000000-0x00040000 : "RedBoot"
0x00040000-0x00080000 : "config"
0x00080000-0x001c0000 : "linux_recovery"
0x001c0000-0x003c0000 : "rootfs_recovery"
0x003c0000-0x00500000 : "linux"
0x00500000-0x00b00000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
0x00b00000-0x00f40000 : "rootfs_data"
0x00f40000-0x00f80000 : "FIS directory"
0x00f80000-0x00f81000 : "RedBoot config"
0x00fc0000-0x01000000 : "Redundant FIS"
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 17
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
VFS: Mounted root (squashfs filesystem) readonly.
Freeing unused kernel memory: 116k freed
Please be patient, while OpenWrt loads ...
Algorithmics/MIPS FPU Emulator v1.5
- preinit -
Press CTRL-C for failsafe
Press and HOLD default button now for reset to factory defaults
mounting jffs2
switching to jffs2
mini_fo: using base directory: /
mini_fo: using storage directory: /jffs
- init -

Please press Enter to activate this console. Dec 31 19:00:04 syslogd started: BusyBox v1.19.4
Dec 31 19:00:04 kernel: klogd started: BusyBox v1.19.4 (2016-10-03 19:26:31 EDT)
Dec 31 19:00:04 kernel: Linux version 2.6.26.8 (dev@netonixdev) (gcc version 4.1.2) #2 Sun Sep 4 20:55:16 EDT 2016
Dec 31 19:00:04 kernel: Invalid kernel arglist - use RedBoot "exec" command to boot kernel.
Dec 31 19:00:04 kernel: Using predefined kernel options.
Dec 31 19:00:04 kernel: CPU revision is: 02019654 (MIPS 24K)
Dec 31 19:00:04 kernel: Determined physical RAM map:
Dec 31 19:00:04 kernel: User-defined physical RAM map:
Dec 31 19:00:04 kernel: memory: 08000000 @ 00000000 (usable)
Dec 31 19:00:04 kernel: Entering add_active_range(0, 0, 32768) 0 entries of 256 used
Dec 31 19:00:04 kernel: Zone PFN ranges:
Dec 31 19:00:04 kernel: Normal 0 -> 32768
Dec 31 19:00:04 kernel: Movable zone start PFN for each node
Dec 31 19:00:04 kernel: early_node_map[1] active PFN ranges
Dec 31 19:00:04 kernel: 0: 0 -> 32768
Dec 31 19:00:04 kernel: On node 0 totalpages: 32768
Dec 31 19:00:04 kernel: Normal zone: 256 pages used for memmap
Dec 31 19:00:04 kernel: Normal zone: 0 pages reserved
Dec 31 19:00:04 kernel: Normal zone: 32512 pages, LIFO batch:7
Dec 31 19:00:04 kernel: Movable zone: 0 pages used for memmap
Dec 31 19:00:04 kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
Dec 31 19:00:04 kernel: Kernel command line: console=ttyS0,115200 init=/etc/preinit mem=128M
Dec 31 19:00:04 kernel: Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
Dec 31 19:00:04 kernel: Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
Dec 31 19:00:04 kernel: Writing ErrCtl register=80000000
Dec 31 19:00:04 kernel: Readback ErrCtl register=80000000
Dec 31 19:00:04 kernel: Cache parity protection enabled
Dec 31 19:00:04 kernel: PID hash table entries: 512 (order: 9, 2048 bytes)
Dec 31 19:00:04 kernel: Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Dec 31 19:00:04 kernel: Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Dec 31 19:00:04 kernel: Memory: 126052k/131072k available (1917k kernel code, 4840k reserved, 417k data, 116k init, 0k highmem)
Dec 31 19:00:04 kernel: SLUB: Genslabs=6, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Dec 31 19:00:04 kernel: Calibrating delay loop... 277.70 BogoMIPS (lpj=1388544)
Dec 31 19:00:04 kernel: Mount-cache hash table entries: 512
Dec 31 19:00:04 kernel: net_namespace: 640 bytes
Dec 31 19:00:04 kernel: NET: Registered protocol family 16
Dec 31 19:00:04 kernel: NET: Registered protocol family 2
Dec 31 19:00:04 kernel: Switched to high resolution mode on CPU 0
Dec 31 19:00:04 kernel: IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
Dec 31 19:00:04 kernel: TCP established hash table entries: 4096 (order: 3, 32768 bytes)
Dec 31 19:00:04 kernel: TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
Dec 31 19:00:04 kernel: TCP: Hash tables configured (established 4096 bind 4096)
Dec 31 19:00:04 kernel: TCP reno registered
Dec 31 19:00:04 kernel: NET: Registered protocol family 1
Dec 31 19:00:04 kernel: squashfs: version 3.0 (2006/03/15) Phillip Lougher
Dec 31 19:00:04 kernel: Registering mini_fo version $Id$
Dec 31 19:00:04 kernel: JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
Dec 31 19:00:04 kernel: msgmni has been set to 246
Dec 31 19:00:04 kernel: io scheduler noop registered
Dec 31 19:00:04 kernel: io scheduler deadline registered (default)
Dec 31 19:00:04 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 1 ports, IRQ sharing disabled
Dec 31 19:00:04 kernel: serial8250.0: ttyS0 at MMIO 0x70100000 (irq = 14) is a 16550A
Dec 31 19:00:04 kernel: console [ttyS0] enabled
Dec 31 19:00:04 kernel: No NAND device found!!!
Dec 31 19:00:04 kernel: m25p80 spi0.0: m25p128 (16384 Kbytes)
Dec 31 19:00:04 kernel: Mapped SPI read area @0x40000000 len 16777216
Dec 31 19:00:04 kernel: Creating 10 MTD partitions on "m25p80":
vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
switch: 'Luton26' board detected
nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
i2c /dev entries driver
i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19

[Eric Stern]

You can try push/holding the reset button on the front while it boots. If that doesn't work you will have to RMA it. Instructions here: viewtopic.php?f=6&t=1259

[dougncbp]

Thanks, I tried that once but perhaps didn't hold the button down long enough. It worked this time, wonderful!

But what caused it to lock up like that? As I said in the original post, it took the firmware update just fine and ran for half a day. Then later, I took the opportunity to change the MTU settings on all the ports thinking it was what was blocking IPv6 traffic. 5 mins later, I got all the pages for that whole POP being down. I did a hard reboot and that's when it got into the failed state. The settings change on the MTU took it from 1518 where someone had set it before, to the more standard 1528.

The real question is, can I trust this switch again? It does show 1.4.6 firmware on the default settings, and the MTU on all the ports is 1528. Does board Rev. C have enough horsepower for well over 100M sustained throughput?

I just re-loaded the config from a backup and it took it. Changed the MTU's again, and we'll see how t does with no load. I have a newer model coming from Streakwave, but that will be next week. Id like to re-deploy this if it will handle that traffic level and IPv6.

Thanks again,

Doug

[Julian]

Hi Doug,

There is no problem with your switch core. A likely explanation of this failure is that the flash got corrupted somehow. There is a very slim probability that this can happen at any time due to a read/write error, a page refresh error, etc. We know that this is an unavoidable circumstance, hence the inclusion of a recovery partition and a hardware method of flash recovery.

As an aside, the reason redboot can't find a NAND device is because there is no NAND device on board; Our flash is SPI, since it offers increased reliability among other things.

I wouldn't start mistrusting the device just yet, sir. If it develops a habit of boogering itself during flash writes, that's another story, but the set of circumstances you're describing give no indication that this is anything other than a glitch. One final thing you can do to rule something else out is upgrade to 1.4.7rc7, default the switch, and re-import your configuration: if there is anything in config that's not agreeing with your switch, we've put measures in place to find them in the latest RC.

Hope this helps,

Julian

[sirhc]

Julian is correct on several things:
1) We do not have NAND Flash, this message is because we use the STOCK bootloader provided by Vitesse that looks for all types of supported flash.

2) The flash probably got corrupted and holding the default button in while applying power for 20 seconds causes the switch to reformat the JFFS2 partition of flash that can correct for any blocks that have gone bad bad on the flash whereas a normal default will not.

Not all factory defaults are created equally

Press the factory default button and hold for 20 seconds when powering up
This feature reformats the writable partition where the config is stored then recreates a new config with the default, this process is actually handled but the boot loader. If you ever see an error or warning on the console during initial boot or in the Linux system log about the JFFS2 filesystem such as a checksum issue this is the only way to correct that issue. Now the JFFS2 file system is pretty resilient and even if it gets corrupted 99.9% of the time you would never have an issue but if you really want the error message to go away you have to do this type of factory default, just remember to backup your config or write it down first as with any factory default your config is lost!

WHEN PERFORMING THIS DEFAULT YOU MUST LET THE SWITCH SIT FOR SEVERAL MINUTES

Does board Rev. C have enough horsepower for well over 100M sustained throughput?

The only things that changes between board Revs are things that have nothing to do with switching Capacity.
ALL OF OUR SWITCHES USE THE SAME SWITCH CORE FROM THE WS-6-MINI TO THE WS-24-400A.

They all use the VSC-7427 switch core from Vitesse < You can Google this switch core and read about it
The switch core is capable of line speed on all ports at the same time which is a 26 Gbps switch core.
https://ethernet.microsemi.com/download ... er=vsc7427

[dougncbp]

Thanks, everyone. I will try the beta and re-deploy as soon as I can! But in any case, we have a WS-12-400 -AC coming. Need a good spare, and the old switch can serve that purpose nicely.

[sirhc]

Since you have switch out of service you could apply the grounding MOD if you're good with soldering iron?

Here is the post describing how to do it. viewtopic.php?f=17&t=2493&p=17453&hilit=MOD#p17448

I will post up pictures Monday on all models missing in that post

[dougncbp]

Yeah, I might look into this. I was a bench tech for 13 years at an electronics plant, so I have solder in my veins...

On the new switch (WS-12-400-AC), how can I tell the manufacturing date? It came with 1.3.8 firmware, which I of course upgraded. Hopefully I can also use the config file from the WS-12-250B since the port count is equal.

[Julian]

manufacture date should be on the barcode label on the box it came in.

Failing that, give us a MAC, we'll give you date of MFG.

YMMV on the config interchangeability.. perform this operation at own risk, and preferably on FW version 1.4.7rc7 or later (not that later's out yet)

[dougncbp]

OK, checked the box and the mfg. date is 2/18/16, so I guess the new switch needs a mod as well. Once I get the mod done on the old switch (whenever Chris posts a shot of it), then I could swap it out again and maybe even RMA it to have the mod made since it's still in warranty.