This document discusses the causes and actions while having a Routing Engine that fails to boot Junos.
Having a Routing Engine e.g. RE-S-MX104-WW-S or RE-S-1800x4-16G from MX platfom and seeing following bootup which ends with Watchdog line messages.
FreeBSD/PowerPC U-Boot bootstrap loader, Revision 2.4
Memory: 3968MB
/boot/init.4th loaded.
Loading /boot/defaults/loader.conf
/kernel data=0xce4124+0xe10b0 syms=[0x4+0xaaf60+0x4+0x10042d]
Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]...
Kernel entry at 0xa00000c0 ...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
platform_early_bootinit: MX-PPC Series Early Boot Initialization
mxppc_set_re_type: hw.board.type is MX104
WDOG initialized
Copyright (c) 1996-2017, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
JUNOS 16.1R5.7 #0: 2017-08-18 17:36:30 UTC
[email protected]:/volume/build/junos/16.1/release/16.1R5.7/obj/powerpc/junos/bsd/kernels/JUNIPER-PPC/kernel
WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant
Timecounter "decrementer" frequency 37500000 Hz quality 0
cpu0: Freescale e5500 core revision 1.2
cpu0: HID0 c0000080<EMCP,EN_L2MMU_MHD,EN_MAS7_UPDATE>
real memory = 4131389440 (3940 MB)
avail memory = 4055101440 (3867 MB)
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
Security policy loaded: Junos MAC/veriexec (mac_veriexec)
MAC/veriexec fingerprint module loaded: SHA256
MAC/veriexec fingerprint module loaded: SHA1
ETHERNET SOCKET BRIDGE initialising
random: <Software, Yarrow> initialized
Initializing M/T/EX platform properties ..
nexus0: <Powerpc Nexus device>
ocpbus0: <on-chip peripheral bus> on nexus0
openpic0: <OpenPIC in on-chip peripheral bus> iomem 0xfe040000-0xfe06ffff on ocpbus0
uart0: <16550 or compatible> iomem 0xfe11c500-0xfe11c50f irq 52 on ocpbus0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> iomem 0xfe11c600-0xfe11c60f irq 52 on ocpbus0
lbc0: <Freescale Local Bus Controller> iomem 0xfe124000-0xfe12410f,0xff000000-0xffffffff irq 16,17,18 on ocpbus0
uart2: <16750 or compatible> iomem 0xff600000-0xff600fff on lbc0
uart3: <16750 or compatible> iomem 0xff500000-0xff500fff on lbc0
cfi0: <AMD/Fujitsu - 4MB> iomem 0xffc00000-0xffffffff on lbc0
amxbcpld0: <MX104 Bootcpld> iomem 0xff700000-0xff700fff on lbc0
amxbcpld0: MX104 Bootcpld Ver 1 Rev 13 (Built 1-5-13)
i2c0: <MPC85XX OnChip i2c Controller> iomem 0xfe118000-0xfe118014 irq 54 on ocpbus0
ds1672 rtc0: <DS1672 RTC> on i2c0
i2c1: <MPC85XX OnChip i2c Controller> iomem 0xfe118100-0xfe118114 irq 54 on ocpbus0
i2c2: <MPC85XX OnChip i2c Controller> iomem 0xfe119000-0xfe119014 irq 55 on ocpbus0
i2c3: <MPC85XX OnChip i2c Controller> iomem 0xfe119100-0xfe119114 irq 55 on ocpbus0
ehci0: <Frescale Integrated USB 2.0 controller> iomem 0xfe210000-0xfe210fff irq 60 on ocpbus0
usb0: EHCI version 1.0
usb0 on ehci0
usb0: USB revision 2.0
uhub0: Freescale EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1: SMSC USB2513Bi, class 9/0, rev 2.00/b.a0, addr 2
uhub1: multiple transaction translators
uhub1: 3 ports with 3 removable, self powered
ehci_activate_qh: unexpected next ptr
QH(0xd613ad80) at 0xf2f40d80:
sqtd=0xd613bde0 inactivesqtd=0xd613bcc0
link=0xf2f493c2<QH>
endp=0x80012102
addr=0x02 inact=0 endpt=1 eps=2 dtc=0 hrecl=0
mpl=0x1 ctl=0 nrl=8
endphub=0x40811c01
smask=0x01 cmask=0x1c huba=0x01 port=1 mult=1
curqtd=0x00000001<T>
Overlay qTD:
next=0xf2f3fcc0<> altnext=0x00000001<T>
status=0x00000000: toggle=0 bytes=0x0 ioc=0 c_page=0x0
cerr=0 pid=0 stat=0x0
buffer[0]=0x00000000
buffer[1]=0x00000000
buffer[2]=0x00000000
buffer[3]=0x00000000
buffer[4]=0x00000000
QTD(0xd613bde0) at 0xf2f3fde0:
next=0xf2f3fcc0<> altnext=0xf2f3fcc0<>
status=0x00208d00: toggle=0 bytes=0x20 ioc=1 c_page=0x0
cerr=3 pid=1 stat=0x0
buffer[0]=0x002477ec
QTD(0xd613bcc0) at 0xf2f3fcc0:
next=0x00000001<T> altnext=0x00000001<T>
umass0: ATP Electronics AF8GSSGH-JP4, rev 2.00/11.00, addr 3
umass0: SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
bman0 iomem 0xfe31a000-0xfe31afff,0xfd800000-0xfd803fff,0xfd804000-0xfd804fff irq 32,121,123,125,127,129,131,133,135,137,139 on ocpbus0
bman0: Hardware version: 1.0.
qman0 iomem 0xfe318000-0xfe318fff,0xfd808000-0xfd80bfff,0xfd80c000-0xfd80cfff irq 32,120,122,124,126,128,130,132,134,136,138 on ocpbus0
qman0: Hardware version: 1.2.
fman0 iomem 0xfe400000-0xfe4fffff irq 112,32 on ocpbus0
fman0: Hardware version: 3.0.
dtsec0 on fman0
dtsec0: Configured for independent mode.
dtsec0: hardware MAC address 02:00:ff:00:00:04
dtsec1 on fman0
dtsec1: Configured for independent mode.
dtsec1: hardware MAC address 02:00:00:00:00:04
dtsec2 on fman0
dtsec2: Configured for independent mode.
dtsec2: hardware MAC address 02:00:01:00:00:04
dtsec3 on fman0
dtsec3: Configured for independent mode.
dtsec3: hardware MAC address 5c:45:27:67:aa:5f
miibus0: <MII bus> on dtsec3
e1000phy0: <Marvell 88E1111 Gigabit PHY> on miibus0
e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
Initializing product: 152 ..
Setting up M/T interface operations and attributes
platform_cookie_read not implemented
Initializing MX-PPC platform mastership..
Registered AMT tunnel Encap with UDP Tunnel!
Loading Redundant LT driver
Loading Redundant Virtual Interface driver
rvt_ifd_attach: Error setting driver API's for family = 59
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
Registering tcp_platform_dependent = tcp_handle_special_ports
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <ATP AF8GSSGH-JP4 1100> Fixed Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 7678MB (15724544 512 byte sectors: 255H 63S/T 978C)
random: unblocking device.
Kernel thread "wkupdaemon" (pid 49) exited prematurely.
Trying to mount root from ufs:/dev/da0s1a
WARNING: /mnt was not properly dismounted
Attaching /packages/jbase via /dev/mdctl...
Mounted jbase package on /dev/md0...
Verified manifest signed by PackageProductionEc_2017 method ECDSA256+SHA256
veriexec: cannot validate /etc/db/manifest/jboot.ecerts: subject issuer mismatch: /C=US/ST=CA/L=Sunnyvale/O=Juniper Networks/OU=Juniper CA/CN=PackageCA/[email protected]
Verified jbase-ppc-16.1R5.7 signed by PackageProductionEc_2017 method ECDSA256+SHA256
/etc/rc.conf: 30: Syntax error: ")" unexpected
WARNING: system watchdog timer still running,
use '/sbin/watchdog -off' to disable
Enter full pathname of shell or 'recovery' for root password recovery or RETURN for /bin/sh: ^@watchdog: scheduling fairness gone. Too much high-priority workload!
watchdog: scheduling fairness gone for 20 seconds now.
watchdog: scheduling fairness gone for 40 seconds now.
The watchdog messages on this scenario point that Routing Engine bootup has failed, and has at least 1 drive not responding.
For this scenario the probable causes are these:
1- Compact flash is lost or not accesible
2- Hard drive not mounting
3- Routing Engine unexpected error
Solutions that can be done:
1- Reload Routing Engine several times (x3 - 4 times) to hope for a Flash discoverage
2- USB recovery install (steps can be found on KB76493 [juniper.net])
3- Lastly, RMA for the unit is needed
*Note: If by doing the 2 solution you find following messages then is pointing that recovery is not proceeding because of permanent hardware failure, from there open a JTAC ticket to process RMA for the unit.
PARTIALLY ALLOCATED INODE I=45453
CLEAR? yes
CANNOT WRITE BLK: 380864