Description

This article describes Error code (0x2c0002) which may be seen on MX FPCs: MPC7E/MPC8E/MPC9E.

Symptoms

When FPC boots up, it may report a Major Alarm with error code: 0x2c0002. This is due to the Linux (running on certain Juniper FPCs) not being able to detect the storage device in the FPC (msata device). Depending on the Junos version running on the device, the error code may or may not be written under the output of 'show chassis alarms' .

test@router1> show chassis alarms 
1 alarms currently active 
Alarm time Class Description 
2017-10-11 01:05:05 BST Major FPC 7 Major Errors
test@router2> show chassis alarms 
1 alarms currently active
Alarm time               Class  Description
2017-08-20 22:52:00 PDT  Major  FPC 4 Major Errors - Storage Error code: 0x2c0002


If the Error code is not shown from the chassis alarms output, you can login to the FPC shell and find the error code:

test@router1> start shell pfe network fpc7
SMPC7(router1 vty)# show cmerror level
Level  Count Occured Cleared  Threshold   R-Limit  Action
-------------------------------------------------------------
Minor      0       0       0          10      10  log
Major      1        1       0           1      10  get-state cmalarm  
Fatal      0       0       0           1      10  reset
 
SMPC7(router1 vty)# show cmerror module 5
 
Module (5) (Storage Device )
 
Error-id  PFE  Level  Threshold  Count  Occured  Cleared  Last-occurred(ms ago)  Description
--------------------------------------------------------------------------------------------
0x2c0002   0     Major   1           1       1        0        22376687               Storage failed/disabled
0x2c0001  0    Minor  1          0      0        0        0                      Storage Pre-Fail error

 
SMPC7(router1 vty)# show cmerror module 5 error 0x2c0002
 
Error-id              : 0x2c0002
Description           : Storage failed/disabled
PFE                   : 0
Level                 : Major
Count                 : 1
Threshold             : 1
Error Limit           : 1
Occur Count           : 1
Clear Count           : 0
Last-occurred(ms ago) : 22387006
 
Logs:
----------------------------------------------------------
Index  Time                 Sub-Err   State    Description
----------------------------------------------------------
0      10/11/17 03:14:52    0         Set       Msata fail alarm
 

Checking the bootenable from the linux shell of the FPC, you'll find that the msata is missing:

test@router1> start shell
% rsh -Ji fpc7
rt002:~$
rt002:/sys/devices/platform/ngpmb-nvram$ cat bootenable
usb
network
<<<<<<<<<<<<<<   msata is missing here
msata2

 

Comparing the bootenable with what it should be from another router:

router3-re0-fpc5:~$ cat /sys/devices/platform/ngpmb-nvram/bootenable
efi
usb
network
msata   
msata2

Note: The same FPC error code can be seen on PTX FPC3-PTX-U2 or FPC3-PTX-U3 while the recovery method is different and is not covered in this KB article.

Solution

The solution for this issue is to add the storage device (msata device) to the bootenable and reboot the FPC. The FPC will then come online with the storage device (msata) detected and alarm will be cleared. This will be done from the linux shell of the FPC:

root@router1-re0>start shell 
%rsh –Ji fpc7
router1-re0-fpc7:~$ cd /sys/devices/platform/ ngpmb-nvram
router1-re0-fpc7:/sys/devices/platform/ ngpmb-nvram $ echo usb network msata msata2 > /sys/devices/platform/ngpmb-nvram/bootenable


After that, restart the FPC and it should come up with the correct bootenable.

In later Junos versions; you may not find out ngpmb-nvram, but instead stout-lpc. In this case, perform the same steps for adding the msata to the bootenable

 root@nanostring-fpc8:~# echo usb network msata msata2 >    /sys/devices/platform/stout-lpc/nvram-bios/bootenable
 root@nanostring-fpc8:~# reboot


Finally, in case for any reason adding the msata device failed due to permission denial, access the shell using root user (you'll need the root password in this case):

root@router1-re0>start shell user root

Modification History

2020-04-17: Updated router and user names with generic naming convention.