Description

This article explains the cause and recommended steps to address an issue where a Flexible PIC Concentrator (FPC) goes offline due to a CLI command. This event typically occurs when an FPC is manually brought offline, either intentionally for maintenance or by an unintended command on the MX or PTX router which is running with  Junos and Junos EVO versions 

Symptoms

When an FPC is taken offline due to a CLI command, you may observe the following:

- The FPC appears in an “Offline” state in the output of `show chassis fpc`.
- System logs show messages such as:

17:42:22.873 Router-re0 mgd[1122]: UI_CMDLINE_READ_LINE: User 'ABCDEFGH', command 'request chassis fpc offline slot 1 '
17:42:22.877 Router-re1 sysman[11681]: SYSTEM_NODE_CONFIG_OFFLINE_EVENT: Node configured offline fpc1
17:42:22.878 Router-re0 hwdre[15454]: fru_state_transition /Chassis[0]/Fpc[1] ONLINE -> TRANSITION_OFFLINE
17:42:36.189 Router-fpc0 evo-aftmand-bt[18442]: [Info] Pfe: controller pending delete for fru:/Chassis[0]/Fpc[1]/Pic[0] i2cid:0xaf8 state:TRANSITION_OFFLINE
17:44:01.204 Router-re1 CROND[27282]: (root) CMD (/bin/bash /usr/sbin/manage_fpc_logs.sh)
17:44:01.205 Router-re1 CROND[27283]: (root) CMD (  test ! -f /tmp/.pkid_sync_done && /usr/sbin/pkid-sync.sh > /dev/null 2>&1)
17:44:01.381 Router-re0 CROND[8029]: (root) CMD (/bin/bash /usr/sbin/manage_fpc_logs.sh)
17:44:01.382 Router-re0 CROND[8030]: (root) CMD (  test ! -f /tmp/.pkid_sync_done && /usr/sbin/pkid-sync.sh > /dev/null 2>&1)
17:44:02.252 Router-re0 hwdre: HWD_FRU_OFFLINE_NOTICE: FRU offline fpc1
17:44:02.252 Router-re0 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruOffline for /Chassis[0]/Fpc[1]
17:44:02.252 Router-re0 hwdre[15454]: fru_state_transition /Chassis[0]/Fpc[1] TRANSITION_OFFLINE -> OFFLINE
17:44:02.252 Router-re0 hwdre[15454]: fru_state_transition /Chassis[0]/Fpc[1] OFFLINE -> POWER_OFF_START
17:44:02.253 Router-re0 kernel: jnx-connector fpc1: active:Feb 3 13:2
17:44:02.266 Router-re0 kernel: jnx-connector fpc1: active:Feb 3 13:2 -> present:Feb 3 13:2
17:44:02.266 Router-re0 hwdre[15454]: process_change_event Fru name /Chassis[0]/Fpc[1] get_attr enable state: 0 fruState: 17
17:44:02.449 Router-re0 kernel: jnx-connector fpc1: present:powerbad_ev
17:44:02.449 Router-re0 kernel: jnx-connector fpc1: present:powerbad_ev -> present:powerbad_ev
17:44:02.254 Router-fpc0 evo-aftmand-bt[18442]: [Info] Pfe: controller modify for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOffStart dmf block mode:1
17:44:02.255 Router-fpc0 evo-aftmand-bt[18442]: [Info] Pfe: controller add for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOffStart dmf block mode:1
17:45:30.071 Router-re0 hwdre: HWD_FRU_SNMP_TRAP_NOTICE: SNMP trap generated: jnxFruPowerOn for /Chassis[0]/Fpc[1]
17:45:30.071 Router-re0 hwdre[15454]: fru_state_transition /Chassis[0]/Fpc[1] POWER_ON_START -> POWER_ON
17:45:30.105 Router-fpc2 hwdfpc[16353]: Unsupported Pmb detected
17:45:30.570 Router-fpc0 evo-aftmand-bt[18442]: [Info] Pfe: controller modify for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOn dmf block mode:1
17:45:30.571 Router-fpc0 evo-aftmand-bt[18442]: [Info] Pfe: controller add for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOn dmf block mode:1
17:45:30.570 Router-fpc2 evo-aftmand-bt[18370]: [Info] Pfe: controller modify for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOn dmf block mode:1
17:45:30.570 Router-fpc2 evo-aftmand-bt[18370]: [Info] Pfe: controller add for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOn dmf block mode:1
17:45:30.570 Router-fpc3 evo-aftmand-bt[18365]: [Info] Pfe: controller modify for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOn dmf block mode:1
17:45:30.570 Router-fpc3 evo-aftmand-bt[18365]: [Info] Pfe: controller add for fru:/Chassis[0]/Fpc[1] i2cid:0xd07 type:Fpc state:PowerOn dmf block mode:1

Solution

An FPC can be brought offline via CLI for various reasons:
- Maintenance: Planned maintenance operations may require taking an FPC offline.
- Diagnostics: For testing or troubleshooting, an FPC may be taken offline to isolate issues.
- Accidental Command Execution: An administrator may accidentally execute the command to bring an FPC offline.

The CLI command responsible is:
>request chassis fpc slot <fpc-slot-number> offline

Step 1: Verify the FPC Status
1. Check the current status of all FPCs to identify if an FPC is offline.
   >show chassis fpc
 Confirm that the affected FPC shows as “Offline.”

Step 2: Check System Logs for Context
2. Review the system logs to identify when and why the FPC was taken offline.
   >show log messages | match "fpcX offline"
 Replace `X` with the FPC slot number to filter relevant entries. Look for any messages or indications of the command setting the FPC offline.

Step 3: Bring the FPC Back Online
3. If the FPC was taken offline unintentionally, you can bring it back online with the following command:
   >request chassis fpc slot <fpc-slot-number> online

   Example:
   >request chassis fpc slot 1 online
This command reinitializes the FPC and restores it to service. The system may prompt for confirmation, so type `yes` to proceed.

Step 4: Confirm FPC Online Status
4. Verify the FPC is online and operational.
   >show chassis fpc
 The output should display the FPC in an “Online” state, confirming successful restoration.
 
Additional Commands
-Monitor System Logs:
  >show log messages

-Check Environmental and Power Conditions:
  >show chassis environment   

Modification History

2024-11-06 : Article Created