The intention of the article is to provide a troubleshooting approach when FPC is stuck in Ready state and how to resolve it. 

show chassis fpc            

Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)

State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer

0  Online            60     34          3       34     35     35    3168       59         29

1  Ready             39    100          0       99     99     99    3168        6         17

FPC1 had minor errors ,  A proactive reboot of the FPC was performed which led to FPC stuck in Ready state and wasn’t transitioning to Online state. Reseat of the FPC card was tried but that didn’t help . A new card was put on the slot but that too didn’t help. However, when the same card was put on a different slot the card came online.

From the captured logs there was no abnormality observed . Below logs were captured during the initial troubleshooting. 

• show log messages 
• show log chassisd 
• start shell pfe network fpc1
• show syslog messages 

To understand the issue further in a separate MW with tech on site . We wanted to collect the console logs from the working and non-working FPC and compare it .  Collect the below logs for reference. 

• Collect the FPC1 VARLOGS first for further review .

https://supportportal.juniper.net/s/article/How-to-collect-linux-based-FPC-logs

• Request the tech on site to reseat the FPC and collect the console logs from the FPC . Upon collecting no abnormal logs were noticed . MPC was initialized . 

show chassis fpc

Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)

State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer

1  Ready   

start shell user root

cty -f fpc1 

[Jan 22 09:03:56.514 LOG: Info] eachip_init(3615): Create EA Background Service Thread.

[Jan 22 09:03:56.517 LOG: Info] cchip_init:478 Initialize center chip 2

[Jan 22 09:03:56.747 LOG: Debug] Allocated memory for 90 channels MQSS(2)

[Jan 22 09:03:56.749 LOG: Info] chan_count 90 num_chan 90 

[Jan 22 09:03:56.749 LOG: Debug] [mqss_ccl_init:214] asic_id[62]

[Jan 22 09:03:56.749 LOG: Debug] [mqss_ccl_init_instance_tbl:107] 

[Jan 22 09:03:56.749 LOG: Debug] [mqss_ccl_init_link_map_tbl:68] 

[Jan 22 09:03:56.752 LOG: Info] Installing CCL vector version: 0x20000

[Jan 22 09:03:56.753 LOG: Info] MQSS(2): Initialized successfully

• Next , check the fabric links and it was noticed that for FPC1 the fabric links were not online. It noticed that FPC1 wasn’t using the fabric link . This meant that there was some communication problem with the SPMB0 and FPC1 . 

start shell pfe network fpc1  

 SMPC platform (1750Mhz Intel(R) Atom(TM) CPU processor, 3168MB memory, 8192KB flash)

 SMPC1(vty)# show fabric state

 Legends ++++++++++++++++++++++++++

 FM_HSL2_LINK_UNUSED        [0]

 FM_HSL2_LINK_OK            [1]

 FM_HSL2_PLANE_ENABLED      [2]

 FM_HSL2_PLANE_DISABLED     [3]

 FM_HSL2_LINK_ERROR         [4]

 FM_HSL2_PLANE_ENABLE_ERROR [5]

 FM_HSL2_PLANE_DEST_ERROR   [6]

 FM_HSL2_PLANE_SELF_PING_OK [7]

 FM_HSL2_PLANE_SELF_PING_ERROR [8]

 plane_status/link_status ++++++++

plane_status/link_status ++++++++

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.943:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.944:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.945:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.947:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.948:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.949:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.951:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.952:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.953:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.955:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.957:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.958:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.960:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.961:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.962:  [0/0] [0/0]

 [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0] [0/0]021530.964:  [0/0] [0/0]

show chassis fabric fpcs

Fabric management FPC state:

FPC 0

  PFE #0

      Plane 0: Plane enabled

      Plane 1: Plane enabled

      Plane 2: Plane enabled

      Plane 3: Plane enabled

      Plane 4: Plane enabled

      Plane 5: Plane enabled

      Plane 6: Plane enabled

      Plane 7: Plane enabled

  PFE #1

      Plane 0: Plane enabled

      Plane 1: Plane enabled

      Plane 2: Plane enabled

      Plane 3: Plane enabled

      Plane 4: Plane enabled

      Plane 5: Plane enabled

      Plane 6: Plane enabled

      Plane 7: Plane enabled

  PFE #2

      Plane 0: Plane enabled

      Plane 1: Plane enabled

      Plane 2: Plane enabled

      Plane 3: Plane enabled

      Plane 4: Plane enabled

      Plane 5: Plane enabled

      Plane 6: Plane enabled

      Plane 7: Plane enabled

  PFE #3

      Plane 0: Plane enabled

      Plane 1: Plane enabled

      Plane 2: Plane enabled

      Plane 3: Plane enabled

      Plane 4: Plane enabled

      Plane 5: Plane enabled

      Plane 6: Plane enabled

      Plane 7: Plane enabled

FPC 2

  PFE #0

• To confirm if the problem is with SPMB 0 perform a non-graceful switchover to RE1 so all the FPC’s will converge to the SPMB1 . Upon doing that we saw FPC1 came online.

user@re0> request chassis routing-engine master switch          

warning: Traffic will be interrupted while the PFE is re-initialized

Toggle mastership between routing engines ? [yes,no] (no) yes 

user@re1> show chassis fpc    

                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)

 Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer

031735.347:   0  Online            47     15          0       16     10      4    3168       14         25

 1  Online            49     96          0       41     11      4    3168       15         25

 2  Online            48     17          0       20      9      3    3168       14         25

 3  Online            51     16          0       24     11      4    3168       14         25

 4  Online            49     16          0       15      9      4    3168       14         25

 5  Online            48     14          0       15      8      3    3168       14         25

 6  Offline         ---Configured power off---

 7  Online            27    100          0       16      3      1    2048       13         21

 8  Online            35     21          0       20      8      3    3584        9         25

 9  Online            36     22          0       21      8      3    3584        9         25

• From the above we were able to confirm that issue was not with the card but with the communication between SPMB0 and FPC1 seems to be broken . 
• Switchover to RE0 again to see if that solves the communication problem or not but first reseat the CB0 .
• Once CB0 came online attempt the  switchover again from RE1 ro RE0 , FPC1 came online with RE0 being the master .

user@re1> request chassis routing-engine master switch          

warning: Traffic will be interrupted while the PFE is re-initialized

Toggle mastership between routing engines ? [yes,no] (no) yes 

user@re0> show chassis fpc    

                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)

 Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer

031735.347:   0  Online            47     15          0       16     10      4    3168       14         25

 1  Online            49     96          0       41     11      4    3168       15         25

 2  Online            48     17          0       20      9      3    3168       14         25

 3  Online            51     16          0       24     11      4    3168       14         25

 4  Online            49     16          0       15      9      4    3168       14         25

 5  Online            48     14          0       15      8      3    3168       14         25

 6  Offline         ---Configured power off---

 7  Online            27    100          0       16      3      1    2048       13         21

 8  Online            35     21          0       20      8      3    3584        9         25

 9  Online            36     22          0       21      8      3    3584        9         25

• If the above steps also doesn’t help to solve the issue then plan to replace the CB .
• If that too doesn’t help then the problem could be related to the midplane.