This article explains why the FPC goes offline, due to unreachable destinations errors and how to recover the services.
Lets consider, we insert a faulty MPC3E card on the slot 5 of the MX chassis, then we might see FPC5 has unreachable destinations errors and the FPC goes offline with the below log messages:
Nov 06 08:44:03 send: yellow alarm set, device CB 0, reason Check CB 0 Fabric Chip 0
Nov 06 08:44:03 send: yellow alarm set, device CB 1, reason Check CB 1 Fabric Chip 0
Nov 06 08:44:06 send: red alarm set, device FPC 5, reason FPC 5 has unreachable destinations
Nov 06 08:44:06 FH: fm_hsl2_mx_detect_fabric_failures: FPC 5 has started blackholing
Nov 06 08:44:06 CHASSISD_FM_FABRIC_DEGRADED: DPCs are seeing grant timeouts; System is blackholing
Nov 06 08:44:06 FH: If fabric does not come up within ten seconds, fabric down actions will be initiated
Nov 06 08:44:21 send: red alarm set, device FPC 5, reason FPC 5 offlined due to unreachable destinations
Nov 06 08:44:21 fpc_disconnect_generic: fpc 5 slc 0 state Online cargs 0x9bf3be0 clean_shutdown 1, offline_reason=FPC offlined due to unreachable destinations
Nov 06 08:44:23 CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 6, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MPC3E NG PQ & Flex Q @ 5/*/*, jnxFruType 3, jnxFruSlot 5, jnxFruOfflineReason 67, jnxFruLastPowerOff 1710383844, jnxFruLastPowerOn 1710363757)
When the fabric has traffic destined to another FPC, it sends out a fabric request to the destination hardware asking for permission or bandwidth to pass the traffic to the destination. Sometimes there will not be any response from the destination hardware due to some error conditions either on the source hardware or the destination. Then the chassis makes an attempt to self-heal the link between the source and the destination FPC.
If the recovery attempt fails, then the alarm might be seen on the device, pointing out the link failure seen on the source and destination hardware.
Nov 06 08:44:06 send: red alarm set, device FPC 3, reason FPC 3 has unreachable destinations
Nov 06 08:44:06 send: red alarm set, device FPC 4, reason FPC 4 has unreachable destinations
Remember that the message is being reported from the source hardware only, and not the destination. The below syslog messages indicate that there is a link error between the source hardware and destination hardware.
Nov 06 08:44:00 <hostname> fpc3 CMTFPC: fabric fab_dest 400 for FPC 3 PFE: 1 fabric plane 7 saw dest-err in stream 148
Nov 06 08:44:00 <hostname> fpc3 MQSS(0): FO: Request timeout error - Number of timeouts 11, RC select 5, Stream 148
Nov 06 08:44:00 <hostname> fpc3 MQSS(1): FO: Request timeout error - Number of timeouts 7, RC select 5, Stream 148
Nov 06 08:44:00 <hostname> fpc4 CMTFPC: Fabric request time out pfe 0 plane 6 fab_stream 148, attempting recovery
Nov 06 08:44:00 <hostname> fpc4 CMTFPC: Fabric request time out pfe 0 plane 5 fab_stream 148, attempting recovery
Nov 06 08:44:00 <hostname> fpc4 CMTFPC: Fabric request time out pfe 0 plane 4 fab_stream 148, attempting recovery
Nov 06 08:44:00 <hostname> fpc4 CMTFPC: fabric fab_dest 400 for FPC 4 PFE: 0 fabric plane 0 saw dest-err in stream 148
Nov 06 08:44:00 <hostname> fpc4 CMTFPC: fabric fab_dest 400 for FPC 4 PFE: 0 fabric plane 2 saw dest-err in stream 148
Nov 06 08:44:00 <hostname> fpc4 CMTFPC: fabric fab_dest 400 for FPC 4 PFE: 0 fabric plane 7 saw dest-err in stream 148
KB35971 [juniper.net]: https://supportportal.juniper.net/s/article/Syslog-message-fpc-CMTFPC-Fabric-request-time-out-pfe-plane-attempting-recovery
During the time of FPC removal/insertion, if the hardware is faulty or when the destination hardware is congested we might see the above issue.
Packet Forwarding Engines (PFEs) usually send traffic to each other using high-speed links (HSLs). When the fabric has traffic destined to another FPC, it sends out a fabric request and if no response from the destination hardware received, then fabric self-healing mechanism will initiate to recover the link between the source and the destination FPC.
In this case, we may need to collect the below logs to check the fabric plane that is affected:
· show chassis fabric fpcs
· show chassis fabric plane
· show chassis fabric plane-location
· show chassis fabric map
· show chassis fabric summary
· show chassis fabric summary extended
· show chassis fabric destinations
When you find the affected plane from the above output collected, then we may need to restart the fabric plane & FPC to resolve the issue.
· >request chassis fabric plane # offline
· >request chassis fabric plane # online
· >request chassis fpc fpc-slotnumber offline
· >request chassis fpc fpc-slotnumber online