The customer is seeing below alarm on Mx10003
Router> show chassis alarms 2 alarms currently active Alarm time Class Description 2023-09-11 19:54:44 UTC Minor Check FPC 0 Fabric Chip
Initially, we saw the alarm.
Later, we saw plane 0 going into a check state. There are multiple planes where there might not be a service impact, but keeping the FPC in this state is not advisable.
The most interesting part is that this is the MX10003, and fabric chips are installed in the FPC itself. CB does not have a fabric chip.
So if you see an alarm like above, then you need to look into FPC and not CB.
1) First, go through the Varlogs. If you find anything there, then investigate in the respective direction; otherwise, collect below command output
2) Below is the list of commands to collect.
++++++++++++ Set 1: Chassis ++++++++++++++ Please save them with the file name: chassis-fabric.txt show chassis alarms | no-more show chassis errors active | no-more show chassis fabric midplane | no-more show chassis fabric summary | no-more show chassis fabric summary extended | no-more show chassis fabric fpcs | no-more show chassis fabric plane | no-more show chassis fabric degradation | no-more show chassis fabric destinations | no-more show chassis fabric plane-location | no-more show chassis fabric reachability | no-more show chassis fabric reachability detail | no-more show chassis environment cb | no-more show chassis environment fpc | no-more show pfe statistics error | no-more show pfe statistics traffic fpc 0 | no-more show pfe statistics traffic fpc 1 | no-more
++++++++++++ Set 2: FPC0 shell ++++++++++++++ Please save them with the file name: FPC0-shell.txt start shell pfe network fpc0 show hsl2 statistics show fabric state show fabric plane_map 0 show fabric plane_map 1 show fabric plane_map 2 show fabric ccl statistics <<< Please repeat the command 3 times @ 1 Min interval show fm-hsl2 ccl statistics <<< Please repeat the command 3 times @ 1 Min interval show pfe statistics traffic <<< Please repeat the command 3 times @ 1 Min interval show pfe statistics error <<< Please repeat the command 3 times @ 1 Min interval show ccl statistics detail show ccl errors show mqss 0 fo stats show mqss 1 fo stats show mqss 2 fo stats show mqss 0 fi stats show mqss 1 fi stats show mqss 2 fi stats
++++++++++++ Set 3: FPC0 syslog ++++++++++++++ Please save them with the file name: FPC0-syslog-shell.txt start shell pfe network fpc0 show syslog messages show nvram
++++++++++++ Set 4: FPC1 shell ++++++++++++++ Please save them with the file name: FPC1-shell.txt start shell pfe network fpc1 show hsl2 statistics show fabric state show fabric plane_map 0 show fabric plane_map 1 show fabric plane_map 2 show fabric ccl statistics <<< Please repeat the command 3 times @ 1 Min interval show fm-hsl2 ccl statistics <<< Please repeat the command 3 times @ 1 Min interval show pfe statistics traffic <<< Please repeat the command 3 times @ 1 Min interval show pfe statistics error <<< Please repeat the command 3 times @ 1 Min interval show ccl statistics detail show ccl errors show mqss 0 fo stats show mqss 1 fo stats show mqss 2 fo stats show mqss 0 fi stats show mqss 1 fi stats show mqss 2 fi stats
++++++++++++ Set 5: FPC1 syslog ++++++++++++++ Please save them with the file name: FPC1-syslog-shell.txt start shell pfe network fpc1 show syslog messages show nvram
3) Below is the area where I saw the problems
>> FPC 0, Plane 0, PFE 1 is set with link error. Plane 0 in check state.
show chassis fabric midplane | no-more Fabric management PLANE state Plane 0 Plane state: ACTIVE FPC 0 PFE 0 :Links ok PFE 1 :Link error (TF) show chassis fabric summary | no-more Plane State Uptime 0 Check 3 days, 1 hour, 39 minutes, 55 seconds 1 Online 8 days, 2 hours, 5 minutes, 21 seconds 2 Online 8 days, 2 hours, 5 minutes, 21 seconds
>> FPC 0, Plane 0, PFE 0, 1 and 2 has fabric destination error.
show chassis fabric destinations | no-more Fabric destinations state: 0: non-existent 2: enabled 3: disabled 6: dest-err and disabled FPC 0 PFE 0 Plane 0 22200000 23200000 PFE 1 Plane 0 22200000 33300000 PFE 2 Plane 0 22200000 23200000 show chassis fabric plane-location | no-more ------------Fabric Plane Locations------------- Plane 0 FPC 0 FPC 1
>> “show fabric state” for fpc 1 has some errors but same command output for fpc 0 is clean. I did not see any “3” there.
Router> start shell pfe network fpc1 Sep 15 05:46:09 SMPC platform (1750Mhz Intel(R) Atom(TM) CPU processor, 3168MB memory, 8192KB flash) SMPC1(Router vty)# show fabric state Legends ++++++++++++++++++++++++++ FM_HSL2_LINK_UNUSED [0] FM_HSL2_LINK_OK [1] FM_HSL2_PLANE_ENABLED [2] FM_HSL2_PLANE_DISABLED [3] FM_HSL2_LINK_ERROR [4] FM_HSL2_PLANE_ENABLE_ERROR [5] FM_HSL2_PLANE_DEST_ERROR [6] FM_HSL2_PLANE_SELF_PING_OK [7] FM_HSL2_PLANE_SELF_PING_ERROR [8] pfe[0] plane [0] 23233333 22233333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 23233333 22233333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 pfe[1] plane [0] 33333333 22233333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 22233333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 pfe[2] plane [0] 23233333 22233333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 23233333 22233333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 33333333 ++++++++ FPC 0 ++++++++++++ SMPC0(Router vty)# show fabric plane_map 1 Platform: Summit MX 3RU fabric Link Link Link Plane EA-FI/FO EA PF-local PF-midplane Midplane Midplane plane Type Speed State State Port port/lane port/lane port/lane link state Train state 00 X1P 25G 1 2 16 02/00 ==> 04/00 09/01 ==> 4 (TF) 0 ++++++++ FPC 1 ++++++++++++ SMPC1(Router vty)# show fabric plane_map 1 Platform: Summit MX 3RU fabric Link Link Link Plane EA-FI/FO EA PF-local PF-midplane Midplane Midplane plane Type Speed State State Port port/lane port/lane port/lane link state Train state 00 X1P 25G 1 2 16 02/00 ==> 04/00 09/01 ==> 1 (TF) 1
4) Above logs highlight issue is with FPC 0.
5) Please follow below action in MW to overcome the issue