This article explains the PIC failure and course of action for PIC failure alarm on MPC4E and MPC3D. This KB is intended only for MPC4E and MPC 3D where the PICs are built-in.
Below are some of the symptoms seen in case if a built-in PIC fails on MPC4E and MPC3D.
The PIC status would show up 'Hardware Error' for the problematic PIC.
router-name> show chassis fpc pic-status 4
Slot 4 Online MPC4E 3D 32XGE
PIC 0 Present 8X10GE SFPP- Hardware Error
PIC 1 Online 8X10GE SFPP
PIC 2 Online 8X10GE SFPP
PIC 3 Online 8X10GE SFPP
Below Major alarm would appear.
router-name> show chassis alarms
1 alarms currently active
Alarm time Class Description
2024-10-18 07:43:56 UTC Major FPC 4 PIC 0 Failure
Oct 18 07:42:15 router-name mib2d[18536]: %DAEMON-3-MIB2D_COUNTER_DECREASING: pfes_stats_delta: counter PFES_TRAFFIC_BAD_RT decreasing for type(27), new(0) < old(408)
Oct 18 07:42:15 router-name mib2d[18536]: %DAEMON-3-MIB2D_COUNTER_DECREASING: pfes_stats_delta: counter PFES_TRAFFIC_FABRIC_INPUT_PACKETS decreasing for type(78), new(1362) < old(242425519)
Oct 18 07:42:15 router-name mib2d[18536]: %DAEMON-3-MIB2D_COUNTER_DECREASING: pfes_stats_delta: counter PFES_TRAFFIC_FABRIC_OUTPUT_PACKETS decreasing for type(81), new(260) < old(242419670)
Oct 18 07:43:56 router-name chassisd[18530]: %DAEMON-4-CHASSISD_PIC_HWERROR: PIC 0 in FPC 4 (PIC type 2641, version 0) had hardware error
Oct 18 07:43:56 router-name alarmd[18697]: %DAEMON-4: Alarm set: PIC id=16778359, color=RED, class=CHASSIS, reason=FPC 4 PIC 0 Failure
Oct 18 07:43:56 router-name craftd[18533]: %DAEMON-4: Major alarm set, FPC 4 PIC 0 Failure
Oct 18 07:43:56 router-name : %PFE-3: fpc4 cmic_plat_firmware_init: CMIC(4/0) NOT all vsc8248s is ready.
Oct 18 07:43:56 router-name : %PFE-3: fpc4 CMT: PIC 0 failed PIC_POST_INIT, XM/MQ chip asic id 32
Oct 18 07:43:56 router-name : %PFE-3: fpc4 cmic_pic_cmd: CMIC(4/0) - does not support offline !!
Oct 18 07:43:56 router-name : %PFE-3: fpc4 PIC: command error on detach (7)
Oct 18 07:43:56 router-name : %PFE-3: fpc4 CMT: Error initializing PIC 0, ID 0x0a51
Oct 18 07:44:05 router-name : %PFE-3: fpc4 XMCHIP(0): xmchip_xxlce_mtip_cge_inst_mdio_c45_read: Couldn\'t get the XXLCE MAC handler - mdio_control_num 0
Oct 18 07:44:05 router-name kernel: %KERN-1-DFW_PFE: msg type 11 subtype 18 opcode 169 failed (err 5)
Oct 18 07:44:05 router-name last message repeated 8 times
Oct 18 07:44:05 router-name : %PFE-3: fpc4 cmic_vsc8248_mdio_rd: CMIC(4/1) failed , xminstance=0, paddr=0xf, dev_addr = 0x1, status = 15. port_group=0
Oct 18 07:44:05 router-name : %PFE-3: fpc4 vsc8248_reg_read: Error in vsc8248 register read
Oct 18 07:44:05 router-name : %PFE-3: fpc4 vsc8248_mode_set: cmic-vsc8248-4/0/2: vsc8248 reg read failed, channel:3 device:0x1 register:0x801b
Oct 18 07:44:05 router-name : %PFE-3: fpc4 vsc8248_init_channel: cmic-vsc8248-4/0/2: error in configuring the clock and data rate channel:3
Oct 18 07:44:05 router-name : %PFE-3: fpc4 CMIC(4/1): Failed to initialize the VSC8248: 2 Channel: 3, corresponding to lane: 0
Oct 18 07:44:05 router-name : %PFE-3: fpc4 cmic_phy_dev_init: CMIC(4/1) -- Failed in cmic_phy_vsc8248_dev_init. ret=0x1.