Description

MPC7E/8E/9E ukernel could crash upon an HMC failure leading to an XTXN idling/timeout because some ukernel thread (the exact thread could be nearly any) holds the CPU for longer than SCHED_MAX_RUN_TIME, so the watchdog aborts the thread and the entire ukernel crashes.

That type of HMC failure could happen in the field and it's handled by CMERROR toolkit, which (depending on the exact CMERROR event) usually disables the PFE.

Because of this watchdog initiated ukernel crash, the failure cannot be handled proprely.

Symptoms

The signature logs are below. FPC will crash and be automatically rebooted.

 

May 8 17:53:43 device_name fpc0 EA[0:0]-1 Congestion Detected, Active Zones:0:0:1:f:f:f:f:0:3:f:f:f:8:e:0:0:0:0:0:0:0:0:0:0 
May 8 17:53:43 device_name fpc0 EA[0:0]-1 Congestion Detected, Active Zones:0:0:1:f:f:f:f:0:3:f:f:f:8:e:0:0:0:0:0:0:0:0:0:0 
May 8 17:53:43 device_name ine0[26806]: Number of tftp connections at max limit (1)
May 8 17:53:43 device_name fpc0 MQSS(0): DRD: Error: Fabric reorder ID timeout error - Valid 1, Reorder ID 79 
May 8 17:53:43 device_name fpc0 listmgr_host_xtxn_idle(1526): EA[0:0].llm: Host XTXN 14 busy waiting for Response, fcode_addr reg 0x000011484ed8bc00. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 0 fcode 0x00001148fc00f7b5 status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 1 fcode 0x00001148fccecd31 status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 2 fcode 0x00001148fc00f79d status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 3 fcode 0x00001148fccf7257 status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 4 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 5 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 6 fcode 0x08208000000002e0 status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 7 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 8 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 9 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 10 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 11 fcode 0x0000000000000000 status 0x00000000 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 12 fcode 0x0000002cfbf80005 status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 13 fcode 0x00001450fc309558 status 0x00000100 wr_posted 0. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 14 fcode 0x000011484ed8bc00 status 0x00000003 wr_posted 1. 
May 8 17:53:43 device_name fpc0 show_llm_host_xtxn_dbg(1303): EA[0:0].llm Host_XTXN 15 fcode 0x00000038fbf800f6 status 0x00000600 wr_posted 0. 
May 8 17:53:43 device_name fpc0 listmgr_host_xtxn_exec_chk_status(1758): (EA[0:0].llm xtxn 14, fcode 0x48, addr 0x4ed8bc00, xmt/resp 1/0) 
May 8 17:53:43 device_name fpc0 luss_write_dmem_internal(1464): listmgr_host_xtxn_exec_chk_status error. 
May 8 17:53:43 device_name fpc0 Failed to write init val to capture buf at 0x4ed8bc00. 
May 8 17:53:43 device_name fpc0 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/1/79, Data = d104f0200000000000000000000000000000 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 72 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 73 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(674): EA[0:0].disp[3] PRIMARY_TIMEOUT (PPE 1 Zone 9). 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 48 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 49 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(674): EA[0:0].disp[2] PRIMARY_TIMEOUT (PPE 1 Zone 11). 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(681): EA[0:0].disp[2] SECONDARY_TIMEOUT (PPE 1 Zone 5). 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 24 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 25 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 26 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 27 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 28 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 29 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 30 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 31 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 32 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 33 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 34 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 35 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 36 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 37 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 38 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 39 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 40 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 41 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 42 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 43 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(674): EA[0:0].disp[1] PRIMARY_TIMEOUT (PPE 19 Zone 31). 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(681): EA[0:0].disp[1] SECONDARY_TIMEOUT (PPE 3 Zone 16). 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 0 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 ppe_error_interrupt(4802): EA[0:0]_PPE 1 Errors thread timeout error 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(674): EA[0:0].disp[0] PRIMARY_TIMEOUT (PPE 1 Zone 30). 
May 8 17:53:43 device_name fpc0 dispatch_event_handler(681): EA[0:0].disp[0] SECONDARY_TIMEOUT (PPE 1 Zone 30). 
May 8 17:53:43 device_name fpc0 Error: /fpc/13/pfe/0/cm/0/EA[0:0]/0/EACHIP_CMERROR_HMCIF_RX_LINK_INT_REG_HMC_FATAL_ERR (0x2400c1), scope: pfe, category: functional, severity: major, module: EA[0:0], type: HMCIF RX link int reg HMC fatal err 
May 8 17:53:43 device_name fpc0 Performing action get-state for error /fpc/13/pfe/0/cm/0/EA[0:0]/0/EACHIP_CMERROR_HMCIF_RX_LINK_INT_REG_HMC_FATAL_ERR (0x2400c1) in module: EA[0:0] with scope: pfe category: functional level: major
May 8 17:53:46 device_name fpc0 jnh_read_cmem_internal(1318): Failed jnh_pread_cmem inst 0, lkup_idx 0 addr 0x40840806.0

Solution

No workaround becasue FPC will crash and automatically rebooted.

Modification History

2024-05-13 : Article Created