[Junos EVO] Destination PFE Errors on PCS0/PSC1 Interface

Description

Interfaces on a "PTX10001-36MR" were down due to LACP timeouts.

Symptoms

FPC stared to report ASIC errors and these errors lead to PFE host path wedge condition. Due to the host path wedge condition on FPC0#PFE0 and FPC0#PFE1, LACP timeouts are reported for interfaces hosted on these PFEs.

Message logs were filled with below error logs. BT Chip 0 and Chip 1 reporting below ASIC errors.

Feb 8 17:15:02 2024 labrouter resiliencyd[11932]: Error: /fpc/0/evo-cda-bt/0/cm/0/btchip/0/dp_1_zfi_0_fifo_intr_pcsif_pcs1_req_dest_pfe_err (0x4511f0), scope: pfe, category: functional, severity: minor, module: btchip, type: dp_1_zfi_0_fifo_intr_pcsif_pcs1_req_dest_pfe_err
Feb 8 17:15:02 2024 labrouter resiliencyd[11932]: Cmerror Op Set: btchip: chip_num:0 intr_full_name:dp_1_zfi_0_fifo_intr_status_pcsif_pcs1_req_dest_pfe_err intr_bit_pos:9 (URI: /fpc/0/evo-cda-bt/0/cm/0/btchip/0/dp_1_zfi_0_fifo_intr_pcsif_pcs1_req_dest_pfe_err)
Feb 8 17:15:03 2024 labrouter resiliencyd[11932]: Performing action log for error /fpc/0/evo-cda-bt/0/cm/0/btchip/0/dp_1_zfi_0_fifo_intr_pcsif_pcs1_req_dest_pfe_err (0x4511f0) in module: btchip with scope: pfe category: functional level: minor
Feb 8 17:15:03 2024 labrouter resiliencyd[11932]: Cmerror Op Clear: btchip: chip_num:0 intr_full_name:dp_1_zfi_0_fifo_intr_status_pcsif_pcs1_req_dest_pfe_err intr_bit_pos:9 (URI: /fpc/0/evo-cda-bt/0/cm/0/btchip/0/dp_1_zfi_0_fifo_intr_pcsif_pcs1_req_dest_pfe_err)
…

Feb 8 17:15:05 2024 labrouter resiliencyd[11932]: Error: /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_zfi_1_fifo_intr_pcsif_pcs1_data_dest_pfe_err (0x451201), scope: pfe, category: functional, severity: minor, module: btchip, type: dp_1_zfi_1_fifo_intr_pcsif_pcs1_data_dest_pfe_err
Feb 8 17:15:05 2024 labrouter resiliencyd[11932]: Performing action log for error /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_zfi_1_fifo_intr_pcsif_pcs1_data_dest_pfe_err (0x451201) in module: btchip with scope: pfe category: functional level: minor
Feb 8 17:15:05 2024 labrouter resiliencyd[11932]: Cmerror Op Set: btchip: chip_num:1 intr_full_name:dp_1_zfi_1_fifo_intr_status_pcsif_pcs1_data_dest_pfe_err intr_bit_pos:5 (URI: /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_zfi_1_fifo_intr_pcsif_pcs1_data_dest_pfe_err)
Feb 8 17:15:06 2024 labrouter resiliencyd[11932]: Performing action log for error /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_zfi_1_fifo_intr_pcsif_pcs1_data_dest_pfe_err (0x451201) in module: btchip with scope: pfe category: functional level: minor
Feb 8 17:15:06 2024 labrouter resiliencyd[11932]: Cmerror Op Clear: btchip: chip_num:1 intr_full_name:dp_1_zfi_1_fifo_intr_status_pcsif_pcs1_data_dest_pfe_err intr_bit_pos:5 (URI: /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_zfi_1_fifo_intr_pcsif_pcs1_data_dest_pfe_err)

Feb 7 11:01:11 2024 labrouter evo-cda-bt[9643]: [Warn] CDA Intr: VIQ wedge detected: chip_num:2 blockInst:0 jsViqFld:0x662 wedged VIQ:0xe2 VIQ RRSN:0x4e60 VIQ TGSN:0x44f1 VIQ RDSN:0x24c0 src ppfe:1 egrSlice:4 pnPhysOq:98 wedged VOQ:0xa0a VOQ TRSN:0x0 VOQ RGSN:0x0 VOQ TDSN:0x0

Feb 9 14:20:20 2024 labrouter packetio[19905]: [Error] Wedge-Detect : Host Loopback Wedge Detected: PFE: 1
Feb 9 14:20:20 2024 labrouter resiliencyd[11932]: Error: /fpc/0/HstLbk:pfe:1/0/cm/0/host_loopback/1/host_loopback_wedge_detected (0x20001), scope: pfe, category: functional, severity: major, module: HstLbk:pfe:1, type: host loopback wedge detected
Feb 9 14:20:20 2024 labrouter resiliencyd[11932]: Performing action get-state for error /fpc/0/HstLbk:pfe:1/0/cm/0/host_loopback/1/host_loopback_wedge_detected (0x20001) in module: HstLbk:pfe:1 with scope: pfe category: functional level: major
Feb 9 14:20:22 2024 labrouter resiliencyd[11932]: Performing action cmalarm for error /fpc/0/HstLbk:pfe:1/0/cm/0/host_loopback/1/host_loopback_wedge_detected (0x20001) in module: HstLbk:pfe:1 with scope: pfe category: functional level: major
Feb 9 14:20:22 2024 labrouter resiliencyd[11932]: Alarm set: FPC color=red, class=CHASSIS, reason=FPC 0 Major Errors

Feb 8 19:27:28 2024 labrouter evo-cda-bt[9859]: [Warn] CDA Intr: VIQ wedge detected: chip_num:0 blockInst:1 jsViqFld:0xe62 wedged VIQ:0x1e2 VIQ RRSN:0x1c20 VIQ TGSN:0x1c20 VIQ RDSN:0xeb8 src ppfe:3 egrSlice:4 pnPhysOq:98 wedged VOQ:0x3aa VOQ TRSN:0x0 VOQ RGSN:0x0 VOQ TDSN:0x0

Feb 12 00:34:29 2024 labrouter packetio[19905]: [Error] Wedge-Detect : Host Loopback Wedge Detected: PFE: 0
Feb 12 00:34:30 2024 labrouter resiliencyd[11932]: Error: /fpc/0/HstLbk:pfe:0/0/cm/0/host_loopback/0/host_loopback_wedge_detected (0x20001), scope: pfe, category: functional, severity: major, module: HstLbk:pfe:0, type: host loopback wedge detected
Feb 12 00:34:30 2024 labrouter resiliencyd[11932]: Performing action get-state for error /fpc/0/HstLbk:pfe:0/0/cm/0/host_loopback/0/host_loopback_wedge_detected (0x20001) in module: HstLbk:pfe:0 with scope: pfe category: functional level: major
Feb 12 00:34:31 2024 labrouter resiliencyd[11932]: Performing action cmalarm for error /fpc/0/HstLbk:pfe:0/0/cm/0/host_loopback/0/host_loopback_wedge_detected (0x20001) in module: HstLbk:pfe:0 with scope: pfe category: functional level: major
Feb 12 00:34:31 2024 labrouter resiliencyd[11932]: Cmerror Op Set: HstLbk:pfe:0: Host loopback timeout for pfe: 0 (URI: /fpc/0/HstLbk:pfe:0/0/cm/0/host_loopback/0/host_loopback_wedge_detected)

Other relevant logs
System Active Errors Detail Information
FPC 0
----------------------------------------------------------------
Error Name : host_loopback_wedge_detected
Identifier : /fpc/0/HstLbk:pfe:0/0/cm/0/host_loopback/0/host_loopback_wedge_detected
Description : host loopback wedge detected
State : enabled
Scope : pfe
Category : functional
Level : major
Threshold : 1
Error limit : 1
Occur count : 1
Clear count : 0
Last occurred(ms ago) : 12696803
System Active Errors Detail Information
FPC 0
----------------------------------------------------------------
Error Name : host_loopback_wedge_detected
Identifier : /fpc/0/HstLbk:pfe:1/0/cm/0/host_loopback/1/host_loopback_wedge_detected
Description : host loopback wedge detected
State : enabled
Scope : pfe
Category : functional
Level : major
Threshold : 1
Error limit : 1
Occur count : 1
Clear count : 0
Last occurred(ms ago) : 222346160
System Active Errors Detail Information
FPC 0
----------------------------------------------------------------
Error Name : dp_1_viqm_intr_gnt_rtm
Identifier : /fpc/0/evo-cda-bt/0/cm/0/btchip/0/dp_1_viqm_intr_gnt_rtm
Description : dp_1_viqm_intr_gnt_rtm
State : enabled
Scope : pfe
Category : functional
Level : minor
Threshold : 10
Error limit : 30
Occur count : 2
Clear count : 1
Last occurred(ms ago) : 12705341
System Active Errors Detail Information
FPC 0
----------------------------------------------------------------
Error Name : dp_1_viqm_intr_gnt_rtm
Identifier : /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_viqm_intr_gnt_rtm
Description : dp_1_viqm_intr_gnt_rtm
State : enabled
Scope : pfe
Category : functional
Level : minor
Threshold : 10
Error limit : 30
Occur count : 2
Clear count : 1
Last occurred(ms ago) : 332126344

labuser@labrouter:pfe> show cmerror module brief

-------------------------------------------------------------------------
Module Name Active Errors PFE Callback ModuleData
Specific Function
-------------------------------------------------------------------------
4 btchip 2 Yes 0x0808fe40 0x00000000
9 HstLbk:pfe:0 1 Yes 0x0808fe40 0x00000000
10 HstLbk:pfe:1 1 Yes 0x0808fe40 0x00000000

labuser@labrouter:pfe> show cmerror module 4 error 0x450bd9

Error-id : 0x450bd9

Error Name : dp_1_viqm_intr_gnt_rtm
Identifier : /fpc/0/evo-cda-bt/0/cm/0/btchip/0/dp_1_viqm_intr_gnt_rtm
Description : dp_1_viqm_intr_gnt_rtm
State : enabled
Scope : pfe
Category : functional
PFE : 0
Configured Level : Minor
Default Level : Minor
Count : 1
Threshold : 1
Error Limit : 30
Occur Count : 2
Clear Count : 1
OverItemThres Occur Count : 2
OverItemThres Clear Count : 1
Last-occurred(ms ago) : 9847668

Logs:
----------------------------------------------------------
Index Time Sub-Err State Description
----------------------------------------------------------
0 02/12/24 00:34:21 0 Set user defined 1
1 02/08/24 19:27:29 0 Clear user defined 1
2 02/08/24 19:27:28 0 Set user defined 1

labuser@labrouter:pfe> show cmerror module 4 error 0x450bd9 pfe 1

Error-id : 0x450bd9

Error Name : dp_1_viqm_intr_gnt_rtm
Identifier : /fpc/0/evo-cda-bt/0/cm/0/btchip/1/dp_1_viqm_intr_gnt_rtm
Description : dp_1_viqm_intr_gnt_rtm
State : enabled
Scope : pfe
Category : functional
PFE : 1
Configured Level : Minor
Default Level : Minor
Count : 1
Threshold : 1
Error Limit : 30
Occur Count : 2
Clear Count : 1
OverItemThres Occur Count : 2
OverItemThres Clear Count : 1
Last-occurred(ms ago) : 329329271

Logs:
----------------------------------------------------------
Index Time Sub-Err State Description
----------------------------------------------------------
0 02/08/24 07:50:40 0 Set user defined 1
1 02/08/24 06:46:53 0 Clear user defined 1
2 02/08/24 06:46:52 0 Set user defined 1

labuser@labrouter:pfe> show cmerror module 10 error 0x020001 pfe 1

Error-id : 0x20001

Error Name : host_loopback_wedge_detected
Identifier : /fpc/0/HstLbk:pfe:1/0/cm/0/host_loopback/1/host_loopback_wedge_detected
Description : host loopback wedge detected
State : enabled
Scope : pfe
Category : functional
PFE : 1
Configured Level : Major
Default Level : Major
Count : 1
Threshold : 1
Error Limit : 1
Occur Count : 1
Clear Count : 0
OverItemThres Occur Count : 1
OverItemThres Clear Count : 0
Last-occurred(ms ago) : 219450274

Logs:
----------------------------------------------------------
Index Time Sub-Err State Description
----------------------------------------------------------
0 02/09/24 14:20:22 0 Set user defined 1

labuser@labrouter:pfe> show cmerror module 9 error 0x020001

Error-id : 0x20001

Error Name : host_loopback_wedge_detected
Identifier : /fpc/0/HstLbk:pfe:0/0/cm/0/host_loopback/0/host_loopback_wedge_detected
Description : host loopback wedge detected
State : enabled
Scope : pfe
Category : functional
PFE : 0
Configured Level : Major
Default Level : Major
Count : 1
Threshold : 1
Error Limit : 1
Occur Count : 1
Clear Count : 0
OverItemThres Occur Count : 1
OverItemThres Clear Count : 0
Last-occurred(ms ago) : 9667821

Logs:
----------------------------------------------------------
Index Time Sub-Err State Description
----------------------------------------------------------
0 02/12/24 00:34:31 0 Set user defined 1

labuser@labrouter:pfe> show host-path app wedge-detect state
Wedge-Detection states:

General Information:
RPC ready? : Yes
SYS ready? : Yes (h/w ready : Yes, ISSU : No)
Stopped? : No
Threshold : 3

PFE Inst : 0 FP Interface : fp0
State : Wedged
Transmit count : 395264
Received count : 384850
Missed count : 10413
Wedge reporting : Registered

PFE Inst : 1 FP Interface : fp1
State : Wedged
Transmit count : 395264
Received count : 175201
Missed count : 220062
Wedge reporting : Registered

PFE Inst : 2 FP Interface : fp2
State : Ok
Transmit count : 395264
Received count : 395264
Missed count : 0
Wedge reporting : Registered

Solution

The request destination PFE Error on PCS0/PSC1 Interfaces are keep showing up with high volume causing VIQ wedged and eventually leading to host loopback wedge. This is an indication of hardware failure. A reboot of the device can be tried for immediate recovery but need to replace the router if errors/logs repeat event after the reboot

Modification History

2024-03-14 : Article Created