Description

Parity errors are seen on one of the FPCs which caused minor alarms for other FPCs on the same device. This article explains how to identify the FPC that caused the errors and alarms.

Symptoms

  1. FPC0 (victim FPC) in this case, reported Remote PFE error and Cell reorder logs:

    Aug 26 17:53:58.696  router-name fpc0 Cmerror Op Set: CM[1]: MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (8:1) 
    Aug 26 17:53:59.233  router-name fpc0 Cmerror Op Set: CM[1]: MPC fabric remote PFE error (aggregate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (8:1) 
    Aug 26 17:53:59.776  router-name fpc0 Cmerror Op Set: CM[3]: MPC fabric remote PFE error (rate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (10:3) 
    Aug 26 17:54:00.313  router-name fpc0 Cmerror Op Set: CM[3]: MPC fabric remote PFE error (aggregate based) (1) exceed raising threshold (1) occurrance (0) for module/pfe (10:3) 
    Aug 26 17:54:00.505  router-name fpc0 XMCHIP(0): XMCHIP(0): FI: Error cell sent to reorder engine - Stream 157, Count 8 

  2. A Minor alarm was raised for FPC0 (and other FPCs):

    root@router-name> show chassis alarms no-forwarding

    10 alarms currently active
    Alarm time               Class  Description
    2021-08-26 17:54:35 PDT  Major  FPC 7 Major Errors
    2021-08-26 17:54:19 PDT  Minor  FPC 0 Minor Errors
    2021-08-26 17:54:19 PDT  Minor  FPC 6 Minor Errors
    2021-08-26 17:54:19 PDT  Minor  FPC 3 Minor Errors
    2021-08-26 17:54:18 PDT  Minor  FPC 4 Minor Errors
    2021-08-26 17:54:17 PDT  Minor  FPC 5 Minor Errors
    2021-08-26 17:54:15 PDT  Minor  FPC 2 Minor Errors
    2021-08-26 17:53:48 PDT  Minor  FPC 7 Minor Errors

Solution

  1. Notice the Stream ID in the following message for FPC0. To identify the source FPC is sending these error parcels, use this calculation: (157-128)/4 = 7.25.
    Round this off to the nearest whole number which is 7, in this case 

    Aug 26 17:54:00.505  router-name fpc0 XMCHIP(0): XMCHIP(0): FI: Error cell sent to reorder engine -  Stream 157 , Count 8 

    The culprit FPC here is FPC7, which is sending these error parcels to other FPCs and other FPCs are generating minor alarm since the rate of reception of these error packets is exceeding threshold.

  2. A further deep dive into the message indicates parity error for FPC7 which is the culprit here:

    Aug 26 17:52:11.905  router-name fpc7 XMCHIP(1): XMCHIP(1): OCM: LO0 parity error - Parity Error Address 0x800c, Bank 0x4000, Data 0x6a6d7f736973686b 
    Aug 26 17:52:12.042  router-name fpc7 XMCHIP(1): XMCHIP(1): OCM: Parity Error Counter 1 

The culprit FPC can be rebooted during a maintenance window as an attempt to recover from the errors. If this does not work, it may be an indication of an underlying hardware issue with the culprit FPC.

Note: As per changes via PR1530244 - A new CMERROR has been added to the system regarding OCM parity errors , a CMERROR would be raised based on the OCM error threshold . With this PR enhancement, if the count of OCM parity errors is less than 5, a minor CMERROR would be raised. And if the OCM parity errors are more than 5, then a major CMERROR would be raised. This major CMERROR should invoke "disable-pfe" action to bring the PFE down automatically to mitigate this issue.