Description

The "DDRIF Checksum error for FO/WO" message reports a transient or permanent hardware issue.

The chassis management alarm (CMALARM/CMERROR) is set as follows:
  • if the rate is less than 6 events per second, these errors are reported only - NO ACTION Needed
  • if the rate is 6 events per second or more, the software is reported as a MINOR alarm. The FPC/MPC with this error code  -XMCHIP_CMERROR_DDRIF_INT_REG_CHKSUM_ERR_MINOR -0x70070 - NO IMPACT to the system. NO ACTION Needed. You may clear the alarm if you wish.
  • if the rate is more than 255 events per second, the software is reported as a MAJOR alarm. The FPC/MPC with this error code - XMCHIP_CMERROR_DDRIF_INT_REG_CHKSUM_ERR_MAJOR - 0x70071. The affected interfaces may be disabled. Recommended ACTION, please RMA the FPC.

This is a Troubleshooting Article for a PFE ASIC Syslog Event.
To view other documented Syslog events related to XMCHIP, XLCHIP, MQCHIP, LUCHIP, EACHIP, and PECHIP, see KB31893 - Master Index of Articles for Troubleshooting PFE ASIC Syslog Events [juniper.net].

.

Symptoms

When a "DDRIF Checksum error" event occurs, messages similar to the following are reported:​

<Host> <FPC#> XMCHIP(x): DDRIF: Checksum error for FO/WO2 - Channel 16, Address 0x9402c, Checksum Errors 1, Checksum Poison Count 0
<Host> <FPC#> XMCHIP(x): DDRIF: Checksum error for WO1 - Channel 16, Address  0x9402c, Checksum Errors 1, Checksum Poison Count 0
<Host> <FPC#> XMCHIP(x): DDRIF: Checksum error for FO1 - Channel 16, Address 0x9402c, Checksum Errors 1, Checksum Poison Count 0

Indications:

  1. Service impact, depending on the number of error counters reported

  2. Once the error rate is higher than 255/sec default, disable-pfe action will bring this PFE out-of-service and shut down all wan and fabric interfaces for the corresponding PFE.

  3. If the error is reported for the WO block only, then exposure is only local to this PFE reporting the error

 

Solution

This is a transient or permanent hardware issue for the WAN side or Fabric side or due to a software defect. A transient DDRIF error usually accompanies by the OCM parity error messages in the Syslog.

A CMALARM has been added to enhancement PR1157937. If the checksum error rate is 5-255/sec, then a MINOR alarm will be raised. A MAJOR alarm will be raised if the rate is more than 255. The error counter illustrates the packets that are dropped. Once this error rate reaches 255/sec, the MPC should be replaced. Don't restart the MPC and restart again. Usually, such MPC boards fail memory diagnostics tests during the reboot. 

PR1166106 will prevent MPC which fails memory BIST tests from becoming operational and exposing further outages.


Perform these steps to determine the cause and resolve the problem (if any).  Continue through each step until the problem is resolved.

  1. Collect the show command output.

    {SYSLOGSERVICENOWTOKEN.EN_US}

    Capture the output to a file (in case you have to open a technical support case). To do this, configure each SSH client/terminal emulator to log your session.

    show chassis alarm
    show log messages
    show log chassisd
    start shell network pfe <fpc#>
    show nvram
    show syslog messages
    exit

  2. Analyze the show command output.

    In the 'show log messages', review the events that occurred at or just before the appearance of the "DDRIF Checksum error" message. Frequently these events help identify the cause.

    • Contact Juniper Support  to RMA the card if the error rate was 255/sec or higher

    • The  generic pfe-disable event script  will detect this condition if error rate is too high (255/sec or higher) and invoke the disable-pfe action

    • Run the CLI command ‘ request chassis fpc slot # offline

      • Then configure the FPC to be powered off until it is replaced.

        • ​​This is needed to prevent the risk of operational impact on the remote PFEs

 

This article is indexed in KB31893 - primary Index of Articles for Troubleshooting PFE ASIC Syslog Events [juniper.net] ; tag XMCHIPTSG


Tip: When looking at an event in the logs, it is important to focus on the first error message in a collection of syslog messages. The first error message is usually the cause of all the follow-on error messages. The follow-on collateral damage error messages can be ignored.

Modification History

2024-01-03: Added link to KB75113 [juniper.net] in Related Information
2022-05-20: updated the content as low rate of DDRIF does not require a RMA anymore unless it repeats after a FPC restart. Since Junos OS 17.3 and higher ucode hardening efforts had been done to protect remote PFEs entering in a lookup wedge conditions once processing corrupted fabric parcels sourced from a PFE reporting DDRIF errors.
2020-08-02: added OCM block PR1530244 to raise major alarm if the parity error events for longer then 5 events
2019-09-30: Article reviewed for accuracy; no changes required.
2017-09-11: Added some clarity around how to offline the fpc and then configure the fpc to be powered off.

Related Information

KB75113 - FPC Minor Error: XMCHIP(): XMCHIP(): DDRIF: Checksum error detected [juniper.net]