Alert Type

PSN - Product Support Notification
HighA SIB3 with multi-bit uncorrectable memory error may cause traffic impact
HighContinued using a SIB3 with multi-bit uncorrectable memory error may cause PFEs to stop forwarding traffic

Product Affected

JNP10008-SF3 or JNP10016-SF3 or JNP10004-SF3 running on PTX Series running Junos Evolved Software

Alert Description

We have made improvements to the Junos Evolved uncorrectable shared memory (SM-D) error handling for the JNP10008/16-SF3 and JNP10004-SF3 boards (SIB3).

PR1625305 improves the error handling by offline SIB3 once the software detects the error - reducing the chance of corrupted packets causing PFEs to stop forwarding.

 

Solution

Recommendation 

On a rare occasion which a SIB3 encounters SM-D uncorrectable memory error:

  • RMA the SIB3 that exhibits the uncorrectable memory error.
  • While waiting for the replacement, we recommend you configure the SIB3 to stay offline via  "set chassis sib slot <slot> power off"  configuration command. This is to prevent the SIB3 to become online after the router reboots.

If unable to upgrade to software with PR1625305

  • Deploy the event script. The event script mimics the improved Junos Evolved SIB error handling (see PR1625305) by
    1. When the script detects an uncorrectable error, the script takes the SIB3 offline immediately
    2. The software notifies operators by syslog and SNMP trap
    3. In addition to the software changes by PR1625305, the script can add a configuration to keep the SIB3 offline - preventing it from becoming online after a system reboot
Upgrade to software with the PR1625305 - SIB3 SM-D multi-bit uncorrectable ECC memory error handling enhancement
We have made improvements to how Junos Evolved handles the JNP10004/8/16-SF3 uncorrectable memory errors.

Currently, the JNP10004-SF3, JNP10008-SF3, and JNP10016-SF3 (aka SIB3) may experience uncorrectable memory errors in the shared memory (SM-D) area. When a SIB3 encounters such errors, JUNOS software is programmed to mitigate this situation by rebooting the suspected SIB3 board. After at most 3 attempts, the suspected SIB3 will be off-line. For the SIB3 board that JUNOS offlines after 3 recovery attempts, if you reboot the router or restart the SIB3, the SIB3 board may boot up, and in a rare case, the memory controller may not report multi-bit errors. Continued usage of the faulty SIB3 may experience SM-D non-detectable memory errors that may cause traffic impact.

PR1625305 improves the SIB3 SM-D multi-bit uncorrectable ECC memory error handling behavior discussed above. Once the software detects the error the SIB3 is offline immediately - reducing the chance of corrupted packets causing traffic impact. The software will no longer attempt to recover the suspected SIB3 by rebooting it.

On the JNP10008-SF3 or JNP10016-SF3 (SIB3), you may check if the SIB3 is offline because of multiple-bit ECC memory error in the shared memory area by using the following command from the Linux shell. In this example, the shared memory experienced memory error:
$ journalctl | egrep "hwdre|fab" | egrep "FATAL ERROR"
Sep 03 01:44:49 re0 resiliencyd[18684]: Cmerror Op Set: zfchip::FATAL ERROR!! from SM-D TYPE0 HDR0_PROTECTION_DETECT (URI: /sib/3/fabspoked-fchip/0/cm/0/zfchip/0/sm_d_type0_protect_hdr0_protection_detect)

In this example the SIB in slot 3 experiencing the shared memory error (shown as SM-D). This is a situation in which there has been a combination of undetected ECC errors, detectable ECC errors in the packet header, and parity errors in the payload shared memory:

Jul 21 13:31:30 re0 fabspoked-fchip[13073]: SIB3_ZF_0: [MINOR] Parity detection for Payload slice 0 INTR in SM-D Type-0 for banks 23-0 (bank-mask 0xc00000 log-address 0x60e)
Jul 21 13:31:30 re0 fabspoked-fchip[13073]: SIB3_ZF_0: [MINOR] Parity detection for Payload slice 1 INTR in SM-D Type-0 for banks 23-0 (bank-mask 0xc00000 log-address 0x60e)
Jul 21 13:31:30 re0 fabspoked-fchip[13073]: SIB3_ZF_0: [MINOR] Parity detection for Payload slice 2 INTR in SM-D Type-0 for banks 23-0 (bank-mask 0xc00000 log-address 0x60e)
Jul 21 13:31:30 re0 fabspoked-fchip[13073]: SIB3_ZF_0: [FATAL] ECC protection detection for Header slice INTR in SM-D Type-0 for banks 23-12 (bank-mask 0xc00 log-address 0x60e)
 

Recommendation - to perform the following steps

  • RMA the SIB3 that exhibits the uncorrectable memory error.
  • While waiting for the replacement, you need to configure the SIB3 to stay offline via  "set chassis sib slot <slot> power off"  configuration command.
  • Deploy the event script. The event script mimics the improved Junos Evolved SIB error handling (see PR1625305). Please contact JTAC if you are interested in deploying the event script.

Modification History

2022-03-05 Update the part number to be JNP10004-SF3 instead of SIB3-JNP10004. Deleted duplicated syslog entries - shorten the length of the article
First publication 2022-02-25