When too many SER(soft-error-recovery) of memory ECC parity error happens, rpd routing protocols, like lacp/isis/ospf/BGP and other protocols, like BFD etc. would become down, and then lead to service outage, customer had to reboot the box manually to restore service.
Below logs will be flooding in Messages log. Following with routing protocols flapping logs.
Jul 11 15:40:00 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=0, unit=0, reset_type=0 Jul 11 15:40:00 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=1, unit=0, reset_type=0 Jul 11 15:40:00 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_ParityErrInt event=13, intr_id=1110, block=0, unit=0, reset_type=0 Jul 11 15:40:00 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_ParityErrInt event=13, intr_id=1110, block=1, unit=0, reset_type=0 Jul 11 15:40:17 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=0, unit=0, reset_type=0 Jul 11 15:40:17 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=1, unit=0, reset_type=0 Jul 11 15:40:17 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_Ecc_2bErrInt event=13, intr_id=1112, block=0, unit=0, reset_type=0 Jul 11 15:40:43 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=0, unit=0, reset_type=0 Jul 11 15:40:43 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=1, unit=0, reset_type=0 Jul 11 15:40:44 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_2bErrInt event=13, intr_id=805, block=1, unit=0, reset_type=0 Jul 11 15:40:52 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=0, unit=0, reset_type=0 Jul 11 15:40:53 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_2bErrInt event=13, intr_id=805, block=0, unit=0, reset_type=0 Jul 11 15:40:53 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=1, unit=0, reset_type=0 Jul 11 15:40:53 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_ParityErrInt event=13, intr_id=1110, block=0, unit=0, reset_type=0 Jul 11 15:40:53 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_ParityErrInt event=13, intr_id=1110, block=1, unit=0, reset_type=0 Jul 11 15:40:56 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_2bErrInt event=13, intr_id=805, block=0, unit=0, reset_type=0 Jul 11 15:40:56 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_1bErrInt event=13, intr_id=804, block=1, unit=0, reset_type=0 Jul 11 15:40:56 2024 fpc0 DNX_SER_ERR: SER:name=EGQ_ECC_Ecc_2bErrInt event=13, intr_id=805, block=1, unit=0, reset_type=0 Jul 11 15:40:56 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_ParityErrInt event=13, intr_id=1110, block=0, unit=0, reset_type=0 Jul 11 15:40:56 2024 fpc0 DNX_SER_ERR: SER:name=SCH_ECC_ParityErrInt event=13, intr_id=1110, block=1, unit=0, reset_type=0
Here are some notes and guidance, based on BCM recommendations:
ECC 1b Error:
- These errors should be ignored, no action required from application
ECC 2b Error:
- If same error is seen (with same memory address) thrice within one hour, it is safe to assume that recovery mechanism is not working, hence device should be rebooted in next planned maintains window
Parity Errors: