Description

ACX5448 sometimes floods error below and caused protocols down and then led to outage in hours or days.
 

May 6 21:49:24 2024 abc fpc0 DNX_SER_ERR: SER:name=CRPS_ECC_Ecc_1bErrInt event=13, intr_id=586, block=0, unit=0, reset_type=0  

May 6 22:23:36 2024 abc fpc0 DNX_SER_ERR: SER:name=CRPS_ECC_Ecc_1bErrInt event=13, intr_id=586, block=0, unit=0, reset_type=0  

May 6 23:01:12 2024 abc fpc0 DNX_SER_ERR: SER:name=CRPS_ECC_Ecc_1bErrInt event=13, intr_id=586, block=0, unit=0, reset_type=0  

Symptoms

Uplinks will be impacted leading to protocol down and impact on forwarding.7
 

May 7 08:16:29 2024 abc fpc0 DNX_SER_ERR: SER:name=CRPS_ECC_Ecc_1bErrInt event=13, intr_id=586, block=0, unit=0, reset_type=0

May 7 09:11:12 2024 abc fpc0 DNX_SER_ERR: SER:name=CRPS_ECC_Ecc_1bErrInt event=13, intr_id=586, block=0, unit=0, reset_type=0

May 7 09:11:12 2024 abx fpc0 sfp_thread: periodic thread awaken prematurely.

May 7 09:11:12 2024 abc fpc0 sfp_thread: periodic thread awaken prematurely.

May 7 09:11:15 2024 abc lacpd[8180]: LACPD_TIMEOUT: et-0/1/0: lacp current while timer expired current Receive State: CURRENT

May 7 09:11:15 2024 abc kernel: lag_bundlestate_ifd_change: bundle ae18: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 40000000000 Number of links (Current : Required) 0 : 1

May 7 09:11:15 2024 abc lacpd[8180]: LACPD_TIMEOUT: et-0/1/1: lacp current while timer expired current Receive State: CURRENT

May 7 09:11:15 2024 abc lacpd[8180]: LACP_INTF_DOWN: ae18: Interface marked down due to lacp timeout on member et-0/1/1

May 7 09:11:15 2024 abc mib2d[8157]: SNMP_TRAP_LINK_DOWN: ifIndex 561, ifAdminStatus up(1), ifOperStatus down(2), ifName ae18

Solution

From the host logs, it is identified that h/w memory corruption happened at memory CRPS_CRPS_12_CNTS_MEM.
 

[May 15 06:18:30.443 LOG: Notice] RIO-MIC(0/0) link 27 SFP receive power low alarm cleared

0:dcmn_interrupt_print_info: name=CRPS_ECC_Ecc_1bErrInt, id=586, index=0, block=0, unit=0, recurring_action=0 | nof_occurences=0002, cnt_overflowf=0x0, memory address=0x000d3f88 memory=CRPS_CRPS_12_CNTS_MEM, array element=0, index=16264 | None
 

Info regarding the memory CRPS_CRPS_12_CNTS_MEM:

  1. It is a less important memory. Immediate precautionary step (reboot the device) is not needed.
  2. It is a dynamic memory. Meaning, We can’t cache this memory on software. (Note: If memory is cached, recurring occurrence of interrupt can be avoided). Only way to recover is power cycle the system (But immediate power cycling is not needed).

Modification History

2024-06-16 : Article Created