Description

For the ams interface configuration (N:1 model), the MS-MPC slot 5 was offline by CLI command due to the user trying to recover the uncorrectable ECC memory errors, it caused the cascaded issue and MS-MPC card in slot 0 was "unexpected shutdown of connection to datapath-traced".

Symptoms

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn chassisd[18380]: %DAEMON-5-CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 5 offline: Offlined by cli command

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn chassisd[18380]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power off (jnxFruContentsIndex 7, jnxFruL1Index 6, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MS-MPC @ 5/*/*, jnxFruType 3, jnxFruSlot 5, jnxFruOfflineReason 7, jnxFruLastPowerOff -1081178813, jnxFruLastPowerOn 5090)

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn chassisd[18380]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 6, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MS-MPC @ 5/*/*, jnxFruType 3, jnxFruSlot 5, jnxFruOfflineReason 7, jnxFruLastPowerOff -1081178813, jnxFruLastPowerOn 5090)

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn chassisd[18380]: %DAEMON-5-CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(5)

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn mib2d[19209]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 866, ifAdminStatus up(1), ifOperStatus down(2), ifName ms-5/1/0

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn mib2d[19209]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 867, ifAdminStatus up(1), ifOperStatus down(2), ifName pc-5/1/0

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn mib2d[19209]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 868, ifAdminStatus up(1), ifOperStatus down(2), ifName mams-5/1/0

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn mib2d[19209]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 869, ifAdminStatus up(1), ifOperStatus down(2), ifName ms-5/2/0

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn mib2d[19209]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 870, ifAdminStatus up(1), ifOperStatus down(2), ifName pc-5/2/0

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn mib2d[19209]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 871, ifAdminStatus up(1), ifOperStatus down(2), ifName mams-5/2/0

Apr 3 14:15:25 2024 rdlj96cgn03-re0.cn : %DAEMON-3: (FPC Slot 0, PIC Slot 0) ms00 mspmand[241]: Unexpected shutdown of connection to datapath-traced, trying to reconnect

 

 

{master}

[email protected]> show interfaces redundancy <<<<< N:1 model, all mams interfaces in slot 0 are down during the problem stage.

Interface State Last change Primary Secondary Current status

ams10 On primary 00:00:12 mams-5/1/0 mams-0/1/0 secondary down

ams11 On primary 00:00:08 mams-5/2/0 mams-0/2/0 secondary down

ams12 On primary 00:00:05 mams-5/3/0 mams-0/3/0 secondary down

ams13 On primary 53w0d 23:21 mams-7/0/0 mams-0/0/0 secondary down

ams14 On primary 53w0d 23:21 mams-7/1/0 mams-0/1/0 secondary down

ams15 On primary 53w0d 23:21 mams-7/2/0 mams-0/2/0 secondary down

ams16 On primary 53w0d 23:21 mams-7/3/0 mams-0/3/0 secondary down

ams17 On primary 53w0d 23:21 mams-10/0/0 mams-0/0/0 secondary down

ams18 On primary 53w0d 23:21 mams-10/1/0 mams-0/1/0 secondary down

ams19 On primary 53w0d 23:21 mams-10/2/0 mams-0/2/0 secondary down

ams20 On primary 53w0d 23:21 mams-10/3/0 mams-0/3/0 secondary down

ams21 On primary 53w0d 23:21 mams-11/0/0 mams-0/0/0 secondary down

ams22 On primary 53w0d 23:21 mams-11/1/0 mams-0/1/0 secondary down

ams23 On primary 53w0d 23:21 mams-11/2/0 mams-0/2/0 secondary down

ams24 On primary 53w0d 23:20 mams-11/3/0 mams-0/3/0 secondary down

ams25 On primary 53w0d 23:21 mams-9/0/0 mams-0/0/0 secondary down

ams26 On primary 53w0d 23:21 mams-9/1/0 mams-0/1/0 secondary down

ams27 On primary 53w0d 23:21 mams-9/2/0 mams-0/2/0 secondary down

ams28 On primary 53w0d 23:21 mams-9/3/0 mams-0/3/0 secondary down

ams5 On primary 53w0d 23:19 mams-1/2/0 mams-0/0/0 secondary down

ams6 On primary 27w6d 00:47 mams-2/2/0 mams-0/0/0 secondary down

ams7 On primary 53w0d 23:19 mams-3/2/0 mams-0/1/0 secondary down

ams8 On primary 37w6d 00:22 mams-4/2/0 mams-0/1/0 secondary down

ams9 On primary 00:00:17 mams-5/0/0 mams-0/0/0 secondary down

 

 

Solution

The FPC 0 went into the error status due to FPC 5 suddenly shutdown by CLI (the user was trying to recover the uncorrectable memory errors based on the notes) and the data-path broken. It caused many internal errors in the FPC 0 afterwards, which caused the second mams interfaces down in slot 0.

 

Reset this FPC 0 card in the maintenance window to recover the ams interfaces issue.

Modification History

2024-05-23 : Article Created