The issue: around 20% loss of the traffic coming from the fabric. The issue observed only when the faulty linecard is booting up while traffic is running. No issues seen if traffic starts to flow after the linecard fully booted. Preliminary, the issue is with the software handling states of the center chip, XM. The same chip being used in MPC4, MPC5, NG-MPC2, NG-MPC3 cards. The issue is not transient, FPC continues to stay in the same state doesn't matter is traffic running or not. The behavior is the same in the recently released 23.4R2-S3.

Major FPC 14 Hard errors

Device has alerted for FPC Major errors:

"Oct 21 06:12:15.853 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:15.853 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:15.869 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:15.876 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail

Oct 21 06:12:18.278 2024 hostname chassisd[8882]: CHASSISD_I2CS_READBACK_ERROR: Readback error from I2C slave for FPC 14 ([0x24, 0xa] -> 0x0)

Oct 21 06:12:18.407 2024 hostname chassisd[8882]: CHASSISD_IPC_WRITE_ERR_NULL_ARGS: FRU has no connection arguments fru_send_msg FPC 14

Oct 21 06:12:19.851 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:19.851 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:19.851 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:19.876 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail

Oct 21 06:12:24.856 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:24.856 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:24.856 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:24.880 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail

Oct 21 06:12:29.860 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:29.860 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:29.861 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:29.884 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail

Oct 21 06:12:35.864 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:35.864 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:35.865 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:35.889 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail

Oct 21 06:12:39.867 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:39.867 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:39.867 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:39.892 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail

Oct 21 06:12:44.871 2024 hostname alarmd[9069]: Alarm cleared: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:44.871 2024 hostname craftd[8885]: Minor alarm cleared, FPC 14 Temp Sensor Fail

Oct 21 06:12:44.871 2024 hostname alarmd[9069]: Alarm set: FPC id=83889768, color=YELLOW, class=CHASSIS, reason=FPC 14 Temp Sensor Fail

Oct 21 06:12:44.895 2024 hostname craftd[8885]: Minor alarm set, FPC 14 Temp Sensor Fail"

On debug of this issue we found that we can not give a point fix for this case.

The possible proposed fixes for this issue would be:

1)	Enhance the cli command "show pfe statistics traffic" to include the fi drop counters.
2)	Introduce a new FI error alarm.
3)	If a minor alarm is seen repeatedly for a FPC; make the alarm to major.

PR 1857373 has been raised for a software enhancement to improve handling of such errors. The workaround for 19.4 is being discussed.