Description

When setting some LMIC ports new chassis config/speed, the LIMIC restarts, but does not return even after waiting 20-30 min.

Symptoms

root@JTAC> show chassis alarms no-forwarding

 

2 alarms currently active

Alarm time Class Description

2025-05-12 15:35:25 UTC Minor FPC 0 Optics hardware init failed

 

root@JTAC> start shell pfe network fpc0

 

root@JTAC-fpc0:pfe> show picd pic fpc 0 pic 0

 

name : pic-0/0

pic_state : PIC_ONLINE

pic fault reason : NONE

<snip>

pic alt-speed-set : Uninitialized

Solution

All MX ULC cards have dependency on xcvr-plugin code to send correct number of serdes_lanes info during Asic IFd attach.

So, Asic Ifd Attach is gated by xcvr-ready state.

 

If a port has Optics HW init failure, Asic IFD attach won't be called for this port, which is correct.

But due to MX ULC architecture and tight coupling among various apps, without this IFD attach, IFD won't be published, no shmem update in aft-ulcd for info aftd-trio is busy spinning on.

As aftd-trio is busy waiting for this info, it halts all other IPCs leading to expired requests in aft-ulcd. As a result, for ports following this faulty port, link stays DOWN & The Card can also stay down due to same reason.

 

Workaround:

  1. Temporary workaround is to reboot the complete FPC, which brings up LMIC also
  2. Permanent solution is to identify the faulty optics from picd logs, remove it & restart LMIC again

 

The internal PR 1873278 provides solution for this, even when the faulty optics is present.

Modification History

2025-05-30 : Article Created