Description

The article explains the behaviour on Junos platforms with MPC10E, the below logs keep showing after frequent interface flaps. The below logs can be obtained from FPC linux var/logs pfe.logs* messages.

<Timestamp> <Platform> ifd_process_flaps_common: IFD et-x/y/z - pfe link UP processing is complete
<Timestamp> <Platform> <FPC> user.alert aftd-trio: [Alert] IF:Ifd new linkState, ifdName: et-x/y/z ifdIndex: 497 state:Up
<Timestamp> <Platform> <FPC> MQSS(2): QUADPORT2: Cleared Ethernet MAC Remote Fault Delta Event for Port 0 (et-x/y/z)
<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete
<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete
<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete
<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete

Symptoms

During the message outputting, the reported interface might in an unstable status, even the message indicates that "Physical link et-x/y/z is now UP". The interface might be in UP state, but not working properly.

Customer may report that the link went down after flapping issue on DWDM transport. The DWDM transport issue may have resolved  but the interface is still remaining down.
During the issue time, the below logs are obtained in FPC linux var/logs pfe.logs* messages and should be continuous.
The interface may go down after few continuous flap.
This is very rare scenario.

pfe.log.8:[Jun 16 15:05:47.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:48.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:49.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:50.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:51.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:52.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:53.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:54.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:55.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 
pfe.log.8:[Jun 16 15:05:56.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete 


Physical interface: et-4/0/1, Enabled, Physical link is Down
  Interface index: 278, SNMP ifIndex: 606, Generation: 281
  Description: Documentation link
  Link-level type: Ethernet, MTU: 9192, MRU: 9200, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running Down
  Interface Specific flags: Internal: 0x100200
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4004000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: 5c:5e:ab:44:6f:cf, Hardware address: 5c:5e:ab:44:6a:8f
  Last flapped   : 2024-06-16 20:34:58 WIB (16:56:03 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :      851574744948304               198984 bps
   Output bytes  :       81592273943619                14136 bps
   Input  packets:         777915631734                  224 pps
   Output packets:         106835461010                    4 pps
   IPv6 transit statistics:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Label-switched interface (LSI) traffic statistics:
   Input  bytes  :                    0                    0 bps
   Input  packets:                    0                    0 pps
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 1, Drops: 0, Framing errors: 1, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 768, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 8 in use

Solution

The issue is related to a race condition between pic periodic and pfeman thread for the MPC10E card.
To recover from this state, perform a restart for the FPC PIC via CLI for the specific port which is et-4/0/1 in the above case.

<<<request chassis pic offline fpc-slot 4 pic-slot 0>>>

 
{master}
> show log messages.0.gz | match et-4/0/1 | match 606
Jun 18 00:00:39
Jun 14 08:59:19.063 2024  mib2d[39019]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName et-4/0/1
Jun 14 08:59:19.733 2024  mib2d[39019]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 606, ifAdminStatus up(1), ifOperStatus up(1), ifName et-4/0/1
Jun 14 09:16:37.803 2024  mib2d[39019]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName et-4/0/1
Jun 14 09:16:39.433 2024  mib2d[39019]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 606, ifAdminStatus up(1), ifOperStatus up(1), ifName et-4/0/1
Jun 14 09:36:52.642 2024  mib2d[39019]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName et-4/0/1
 
Jun 18 00:26:23.940 2024  chassisd[35606]: %DAEMON-5-CHASSISD_PIC_OFFLINE_NOTICE: Taking PIC 0 in FPC 4 offline: Offlined by cli command
 
Jun 18 00:30:02.508 2024  chassisd[35606]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 5, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: MRATE-5xQSFPP @ 4/0/*, jnxFruType 11, jnxFruSlot 4, jnxFruOfflineReason 2, jnxFruLastPowerOff 370906347, jnxFruLastPowerOn 370927909)
Jun 18 00:30:02.509 2024  chassisd[35606]: %DAEMON-5-CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 138, entStateAdmin 4, entStateAlarm 128)
Jun 18 00:30:02.509 2024  chassisd[35606]: %DAEMON-5-CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged
 
Jun 18 00:30:41.000 2024  : %PFE-1: fpc4 user.alert aftd-trio: [t:2351] [Alert] IF:Ifd new linkState, ifdName:et-4/0/1 ifdIndex:412 state:Up
Jun 18 00:30:41.510 2024  lacpd[35671]: %DAEMON-5-LACP_INTF_MUX_STATE_CHANGED: ae23: et-4/0/1: Lacp state changed from DETACHED to WAITING, actor port state : |-|-|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|, partner port state : |-|-|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|

The above issue is related to PR1735539.
To obtain a permanent fix, kindly upgrade to fix release.
 

https://prsearch.juniper.net/problemreport/PR1735539
Resolved-In

evo:24.2R2-EVO junos:21.2R3-S8 junos:22.2R3-S4 junos:22.4R3-S3 junos:23.2R2-S2 junos:23.4R1-S1-J3 junos:23.4R2 junos:24.2R1 junos:24.2R2 junos:24.3R1

Modification History

Initial Draft : 21/06/2024

Related Information

The issue is related to PR1735539.
To obtain a permanent fix, kindly upgrade to fix release.
 
Resolved-In

evo:24.2R2-EVO junos:21.2R3-S8 junos:22.2R3-S4 junos:22.4R3-S3 junos:23.2R2-S2 junos:23.4R1-S1-J3 junos:23.4R2 junos:24.2R1 junos:24.2R2 junos:24.3R1