The article explains the behaviour on Junos platforms with MPC10E, the below logs keep showing after frequent interface flaps. The below logs can be obtained from FPC linux var/logs pfe.logs* messages.<Timestamp> <Platform> ifd_process_flaps_common: IFD et-x/y/z - pfe link UP processing is complete<Timestamp> <Platform> <FPC> user.alert aftd-trio: [Alert] IF:Ifd new linkState, ifdName: et-x/y/z ifdIndex: 497 state:Up<Timestamp> <Platform> <FPC> MQSS(2): QUADPORT2: Cleared Ethernet MAC Remote Fault Delta Event for Port 0 (et-x/y/z)<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete<Timestamp> <Platform> <FPC> smic_filter_link_state_zt: Physical link et-x/y/z is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete
During the message outputting, the reported interface might in an unstable status, even the message indicates that "Physical link et-x/y/z is now UP". The interface might be in UP state, but not working properly.
Customer may report that the link went down after flapping issue on DWDM transport. The DWDM transport issue may have resolved but the interface is still remaining down.During the issue time, the below logs are obtained in FPC linux var/logs pfe.logs* messages and should be continuous.The interface may go down after few continuous flap.This is very rare scenario.pfe.log.8:[Jun 16 15:05:47.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:48.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:49.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:50.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:51.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:52.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:53.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:54.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:55.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete pfe.log.8:[Jun 16 15:05:56.199 LOG: Info] smic_filter_link_state_zt: Physical link et-4/0/1 is now UP but the previous link down processing is still in progress Will defer interface UP until link down is complete Physical interface: et-4/0/1, Enabled, Physical link is Down Interface index: 278, SNMP ifIndex: 606, Generation: 281 Description: Documentation link Link-level type: Ethernet, MTU: 9192, MRU: 9200, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled Pad to minimum frame size: Disabled Device flags : Present Running Down Interface Specific flags: Internal: 0x100200 Interface flags: Hardware-Down SNMP-Traps Internal: 0x4004000 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Schedulers : 0 Hold-times : Up 0 ms, Down 0 ms Damping : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed Current address: 5c:5e:ab:44:6f:cf, Hardware address: 5c:5e:ab:44:6a:8f Last flapped : 2024-06-16 20:34:58 WIB (16:56:03 ago) Statistics last cleared: Never Traffic statistics: Input bytes : 851574744948304 198984 bps Output bytes : 81592273943619 14136 bps Input packets: 777915631734 224 pps Output packets: 106835461010 4 pps IPv6 transit statistics: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Label-switched interface (LSI) traffic statistics: Input bytes : 0 0 bps Input packets: 0 0 pps Dropped traffic statistics due to STP State: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Input errors: Errors: 1, Drops: 0, Framing errors: 1, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0 Output errors: Carrier transitions: 768, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0 Egress queues: 8 supported, 8 in use
The issue is related to a race condition between pic periodic and pfeman thread for the MPC10E card.To recover from this state, perform a restart for the FPC PIC via CLI for the specific port which is et-4/0/1 in the above case.
<<<request chassis pic offline fpc-slot 4 pic-slot 0>>>
{master}> show log messages.0.gz | match et-4/0/1 | match 606Jun 18 00:00:39Jun 14 08:59:19.063 2024 mib2d[39019]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName et-4/0/1Jun 14 08:59:19.733 2024 mib2d[39019]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 606, ifAdminStatus up(1), ifOperStatus up(1), ifName et-4/0/1Jun 14 09:16:37.803 2024 mib2d[39019]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName et-4/0/1Jun 14 09:16:39.433 2024 mib2d[39019]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 606, ifAdminStatus up(1), ifOperStatus up(1), ifName et-4/0/1Jun 14 09:36:52.642 2024 mib2d[39019]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName et-4/0/1 Jun 18 00:26:23.940 2024 chassisd[35606]: %DAEMON-5-CHASSISD_PIC_OFFLINE_NOTICE: Taking PIC 0 in FPC 4 offline: Offlined by cli command Jun 18 00:30:02.508 2024 chassisd[35606]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 5, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: MRATE-5xQSFPP @ 4/0/*, jnxFruType 11, jnxFruSlot 4, jnxFruOfflineReason 2, jnxFruLastPowerOff 370906347, jnxFruLastPowerOn 370927909)Jun 18 00:30:02.509 2024 chassisd[35606]: %DAEMON-5-CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 138, entStateAdmin 4, entStateAlarm 128)Jun 18 00:30:02.509 2024 chassisd[35606]: %DAEMON-5-CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged Jun 18 00:30:41.000 2024 : %PFE-1: fpc4 user.alert aftd-trio: [t:2351] [Alert] IF:Ifd new linkState, ifdName:et-4/0/1 ifdIndex:412 state:UpJun 18 00:30:41.510 2024 lacpd[35671]: %DAEMON-5-LACP_INTF_MUX_STATE_CHANGED: ae23: et-4/0/1: Lacp state changed from DETACHED to WAITING, actor port state : |-|-|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|, partner port state : |-|-|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|
The above issue is related to PR1735539.To obtain a permanent fix, kindly upgrade to fix release.
https://prsearch.juniper.net/problemreport/PR1735539Resolved-Inevo:24.2R2-EVO junos:21.2R3-S8 junos:22.2R3-S4 junos:22.4R3-S3 junos:23.2R2-S2 junos:23.4R1-S1-J3 junos:23.4R2 junos:24.2R1 junos:24.2R2 junos:24.3R1
Initial Draft : 21/06/2024
The issue is related to PR1735539.To obtain a permanent fix, kindly upgrade to fix release. Resolved-Inevo:24.2R2-EVO junos:21.2R3-S8 junos:22.2R3-S4 junos:22.4R3-S3 junos:23.2R2-S2 junos:23.4R1-S1-J3 junos:23.4R2 junos:24.2R1 junos:24.2R2 junos:24.3R1