Description

After reboot of QFX5100, some of the interfaces that are directly connected to servers may continuously flap. This article suggests a workaround and also recommends upgrading the NIC firmware on the server side to resolve the problem.

 

Symptoms

Observe the following output: 

root@JTAC> show interfaces terse xe-0/0/17 
Dec 06 02:21:03
Interface               Admin Link Proto    Local                 Remote
xe-0/0/17               up    down
xe-0/0/17.0             up    down inet     10.10.10.2/24   

{master:0}
root@JTAC> show interfaces terse xe-0/0/17    
Dec 06 02:21:04
Interface               Admin Link Proto    Local                 Remote
xe-0/0/17               up    down
xe-0/0/17.0             up    down inet     10.10.10.2/24   

{master:0}
root@JTAC> show interfaces terse xe-0/0/17    
Dec 06 02:21:06
Interface               Admin Link Proto    Local                 Remote
xe-0/0/17               up    down
xe-0/0/17.0             up    down inet     10.10.10.2/24   

{master:0}
root@JTAC> show interfaces terse xe-0/0/17    
Dec 06 02:21:07
Interface               Admin Link Proto    Local                 Remote
xe-0/0/17               up    up
xe-0/0/17.0             up    up   inet     10.10.10.2/24   

{master:0}
root@JTAC> show interfaces terse xe-0/0/17    
Dec 06 02:21:08
Interface               Admin Link Proto    Local                 Remote
xe-0/0/17               up    down
xe-0/0/17.0             up    down inet     10.10.10.2/24   

{master:0}
root@JTAC> show interfaces terse xe-0/0/17    
Dec 06 02:21:09
Interface               Admin Link Proto    Local                 Remote
xe-0/0/17               up    down
xe-0/0/17.0             up    down inet     10.10.10.2/24

In the logs, we can see the link flapping as shown below: 

root@JTAC> show log messages | last 100 
Dec 06 02:21:25
Dec  6 02:20:16  JTAC mib2d[4770]: SNMP_TRAP_LINK_UP: ifIndex 672, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/17
Dec  6 02:20:17  JTAC mib2d[4770]: SNMP_TRAP_LINK_DOWN: ifIndex 672, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/0/17
Dec  6 02:20:18  JTAC mib2d[4770]: SNMP_TRAP_LINK_UP: ifIndex 672, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/17
Dec  6 02:20:19  JTAC mib2d[4770]: SNMP_TRAP_LINK_DOWN: ifIndex 672, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/0/17
Dec  6 02:20:21  JTAC mib2d[4770]: SNMP_TRAP_LINK_UP: ifIndex 672, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/17
Dec  6 02:20:22  JTAC mib2d[4770]: SNMP_TRAP_LINK_DOWN: ifIndex 672, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/0/17
Dec  6 02:20:24  JTAC mib2d[4770]: SNMP_TRAP_LINK_UP: ifIndex 672, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/17
Dec  6 02:20:25  JTAC mib2d[4770]: SNMP_TRAP_LINK_DOWN: ifIndex 672, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/0/17
Dec  6 02:20:26  JTAC mib2d[4770]: SNMP_TRAP_LINK_UP: ifIndex 672, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/17

On the server side, we can see the following interface status: 

[root@localhost net]# ethtool enp17s0f0
Settings for enp17s0f1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                1000baseKX/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseT/Full
                                1000baseKX/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no                        <<<<<<<<Link showing down on the server</span>

 

Solution

This issue is related to the Server NIC firmware (INTEL NIC) auto-negotiation stability issue.

 

To resolve the issue, upgrade the NIC firmware on the server side.

Meanwhile, a workaround is to restart auto-negotiation on the server side by using the following command: 

[root@localhost ~]# ethtool -r <server interface>