Description

QFX5130-32CD | 23.4R2.10-EVO | Interface not going down when connected device port (NVIDIA Server) port is made down

Symptoms

After executing "ip link set dev ens3f0np0 down" CLI on the server, the server interface status is down. 

If we check the interface status on the switch is always up.

Topology:

QFX5130-32CD-------QDD-2X200G-AOC-3M-------NVIDIA CX-7 200GE/NDR200

Data collection:

Following outputs were collected from customer device; where et-0/0/4:1 is the connecting interface:

root@router> show interfaces et-0/0/4:1 terse 
root@router> show interfaces extensive et-0/0/4:1
root@router> show interfaces diagnostics optics et-0/0/4

root@router> start shell 
[vrf:none] root@router:~# date
Tue Jun  3 16:02:38 UTC 2025
[vrf:none] root@router:~# pidof picd
22164 --> process number identified for picd
[vrf:none] root@router:~# pidof evo-pfemand
12706 --> process number identified for evo-pfemand
[vrf:none] root@router:~#cli

root@router> show trace application picd pid 22164 --> used the pid derived above.
root@router> show trace application evo-pfemand pid 12706 --> used the pid derived above.

root@router>start shell pfe network fpc0
root@router:pfe> show picd link-history fpc 0 pic 0 port 4 chan 1 | no-more

Solution

Output from the device

 

root@router> show picd link-history fpc 0 pic 0 port 4 chan 1 | no-more    

    description     : List of last 50 events, latest first
    channel-0/0/4:1 :
    Index   Time Stamp                   Event
    -----   --------------------------   --------------------------------------------
    1   2025-05-22 09:30:11.980862   Mac RF Tx clear
    2   2025-05-22 09:29:56.336538   Mac RF Tx clear
    3   2025-05-22 09:29:56.336527   Serdes Tune Ok
    4   2025-05-22 09:29:56.336270   Soaked Xcvr rx up
    5   2025-05-22 09:29:46.837915   Mac RF Tx set
    6   2025-05-22 09:29:46.837166   Raw status change down -> up lf=0 rf=0 pcs=1 ----------------------> Loca fault "0" that means link is still receiving the MAC Signal
    7   2025-05-22 09:29:46.538304   Pfe PortE state Up
    8   2025-05-22 09:29:46.336749   Xcvr Rx OK change (0 -> 1)


Further issue was identified on customer device:

 

The customer NVIDIA Server had a keepalive feature enabled:
 
The cause of the problem has been found, the Mellanox NIC firmware has a feature called "KEEP_ETH_LINK_UP" which will always keep the physical link to the NIC port active, 
even if the port has been shutdown at the OS logical level. 

This will always keep the physical link to the NIC port active, even if the port has been shutdown at the OS logical level.This feature is enabled by default.

Use the following cli can change the default action on the NVIDIA server:
sudo mlxconfig -d /dev/mst/mt4129_pciconf0 set KEEP_ETH_LINK_UP_P1=0 
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf0.1 set KEEP_ETH_LINK_UP_P1=0 
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf1 set KEEP_ETH_LINK_UP_P1=0
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf1.1 set KEEP_ETH_LINK_UP_P1=0
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf0 set KEEP_ETH_LINK_UP_P2=0
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf0.1 set KEEP_ETH_LINK_UP_P2=0
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf1 set KEEP_ETH_LINK_UP_P2=0
 
sudo mlxconfig -d /dev/mst/mt4129_pciconf1.1 set KEEP_ETH_LINK_UP_P2=0

Modification History

2025-06-06 : Article Created

2026-08-02 : Solution, Symptoms updated.