Description

The article explains a scenario where Evo-pfemand coredump

Symptoms

Sometimes customers might experience Evo-pfemand coredump on QFX5130-32CD. This might result in all the interfaces being down.

 show system core-dumps 
re0:
--------------------------------------------------------------------------
-rw-------  1 root  root    24985936 Jun 29 22:18 /var/core/re0/evo-pfemand.re.re0.7182.2024_06_29.22_17_03.tar.gz
-rw-------  1 root  root    24957472 Jun 29 22:18 /var/core/re0/evo-pfemand.re.re0.8599.2024_06_29.22_17_14.tar.gz


 

On executing below command, we could see the application failure for evo-pfemand

 

root@QFX5130-32CD> show system errors active detail

System Active Errors Detail Information
RE 0
----------------------------------------------------------------
Error Name            : application_major_failure
Identifier            : /re/0/emfd-fpa/0/evo-pfemand:0/0/sysman/0/application_major_failure
Description           : application_major_failure
State                 : enabled
Scope                 : board
Category              : processing
Level                 : major
Threshold             : 1
Error limit           : 1
Occur count           : 1
Clear count           : 0

show log message
Jun 29 22:17:15  jtac-qfx5130-32cd mgd[4474]: UI_CMDLINE_READ_LINE: User 'root', command 'show platform binding-queue pending '
Jun 29 22:17:16  jtac-qfx5130-32cd root[9944]: savecore.sh: Started saving core for pid 8599
Jun 29 22:17:18  jtac-qfx5130-32cd root[9951]: savecore.sh: Finished saving core for pid 8599
Jun 29 22:17:18  jtac-qfx5130-32cd root[9952]: savecore.sh: Starting copy of /dev/shm/EvoSharedObjStore to /var/core/.staging/jtac-qfx5130-32cd-r2007/evo-pfemand.re.re0.8599.2024_06_29.22_17_14.sdb
Jun 29 22:17:18  jtac-qfx5130-32cd root[9954]: savecore.sh: Finished copy of /dev/shm/EvoSharedObjStore to /var/core/.staging/jtac-qfx5130-32cd-r2007/evo-pfemand.re.re0.8599.2024_06_29.22_17_14.sdb
Jun 29 22:17:18  jtac-qfx5130-32cd ppman[8642]: [t:8782] [Error] PPM:RPC: PpmRpcService::handleClientStatus: Connection failure server: AFTMAN reason: Connection Reset
Jun 29 22:17:18  jtac-qfx5130-32cd systemd[1]: evo-pfemand.service: Main process exited, code=dumped, status=11/SEGV
Jun 29 22:17:18  jtac-qfx5130-32cd sysman[8285]: SYSTEM_APP_OFFLINE_EVENT: App went offline re0-evo-pfemand
Jun 29 22:17:19  jtac-qfx5130-32cd systemd[1]: evo-pfemand.service: Failed with result 'core-dump'.
Jun 29 22:17:20  jtac-qfx5130-32cd sysman[8285]: SYSTEM_APP_OFFLINE_EVENT: App went offline re0-ppman
Jun 29 22:17:20  jtac-qfx5130-32cd sysman[8285]: SYSTEM_APP_OFFLINE_EVENT: App went offline re0-evoaft-jvisiond-brcm
Jun 29 22:17:20  jtac-qfx5130-32cd sysman[8285]: SYSTEM_APP_OFFLINE_EVENT: App went offline re0-packetio-brcm
Jun 29 22:17:20  jtac-qfx5130-32cd systemd[1]: evo-pfemand.service: Start request repeated too quickly.


Even after gracefully restarting the evo-pfemand process using 'restarting Evo-pfemand gracefully', the issue can persists.

 

Solution

The issue happens when a firewall filter that has routing-instance in the action is applied on irb interfaces that do not have 'l3-interface' knob configured on vlan could result in evo-pfemand coredump.

firewall {
    family inet {
        filter TEST {
            term 1 {
                from {
                    source-address {
                        10.1.1.2/32;
                    }
                }                       
                then {
                             routing-instance TEST;
                }
            }
        }
    }
routing-instances {
    TEST {
        instance-type vrf;
        interface irb.600;
        route-distinguisher 10.1.1.10:655;
        vrf-target target:10.1.1.10:655;
    }
}
vlan600 {
  vlan-id 600;

}

{master:0}

root@QFX5130-32CD> show configuration interfaces irb 

unit 600 {

  family inet {

    filter {

      input TEST;

    }

    address 10.1.1.1/24;

  }

}

 

The solution is to make sure the vlan has l3-interface knob enabled. In the above example, make the following configuration changes

 

set vlans vlan600 l3-interface irb.600

After the configuration change is performed, remove and restart the evo-pfemand using 'restart evo-pfemand gracefully'.
Whenever a vlan is associated with an irb, it is a must to configure 'l3-interface' knob for the vlan. 

Modification History

2024-06-25 : Article Created
2024-06-29: Approved