Description

For some type of MPCs , if it enabled "inline-services flex-flow-sizing" with large scaled routes , then it might encountered crash due to "Assertion failed" in a rare situation.

 

Symptoms

With inline-services flex-flow-sizing, the programmed flow-table-size for each family would be 64*256K .
With large scale routes , the route-record would be large too.
user@router# run show services accounting status inline-jflow fpc-slot 1    
  Status information
    FPC Slot: 1
    IPV4 export format: Version-IPFIX, IPV6 export format: Not set
    BRIDGE export format: Not set, MPLS export format: Not set
    IPv4 Route Record Count: 4000020, IPv6 Route Record Count: 0, MPLS Route Record Count: 0
    Route Record Count: 4000020, AS Record Count: 5
    Route-Records Set: Yes, Config Set: Yes
    Service Status: PFE-0: Steady PFE-1: Steady 
    Using Extended Flow Memory?: PFE-0: No PFE-1: No 
    Flex Flow Sizing ENABLED?: PFE-0: Yes PFE-1: Yes 
    IPv4 MAX FLOW Count: 16777220, IPv6 MAX FLOW Count: 16777220
    BRIDGE MAX FLOW Count: 16777220, MPLS MAX FLOW Count: 16777220

    MAX supported Flow Table size: 48


When PFE memory pool utilization is high , which shared by flow-table and sample route-record , the FPC might encounter a crash while handling memory related operations. Coredump files can be seen as below:

user@router# run show system core-dumps 
-rw-r--r--  1 root  wheel  527600584 Apr 25 12:25 /var/crash/core-router-fpc1-smpc.elf.0.tgz
-rw-r--r--  1 root  wheel  572614190 Apr 26 07:48 /var/crash/core-router-fpc1-smpc.elf.1.tgz
-rw-r--r--  1 root  wheel  523528863 Apr 26 08:58 /var/crash/core-router-fpc1-smpc.elf.2.tgz

Solution

The problem has been fixed in PR1798466.

Impacted versions:
evo:21.2R3-EVO evo:21.3R2-EVO evo:21.4R2-EVO evo:22.1R1-EVO junos:20.2R3-S4 junos:20.3X75-D40 junos:21.1R3 junos:21.2R3 junos:21.3R2 junos:21.4R1 junos:21.4R2 junos:22.1R1 and later

Resolved-in versions:
evo:24.2R1-EVO evo:24.3R1-EVO junos:23.4R2 junos:24.2R1

All Trio based line cards except below are affected:
MPC3D, MPC Type 1, MPC Type 2, MPCE Type 2, MPCE Type 3

Workaround:
Either reduce the routes scale, or JFLOW/Flex flow configuration needs to be set to acceptable limits. For example, not using flex-flow-sizing, but static configure like below 
inline-services {
    flow-table-size {
        ipv4-flow-table-size 24;
        ipv6-flow-table-size 8;         
        mpls-flow-table-size 3;
    }
}

 

Modification History

2024-05-06 : Article Created