Description

Customer's Junos: 23.4R2-S2.1
Hardware: EX4100-48MP (5-member VC)

Symptoms

We observed core-dumps on the switch and pfe crashed:

root@router> show system core-dumps 
fpc0:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-------  1 root  wheel   28862872 Nov 8  08:36 /var/tmp/fpc1_fxpc.core.0.gz
-rw-------  1 root  wheel   25856874 Nov 8  08:36 /var/tmp/fpc1_fxpc.core.1.gz
-rw-------  1 root  wheel   25781129 Nov 8  08:36 /var/tmp/fpc1_fxpc.core.2.gz
-rw-------  1 root  wheel   25800897 Nov 8  08:36 /var/tmp/fpc2_fxpc.core.0.gz
-rw-------  1 root  wheel   25565509 Nov 8  08:36 /var/tmp/fpc3_fxpc.core.0.gz
-rw-------  1 root  wheel   26126890 Oct 19 20:27 /var/tmp/fxpc.core.0.gz
-rw-------  1 root  wheel   26088665 Oct 23 22:01 /var/tmp/fxpc.core.1.gz
-rw-------  1 root  wheel   26330298 Nov 9  09:58 /var/tmp/fxpc.core.2.gz

 

Following logs observed:

Nov  8 00:45:49  USHO-cLAN-CRB-A01 fpc2 (pkt tx) allocation of bcm pkt failed
Nov  8 00:45:49  USHO-cLAN-CRB-A01 last message repeated 6 times
Nov  8 00:45:50  USHO-cLAN-CRB-A01 fpc2 CLKSYNC: failed to create socket, port  
Nov  8 00:45:54  USHO-cLAN-CRB-A01 last message repeated 4 times
Nov  8 00:45:54  USHO-cLAN-CRB-A01 fpc2 (pkt tx) allocation of bcm pkt failed

Nov  8 00:46:41  USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket 
Nov  8 00:46:41  USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket 
Nov  8 00:46:41  USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket 

Nov  8 00:55:03  USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket 
Nov  8 00:55:03  USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket 
Nov  8 00:55:04  USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket 
Nov  8 00:55:04  USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket 
Nov  8 00:55:04  USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket 
Nov  8 00:55:04  USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket 

Configuration on the switch:

forwarding-options {
    analyzer {
        erspan {
            input {
                ingress {
                    interface all;
                }
            }
            output {
                ip-address 10.2.0.222;
            }
        }
    }
}


If interfaces keep flapping; at some point we observe memory leak on the device. For version; 23.4R2-S2.1 signal to check is this bcb898:
 
root@router> start shell
%

% cprod -A fpc0 -c "show heap 0 accounting pc" | grep bcb898
 00bcb898        23       3640
% cprod -A fpc0 -c "show heap 0 accounting pc" | grep bcb898
 00bcb898        23       3640
% 

Solution

Root Cause:

 

A memory leak occurs with ERSPAN when more than one input interface is configured and an input interface or analyzer undergoes any state change. 
When ERSPAN is set to use all interfaces as inputs, the leak increases as interface flapping becomes more frequent.

Problem is fixed in below versions:
junos:23.2R2-S8 junos:23.4R2-S7 junos:24.2R2-S4 junos:24.4R2-S3 junos:24.4R2-S4 junos:25.2R2 junos:25.2R2-S1 junos:25.2R2-S2 junos:25.4R1 junos:25.4R1-S3 junos:25.4R2 junos:26.1R1

Modification History

2025-12-18 : Article Created