Customer's Junos: 23.4R2-S2.1 Hardware: EX4100-48MP (5-member VC)
We observed core-dumps on the switch and pfe crashed:
root@router> show system core-dumps fpc0: -------------------------------------------------------------------------- /var/crash/*core*: No such file or directory -rw------- 1 root wheel 28862872 Nov 8 08:36 /var/tmp/fpc1_fxpc.core.0.gz -rw------- 1 root wheel 25856874 Nov 8 08:36 /var/tmp/fpc1_fxpc.core.1.gz -rw------- 1 root wheel 25781129 Nov 8 08:36 /var/tmp/fpc1_fxpc.core.2.gz -rw------- 1 root wheel 25800897 Nov 8 08:36 /var/tmp/fpc2_fxpc.core.0.gz -rw------- 1 root wheel 25565509 Nov 8 08:36 /var/tmp/fpc3_fxpc.core.0.gz -rw------- 1 root wheel 26126890 Oct 19 20:27 /var/tmp/fxpc.core.0.gz -rw------- 1 root wheel 26088665 Oct 23 22:01 /var/tmp/fxpc.core.1.gz -rw------- 1 root wheel 26330298 Nov 9 09:58 /var/tmp/fxpc.core.2.gz
Following logs observed:
Nov 8 00:45:49 USHO-cLAN-CRB-A01 fpc2 (pkt tx) allocation of bcm pkt failed Nov 8 00:45:49 USHO-cLAN-CRB-A01 last message repeated 6 times Nov 8 00:45:50 USHO-cLAN-CRB-A01 fpc2 CLKSYNC: failed to create socket, port Nov 8 00:45:54 USHO-cLAN-CRB-A01 last message repeated 4 times Nov 8 00:45:54 USHO-cLAN-CRB-A01 fpc2 (pkt tx) allocation of bcm pkt failed Nov 8 00:46:41 USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket Nov 8 00:46:41 USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket Nov 8 00:46:41 USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket Nov 8 00:55:03 USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket Nov 8 00:55:03 USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket Nov 8 00:55:04 USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket Nov 8 00:55:04 USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket Nov 8 00:55:04 USHO-cLAN-CRB-A01 fpc2 bulkget_reconnect_timer_event:1197 BULKGET: failed to create tnp socket Nov 8 00:55:04 USHO-cLAN-CRB-A01 fpc2 DDOS: failed to create tnp socket
Configuration on the switch:
forwarding-options { analyzer { erspan { input { ingress { interface all; } } output { ip-address 10.2.0.222; } } } }
If interfaces keep flapping; at some point we observe memory leak on the device. For version; 23.4R2-S2.1 signal to check is this bcb898:
root@router> start shell % % cprod -A fpc0 -c "show heap 0 accounting pc" | grep bcb898 00bcb898 23 3640 % cprod -A fpc0 -c "show heap 0 accounting pc" | grep bcb898 00bcb898 23 3640 %
Root Cause:
A memory leak occurs with ERSPAN when more than one input interface is configured and an input interface or analyzer undergoes any state change. When ERSPAN is set to use all interfaces as inputs, the leak increases as interface flapping becomes more frequent.Problem is fixed in below versions:
junos:23.2R2-S8 junos:23.4R2-S7 junos:24.2R2-S4 junos:24.4R2-S3 junos:24.4R2-S4 junos:25.2R2 junos:25.2R2-S1 junos:25.2R2-S2 junos:25.4R1 junos:25.4R1-S3 junos:25.4R2 junos:26.1R1