This article describes a memory leak in "bcm_sdk" that occurs when the filter group used for 802.1X is created and then deleted. PFE crashes due to memory exhaustion and creates an fxpc.core core-dump.
Symptoms
Following logs can be seen showing a memory leak:
Feb 2 13:08:50 user dc-pfe[70602]: Could not allocate 338112 memory port schedule state mem_ptr 0x0
Feb 2 13:08:50 user fpc0 Could not allocate 338112 memory port schedule state mem_ptr 0x0
PFE logs indicate issues when creating the dynamic filter group used for 802.1X:
[Thu Feb 2 13:37:54.476 LOG: Err] ERROR (dfw): No resources for operation:brcm_dfw_field_group_create(gid:181) Failed in unit/pipe 0/0
[Thu Feb 2 13:37:54.477 LOG: Err] ERROR (dfw): failed to create group 181 in unit 0
[Thu Feb 2 13:37:54.477 LOG: Err] ERROR (dfw): rv -14 error "No resources for operation"
[Thu Feb 2 13:37:54.478 LOG: Err] ERROR (dfw): No resources for operation: group(181) creation failedfor unit 0
[Thu Feb 2 13:37:54.478 LOG: Err] ERROR (dfw): dynamic group init failed for IFP-DYN-DOT1X-GROUP:181 for unit 0
[Thu Feb 2 13:38:45.999 LOG: Err] ERROR (dfw): No resources for operation:brcm_dfw_field_group_create(gid:181) Failed in unit/pipe 0/0
[Thu Feb 2 13:38:46.000 LOG: Err] ERROR (dfw): failed to create group 181 in unit 0
[Thu Feb 2 13:38:46.000 LOG: Err] ERROR (dfw): rv -14 error "No resources for operation"
[Thu Feb 2 13:38:46.000 LOG: Err] ERROR (dfw): No resources for operation: group(181) creation failedfor unit 0
[Thu Feb 2 13:38:46.001 LOG: Err] ERROR (dfw): dynamic group init failed for IFP-DYN-DOT1X-GROUP:181 for unit 0
[Thu Feb 2 13:39:36.831 LOG: Err] ERROR (dfw): No resources for operation:brcm_dfw_field_group_create(gid:181) Failed in unit/pipe 0/0
[Thu Feb 2 13:39:36.831 LOG: Err] ERROR (dfw): failed to create group 181 in unit 0
[Thu Feb 2 13:39:36.832 LOG: Err] ERROR (dfw): rv -14 error "No resources for operation"
[Thu Feb 2 13:39:36.832 LOG: Err] ERROR (dfw): No resources for operation: group(181) creation failedfor unit 0
[Thu Feb 2 13:39:36.833 LOG: Err] ERROR (dfw): dynamic group init failed for IFP-DYN-DOT1X-GROUP:181 for unit 0
[Thu Feb 2 13:39:36.913 LOG: Debug] brcm_as_mbv_add_prefix:256 bcm_vlan_mac_add unit:0 hw_token:3 bd_index: 2mac = 0:c5:85:13:db:ed
"Bcm_sdk" heap memory utilization is high:
TFXPC0(user vty)# show heap
ID Base Total(b) Free(b) Used(b) % Name
-- ---------- ----------- ----------- ----------- --- -----------
0 22600468 536870488 352165096 184705392 34 Kernel
1 4e800000 8388608 5906456 2482152 29 DMA
2 4f000000 12582912 12278368 304544 2 PKT DMA DESC
3 42601240 201326592 37956120 163370472 81 Bcm_sdk
4 4fc00000 46137344 23708808 22428536 48 Packet DMA
Device generates fxpc.cores core-dumps:
root@user> show system core-dumps no-forwarding
/var/crash/*core*: No such file or directory
-rw------- 1 root wheel 13485450 Dec 1 08:14 /var/tmp/fxpc.core.0.gz
-rw------- 1 root wheel 13577051 Dec 16 10:14 /var/tmp/fxpc.core.1.gz
-rw------- 1 root wheel 13071026 Jan 6 07:54 /var/tmp/fxpc.core.2.gz
-rw------- 1 root wheel 13227849 Jan 20 08:47 /var/tmp/fxpc.core.3.gz
-rw------- 1 root wheel 13249118 Feb 3 09:57 /var/tmp/fxpc.core.4.gz
total files: 5
On Junos EX4100/4400/4650 and QFX5120 platforms, when dot1x authentication is used, for each client that is authenticated, dynamic filters are created under a shared filter group. These filters are removed when the client de-authenticates, and once the last client disconnects, the entire filter group is deleted. The memory leak happens specifically at this point when the filter group is deleted after the final client logs off. As a result, each full cycle of client disconnection causes a small amount of memory to be lost. Over time, repeated connections and disconnections lead to memory exhaustion, which causes a PFE crash impacting the traffic.
The root cause is that memory is leaked whenever the last authenticated dot1x client on the system de-authenticates, triggering the deletion of the associated dynamic filter group. This is a system-wide behavior the leak does not depend on which client or port is involved. Over time, repeated authentication/de-authentication cycles lead to cumulative memory loss. The memory leak is not tied to any fixed time interval or specific client/port.
This is address under PR1875983:
https://prsearch.juniper.net/problemreport/PR1875983
Restarting the PFE or rebooting the device is the workaround to recover from this state. However, the issue is resolved in the following versions:
Resolved-In