EX4400 virtual chassis running 24.2R2-S1 stopped forwarding traffic. The following logs could be observed:

fpc3 ERROR (dfw): rv -14 error "No resources for operation"

fpc3 ERROR (dfw): Could not create field entry, unit 0, group 181

fpc3 ERROR (dfw): status [-8] from brcm_dfw_rule_create_exp term (__dot1x_mge-3/0/47_34c91c9f8427-Onboarding-discardLinkLocal) dfw (dot1x_mge-3/0/47) unit 0

fpc3 ERROR (dfw): Cannot program filter "dot1x_mge-3/0/47" (type DOT1X) - TCAM has 23574 free entries and the filter requires 0 free entries

the switch seems to have self recovered. no core dumps were generated but the switch showed offline.

fpc3 ERROR (dfw): rv -14 error "No resources for operation"

fpc3 ERROR (dfw): Could not create field entry, unit 0, group 181

fpc3 ERROR (dfw): status [-8] from brcm_dfw_rule_create_exp term (__dot1x_mge-3/0/47_34c91c9f8427-Onboarding-discardLinkLocal) dfw (dot1x_mge-3/0/47) unit 0

fpc3 ERROR (dfw): Cannot program filter "dot1x_mge-3/0/47" (type DOT1X) - TCAM has 23574 free entries and the filter requires 0 free entries

This error indicates a TCAM resource exhaustion on FPC3. The DFW (Data plane Firewall) subsystem failed to install the filter due to insufficient TCAM entries (only 23,301 free reported). Analyzing the logs was determined that a catastrophic failure initiated by TCAM resource exhaustion led to the hang of the PFE, causing the traffic to stop forwarding. the TCAM resource exhaustion for 802.1X Filter Programming Failure was determined by these logs:

fpc3 ERROR (dfw): rv -14 error "No resources for operation" Cannot program filter "dot1x_mge-3/0/47" (type DOT1X) - TCAM has 23,301 free entries

fpc3 DFWE ERROR DFW: Filter : "dot1x_mge-3/0/47" is NOT programmed in HW

The TCAM programming issue led to Memory pressure begins escalating, these logs can be observed: JSYSMOND_LOW_MEM_MINOR_EVENT

Memory crisis intensifies: JSYSMOND_LOW_MEM_MAJOR_EVENT

KERNEL INTERVENTION: Memory exhaustion reaches critical threshold (3450 free pages vs 3463 threshold) with event processing backlogs (mib2d: 121,000 unprocessed events, dfwd: 46,000 unprocessed events) → kernel forcibly kills dfwd (PID 35826) and pfe-manager (PID 28750) to reclaim resources

Main initial trigger was a flapping condition of the physical interface that had the TCAM program issue, with the logs we were able to identify several link flaps of the same interface prior to the TCAM errors which then led to memory exhaustion.

Possible troubleshooting Steps for these scenarios

• Verify TCAM usage: Check the filters created and verify TCAM usage to confirm resource exhaustion on the affected FPC.
• Check filter count: Check the firewalls configuration for the protocol in question that could be affecting, like in this case to list active DOT1X filters and identify high consumers.
• Consolidate or remove unnecessary filters or extra terms in filters to reduce TCAM program.
• Identify if there is an specific interface in question, check for errors, carrier transitions, flaps constantly occurring. Change hardware if needed for the port.
• Restart affected FPC as a temporary workaround, but address root cause first.