On SRX3xx devices where VRRP is configured on multiple IRB interfaces, the Virtual IP (VIP) for certain VLANs may become unreachable. While the physical interface IP remains reachable, traffic destined for the VIP fails, leading to disrupted communication for the affected VLAN.
This issue arises due to a limitation in the Broadcom (BCM) L2 cache, which has a maximum capacity of 128 entries. The VRRP virtual MAC addresses are not guaranteed to be installed into the L2 cache when this limit is reached, causing the VIP to be non-functional for some VLANs.
Ping to the VIP fails, while the local interface IP remains reachable.
ARP resolution for the VIP MAC address is successful, but the traffic does not flow.
Example:
srx> show vrrp brief
irb.504 up 100 master Active A 0.339 lcl 10.10.50.3 <-- Ping works.
vip 10.10.50.1 <-- Ping fails.
L2 Cache Diagnostics:
The VRRP MAC for the non-functional VLAN is missing in the L2 cache.
In order to retrieve the vmac for irbs (it's the same for all irbs)
srx> show vrrp interface irb.504 | match mac <<< non working vlan
Virtual Mac: 00:00:5e:00:01:64
srx> show vrrp interface irb.878 | match mac <<< working vlan
Once you find the vmac address, you will look for it under the l2 cache list on the bcm level.
srx> request pfe execute target fwdd command "set jbcm command \"l2 cache show\"" | grep 504 <<< use root or any super-user class to issue this command
<<<<< no output found for vlan 504
whereas a random working vlan vmac address is present
srx>request pfe execute target fwdd command "set jbcm command \"l2 cache show\"" | grep 878
118 : mac=00:00:5e:00:01:64 vlan=878 trunk=40 CPU lookup_class =0
You may find other entries for non working vlan on "l2 list" (not "l2 cache show") but not the vmac that we are looking for
srx> request pfe execute target fwdd command "set jbcm command \"l2 show\"" | grep 504
mac=00:40:58:14:41:01 vlan=912 GPORT=0x0 Trunk=10 Hit <<<<<<<< this is a different vmac from the VIP
Errors in log messages indicate failure to add or delete VRRP MAC addresses in the BCM L2 cache.
Nov 15 04:58:34 fw1 : %PFE-3: :jbcm_irb_acl_delete_mac: JBCM_DEBUG: delete irb address for vlan : 504 mac: 01:00:5e:00:00:12
Nov 15 04:58:34 fw1 : %PFE-3: JBCM(0/0):jbcm_irb_acl_delete_mac: delete address from l2 cache error: Entry not found <-- failed
Or
Nov 15 05:01:58 fw1 vrrpd[6883]: %DAEMON-4-VRRPD_NEW_MASTER: Interface irb.504 (local address 10.10.50.3) became VRRP master for group 100 with master reason notMaster
Nov 15 05:01:58 fw1 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:504, lower 2bytes mac:01:64. <-- failed
Temporary Workarounds:
Restart VRRP gracefully:
root@fw1> restart vrrp gracefully
Note: This may temporarily resolve the issue for the affected VLAN but lead to other VLANs experiencing similar issues.
Reboot the device:
If restarting VRRP does not resolve the issue, rebooting the device may help.
Permanent Mitigation:
Reduce the number of IRB interfaces with VRRP:
Keep the number of VLANs installed under l2 cache under 128 to prevent exceeding the L2 cache limit. 128 is the number of entries for vrrp vlans along with other l2 protocols BPDU/STP
Monitor Failures Using SNMP Traps:
Configure event policies to generate SNMP traps when L2 cache issues occur.
Example Configuration:
root@fw1> show configuration event-options
policy PFE {
events PFE;
attributes-match {
PFE.message matches "JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:";
}
then {
raise-trap;
traceoptions {
file event-log-l2 size 10m files 10;
flag all;
Diagnostic Commands:
when the issue happens you will find the below log messages and snmp traps generated
root@fw1_0> restart vrrp gracefully
Nov 20 16:11:43
Virtual Router Redundancy Protocol process signalled but still running, waiting 28 seconds more
Virtual Router Redundancy Protocol process started, pid 11048
root@fw1_0> show log messages | match vid | last 20
Nov 20 16:12:39
Nov 20 16:11:59 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:900, lower 2bytes mac:01:64
Nov 20 16:11:59 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:994, lower 2bytes mac:01:64
Nov 20 16:12:05 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:904, lower 2bytes mac:01:64
Nov 20 16:12:05 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:912, lower 2bytes mac:01:64
Nov 20 16:12:06 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:990, lower 2bytes mac:01:64
Nov 20 16:12:06 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:997, lower 2bytes mac:01:64
Nov 20 16:12:06 fw1_0 : %PFE-3: JBCM:jbcm_jvlan_mac_control: unit 0 add VRRP mac failed, vid:791, lower 2bytes mac:01:64
root@fw1_0> show log event-log-l2 | match vid
Nov 20 16:12:27
Nov 20 16:11:59 Trap Object/Value <jnxSyslogEventName[730] = jnxSyslogHostName[730] = fw1_0,jnxSyslogMessage[730] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:900\,\ lower\ 2bytes\ mac:01:64
Nov 20 16:12:00 Trap Object/Value <jnxSyslogEventName[731] = jnxSyslogHostName[731] = fw1_0,jnxSyslogMessage[731] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:994\,\ lower\ 2bytes\ mac:01:64
Nov 20 16:12:05 Trap Object/Value <jnxSyslogEventName[732] = jnxSyslogHostName[732] = fw1_0,jnxSyslogMessage[732] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:904\,\ lower\ 2bytes\ mac:01:64
Nov 20 16:12:06 Trap Object/Value <jnxSyslogEventName[733] = jnxSyslogHostName[733] = fw1_0,jnxSyslogMessage[733] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:912\,\ lower\ 2bytes\ mac:01:64
Nov 20 16:12:06 Trap Object/Value <jnxSyslogEventName[734] = jnxSyslogHostName[734] = fw1_0,jnxSyslogMessage[734] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:990\,\ lower\ 2bytes\ mac:01:64
Nov 20 16:12:06 Trap Object/Value <jnxSyslogEventName[735] = jnxSyslogHostName[735] = fw1_0,jnxSyslogMessage[735] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:997\,\ lower\ 2bytes\ mac:01:64
Nov 20 16:12:06 Trap Object/Value <jnxSyslogEventName[736] = jnxSyslogHostName[737] = fw1_0,jnxSyslogMessage[736] = JBCM:jbcm_jvlan_mac_control:\ unit\ 0\ add\ VRRP\ mac\ failed\,\ vid:791\,\ lower\ 2bytes\ mac:01:64