This article explains the HA data forwarding behavior change of the integrated routing and bridging (IRB) interface in Chassis Cluster after the fix of PR1042588 - High PFE CPU utilization appears in the secondary node.
In a scenario where an Address Resolution Protocol (ARP) request packet with the destination media access control (MAC) as IRB’s MAC address, the ARP packet is forwarded to the Primary Node, then broadcasted in the security zone. This ARP packet may cause a traffic loop which results in high PFE CPU utilization in the Secondary Node. With the fix of PR1042588 , the HA data forwarding behavior has changed. Packets whose destination is IRB interface's MAC address that arrives on the Secondary Node no longer forwards to the Primary Node. PR1042588 is fixed in Junos 12.1X44-D60, 12.1X46-D45 and 15.1X49-D30. Please note that 12.1X46-D45 is still missing in "Resolved in" field when this KB is created.
The following setup shows SRX240 is running L2 mode with Chassis Cluster and the IRB interface is defined to facilitate in-band management traffic. Both RG0 and RG1 are active in SRX240A (Node0). SRX240B (Node1) is the secondary node. Note that SRX is running 12.1X44-D50 without the fix of PR1042588 .
+-----+-+ +-------+ +-----+-+ |Host +------ge-0/0/8+SRX240A+ge-0/0/9-----eth3+Linux | +-----+-+ +---+---+ +-----+-+ | +-------+ +---+---+ | PC +------ge-5/0/8+SRX240B| +-------+ +-------+
The following example of tcpdump showing the PC is sending an ARP Request destined to IRB MAC(00:24:dc:1c:50:af) to IRB interface in Node1 via reth0 in V1-Trust zone. The ARP Request forwarded from the Secondary node(node1) is flooded out to reth1 of V1-Untrust in Node0.
tcpdump
[root@Saturn ~]# tcpdump -i eth3 -n -e tcpdump: WARNING: eth3: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3, link-type EN10MB (Ethernet), capture size 96 bytes 21:28:21.546382 00:11:43:00:00:01 > 00:24:dc:1c:50:af , ethertype ARP (0x0806), length 60: arp who-has 9.9.9.100 tell 9.9.9.11 21:28:21.546616 00:11:43:00:00:01 > 00:24:dc:1c:50:af , ethertype ARP (0x0806), length 60: arp who-has 9.9.9.100 tell 9.9.9.11
The following SRX interface counter shows ARP Request is received in Node0 from Node1 via fabric interface, and then sending out to forwarding interface in Node0. ge-0/0/8 & ge-5/0/8 = reth0 in V1-Trust ge-0/0/9 & ge-5/0/9 = reth1 in V1-Untrust
{primary:node0} [email protected]> show interfaces ge-0/0/9 | match pps Input rate : 0 bps (0 pps) Output rate : 593176 bps (1235 pps) <==ARP request send out via Untrust interface in Node0 {primary:node0} [email protected]> show interfaces ge-0/0/8 | match pps Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) <==No Packet send out via Trust interface in Node0 {primary:node0} [email protected]> show interfaces fab0 | match pps Input rate : 2079896 bps (1733 pps) <==HA data link in Node0 is receiving ARP request from Node1 Output rate : 2264 bps (1 pps) {primary:node0} [email protected]> show interfaces fab1 | match pps Input rate : 2328 bps (1 pps) Output rate : 1304896 bps (1117 pps) <==HA data link in Node1 is sending ARP request to Node0
The fix for PR1042588 is included in 12.1X44-D60, 12.1X46-D45 and 15.1X49-D30. After the fix, HA data forwarding behavior has changed and packets whose destination is IRB interface's MAC address that arrives on the Secondary Node no longer forwards to the Primary Node. The following test shows HA data forwarding no longer happens with ARP request packet destined to IRB interface arriving in the Secondary Node. Note that it was tested with the internal build including the fix for PR104588. Primary Node - No ARP traffic received from Secondary Node
{primary:node0} [email protected]> show interfaces ge-0/0/8 | match pps Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) {primary:node0} [email protected]> show interfaces ge-0/0/9 | match pps Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) {primary:node0} [email protected]> show interfaces ge-5/0/9 | match pps Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) {primary:node0} [email protected]> show interfaces ge-5/0/8 | match pps Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) {primary:node0} [email protected]> show interfaces fab0 | match pps Input rate : 2320 bps (1 pps) <==HA data link in Node0 no longer receive packet Output rate : 2264 bps (1 pps) Statistics Packets pps Bytes bps {primary:node0} [email protected]> show interfaces fab1 | match pps Input rate : 2328 bps (1 pps) Output rate : 2264 bps (1 pps) <==HA data link in Node1 no longer forward packet Statistics Packets pps Bytes bps
Secondary Node - ARP Request arrivied on ge-5/0/8 (i.e. GRPKT.ge11 ) at around 2k pps but not forwarded to Node0. The Switch chip counter in Node1 shows that GE5( fab1 GTPKT.ge5 ) is not sending any packets despite GE11(ge-5/0/8) is receiving ARP packets at 2K pps.
{secondary:node1} [email protected]> request pfe execute target fwdd command "set jbcm command \"show counter\"" | match PKT GOT: GRPKT.ge3 : 1,011 +1,011 1/s GOT: GTPKT.ge3 : 23 +23 GOT: GRPKT.ge4 : 31,207 +31,207 29/s GOT: GTPKT.ge4 : 21,458 +21,458 30/s GOT: GRPKT.ge5 : 861 +861 1/s GOT: GTPKT.ge5 : 859 +859 1/s GOT: GRPKT.ge11 : 845,279 +845,279 1,999/s GOT: GRPKT.ge23 : 23,373 +23,373 33/s GOT: GTPKT.ge23 : 32,691 +32,691 33/s GOT: GTPKT.ge24 : 845,692 +845,692 1,986/s