Customer uses ACX5448 as EVPN-MPLS PE. EVPN-MPLS ACX5448-PE1 is in a pair via ae46 bundle crosslink to EVPN-MPLS ACX5448-PE2. Both ACX5448s have an uplink to an MX960 P router (interface ae47). This ACX pair is hosting EVPN services, with the service in question using an ERB design with anycast IP addressing (IRB on both ACXs have the same IP and MAC addresses).

The major issue is, for any EVPN traffic that needs to traverse the MPLS crosslink from ACX5448-PE1 to ACX5448-PE2, it seems like this ACX5448-PE1 is incorrectly formatting the EVPN-MPLS packet.

/* Egress packet via ae46 towards ACX5448-PE2 */

DNX: Packet Len 110, Data:

000000: c8 13 37 7f 6f 22 d4 99 6c 22 87 ec 08 00 fb dd <-- Ethertype 0x0800 - wrong

000010: 40 ff 00 01 41 ff 45 00 00 54 45 00 00 54 33 dd <-- Duplicated ip header

000020: 00 00 3f 01 1f f7 0a 7e 80 01 0a 13 93 43 08 00

000030: 8a e5 03 a3 17 0c 66 1c 9c 45 00 0d ec 98 de ad

000040: be ef de ad be ef de ad be ef de ad be ef de ad

000050: be ef de ad be ef de ad be ef de ad be ef de ad

000060: be ef de ad be ef de ad be ef de ad be ef

From this output, you can see:

a. the Ethernet Frame’s Ethertype field is 0x0800 which is incorrect. Next Header is MPLS. Thus, should be 0x8847 (first line).

b. You can see that there are two labels being attached. The first label is completely incorrect. Second label is the VPN label 20, which is correct for the service on ACX5448-PE2. (first and second lines).

c. You can see, part of the IP header is repeated (second line - 45 00 00 54 and then repeated again 45 00 00 54, finally with source and destination IP fields).

ACX5448 EVPN-MPLS PE corrupting EVPN L2 packets via cross-link to secondary ACX5448 EVPN-MPLS PE

DNX: Packet Len 110, Data:

000000: c8 13 37 7f 6f 22 d4 99 6c 22 87 ec 08 00 fb dd <-- Ethertype 0x0800 - wrong

000010: 40 ff 00 01 41 ff 45 00 00 54 45 00 00 54 33 dd 

000020: 00 00 3f 01 1f f7 0a 7e 80 01 0a 13 93 43 08 00 

000030: 8a e5 03 a3 17 0c 66 1c 9c 45 00 0d ec 98 de ad 

000040: be ef de ad be ef de ad be ef de ad be ef de ad 

000050: be ef de ad be ef de ad be ef de ad be ef de ad 

000060: be ef de ad be ef de ad be ef de ad be ef

This problem is seen on ACX5448 with JUNOS releases having PR1566828 fix, that due to a bug in Broadcom(BCM) chipset. With this bug:

1. If the point-to-point link between the pair of EVPN PEs/ACXs uses a specific MAC bit pattern that the BCM program selection doesn't interpret correctly (i.e. incorrect BCM program is selected due to the interface MAC address of the adjacent router).
   1. this issue was seen for outer interface ethernet mac of neighboring ACX, not the payload mac address.

This problem had been fixed via PR1690538 after below release:

junos:21.2R3-S5

junos:21.4R3-S2

junos:22.2R3

junos:22.3R2

junos:22.3R3

junos:22.4R1

junos:22.4R2

junos:22.4R3

junos:23.1R1