This article explains the behavior change in ESI label encoding for ensuring RFC compliance and to avoid interoperability issues.
Starting from Junos OS release 17.4R2-S5, 17.4R3, 18.1R3-S3, 18.2R2-S3, 18.2R3, 18.3R2, 18.4R1-S3, 18.4R2, 19.1R1 and later, ESI label encoding has changed to a higher (first) 20 bit in a 24-bit field to ensure RFC compliance and to avoid interoperability issues.
A quote from section 7 titled "BGP EVPN Routes" in RFC-7432 titled "BGP MPLS-Based Ethernet VPN," which indicates this change, is as follows:
7.5. ESI Label Extended Community The ESI Label field represents an ES by the advertising PE, and it is used in split-horizon filtering by other PEs that are connected to the same multihomed Ethernet segment. Each ESI Label extended community is encoded as an 8-octet value, as follows: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| Type=0x06 | Sub-Type=0x01 | Flags(1 octet)| Reserved=0 |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| Reserved=0 | ESI Label |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ <...>
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| Type=0x06 | Sub-Type=0x01 | Flags(1 octet)| Reserved=0 |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+| Reserved=0 | ESI Label |+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Unfortunately the RFC does not explicitly dictate how the 20-bit long MPLS label should be encoded into and carved out of the 24-bit long ESI label field, and this ambiguity seems to be the reason for the interoperability issue that is described in this section.
Usually EVPN related IETF documents explicitly specify that *IF* an MPLS label is carried in a 3-bytes field, *THEN* the high-order 20 bits of the field contain the MPLS label, for example:
RFC-7432, section '9.2.1. Constructing MAC/IP Address Advertisement' <...> The MPLS Label1 field is encoded as 3 octets, where the high-order 20 bits contain the label value.<...> The MPLS Label2 field is an optional field. If it is present, then it is encoded as 3 octets, where the high-order 20 bits contain the label value.<...> ";
RFC-8317, section '6.1. E-Tree Extended Community': " <...> The value of the 20-bit MPLS label is encoded in the high-order 20 bits of the Leaf label field.<...> ".
For the ESI label value (for example, 0x04982f) extcommunity, only 20 high-order bits (0x4982) were supposed to be used for encoding the label value. This assumption also implies that the receiving side should read the same 20 high-order bits while carving the label value out of the received extcommunity.
However, the problem is that the Junos OS implementation uses 20 low-order bits (0x4982f, or maybe it operates with 3/4 bytes structure in reality) to encode/decode the MPLS bits, whereas a third party one uses 20 high-order bits (0x04982). This creates the interoperability issue.
A sample synopsis of the interoperability problem is given below, using the following example topology:
+----------+ +--+ LN_1 | | +----------+ | | .------. | +----------+ | CPE1 +--+--+ LN_2 | '------' | +----------+ | | | +----------+ +--+ LN_3 +----+ | +----------+ | | | .--------. | +---+ CPE2 | | +----------+ | '--------' +--+ LN_4 +----+ +----------+
+----------+
+--+ LN_1 |
| +----------+
|
.------. | +----------+
| CPE1 +--+--+ LN_2 |
'------' | +----------+
+--+ LN_3 +----+
| +----------+ |
| | .--------.
| +---+ CPE2 |
| +----------+ | '--------'
+--+ LN_4 +----+
In the above topology:
CPE - CPE1 has four multihoming upstream leaf nodes.
CPE - CPE2 has two multihoming upstream leaf nodes.
All four leaf nodes connect to the spine routers, which is not shown in the topology.
The non-DF LN_1 receives the CPE1 OSPF hello packets. It is then expected to push the SH label 301103 (learnt from DF LN_4) and forward the frame to DF LN_4 via the spine, and also flood the frame to other leaf nodes. While DF receives the frame, it will not send the packet back to CPE1 due to the SH label, but it will flood the frame to CPE2 since it connects to the different ESI interface.
Before upgrade, all four leaf nodes run Junos OS 15.1F6 (all spine routers, not shown, also running Junos OS 15.1F6) and the network works fine.
After upgrade (LN_1 and LN_2 are upgraded from Junos OS 15.1F6 to Junos OS 18.2R2), the non-DF leaf node (LN_1) interprets the SH label 301103 (0x4982f), which is advertised from the DF node (LN_4), into 18818 (0x4982), resulting in the OSPF to go down. It seems that the non-DF LN only reads the first 16 bits of the label (actually, the system reads the first/high-end 20 bits) and interprets it as an SH label. Other labels are all right.
After doing a packet dump on Leaf_1 (LN_1 non-DF) on FPCs carrying Spine links ae1 and ae5, the OSPF multicast packet from CPE is sent towards the Spine on ae5.
It is confirmed that Leaf_1 is likely forwarding this frame towards Leaf_4 (LN_4 DF).
Wallclock: 0xfb7e707b
Sent 239 byte parcel:
0xc5 0x7d 0xed 0xcc 0x00 0x1f 0xae 0x1f 0x05 0xb1 0x00 0x02 0x00 0x38 0x00 0x8e 0x3a 0x00 0x00 0x28 0x8a 0x1c 0xee 0xe7 0xc3 0x28 0x8a 0x1c 0xee 0xdf 0xc2 0x88 0x47 0x49 0x36 0x7e 0xff 0x49 0x73 0xae 0xff 0x04 0x98 0x2f 0xff 0x01 0x00 0x5e 0x00 0x00 0x05 0x78 0xfe 0x3d 0x54 0x30 0x12 0x81 0x00 0xc6 0xa7 0x08 0x00 0x45 0xc0 0x00 0xb0 0x3d 0x53 0x00 0x00 0x01 0x59 0x4f 0xec 0x0a 0xf5 0x3f 0xfc 0xe0 0x00 0x00 0x05 0x02 0x01 0x00 0x68 0x90 0x82 0xe4 0x64 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x01 0x10 0x5e 0x6a 0x2d 0x90 0xff 0xff 0xff 0x00 0x00 0x0a 0x12 0x7f 0x00 0x00 0x00 0x28 0x0a 0xf5 0x3f 0xfb 0x0a 0xf5 0x3f 0xfc 0x3a 0xa2 0x1a 0xcc 0x3a 0xa2 0x1a 0xc4 0x3a 0xa2 0x1a 0xc9 0x3a 0xa2 0x1a 0xc2 0x3a 0xa2 0x1a 0xc8 0x3a 0xa2 0x1a 0xc3 0x3a 0xa2 0x1a 0xc5 0x3a 0xa2 0x1a 0xca 0x3a 0xa2 0x1a 0xc6 0x3a 0xa2 0x1a 0xc7 0x3a 0xa2 0x1a 0xcb 0x90 0x82 0xe4 0x63 0x3a 0xa2 0x1a 0xc1 0x90 0x82 0xe4 0x56 0x90 0x82 0xe4 0x57 0x05 0x58 0x10 0x09 0x70 0xcd 0x8c 0x49 0xa8 0x01 0xa0 0x1c 0xb1 0xca 0x0f 0xd4 0x00 0x00 0x00 0x09 0x00 0x01 0x00 0x04 0x00 0x00 0x00 0x01 0x00 0x02 0x00 0x14 0x5e 0x6a 0x2d 0x90 0x7b 0x63 0xb1 0x5c 0xa6 0xbe 0xcb 0xde 0xcc 0xa7 0xae 0x0d 0x5d 0xd9 0xb1 0x63
The packet above is the packet destined towards Leaf_4 captured on ae5 Leaf_1. If we decode this, we get the following:
Ethernet II
Destination: JuniperN_ee:e7:c3 (28:8a:1c:ee:e7:c3) Source: JuniperN_ee:df:c2 (28:8a:1c:ee:df:c2) Type: MPLS label switched packet (0x8847) MultiProtocol Label Switching Header 0100 1001 0011 0110 0111 .... .... .... = MPLS Label: 299879 <<<<< MPLS Transport Label to Spine; ok . .... .... .... .... 111. .... .... = MPLS Experimental Bits: 7 . .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0 . .... .... .... .... .... 1111 1111 = MPLS TTL: 255 MultiProtocol Label Switching Header 0100 1001 0111 0011 1010 .... .... .... = MPLS Label: 300858 <<<<<< Multicast Label for Leaf_4; ok . .... .... .... .... 111. .... .... = MPLS Experimental Bits: 7 . .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0 . .... .... .... .... .... 1111 1111 = MPLS TTL: 255 MultiProtocol Label Switching Header 0000 0100 1001 1000 0010 .... .... .... = MPLS Label: 18818 <<<<< SH Label (0x4982); due to high 20bit read !! should be 301103 (0x4982f) . .... .... .... .... 111. .... .... = MPLS Experimental Bits: 7 . .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1 . .... .... .... .... .... 1111 1111 = MPLS TTL: 255
Destination: IPv4mcast_05 (01:00:5e:00:00:05) Source: JuniperN_54:30:12 (78:fe:3d:54:30:12) Type: 802.1Q Virtual LAN (0x8100) 802.1Q Virtual LAN 110. .... .... .... = Priority: Internetwork Control (6) .0 .... .... .... = DEI: Ineligible . 0110 1010 0111 = ID: 1703 Type: IPv4 (0x0800) Internet Protocol Version 4 0100 .... = Version: 4 . 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT) Total Length: 176 Identification: 0x3d53 (15699) Flags: 0x0000 Time to live: 1 Protocol: OSPF IGP (89) Header checksum: 0x4fec Header checksum status: Unverified Source: 10.245.63.252 Destination: 224.0.0.5 Open Shortest Path First
From the above, we see that a majority of this packet is correct. However, the SH label used by Leaf_1 (LN_1) does not match that advertised by Leaf_4 (LN_4):
admin@LN_4> show evpn instance EVPN1 extensive esi 00:00:17:00:2a:8a:1c:ee:d8:4b | find "Number of ethernet" Number of ethernet segments: 44 ESI: 00:00:17:00:2a:8a:1c:ee:d8:4b Status: Resolved by IFL ae141.1700 Local interface: ae141.1700, Status: Up/Forwarding Number of remote PEs connected: 3 Remote PE MAC label Aliasing label Mode 10.191.8.16 513 513 all-active 10.191.8.18 513 513 all-active 10.191.8.20 300315 300315 all-active Designated forwarder: 10.191.8.18 Backup forwarder: 10.191.8.16 Backup forwarder: 10.191.8.20 Backup forwarder: 10.191.8.22 Advertised MAC label: 301102 Advertised aliasing label: 301102 Advertised split horizon label: 301103 <<<<<<< SH label advertised by DF node - Leaf_4 (LN_4 - 15.1F6) admin@LN_4> show route advertising-protocol bgp 10.191.8.0 table EVPN1.evpn.0 extensive | find "1:10.191.8.22.*d84b" * 1:10.191.8.22:1700::17002a8a1ceed84b::0/304 (1 entry, 1 announced) ? Advertisement is sent out to Spine. BGP group ROUTE_REFLECTORS type Internal Route Distinguisher: 10.191.8.22:1700 Route Label: 301102 Nexthop: Self Flags: Nexthop Change Localpref: 100 AS path: [65129] I Communities: target:65129:1700
If you look at any spine (MX, 15.1F6), it is seen to be receiving the correct label from Leaf_4:
admin@spine> show route receive-protocol bgp 10.191.8.22 table bgp.evpn.0 extensive | find "1:10.191.8.22.*d84b::FF" * 1:10.191.8.22:0::17002a8a1ceed84b::FFFF:FFFF/304 (1 entry, 1 announced) Accepted Route Distinguisher: 10.191.8.22:0 Nexthop: 10.191.8.22 Localpref: 100 AS path: I Communities: target:65129:1700 esi-label:all-active (label 301103) <<<<< correct
If you look at what the spine (MX, 15.1F6) is advertising to Junos 18.2R2-S3 leaves 1 and 2, you see the follows:
admin@spine> show route advertising-protocol bgp 10.191.8.16 table bgp.evpn.0 extensive | find "1:10.191.8.22.*d84b::FF" * 1:10.191.8.22:0::17002a8a1ceed84b::FFFF:FFFF/304 (1 entry, 1 announced) BGP group RR_CLIENTS type Internal Route Distinguisher: 10.191.8.22:0 Nexthop: 10.191.8.22 Localpref: 100 AS path: [65129] I Communities: target:65129:1700 esi-label:all-active (label 301103) <<<<< correct Cluster ID: 10.191.8.0 Originator ID: 10.191.8.22
If you look at what Leaf_1 (non-DF, 18.2R2-S4.3) sees:
admin@LN_1> show route receive-protocol bgp 10.191.8.0 table EVPN1.evpn.0 extensive | find "1:10.191.8.22.*d84b::FF" * 1:10.191.8.22:0::17002a8a1ceed84b::FFFF:FFFF/192 AD/ESI (2 entries, 1 announced) Import Accepted Route Distinguisher: 10.191.8.22:0 Route Label: 0 Nexthop: 10.191.8.22 Localpref: 100 AS path: I (Originator) Cluster list: 10.191.8.2 Originator ID: 10.191.8.22 Communities: target:65129:1700 esi-label:0x0:all-active (label 18818) <<<<<< Value differs, only first 16 bits matched
As can be seen above, there is a difference in the interpretation/decoding of the Split Horizon Label between the different software versions on the leaves. This may be the reason for the multicast packet drops, which introduces some possible incompatibility when CEs/customers are exist on all four leaves (using single ESI) with mismatched versions. Hence, due to this SH label interpretation difference, when the multicast frame is sent out from CPE1, it uses the SH label 18818 as we saw above. We can confirm that this was received on DF node - Leaf_4:
admin@LN_4> show firewall filter spine-ae5 Filter: spine-ae5 Counters: Name Bytes Packets spine-ae5-4-others 85076003898 319398584 spine-ae5-5-others 0 0 spine-ae5-6-others 0 0 spine-ae5-all-others 0 0 spine-ae5-other-vlans 28588 299 spine-ae5-v1703-others 1114428558512 821796863 spine-ae5-v1703-sh18802 0 0 spine-ae5-v1703-sh18818 59466 452 <<<<< LN_4 received OSPF Hello packets with SH label: 18818 spine-ae5-v1703-sh300843 0 0 spine-ae5-v1703-sh301103 0 0 admin@LN_4> show firewall filter OSPF_HELLO_2 Filter: OSPF_HELLO_2 Counters: Name Bytes Packets OSPF_Hello_IN_1703_CPE1 0 0 <<<<< Not forwarded to CPE2 Other_IN_CPE1 18072592446206 13918623100
For compatibility with older versions of the Junos OS, use the following knob in the upgrade leaf nodes.
set protocols evpn es-label-oldstyle