When MPLS is configured on a device, the ingress provider edge (PE) router pushes one MPLS label (IPv6 Explicit-Null) and reserves three MPLS label lengths for MTU calculation by default. On the other hand, on the transit P router (label swap), no label length is reserved for MTU calculation.
Test scenario PE router: ICMP CE1-PE1-PE2
The ingress PE1 pushes one MPLS label, and reserves three MPLS label lengths for MTU calculation by default.
+---------+ |SFLOW | |collector| +----+----+ et-0/3/0| | | +------+ +------+ +----+----+ +------+ +------+ |CE1 | |PE1 |xe-0/0/0:2 xe-0/0/2|PE2 |xe-0/2/0 xe-0/2/1|PE3 | |CE2 | | +----------+ +--------------------+ <sflow +------------------+ +----------+ |<br /> | | | |mtu1518 mtu1518| capture | | | | | +------+<- eBGP ->+------+<------ iBGP ------>+---------+ +------+<- eBGP ->+------+ <-----------------------iBGP-------------------->
CE1 lo0=1:1:1::1/128 PE1 lo0=1:1:1::2/128, 1.1.1.2/32 PE2 lo0=1:1:1::3/128, 1.1.1.3/32 PE3 lo0=1:1:1::4/128, 1.1.1.4/32 CE2 lo0=1:1:1::5/128
MTU (IFD, INET, INET6, MPLS) = (1518B, 1500B, 1500B, 1504B)
User@PE1> show configuration interfaces xe-0/0/0:2 mtu 1518; unit 0 { family inet { mtu 1500; address 10.1.1.5/30; } family inet6 { mtu 1500; } family mpls { mtu 1504; } } User@PE2> show configuration interfaces xe-0/0/2 mtu 1518; unit 0 { family inet { mtu 1500; address 10.1.1.6/30; } family inet6 { mtu 1500; } family mpls { mtu 1504; } }
ICMPv6 size 1444 B succeeds.
EtherHeader[14B]+ MPLS label[4B] x 3 +IPv6 Header[40B]+ICMP Header[8B]+ICMPv6[1444B] = 1518B <---- Same as MTU IFD ^^^^^^^^^^^^^^^^^^ 3 mpls labels are taken into consideration. User@CE1> ping 1:1:1::3 source 1:1:1::1 count 1 do-not-fragment size 1444 PING6(1492=40+8+1444 bytes) 1:1:1::1 --> 1:1:1::3 1452 bytes from 1:1:1::3, icmp_seq=0 hlim=63 time=1.390 ms --- 1:1:1::3 ping6 statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/std-dev = 1.390/1.390/1.390/0.000 ms User@SFLOW> monitor traffic interface et-0/3/0 size 4000 write-file mpls_packet.cap Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay. Address resolution timeout is 4s. Listening on et-0/3/0, capture size 4000 bytes ^C == "mpls_packet.cap" == No. Time Source Destination Protocol Length Info 1 37.555122 10.0.0.1 10.0.5.2 sFlow 222 V5, agent 10.0.0.1, sub-agent ID 0, seq 48993, 1 samples Frame 1: 222 bytes on wire (1776 bits), 222 bytes captured (1776 bits) Juniper Ethernet Internet Protocol Version 4, Src: 10.0.0.1, Dst: 10.0.5.2 User Datagram Protocol, Src Port: 52047, Dst Port: 6343 InMon sFlow Datagram version: 5 Agent address type: IPv4 (1) Agent address: 10.0.0.1 Sub-agent ID: 0 Sequence number: 48993 SysUptime: 3 days, 18 hours, 8 minutes, 46 seconds (324526660ms) NumSamples: 1 Flow sample, seq 1195353 0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0) .... .... .... .... .... 0000 0000 0001 = sFlow sample type: Flow sample (1) Sample length (byte): 132 Sequence number: 1195353 0000 0000 .... .... .... .... .... .... = Source ID class: 0 .... .... 0000 0000 0000 1011 1110 1110 = Index: 3054 Sampling rate: 1 out of 65536 packets Sample pool: 1029308416 total packets Dropped packets: 0 Input interface (ifIndex): 3054 Output interface: 0x00000000 Flow record: 1 Raw packet header 0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0) Format: Raw packet header (1) Flow data length (byte): 92 Header protocol: Ethernet (1) Frame Length: 78 Payload removed: 4 Original packet length: 74 Header of sampled packet: 28c0dab0e002045c6cf68f7188470000213f6000000000103a3f00010001000100000000? Ethernet II, Src: JuniperN_f6:8f:71 (04:5c:6c:f6:8f:71), Dst: JuniperN_b0:e0:02 (28:c0:da:b0:e0:02) MultiProtocol Label Switching Header, Label: 2 (IPv6 Explicit-Null), Exp: 0, S: 1, TTL: 63 <-- 1 Label pushed Internet Protocol Version 6, Src: 1:1:1::1, Dst: 1:1:1::3 Internet Control Message Protocol v6
EtherHeader[14B]+ MPLS label[4B] x 3 +IPv6 Header[40B]+ICMP Header[8B]+ICMPv6[1444B] = 1518B <---- Same as MTU IFD ^^^^^^^^^^^^^^^^^^ 3 mpls labels are taken into consideration.
User@CE1> ping 1:1:1::3 source 1:1:1::1 count 1 do-not-fragment size 1444 PING6(1492=40+8+1444 bytes) 1:1:1::1 --> 1:1:1::3 1452 bytes from 1:1:1::3, icmp_seq=0 hlim=63 time=1.390 ms --- 1:1:1::3 ping6 statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/std-dev = 1.390/1.390/1.390/0.000 ms User@SFLOW> monitor traffic interface et-0/3/0 size 4000 write-file mpls_packet.cap Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay. Address resolution timeout is 4s. Listening on et-0/3/0, capture size 4000 bytes ^C == "mpls_packet.cap" == No. Time Source Destination Protocol Length Info 1 37.555122 10.0.0.1 10.0.5.2 sFlow 222 V5, agent 10.0.0.1, sub-agent ID 0, seq 48993, 1 samples Frame 1: 222 bytes on wire (1776 bits), 222 bytes captured (1776 bits) Juniper Ethernet Internet Protocol Version 4, Src: 10.0.0.1, Dst: 10.0.5.2 User Datagram Protocol, Src Port: 52047, Dst Port: 6343 InMon sFlow Datagram version: 5 Agent address type: IPv4 (1) Agent address: 10.0.0.1 Sub-agent ID: 0 Sequence number: 48993 SysUptime: 3 days, 18 hours, 8 minutes, 46 seconds (324526660ms) NumSamples: 1 Flow sample, seq 1195353 0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0) .... .... .... .... .... 0000 0000 0001 = sFlow sample type: Flow sample (1) Sample length (byte): 132 Sequence number: 1195353 0000 0000 .... .... .... .... .... .... = Source ID class: 0 .... .... 0000 0000 0000 1011 1110 1110 = Index: 3054 Sampling rate: 1 out of 65536 packets Sample pool: 1029308416 total packets Dropped packets: 0 Input interface (ifIndex): 3054 Output interface: 0x00000000 Flow record: 1 Raw packet header 0000 0000 0000 0000 0000 .... .... .... = Enterprise: standard sFlow (0) Format: Raw packet header (1) Flow data length (byte): 92 Header protocol: Ethernet (1) Frame Length: 78 Payload removed: 4 Original packet length: 74 Header of sampled packet: 28c0dab0e002045c6cf68f7188470000213f6000000000103a3f00010001000100000000? Ethernet II, Src: JuniperN_f6:8f:71 (04:5c:6c:f6:8f:71), Dst: JuniperN_b0:e0:02 (28:c0:da:b0:e0:02) MultiProtocol Label Switching Header, Label: 2 (IPv6 Explicit-Null), Exp: 0, S: 1, TTL: 63 <-- 1 Label pushed Internet Protocol Version 6, Src: 1:1:1::1, Dst: 1:1:1::3 Internet Control Message Protocol v6
ICMPv6 size 1445B fails.
EtherHeader[14B]+ MPLS label[4B] x 3 +IPv6 Header[40B]+ICMP Header[8B]+ICMPv6[1445B] = 1519B <---- 1B greater than MTU IFD ^^^^^^^^^^^^^^^^^^ 3 mpls labels are taken into consideration.
User@CE1> ping 1:1:1::3 source 1:1:1::1 count 1 do-not-fragment size 1445 PING6(1493=40+8+1445 bytes) 1:1:1::1 --> 1:1:1::3 1240 bytes from 2001:db8::84:0: Packet too big mtu = 1492 Vr TC Flow Plen Nxt Hlim 6 00 00000 05ad 3a 40 1:1:1::1->1:1:1::3 ICMP6: type = 128, code = 0 --- 1:1:1::3 ping6 statistics --- 1 packets transmitted, 0 packets received, 100% packet loss
Test scenario P router : ICMP CE1-PE1-P-PE2
PE1 pushes one MPLS label. On the transit P (label swap), no label is reserved for MTU calculation.
+------+ +------+ +---------+ +------+ +------+ |CE1 | |PE1 |xe-0/0/0:2 xe-0/0/2|P |xe-0/2/0 xe-0/2/1|PE2 | |CE2 | | +----------+ +--------------------+ +------------------+ +----------+ | | | | |mtu9000 mtu9000| |mtu1518 mtu1518| | | | +------+ +------+ +---------+ +------+ +------+ <- eBGP -> <-----------------------iBGP--------------------> <- eBGP ->
CE1 lo0=1:1:1::1/128 PE1 lo0=1:1:1::2/128, 1.1.1.2/32 P lo0=1:1:1::3/128, 1.1.1.3/32 PE2 lo0=1:1:1::4/128, 1.1.1.4/32 CE2 lo0=1:1:1::5/128
A large MTU is set on xe-0/0/0:2 and xe-0/0/2 for long packets to go through.
User@PE1> show configuration interfaces xe-0/0/0:2 mtu 9000; unit 0 { family inet { mtu 8000; address 10.1.1.5/30; } family inet6 { mtu 8000; } family mpls { mtu 8000; } } User@P> show configuration interfaces xe-0/0/2 mtu 9000; unit 0 { family inet { mtu 8000; address 10.1.1.6/30; } family inet6 { mtu 8000; } family mpls { mtu 8000; } }
User@P> show configuration interfaces xe-0/2/0 mtu 1518; unit 0 { family inet { mtu 1500; address 10.1.1.9/30; } family inet6 { mtu 1500; } family mpls { mtu 1504; } } User@PE2> show configuration interfaces xe-0/2/1 mtu 1518; unit 0 { family inet { mtu 1500; address 10.1.1.10/30; } family inet6 { mtu 1500; } family mpls { mtu 1504; } }
ICMPv6 size 1452B succeeds.
EtherHeader[14B]+ MPLS label[4B] x 1 +IPv6 Header[40B]+ICMP Header[8B]+ICMPv6[1452B] = 1518B <---- Same as MTU IFD ^^^^^^^^^^^^^^^^^^ 1 mpls label is taken into consideration. User@CE1> ping 1:1:1::4 source 1:1:1::1 count 1 do-not-fragment size 1452 PING6(1500=40+8+1452 bytes) 1:1:1::1 --> 1:1:1::4 1460 bytes from 1:1:1::4, icmp_seq=0 hlim=61 time=1.520 ms --- 1:1:1::4 ping6 statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/std-dev = 1.520/1.520/1.520/0.000 ms
EtherHeader[14B]+ MPLS label[4B] x 1 +IPv6 Header[40B]+ICMP Header[8B]+ICMPv6[1452B] = 1518B <---- Same as MTU IFD ^^^^^^^^^^^^^^^^^^ 1 mpls label is taken into consideration.
User@CE1> ping 1:1:1::4 source 1:1:1::1 count 1 do-not-fragment size 1452 PING6(1500=40+8+1452 bytes) 1:1:1::1 --> 1:1:1::4 1460 bytes from 1:1:1::4, icmp_seq=0 hlim=61 time=1.520 ms --- 1:1:1::4 ping6 statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/std-dev = 1.520/1.520/1.520/0.000 ms
ICMPv6 size 1453B fails.
EtherHeader[14B]+ MPLS label[4B] x 1 +IPv6 Header[40B]+ICMP Header[8B]+ICMPv6[1453B] = 1519B <---- 1B greater than MTU IFD ^^^^^^^^^^^^^^^^^^ 1 mpls label is taken into consideration. User@CE1> ping 1:1:1::4 source 1:1:1::1 count 1 do-not-fragment size 1453 PING6(1501=40+8+1453 bytes) 1:1:1::1 --> 1:1:1::4 ping: sendmsg: Message too long ping6: wrote 1:1:1::4 1461 chars, ret=-1 --- 1:1:1::4 ping6 statistics --- 1 packets transmitted, 0 packets received, 100% packet loss
In the ingress PE router, the packet received is IP and MPLS labels are being pushed, so we reserve space for "max-labels : 3," considering the end-to-end MPLS path.
In the case of a transit router (label swap), the packet protocol is already MPLS and "max-labels : 3" is not honored.
This is as per design on MX Trio based MPC cards.