GRE MTU behavior with and without MTU manual settings on tunnel interface
Directly connected links R1 and R2
ge- 10.10.10.3/24 ge- 10.10.10.4/24 R1 --------------------------------------------------------------------- R2 gr- 172.24.1.1/24 gr- 172.24.1.2/24
Ping above size 1448 fails with MTU 1500 set on the tunnel interface gr-0/0/0.
user@R1_re> show configuration interfaces | display set set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.3/24 set interfaces gr-0/0/0 unit 0 tunnel source 10.10.10.3 set interfaces gr-0/0/0 unit 0 tunnel destination 10.10.10.4 set interfaces gr-0/0/0 unit 0 tunnel allow-fragmentation set interfaces gr-0/0/0 unit 0 family inet mtu 1500 set interfaces gr-0/0/0 unit 0 family inet address 172.24.1.1/24 user@R2_re> show configuration interfaces | display set set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.4/24 set interfaces gr-0/0/0 unit 0 tunnel source 10.10.10.4 set interfaces gr-0/0/0 unit 0 tunnel destination 10.10.10.3 set interfaces gr-0/0/0 unit 0 tunnel allow-fragmentation set interfaces gr-0/0/0 unit 0 family inet address 172.24.1.2/24
user@R1_re> ping 172.24.1.2 size 1449 detail PING 172.24.1.2 (172.24.1.2): 1449 data bytes ^C --- 172.24.1.2 ping statistics --- 46 packets transmitted, 0 packets received, 100% packet loss user@R1_re> show interfaces gr-0/0/0.0 Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 585) Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 10.10.10.4:10.10.10.3:47:df:64:0000000000000000 Encapsulation: GRE-NULL Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off Gre keepalives configured: Off, Gre keepalives adjacency state: down Input packets : 2 Output packets: 2 Protocol inet, MTU: 1500 Max nh cache: 0, New hold nh limit: 0, Curr nh cnt: 0, Curr new hold cnt: 0, NH drop cnt: 0 Flags: Sendbcast-pkt-to-re, User-MTU Addresses, Flags: Is-Preferred Is-Primary Destination: 172.24.1/24, Local: 172.24.1.1, Broadcast: 172.24.1.255
In case the MTU is hardcoded on the tunnel interface, fragmentation happens after the packet is GRE encapsulated. MTU will be considered for the outer IP header + payload.
Inet MTU calculation (1500) (Outer header + Payload) = 1448 Bytes Payload + 20 Bytes Outer IP header + 20 Bytes Inner IP Header + 4 Bytes GRE header + 8 Bytes ICMP header (therefore 1448 bytes of payload can be sent without fragmentation).
In this case because “allow-fragmentation” knob is configured , the ICMP packets exceeding the payload size 1448 will get fragmented for the whole packet (i.e., after GRE encapsulation) and will be sent out of the router with two packets.
For example, ICMP packet with payload size 1449 will get fragmented into 2 packets.
Packet 1 :- 1448 Bytes Payload + 20 Bytes Outer IP header + 20 Bytes Inner IP Header + 4 Bytes GRE header + 8Bytes ICMP header. (Total length = 1500) Packet 2 :- 1 byte Payload + outer IP header (Total Length = 21)
+++++++ Packet 1 ++++++++++
Internet Protocol Version 4
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 1500
Identification: 0xf2c1 (62145) <<< Notice the Identification field.
001. .... = Flags: 0x1, More fragments <<< Notice fragmentation fields set on the Outer IP header.
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 64
Protocol: Generic Routing Encapsulation (47)
Header Checksum: 0x3a17
Header checksum status: Unverified
Source Address: 10.10.10.3
Destination Address: 10.10.10.4
Generic Routing Encapsulation (IP)
Total Length: 1477
Identification: 0xf2c0 (62144)
000. .... = Flags: 0x0 < Notice no fragmentation fields set for Inner Header
Protocol: ICMP (1)
Header Checksum: 0x2844
Source Address: 172.24.1.1
Destination Address: 172.24.1.2
Internet Control Message Protocol
+++++++ Packet 2 ++++++++++
Total Length: 21
Identification: 0xf2c1 (62145) < Notice the Identification field & compare it with first packet Outer IP header.
000. .... = Flags: 0x0
...0 0000 1011 1001 = Fragment Offset: 1480
Header Checksum: 0x5f25
Ping above size 1448 does not fail when MTU is not set on the tunnel interface gr-0/0/0.
user@R1_re> ping 172.24.1.2 size 1449 detail count 1 PING 172.24.1.2 (172.24.1.2): 1449 data bytes 1457 bytes from 172.24.1.2 via gr-0/0/0.0: icmp_seq=0 ttl=64 time=5.176 ms --- 172.24.1.2 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/stddev = 5.176/5.176/5.176/0.000 ms user@R1_re> show configuration interfaces | display set set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.3/24 set interfaces gr-0/0/0 unit 0 tunnel source 10.10.10.3 set interfaces gr-0/0/0 unit 0 tunnel destination 10.10.10.4 set interfaces gr-0/0/0 unit 0 tunnel allow-fragmentation set interfaces gr-0/0/0 unit 0 family inet address 172.24.1.1/24 user@R2_re> show configuration interfaces | display set set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.4/24 set interfaces gr-0/0/0 unit 0 tunnel source 10.10.10.4 set interfaces gr-0/0/0 unit 0 tunnel destination 10.10.10.3 set interfaces gr-0/0/0 unit 0 tunnel allow-fragmentation set interfaces gr-0/0/0 unit 0 family inet address 172.24.1.2/24 user@R1_re> show interfaces gr-0/0/0.0 Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 585) Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 10.10.10.4:10.10.10.3:47:df:64:0000000000000000 Encapsulation: GRE-NULL Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off Gre keepalives configured: Off, Gre keepalives adjacency state: down Input packets : 2 Output packets: 2 Protocol inet, MTU: 1476 Max nh cache: 0, New hold nh limit: 0, Curr nh cnt: 0, Curr new hold cnt: 0, NH drop cnt: 0 Flags: Sendbcast-pkt-to-re Addresses, Flags: Is-Preferred Is-Primary Destination: 172.24.1/24, Local: 172.24.1.1, Broadcast: 172.24.1.255
Inet MTU calculation (1476) (Inner IP + Payload) = 1448 Bytes Payload + 20 Bytes Inner IP Header + 8 Bytes ICMP header (1448 bytes of payload can be sent without fragmentation).
In this case, the ICMP packets exceeding the payload size 1448 will get fragmented for the inner packet (i.e., before GRE encapsulation) and will be sent out of the router with two packets.
Packet 1 :- 1448 Bytes Payload + 20 Bytes Outer IP header + 20 Bytes Inner IP Header + 4 Bytes GRE header + 8 Bytes ICMP header. (Total length = 1500) Packet 2 :- 1 byte Payload + 20 Bytes Inner IP header + 20 bytes Outer IP header (Total Length = 45)
Identification: 0x36f4 (14068)
000. .... = Flags: 0x0 << Notice no fragmentation fields set on the Outer IP header.
Header Checksum: 0x15e5
Total Length: 1476 << Notice Total length not exceeding 1476 which equals inet MTU for inner IP header + payload.
Identification: 0x36f3 (14067) << Notice Identification field of Inner IP Header.
001. .... = Flags: 0x1, More fragments << Notice fragmentation fields set on the Inner IP header.
Header Checksum: 0xc412
Total Length: 45
Identification: 0x36f7 (14071)
Header Checksum: 0x1b91
Identification: 0x36f3 (14067) < Notice the Identification field and compare it with first packet Inner IP header.
...0 0000 1011 0110 = Fragment Offset: 1456
Header Checksum: 0xe90b
To conclude, when MTU is manually set, MTU checks will be done after GRE Encapsulation. When MTU is not manually set, MTU checks will be done before GRE Encapsulation.
Additionally, in Scenario 1, there will be packet drops on the destination, because the received fragmented packets will not be re-assembled at the receiving router. To re-assemble packets on the receiving router, configure “reassemble-packets”.
user@R2_re> show configuration interfaces gr-0/0/0 | display set set interfaces gr-0/0/0 unit 0 reassemble-packets set interfaces gr-0/0/0 unit 0 tunnel source 10.10.10.4 set interfaces gr-0/0/0 unit 0 tunnel destination 10.10.10.3 set interfaces gr-0/0/0 unit 0 tunnel allow-fragmentation set interfaces gr-0/0/0 unit 0 family inet address 172.24.1.2/24
Enable reassembly of fragmented generic routing encapsulation (GRE) encapsulated packets on GRE tunnel interfaces at the endpoint of theGRE tunnel.
If you do not include the reassemble-packets statement, the GRE tunnel interface does not reassemble fragmented GRE-encapsulated packets.
user@R1_re> ping 172.24.1.2 size 1449 detail PING 172.24.1.2 (172.24.1.2): 1449 data bytes 1457 bytes from 172.24.1.2 via gr-0/0/0.0: icmp_seq=284 ttl=64 time=2.761 ms 1457 bytes from 172.24.1.2 via gr-0/0/0.0: icmp_seq=285 ttl=64 time=2.354 ms 1457 bytes from 172.24.1.2 via gr-0/0/0.0: icmp_seq=286 ttl=64 time=2.629 ms 1457 bytes from 172.24.1.2 via gr-0/0/0.0: icmp_seq=287 ttl=64 time=2.739 ms 1457 bytes from 172.24.1.2 via gr-0/0/0.0: icmp_seq=288 ttl=64 time=2.586 ms ^C --- 172.24.1.2 ping statistics --- 289 packets transmitted, 5 packets received, 98% packet loss round-trip min/avg/max/stddev = 2.354/2.614/2.761/0.145 ms
2023-11-02 : Minor formatting changes