Description

IPv6 traffic uses default MTU 1514 byte for 10 minutes after changing interface MTU size. The default MTU 1514 byte works for ICMPv6 during this 10 minutes, and the newly configured MTU starts working after the 10 minutes.

By default, IPv6-path-mtu-discovery is enabled, and IPv6 Path MTU Discovery timeout is by default 10-mins configured. So when we change MTU size, it will take effect after MTU Discovery timeout (i.e 10-mins).

Symptoms

In the test below, when we delete MTU and set MTU=9114 on an interface, ICMPv4 size 9072B succeeds immediately, but ICMPv6 size 9052B keeps failing.

ICMPv4 = 9114byte -(Ether Header[14byte]＋IPv4 Header [20byte]＋ICMP Header[8byte]) = 9114 - 42 = 9072byte
ICMPv6 = 9114byte -(Ether Header[14byte]＋IPv6 Header [40byte]＋ICMP Header[8byte]) = 9114 - 62 = 9052byte

ICMPv6 size 9052B will become okay 10 minutes later. During the 10 minutes, the default MTU 1514 works for ICMPv6. 

Test topology:

xe-1/0/0 ----------------------- xe-1/1/0 [LSYS:Other_MX]
mtu=9114                         mtu=9114

Base config:
Both xe-1/0/0 and xe-1/1/0 are MTU 9114 

set logical-systems Other_MX interfaces xe-1/1/0 unit 0 family inet address 150.87.46.1/30
set logical-systems Other_MX interfaces xe-1/1/0 unit 0 family inet6 address 2160:11b7:1aa::1/126
set chassis network-services enhanced-ip
set interfaces xe-1/0/0 mtu 9114
set interfaces xe-1/0/0 unit 0 family inet address 150.87.46.2/30
set interfaces xe-1/0/0 unit 0 family inet6 address 2160:11b7:1aa::2/126
set interfaces xe-1/1/0 mtu 9114

  1. MTU is 9114B on both interfaces 

    User@MX> clear interfaces statistics all

User@MX> show interfaces xe-1/0/0 extensive |no-more
Physical interface: xe-1/0/0, Enabled, Physical link is Up
  Interface index: 149, SNMP ifIndex: 545, Generation: 152
  Link-level type: Ethernet, MTU: 9114, MRU: 9122, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, <-- MTU=9114
  Loop Detect PDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled, Speed Configuration: Auto
  Pad to minimum frame size: Disabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: f4:cc:55:83:e8:a5, Hardware address: f4:cc:55:83:e8:a5
  Last flapped   : 2020-06-04 08:43:17 IST (02:10:56 ago)
  Statistics last cleared: 2020-06-04 10:54:04 IST (00:00:09 ago)
-- snip --

User@MX> show interfaces xe-1/1/0 extensive |no-more
Physical interface: xe-1/1/0, Enabled, Physical link is Up
  Interface index: 151, SNMP ifIndex: 750, Generation: 154
  Link-level type: Ethernet, MTU: 9114, MRU: 9122, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, <-- MTU=9114
  Loop Detect PDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled, Speed Configuration: Auto
  Pad to minimum frame size: Disabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: f4:cc:55:83:e8:ce, Hardware address: f4:cc:55:83:e8:ce
  Last flapped   : 2020-06-04 08:43:16 IST (02:11:06 ago)
  Statistics last cleared: 2020-06-04 10:54:04 IST (00:00:18 ago)
-- snip --

  2. ICMP test.

    2-1. ICMPv4 size 9072B is successful and ICMPv4 size 9073B is NOT successful.

    2-2. ICMPv6 size 9052B is successful and ICMPv6 size 9053B is NOT successful.

    IPv4 = 9114byte -(Ether Header[14byte]＋IPv4 Header [20byte]＋ICMP Header[8byte]) = 9114 - 42 = 9072byte
    IPv6 = 9114byte -(Ether Header[14byte]＋IPv6 Header [40byte]＋ICMP Header[8byte]) = 9114 - 62 = 9052byte 
    User@MX> ping 150.87.46.1 size 9072 do-not-fragment count 3
PING 150.87.46.1 (150.87.46.1): 9072 data bytes
9080 bytes from 150.87.46.1: icmp_seq=0 ttl=64 time=2.852 ms
9080 bytes from 150.87.46.1: icmp_seq=1 ttl=64 time=2.703 ms
9080 bytes from 150.87.46.1: icmp_seq=2 ttl=64 time=2.605 ms

    150.87.46.1 ping statistics:

    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max/stddev = 2.605/2.720/2.852/0.102 ms 
    User@MX> ping 150.87.46.1 size 9073 do-not-fragment count 3
PING 150.87.46.1 (150.87.46.1): 9073 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

    150.87.46.1 ping statistics

    3 packets transmitted, 0 packets received, 100% packet loss 
    User@MX> ping 2160:11b7:1aa::1 size 9052 do-not-fragment count 3
Jun 04 10:55:45
PING6(9100=40+8+9052 bytes) 2160:11b7:1aa::2 --> 2160:11b7:1aa::1
9060 bytes from 2160:11b7:1aa::1, icmp_seq=0 hlim=64 time=5.501 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=1 hlim=64 time=2.602 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=2 hlim=64 time=3.448 ms

    2160:11b7:1aa::1 ping6 statistics 
    3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 2.602/3.850/5.501/1.217 ms

User@MX> ping 2160:11b7:1aa::1 size 9053 do-not-fragment count 3
Jun 04 10:55:53
PING6(9101=40+8+9053 bytes) 2160:11b7:1aa::2 --> 2160:11b7:1aa::1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9061 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9061 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9061 chars, ret=-1

    2160:11b7:1aa::1 ping6 statistics

    3 packets transmitted, 0 packets received, 100% packet loss

  3. Continuous ICMP test

    3-1. Continuous ICMP_v4 size 9072B. 

    User@MX> ping 150.87.46.1 size 9072 do-not-fragment
PING 150.87.46.1 (150.87.46.1): 9072 data bytes
9080 bytes from 150.87.46.1: icmp_seq=0 ttl=64 time=2.622 ms
9080 bytes from 150.87.46.1: icmp_seq=1 ttl=64 time=2.634 ms
9080 bytes from 150.87.46.1: icmp_seq=2 ttl=64 time=3.266 ms
9080 bytes from 150.87.46.1: icmp_seq=3 ttl=64 time=2.574 ms
9080 bytes from 150.87.46.1: icmp_seq=4 ttl=64 time=2.586 ms
-- snip --

    3-2. Continuous ICMP_v6 size 9052B. 

    User@MX> ping 2160:11b7:1aa::1 size 9052 do-not-fragment
Jun 04 10:57:14
PING6(9100=40+8+9052 bytes) 2160:11b7:1aa::2 --> 2160:11b7:1aa::1
9060 bytes from 2160:11b7:1aa::1, icmp_seq=0 hlim=64 time=2.700 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=1 hlim=64 time=2.596 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=2 hlim=64 time=2.635 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=3 hlim=64 time=2.479 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=4 hlim=64 time=2.584 ms

  4. Delete mtu 9114 on xe-1/0/0. 

    [edit]
User@MX# delete interfaces xe-1/0/0 mtu 9114
Jun 04 10:57:44

[edit]
User@MX# show |compare
Jun 04 10:57:48
[edit interfaces xe-1/0/0]
-   mtu 9114;

[edit]
User@MX# commit
Jun 04 10:57:52
commit complete

    ICMPv4 fails. This is expected. 

    9080 bytes from 150.87.46.1: icmp_seq=33 ttl=64 time=2.644 ms
9080 bytes from 150.87.46.1: icmp_seq=34 ttl=64 time=3.061 ms
9080 bytes from 150.87.46.1: icmp_seq=35 ttl=64 time=2.567 ms
9080 bytes from 150.87.46.1: icmp_seq=36 ttl=64 time=2.570 ms
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

    ICMPv6 fails. This is expected. 

    9060 bytes from 2160:11b7:1aa::1, icmp_seq=35 hlim=64 time=2.615 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=36 hlim=64 time=2.590 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=37 hlim=64 time=4.942 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=38 hlim=64 time=15.623 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=39 hlim=64 time=2.549 ms
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1

  5. Set mtu 9114 on xe-1/0/0. 

    [edit]
User@MX# set interfaces xe-1/0/0 mtu 9114
Jun 04 10:58:14

[edit]
User@MX# show |compare
Jun 04 10:58:18
[edit interfaces xe-1/0/0]
+   mtu 9114;

[edit]
User@MX# commit
Jun 04 10:58:21
commit complete

    ICMPv4 restored immediately. 

    ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
9080 bytes from 150.87.46.1: icmp_seq=66 ttl=64 time=3.730 ms
9080 bytes from 150.87.46.1: icmp_seq=67 ttl=64 time=2.569 ms
9080 bytes from 150.87.46.1: icmp_seq=68 ttl=64 time=2.638 ms
9080 bytes from 150.87.46.1: icmp_seq=69 ttl=64 time=3.858 ms
9080 bytes from 150.87.46.1: icmp_seq=70 ttl=64 time=3.407 ms
9080 bytes from 150.87.46.1: icmp_seq=71 ttl=64 time=2.535 ms
9080 bytes from 150.87.46.1: icmp_seq=72 ttl=64 time=2.624 ms

    ICMPv6 still fails. 

    ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Can't assign requested address
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1

User@MX> show interfaces xe-1/0/0 extensive |no-more
Jun 04 11:00:40
Physical interface: xe-1/0/0, Enabled, Physical link is Up
  Interface index: 149, SNMP ifIndex: 545, Generation: 152
  Link-level type: Ethernet, MTU: 9114, MRU: 9122, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, <----- MTU=9114
  Loop Detect PDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled, Speed Configuration: Auto
  Pad to minimum frame size: Disabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: f4:cc:55:83:e8:a5, Hardware address: f4:cc:55:83:e8:a5
  Last flapped   : 2020-06-04 08:43:17 IST (02:17:23 ago)
  Statistics last cleared: 2020-06-04 10:54:04 IST (00:06:36 ago)

User@MX> show interfaces xe-1/1/0 extensive |no-more
Jun 04 11:00:51
Physical interface: xe-1/1/0, Enabled, Physical link is Up
  Interface index: 151, SNMP ifIndex: 750, Generation: 154
  Link-level type: Ethernet, MTU: 9114, MRU: 9122, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, <----- MTU=9114
  Loop Detect PDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled, Speed Configuration: Auto
  Pad to minimum frame size: Disabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: f4:cc:55:83:e8:ce, Hardware address: f4:cc:55:83:e8:ce
  Last flapped   : 2020-06-04 08:43:16 IST (02:17:35 ago)
  Statistics last cleared: 2020-06-04 10:54:04 IST (00:06:47 ago)

    Within 10 minutes after MTU 9114 is reconfigured Jun 04 10:58:21,

    ICMPv6 size < 1452B works, and ICMPv6 size > 1453B fails.
                =                              = 
    User@mx240-re0> ping 2160:11b7:1aa::1 size 1452 do-not-fragment <----------- OK
PING6(1500=40+8+1452 bytes) 2160:11b7:1aa::2 --> 2160:11b7:1aa::1
1460 bytes from 2160:11b7:1aa::1, icmp_seq=0 hlim=64 time=1.209 ms
1460 bytes from 2160:11b7:1aa::1, icmp_seq=1 hlim=64 time=1.370 ms
1460 bytes from 2160:11b7:1aa::1, icmp_seq=2 hlim=64 time=1.119 ms
^C
--- 2160:11b7:1aa::1 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 1.119/1.233/1.370/0.104 ms

User@mx240-re0> ping 2160:11b7:1aa::1 size 1453 do-not-fragment <----------- Fail
PING6(1501=40+8+1453 bytes) 2160:11b7:1aa::2 --> 2160:11b7:1aa::1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 1461 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 1461 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 1461 chars, ret=-1
^C
--- 2160:11b7:1aa::1 ping6 statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

    ICMPv6 = MTU -(Ether Header[14byte]+IPv6 Header [40byte]+ICMP Header[8byte]) = MTU - 62B = 1452byte
    MTU = 1452B + 62B = 1514B. This is the default MTU on Ethernet interfaces. 
    User@MX> show interfaces xe-1/1/1 media
Physical interface: xe-1/1/1, Enabled, Physical link is Up
  Interface index: 143, SNMP ifIndex: 984
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, <----- default MTU=1514
  Loop Detect PDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled, Speed Configuration: Auto

    10 minutes after MTU change, ICMPv6 works for the default MTU 1514 byte, then newly configured MTU starts working.

    10 minutes later, ICMPv6 starts working. 
    User@MX>
Jun 04 11:09:15

ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
ping: sendmsg: Message too long
ping6: wrote 2160:11b7:1aa::1 9060 chars, ret=-1
9060 bytes from 2160:11b7:1aa::1, icmp_seq=708 hlim=64 time=67.535 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=709 hlim=64 time=2.583 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=710 hlim=64 time=2.599 ms
9060 bytes from 2160:11b7:1aa::1, icmp_seq=711 hlim=64 time=2.593 ms

Solution

By default, IPv6-path-mtu-discovery is enabled, and IPv6 Path MTU Discovery timeout is by default 10-mins configured. So when we change MTU size, it will take effect after MTU Discovery timeout (i.e 10-mins).

This is expected as per RFC-8201 (refer section-4):

"Nodes may detect increases in PMTU, but because doing so requires sending packets larger than the current estimated PMTU, and because the likelihood is that the PMTU will not have increased, this must be
done at infrequent intervals.  An attempt to detect an increase (by sending a packet larger than the current estimate) must not be done less than 5 minutes after a Packet Too Big message has been received for the given path.  The recommended setting for this timer is twice its minimum value (10 minutes)." 
[edit system internet-options]
User@MX# set ipv6-path-mtu-discovery-timeout ?   
Possible completions:
  <ipv6-path-mtu-discovery-timeout>  IPv6 Path MTU Discovery timeout (5..71582788 minutes)
[edit system internet-options]
User@MX#