This article describes the TTL behavior when different levels of encapsulation and decapsulation occur at various layers of the network hierarchy.

In the customer's environment, there are different levels of encapsulation and decapsulation, such as MPLS-over-UDP, followed by MPLS with LDP over RSVP as the transport, and then MPLS-over-UDP encapsulation and decapsulation again. They wanted to analyse the behavior at each hop. This article explains the observed behavior, along with packet captures and route outputs at various levels.

Referring to the topology above, where encapsulation and decapsulation occur at multiple levels, the behaviour observed during a ping from 1.1.1.1 to 11.11.11.11 is as follows:

• As indicated by the topology, the source is 1.1.1.1, and the destination is 11.11.11.11. When the packet first arrives at MX240-r002 from MX104-r017, it is in its pure IP form. As the packet is forwarded to MX480-r164, the IP packet gets encapsulated with an MPLSoUDP header, and a VPN label is imposed.

Static Route for Destination towards MX240-R002:

labroot@jtac-MX104-r017> show route 11.11.11.11

11.11.11.11/32    *[Static/5] 1w2d 23:17:47

                   > to 192.168.10.2 via xe-2/0/0.0

VPN Route for Destination over MPLSoUDP towards MX480-R164:

labroot@jtac-MX240-r002> show route 11.11.11.11

11.11.11.11/32    *[BGP/170] 14:11:52, localpref 100, from 3.3.3.3

                     AS path: I, validation-state: unverified

                   > via Tunnel Composite, UDP (src 2.2.2.2 dest 3.3.3.3), Push 29

On MX480-r164, the MPLSoUDP packet is decapsulated, and the VPN label is swapped with the MPLS transport label. The forwarding table for label 29 indicates a push operation for the transport label and a swap operation for the VPN label.

VPN Route for Destination over RSVP towards MX480-R172:

 

labroot@jtac-MX480-r164> show route 11.11.11.11

11.11.11.11/32    *[BGP/170] 16:04:19, localpref 100, from 6.6.6.6

                     AS path: I, validation-state: unverified

                   > to 192.168.30.2 via et-5/1/0.0, label-switched-path MX480-MX480

Forwarding Table for Label 29 on MX480-R164:

labroot@jtac-MX480-r164> show route forwarding-table label 29

Routing table: default.mpls

MPLS:

Destination       Type RtRef Next hop          Type Index   NhRef Netif

29                user    0                   indr 1048576    2

                             192.168.30.2     Swap 26, Push 20(top)     644    2 et-5/1/0.0

On MX480-R172, the packet arrives with only one label. The transport label is popped one hop back at MX240-r041. On MX480-R172, the packet is encapsulated again with MPLSoUDP, a VPN label (29) is pushed, and the packet is forwarded to MX240-r037. On MX240-r037, the MPLSoUDP is stripped, and the packet arrives at the destination as a pure IP packet on MX104-r0120.

VPN Route for Destination over MPLSoUDP towards MX240-R037:

11.11.11.11/32    *[BGP/170] 16:10:51, localpref 100, from 7.7.7.7

                     AS path: I, validation-state: unverified

                   > via Tunnel Composite, UDP (src 6.6.6.6 dest 7.7.7.7), Push 17
IP Route for Destination towards MX104-r020:

labroot@jtac-MX240-r037> show route 11.11.11.11

11.11.11.11/32    *[Static/5] 16:14:15

                   > to 192.168.70.1 via ge-1/0/0.0

Actual TTL Behavior Hop-by-Hop

1. Observation on MX104-r017 and MX240-r002

• When the ICMP packet egressed from MX104-r017, a packet capture performed at the ingress of MX240-r002 showed the TTL as 64, which is the default behaviour for ICMP packets in Junos.
• As the packet exited MX240-r002, the TTL value was decremented by one, as expected. This resulted in the inner IP packet TTL being 63. Additionally, due to MPLSoUDP encapsulation, a VPN label was imposed along with the outer UDP header.
• The inner IP packet TTL was copied to the VPN label, as per default junos behaviour. However, the outer UDP header's TTL was 255, which is independent of the inner IP packet or VPN label TTL. This was confirmed via packet captures.

#### Incoming Packet on jtac-MX240-r002 (Incoming as IP with IP TTL of 64)

C2 00 68 B3 00 01 C2 01 68 B3 00 01 08 00 45 00 00 54 53 86 00 00 40 01 0F 0C 01 01 01 01 0B 0B 0B 0B 08 00 BA 25 9A AA 32 D6 67 19 20 FE 00 0C FD 32 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37

------------------------------------------------------------------------------------------

Frame 1: 98 bytes on wire (784 bits)

Ethernet II

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0x5386 (21382)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 64

Protocol: ICMP (1)

Header Checksum: 0x0f0c

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 0

Internet Control Message Protocol

Type: 8 (Echo (ping) request)

Code: 0

Checksum: 0xba25 [correct]

Checksum Status: Good

Identifier (BE): 39594 (0x9aaa)

Identifier (LE): 43674 (0xaa9a)

Sequence Number (BE): 13014 (0x32d6)

Sequence Number (LE): 54834 (0xd632)

Timestamp from icmp data: Oct 23, 2024 18:14:54.851250000 CEST

Timestamp from icmp data (relative): 690.148751000 seconds

Data ????

2. MPLSoUDP Considerations

• In this setup, MPLSoUDP exists between two directly connected routers. However, in the customer’s case, there may be multiple intermediate devices. This means that only the outer UDP header TTL will decrement (e.g., 255 > 254 > 253), while the inner VPN label TTL and IP header TTL remain unchanged.

#### Outgoing Packet from jtac-MX240-r002 (Inner IP TTL = 63, MPLS VPN Label TTL – 63, Outer IP TTL = 255)

45 00 00 74 4f d7 00 00 ff 11 61 98 02 02 02 02 03 03 03 03 f6 60 19 eb 00 60 00 00 00 01 d1 3f 45 00 00 54 ce 6e 00 00 3f 01 95 23 01 01 01 01 0b 0b 0b 0b 08 00 89 2e 9d 39 29 c0 67 19 b2 eb 00 05 a2 ca 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 116

Identification: 0x4fd7 (20439)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 255

Protocol: UDP (17)

Header Checksum: 0x6198

Header checksum status: Unverified

Source Address: 2.2.2.2

Destination Address: 3.3.3.3

Stream index: 0

User Datagram Protocol

Source Port: 63072

Destination Port: 6635

Length: 96

Checksum: 0x0000 [zero-value ignored]

Stream index: 0

Stream Packet Number: 1

Timestamps

UDP payload (88 bytes)

MultiProtocol Label Switching Header

0000 0000 0000 0001 1101 .... .... .... = MPLS Label: 29 (0x0001d)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1

.... .... .... .... .... .... 0011 1111 = MPLS TTL: 63

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0xce6e (52846)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 63

Protocol: ICMP (1)

Header Checksum: 0x9523

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 1

Internet Control Message Protocol

3. Behaviour on MX480-r164

• When the packet reached MX480-r164, MPLSoUDP was decapsulated, meaning the UDP header was removed. For forwarding, a lookup was performed on the VPN label (Label 29).
• The forwarding table indicated both a push and swap operation:
  • The VPN label TTL was decremented by one.
  • VPN Label 29 was swapped with Label 26, and a new transport label (Label 20) was pushed.
  • The TTL value of 62 was copied to both the new VPN label and the transport label.

#### Incoming Packets on jtac-MX240-r025 ( Inner IP TTL = 63, VPN MPLS label TTL = 62, RSVP MPLS TTL = 62)

88 47 00 01 40 3e 00 01 a1 3e 45 00 00 54 fa b2 00 00 3f 01 68 df 01 01 01 01 0b 0b 0b 0b 08 00 a4 98 9d 39 56 c4 67 19 c2 fc 00 05 4a 4b 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

--------------------------------------------------------

Frame 1: 106 bytes on wire (848 bits)

Ethernet II

MultiProtocol Label Switching Header

0000 0000 0000 0001 0100 .... .... .... = MPLS Label: 20 (0x00014)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0

.... .... .... .... .... .... 0011 1110 = MPLS TTL: 62

MultiProtocol Label Switching Header

0000 0000 0000 0001 1010 .... .... .... = MPLS Label: 26 (0x0001a)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1

.... .... .... .... .... .... 0011 1110 = MPLS TTL: 62

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0xfab2 (64178)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 63

Protocol: ICMP (1)

Header Checksum: 0x68df

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 0

Internet Control Message Protocol

Type: 8 (Echo (ping) request)

Code: 0

Checksum: 0xa498 [correct]

Checksum Status: Good

Identifier (BE): 40249 (0x9d39)

Identifier (LE): 14749 (0x399d)

Sequence Number (BE): 22212 (0x56c4)

Sequence Number (LE): 50262 (0xc456)

Timestamp from icmp data: Oct 24, 2024 05:46:04.346699000 CEST

Timestamp from icmp data (relative): 125.653302000 seconds

Data ????

4. Processing by MX240-r025 and MX204-r041

• MX240-r025 and MX204-r041 are pure MPLS routers without any services.
• At MX240-r025, a lookup was performed on Label 20, and only the top label’s TTL was decremented. The VPN label and inner IP packet TTLs remained unchanged.
• At MX204-r041, RSVP termination occurred. Since explicit null was not used, the top label was popped off, and the TTL value decremented to 60. This TTL value was then copied to the next available label (VPN Label 26).

#### Incoming packet on MX204-r041 (Inner IP TTL = 63, Inner VPN MPLS label TTL = 62, RSVP MPLS label TTL = 61)

88 47 00 01 40 3d 00 01 a1 3e 45 00 00 54 e7 b1 00 00 3f 01 7b e0 01 01 01 01 0b 0b 0b 0b 08 00 43 4d 9d 39 e6 bb 67 19 cf ae 00 07 0e eb 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

--------------------------------------------------------------------

Frame 1: 106 bytes on wire (848 bits)

Ethernet II

MultiProtocol Label Switching Header

0000 0000 0000 0001 0100 .... .... .... = MPLS Label: 20 (0x00014)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0

.... .... .... .... .... .... 0011 1101 = MPLS TTL: 61

MultiProtocol Label Switching Header

0000 0000 0000 0001 1010 .... .... .... = MPLS Label: 26 (0x0001a)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1

.... .... .... .... .... .... 0011 1110 = MPLS TTL: 62

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0xe7b1 (59313)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 63

Protocol: ICMP (1)

Header Checksum: 0x7be0

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 0

Internet Control Message Protocol





#### Outgoing packet on MX204-r041 (Inner IP TTL = 63, VPN MPLS label TTL = 60, RSVP label TTL in incoming packet was 61 which was copied onto VPN label when RSVP label was popped. And when sending out, VPN label TTL is now decremented by 1 [61-1 = 60])

88 47 00 01 a1 3c 45 00 00 54 e7 b1 00 00 3f 01 7b e0 01 01 01 01 0b 0b 0b 0b 08 00 43 4d 9d 39 e6 bb 67 19 cf ae 00 07 0e eb 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

--------------------------------------------------------------------

Frame 1: 102 bytes on wire (816 bits)

Ethernet II

MultiProtocol Label Switching Header

0000 0000 0000 0001 1010 .... .... .... = MPLS Label: 26 (0x0001a)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1

.... .... .... .... .... .... 0011 1100 = MPLS TTL: 60

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0xe7b1 (59313)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 63

Protocol: ICMP (1)

Header Checksum: 0x7be0

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 0

Internet Control Message Protocol

5. Behavior on MX480-r172

• When the packet arrived at MX480-r172, the top MPLS label was popped, and the TTL value (60) was copied to the inner IP header, resulting in the inner IP packet TTL being 60.
• At this point, a lookup was performed for the destination 11.11.11.11, leading to MPLSoUDP encapsulation with a new VPN label and UDP header:
  • The inner IP packet TTL decremented to 59, which was copied to the VPN label (Label 17).
  • The outer UDP header TTL was 255, as previously observed.

#### Outgoing Packet from jtac-MX480-r172 (Inner IP TTL = 59, VPN Label TTL = 59, Outer IP TTL = 255. VPN label was popped here and since IP is exposed, it copies VPN TTL of 59 on Inner IP, Same TTL of 59 is copied on newly imposed VPN label and outer IP TTL is 255)

08 00 45 00 00 74 c6 27 00 00 ff 11 db 37 06 06 06 06 07 07 07 07 c3 2c 19 eb 00 60 00 00 00 01 11 3b 45 00 00 54 64 bc 00 00 3b 01 02 d6 01 01 01 01 0b 0b 0b 0b 08 00 a8 9c 9d 39 56 27 67 19 d1 e8 00 01 37 fc 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 116

Identification: 0xc627 (50727)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 255

Protocol: UDP (17)

Header Checksum: 0xdb37

Header checksum status: Unverified

Source Address: 6.6.6.6

Destination Address: 7.7.7.7

Stream index: 0

User Datagram Protocol

Source Port: 49964

Destination Port: 6635

Length: 96

Checksum: 0x0000 [zero-value ignored]

Stream index: 0

Stream Packet Number: 1

Timestamps

UDP payload (88 bytes)

MultiProtocol Label Switching Header

0000 0000 0000 0001 0001 .... .... .... = MPLS Label: 17 (0x00011)

.... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0

.... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1

.... .... .... .... .... .... 0011 1011 = MPLS TTL: 59

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0x64bc (25788)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 59

Protocol: ICMP (1)

Header Checksum: 0x02d6

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 1

Internet Control Message Protocol

6. Final Observation at MX240-r037

• At MX240-r037, a lookup was performed for the VPN label, which was then popped as MPLSoUDP was decapsulated. The TTL value decremented to 58.
• A subsequent lookup for the destination 11.11.11.11 resulted in the TTL value (58) being copied to the IP packet. This was confirmed via packet captures and the ping response from source to destination or vice versa.

#### Outgoing from jtac-MX240-r037 ( Final Inner IP TTL = 58)

 

08 00 45 00 00 54 26 23 00 00 3a 01 42 6f 01 01 01 01 0b 0b 0b 0b 08 00 0c 1a 9d 39 1c 64 67 19 d4 f1 00 05 0b 35 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

Frame 1: 98 bytes on wire (784 bits)

Ethernet II

Internet Protocol Version 4

0100 .... = Version: 4

.... 0101 = Header Length: 20 bytes (5)

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

Total Length: 84

Identification: 0x2623 (9763)

000. .... = Flags: 0x0

...0 0000 0000 0000 = Fragment Offset: 0

Time to Live: 58

Protocol: ICMP (1)

Header Checksum: 0x426f

Header checksum status: Unverified

Source Address: 1.1.1.1

Destination Address: 11.11.11.11

Stream index: 0

Internet Control Message Protocol

Type: 8 (Echo (ping) request)

Code: 0

Checksum: 0x0c1a [correct]

Checksum Status: Good

Identifier (BE): 40249 (0x9d39)

Identifier (LE): 14749 (0x399d)

Sequence Number (BE): 7268 (0x1c64)

Sequence Number (LE): 25628 (0x641c)

Timestamp from icmp data: Oct 24, 2024 07:02:41.330549000 CEST

Timestamp from icmp data (relative): 176.669452000 seconds

Data ????




labroot@jtac-MX104-r017> ping 11.11.11.11 source 1.1.1.1   

PING 11.11.11.11 (11.11.11.11): 56 data bytes

64 bytes from 11.11.11.11: icmp_seq=0 ttl=58 time=1.502 ms

64 bytes from 11.11.11.11: icmp_seq=1 ttl=58 time=2.891 ms

^C

--- 11.11.11.11 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max/stddev = 1.502/2.196/2.891/0.695 ms