On Junos OS platforms, When "dynamic tunnels" configured and "set chassis loopback-dynamic-tunnel" knob is used and when TCP (Transmission Control Protocol) traffic passed via MPLSoUDP (Multi-Protocol Label Switching Over User Datagram Protocol) tunnel through an outgoing AE (Aggregated Ethernet) bundle interface having member interfaces, use of either inner or outer header hash calculations lead to out-of-order packets at the egress. It causes service impact on related flow of traffic due to out-of-order packets.
Observing TCP sequence number out of order
Below example >> shows packet with id 61632 arriving before packet id 61631
11:01:26, 202 §14:01:25.844011 IP (tos 0xc0, ttl 253, id 61632, offset 0, flags [none], proto TCP (6), length 368)
11:01:26, 202 § 10.0.0.203.43305 > 10.0.0.38.ssh: Flags [P.], cksum 0xb346 (correct), seq 7345:7673, ack 72, win 4056, length 328
11:01:26, 202 §14:01:25.844023 IP (tos 0xc0, ttl 253, id 61631, offset 0, flags [none], proto TCP (6), length 576)
11:01:26, 202 § 10.0.0.203.43305 > 10.0.0.38.ssh: Flags [.], cksum 0x51ac (correct), seq 6809:7345, ack 72, win 4056, length 536
On Junos OS platforms, TCP traffic flow over MPLSoUDP tunnel egressing through AE bundle interface which has multiple member interfaces cause packets to go out-of-order at the egress. This issue can be seen when "dynamic tunnels" configured and "set chassis loopback-dynamic-tunnel" knob is used which leads the packet to get processed in two passes. In a first pass, inner packet is looked up in routing table and tunnel encapsulation is added. Post this packet is put on loopback. In second pass, packet is received from loopback where lookup on outer tunnel header is done, and packet is forwarded on to outgoing AE. Inner payload hash is computed on the packet in first pass, and it is passed over the loopback path. It is expected that the second pass uses this hash to decide which link on the outgoing AE bundle gets used. In certain situations, because of an un-initialized memory value rather than reusing existing hash, new hash is recomputed on the outer header for further use. This would mean that same flow could choose link of outgoing AE based on either of the inner or outer hash at times. This leads to spraying of packets on egress AE for the same flow eventually leading to out-of-order packets.
The out-of-order packets can be seen by the packet capture on outgoing AE interface.
when random TCP src and dst port of same flow are sent towards destination, traffic is split across the ae10 bundle member links.
Issue is not seen for every flow, but only for some flows.
Packets of the same flow(1000 pps) is split across the outgoing interface member links.
Incoming ae0 member link
{master}
lab@labdevice> show interfaces xe-1/3/0 | match rate
Input rate : 581560 bps (1012 pps)
Output rate : 0 bps (0 pps)
Outgoing ae10 member links:
lab@labdevice> show interfaces xe-0/2/0 | match rate
Feb 07 11:31:17
Input rate : 1472 bps (2 pps)
Output rate : 263064 bps (338 pps)
lab@labdevice> show interfaces xe-1/2/0 | match rate
Feb 07 11:32:24
Input rate : 992 bps (1 pps)
Output rate : 523592 bps (654 pps)
This issue is fixed via PR- https://prsearch.juniper.net/problemreport/PR1864237
Workaround: Use of single member on AE link where MPLSoUDP tunnel is riding on.