Description

This article explains packet reorder behavior between multiple flows. MX series routers guaranteed packets order in each single flow. However they do not guarantee packets order between multiple flows.

Symptoms

Flow determination

To understand packets reorder behavior we need to understand what “flow” is on MX series routers.
When any packets come into MX series routers, they are classified according to a certain rule. These classified packets are called flows.  

A flow is determined by 5-tuple, which is src/dest IP addresses, IP protocol number, and src/dest ports.* Packets which have the same 5-tuple components will be classified as a the same flow. Packets with different parameters for the tuples in concern are classified as a different flow.

Why do packets reordering happen between flows?

The order of sending packets from MX series routers is dependent on how the packets are processed in PFE (packet forwarding engine). PFE sends packets in the order of packets that has finished processing. Note that it is not dependent on receiving packets order. Typically, incoming packets will be processed according to the receiving order. However, especially under short packets and high rate traffic conditions, some packets might exist on a PFE at the same time, and packets processing order might be changed. MX series routers guarantees packets order in each single flow, but not for multiple flows. This helps avoid flow stack by other flows.

Examples

Test packets include sequence number in their first byte data field by Tester. 

Topology

           UDP, 10G rate, L3 64byte, 10 packets
          ------------------------------------>

                                           Capture point
+--------+             +--------+            |+--------+
| Tester |     xe-0/0/0|   DUT  |xe-0/0/1    v| Tester |
|        |-------------|        |-------------|        |
|        |.2         .1|        |.1         .2|        |
+--------+             +--------+             +--------+
           10.0.0.0/24            20.0.0.0/24

DUT  : MX480
Junos: 15.1R1
FPC  : MPC4E

  1. Single flow pattern

    Traffic
    Src IP address    : 10.0.0.2
    Dest IP address   : 20.0.0.2
    IP protocol number: 17(UDP)
    Src port          : 63
    Dest port         : 63

    Test result

    Packets reordering did not happen.

  2. Multiple flows (10 flows) pattern

    Traffic
    Src IP address    : 10.0.0.2
    Dest IP address   : 20.0.0.2
    IP protocol number: 17(UDP)
    Src port          : 63
    Dest port         : 50001-50010

    Test result

    Packets (flows) reordering happened.

  3. Multiple flows (2 flows) pattern

    Traffic
    Src IP address    : 10.0.0.2
    Dest IP address   : 20.0.0.2
    IP protocol number: 17(UDP)
    Src port          : 63
    Dest port         : 50001-50002

    In this pattern, each flows packets were sent out alternately by Tester.

    Test result

    Packets reordering happened between flows. However, packets reordering did not happen in each single flow.

* Pure IP packets which do not have src/dest ports should determine a flow without src/dest ports. There is a logic flaw in MX routers where packets with random data on the same off-set field on src/dest ports will be handled as different flows. This behavior has reported in some Junos releases and it was resolved by PR1177418 - The OOS (Out-Of-Sequence) issue might be observed for non-tcp/non-udp packets with IP encapsulation

Solution

This is behavior is by design and is part of the specification.