A customer reported an issue with TCP sessions being disrupted when using an ECMP (Equal-Cost Multi-Path) environment on the PTX10k series routers. The problem occurred due to a new hashing algorithm that included the Source and destination MAC address as a factor in next-hop selection. This caused TCP sessions to devices behind the PTX router using anycast to break, as the PTX router had multiple next hops for IPs advertised by those hosts.
The customer’s environment included a BGP multipath, with each BGP's next hop reachable via multiple label-switched paths (LSPs). With TE++ (Traffic Engineering), multiple tunnels were signalled and re-signalled between PEs (Provider Edges). This resulted in traffic from the same flow arriving at a PE from different P routers, leading to different outbound interfaces. This, in turn, led to the flow being rehashed to a different load balancer. Since load balancers expect symmetrical traffic, this caused the flow to be dropped.
Root Cause:
In the PTX10k, the source and destination MAC address was being considered during ECMP hashing, which led to the traffic being forwarded to different load balancers when the PE router received traffic via multiple paths. This behaviour was breaking TCP sessions because the load balancers were expecting symmetrical traffic, which was not the case due to the rehashing.
Solution/Workaround:
To mitigate the issue, the solution was to remove the MAC address from the hashing calculation in the ECMP environment. Specifically, the workaround involved configuring the PTX10k to use the same MAC address for its backbone-facing interfaces, effectively removing this variable from the ECMP calculation. This workaround was found to work on routers with EVO configurations.
For EVO-based routers, the workaround was implemented by applying hidden knobs to disable consideration of the source and destination MAC addresses in hash calculations.
Topology:
Configuration Steps:
1. Configure hidden knobs on the DUT (PTX10001-36MR) to remove SMAC and DMAC from hash calculation:
root@ROUTER-001> show configuration forwarding-options enhanced-hash-key | display set
set forwarding-options enhanced-hash-key family any no-source-mac
set forwarding-options enhanced-hash-key family any no-destination-mac
2. Verify interface configurations:
With the Ingress interface and Mac being changed on the peer P router (MX960 as per the above topology).
TEST-1
-> Modify the MAC address on the output port on the transient P router (MX960) and observe the traffic behaviour.
labroot@jtac-mx960-r2040-re0# set interfaces et-10/1/2 mac aa:11:22:33:44:55
et-0/0/0 Up 478895449 (1221411) 7261 (3) ----- > Ingress interface connects to P router jtac-mx960
et-0/0/1 Up 0 (0) 15468 (1)
et-0/0/2 Up 0 (0) 15461 (1)
et-0/0/3 Up 0 (0) 15459 (1)
et-0/0/4:0 Up 5247 (2) 6660 (1)
et-0/0/4:1 Up 5230 (0) 27155001 (1220226) ---- > Traffic moved to a different Egress Interface
et-0/0/4:2 Up 5147 (0) 451745778 (0)
et-0/0/4:3 Up 6800 (0) 17580791260 (0)
et-0/0/6:0 Up 541 (0) 2720 (0)
et-0/0/6:1 Up 540 (0) 0 (0)
et-0/0/6:2 Up 17581622107 (0) 9158 (0)
et-0/0/6:3 Up 542 (0) 2729 (0)
TEST-2
-> Remove the MAC address configured on the output port, which leads to rehashing. Traffic continues to flow on the egress has changed.
Changed the MAC again on MX960
labroot@jtac-mx960-r2040-re0# run show interfaces et-10/1/2 | match hard
Current address: 30:b6:4f:de:c9:db, Hardware address: 30:b6:4f:de:c9:db
et-0/0/0 Up 671691607 (1219913) 7442 (2) ----- > Ingress interface connects to P router jtac-mx960-
et-0/0/1 Up 0 (0) 15629 (1)
et-0/0/2 Up 0 (0) 15623 (1)
et-0/0/3 Up 0 (0) 15625 (1)
et-0/0/4:0 Up 5302 (0) 6734 (0)
et-0/0/4:1 Up 5286 (0) 177726921 (1)
et-0/0/4:2 Up 5202 (1) 496368379 (1221455) ---- > Traffic moved to a different Egress Interface
et-0/0/4:3 Up 6873 (0) 17580791315 (1)
et-0/0/6:0 Up 547 (0) 2749 (0)
et-0/0/6:1 Up 546 (0) 0 (0)
et-0/0/6:2 Up 17581622135 (0) 9164 (0)
et-0/0/6:3 Up 548 (0) 2757 (0)
Result: Based on the two tests conducted, we conclude that after configuring the knobs on the DUT (Egress router), the traffic is forwarded to the egress router regardless of any changes in the SMAC or DMAC. The egress port has changed on the Egress router.
TEST 3:
Change the incoming interface, and observe the behavior when traffic switches over to another P router.
et-0/0/0 Up 45716500270 (1220852) 20055 (1) ----- > Ingress interface is et-0/0/0
et-0/0/1 Up 1357 (0) 38944 (1)
et-0/0/2 Up 0 (0) 38928 (1)
et-0/0/3 Up 0 (0) 38927 (1)
et-0/0/4:0 Up 13142 (0) 7813037104 (1219879) ----- > Egress interface is et-0/0/0
et-0/0/4:1 Up 13104 (0) 1099746318 (0)
et-0/0/4:2 Up 12937 (0) 36803738859 (0)
et-0/0/4:3 Up 17110 (2) 12574 (1)
et-0/0/6:0 Down 0 (0) 0 (0)
The interface on the PE (et-0/0/0), which is connected to the P router (MX960), was down. As a result, the traffic is now being received from another transient router, the PTX1000, on interface et-0/0/6:2.
et-0/0/0 Down 45758150261 (0) 20072 (0)
et-0/0/1 Up 1366 (0) 39188 (1)
et-0/0/2 Up 0 (0) 39171 (1)
et-0/0/3 Up 0 (0) 39170 (1)
et-0/0/4:0 Up 13229 (1) 7864660044 (1220446) ----- > Egress interface is same as earlier
et-0/0/4:1 Up 13193 (0) 1099746401 (0)
et-0/0/4:2 Up 13021 (0) 36803738939 (1)
et-0/0/4:3 Up 17217 (0) 12653 (0)
et-0/0/6:1 Up 6896 (0) 0 (0)
et-0/0/6:2 Up 7582734 (1220600) 294 (0) ----- > Ingress interface is now et-0/0/6:2
TEST:4
We have re-enabled the interface et-0/0/0 as the ingress interface. However, we still observe that the traffic has shifted back, and it continues to be forwarded on et-0/0/4:0 as the egress interface.
et-0/0/0 Up 45765489063 (1220237) 20104 (0) ----- > Ingress interface is et-0/0/0
et-0/0/1 Up 1368 (0) 39261 (0)
et-0/0/2 Up 0 (0) 39244 (0)
et-0/0/3 Up 0 (0) 39243 (0)
et-0/0/4:0 Up 13256 (0) 7943610136 (1221419) ----- > Egress interface is same as earlier
et-0/0/4:1 Up 13224 (0) 1099746431 (0)
et-0/0/4:2 Up 13045 (0) 36803738965 (0)
et-0/0/4:3 Up 17250 (0) 12682 (0)
et-0/0/6:1 Up 6911 (1) 0 (0)
et-0/0/6:2 Down 81584739 (0) 337 (0)
Result: If any transient router or input interface on the egress goes down, the traffic will still be forwarded to the same egress interfaces from the alternative port, ensuring it reaches the same load balancer.
Conclusion:
After configuring the hidden knobs to disable consideration of the source and destination MAC addresses in the hash calculation, the traffic is consistently forwarded to the correct egress router regardless of any changes in the MAC address or interface. The traffic continues to be forwarded on the same egress interface, ensuring it reaches the correct load balancer and maintains TCP session integrity.