When troubleshooting a vrouter flow related issue, it is important to determine the correct flow next-hop that is being used. For equal-cost multipath (ECMP), a flag is set in the ECMP flow entries to indicate the next-hop that is being selected among others, which is called the "ECMP index."
This article explains the function of this flag and how it works.
Diagram
To demonstrate what the ECMP index is and how it works, in a lab setup, we created ECMP flows by using an allowed address pair (AAP). In the following diagram, two AAP IP addresses are configured:
AAP1: Applied to VMI/port 4.4.4.10 (bcomp79) and 4.4.4.12 (bcomp80). This is not used in the article.
AAP2: Applied to VMI/port 4.4.4.4 (bcomp79), 4.4.4.6 (bcomp79), and 4.4.4.11 (bcomp80). This is the one being used in this article.
JTAC Setup Diagram
contrail controllers computes VNF ............................... ................. .............. . +-------+ . . +-------+ . . +-------+ . . | | XMPP . . | | . . | 4.10 | .4.4.4.100 . |cont101+-------------------------+bcomp79+-----------+ +-.--- . | | . . | |AAP1. . | | . . +-------+ BGP . ___.__+-------+ . . +-------+ . . __./ . \ . . +-------+ . . +-------+/XMPP . ---------|4.4 |----- . | | . . AAP2\. . +-------+ .4.4.4.200 . | | . . \ . +-------+ . . | | . . .----- |4.6 | . . | | . . +-------+ . . +-------+ . . | | . . | | . . +-------+ . . |cont103| . . |comp106+-----------|4.14 | . . | | . . | | . . +-------+ . . | | . . +-------+ . . . . | | . . . . . . | | . . . . . . | | . . AAP2 . . +-------+ .4.4.4.200 . +-------+_ XMPP . ----------|4.11 |----- . \_._ . / . . +-------+ . . +-------+ BGP . \__.__+-------+ . . +-------+ . . | | . . | | . . | | .4.4.4.100 . |cont102+------------------------+bcomp80+-----------+ +-.---- . | | XMPP . . | |AAP1. . | 4.12 | . . +-------+ . . +-------+ . . +-------+ . . . . . . . ............................... ................. ..............
Test Method
To create an ECMP TCP flow, we start an SSH session from another VM on comp79 to one of the AAP IP addresses: 4.4.4.200. Because the AAP IP address is configured in active-active mode on multiple VMIs/ports, vrouter can forward the traffic to any of the destinations. In the test, the SSH session is established toward a VM interface (VMI) in the remote compute node, bcomp80.
On comp79, where the source VM is located, the flow entry generated by the SSH traffic is as follows:
root@bcomp79:~# flow --match 4.4.4.200:22 Flow table(size 80609280, entries 629760) Entries: Created 497 Added 485 Deleted 904 Changed 904 Processed 497 Used Overflow entries 0 (Created Flows/CPU: 42 36 29 19 26 13 31 27 13 20 28 34 0 2 1 0 10 5 4 3 10 2 12 3 0 20 8 3 1 9 1 0 18 1 6 46 0 6 0 3 1 1 1 1 1 0 0 0)(oflows 0) Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port) Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead Listing flows matching ([4.4.4.200]:22) Index Source:Port/Destination:Port Proto(V) ----------------------------------------------------------------------------------- 387368<=>516872 4.4.4.5:60139 6 (1) 4.4.4.200:22 (Gen: 1, K(nh):13, Action:F, Flags:, TCP:SSrEEr, E:1, QOS:-1, S(nh):0, #<----- ^^^ Stats:759/53415, SPort 62432, TTL 0, Sinfo 23.0.0.0) 516872<=>387368 4.4.4.200:22 6 (1) 4.4.4.5:60139 (Gen: 1, K(nh):13, Action:F, Flags:, TCP:SSrEEr, QOS:-1, S(nh):0, Stats:750/80453, SPort 56830, TTL 0, Sinfo 172.18.102.80)
In the above output, we see the ECMP index: "E:1". This indicates that the flow is going through the second "sub next-hop" of the composite next-hop index 160 in the inet table, as shown below:
inet
root@bcomp79:~# rt --dump 1 | grep -iE "flag|4.4.4.200" Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP Destination PPL Flags Label Nexthop Stitched MAC(Index) 4.4.4.200/32 32 P - 160 - ^^^ root@bcomp79:~# nh --get 160 Id:160 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Policy, Ecmp, Valid Hash Key Parameters: Proto,SrcIP,SrcPort,DstIp,DstPort Sub NH(label): 123(57) 18(28) 25(19) Id:123 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:1 #< 1st sub nh Flags:Valid, EncapFmly:0806 Oif:5 Len:14 Encap Data: 02 1c 6a 93 ae c2 00 00 5e 00 01 00 08 00 Id:18 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:40 Vrf:0 #< 2nd sub nh Flags:Valid, MPLSoUDP, Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:f0 1c 2d 41 91 4f 14 02 ec 66 b1 25 08 00 Vrf:0 Sip:172.18.79.79 Dip:172.18.102.80 Id:25 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:1 #< 3rd sub nh Flags:Valid, EncapFmly:0806 Oif:15 Len:14
Here the ECMP index "E:1" refers to the "second" sub next-hop index 18, which is where the forward flow passes the SSH traffic to. Similarly, index "E:0" would refer to the "first" sub next-hop and index "E:2" would refer to the "third" sub next-hop, and so forth.
Because the source and destination IP addresses belong to the same subnet, to reach the remote compute node, the bridge table will be used for making the final forwarding decision:
root@bcomp79:~# rt --dump 1 --family bridge | grep -iE "flag|1d:3f" Flags: L=Label Valid, Df=DHCP flood Index DestMac Flags Label/VNID Nexthop 46120 2:c0:62:15:1d:3f LDf 29 18
So, essentially the traffic will be forwarded to the underlay fabric via the vrouter bridge table with MPLS label 29.