Description

A race-condition during Virtual Routing Function (VRF) initialization/programming can cause the Virtual Router Redundancy Protocol (VRRP) vRouter MAC 00:00:5e:00:01:00 entry to be missing from the bridge table.
The VRRP MAC address is the one used locally as L2 address for the Virtual Network default gateway - layer 2 communication between a VM and vRouter.
In this scenario, any traffic bounded to the vRouter's VRRP will be dropped with reason "No L2 Route".

Symptoms

Traffic impact is noticed, as vRouter's ingress traffic from workloads (packets entering vRouter from vif interfaces) having the VRRP as destination address will be dropped.
The counter "No L2 Route" will keep increasing during problematic state, accounting for each packet dropped.
Such counter can be verified by checking the global value using the dropstats command output or on a per vif basis, as shown on the output below.
  
root@contrail-compute:~# vif --get 23 --get-drop-stats
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, HbsL=HBS Left Intf
       HbsR=HBS Right Intf, Ig=Igmp Trap Enabled, Ml=MAC-IP Learning Enabled

vif0/23     PMD: tap48cbe58a-7f NH: 24
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:200.200.200.204
            Vrf:4 Mcast Vrf:4 Flags:L3L2DEr QOS:-1 Ref:24
            RX port   packets:8255 errors:0 syscalls:1
            RX queue  packets:7802 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:8255  bytes:778238 errors:0
            TX packets:9291  bytes:839663 errors:0
            ISID: 0 Bmac: 02:48:cb:e5:8a:7f
            Drops:8244
            TX port   packets:9271 errors:20 syscalls:9229

No L2 Route                   8223
 
The following snippet shows the bridge table of VRF 4 on a setup in problematic state. As the output shows, there is no entry for the destination MAC 00:00:5e:00:01:00.  
(contrail-tools)[root@contrail-compute]$ rt --dump 4 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/4
Index       DestMac                  Flags           Label/VNID      Nexthop           Stats
26768       2:dd:1a:9e:11:88           LDf                   33           18               0
28920       2:dd:e6:90:68:1a           LDf                   70           18               0
37496       2:8f:86:c:3b:ce            LDf                   51           17               0
66156       2:ec:3a:4c:7d:8a           LDf                 1014           17               0
79920       2:ff:5f:41:99:f5                                  -          200               0
107556      2:c0:3d:ea:40:6a           LDf                   36           17               0
122836      2:fa:c1:59:19:61           LDf                   37           17               0
130472      2:8f:35:57:50:ab           LDf                   25           19               0
139748      2:56:25:e4:70:4f           LDf                  993           17               0
177984      2:38:ab:99:c8:8            LDf                   74           19               0
180628      2:46:33:19:2:2a                                   -          151               0
191148      2:98:d9:42:59:9c           LDf                   42           17               0
204540      2:48:cb:e5:8a:7f                                  -           35               0
206596      ff:ff:ff:ff:ff:ff          LDf                    0           43             192
236748      2:cb:c0:5:b6:3e                                   -          125               0
243124      2:d6:72:17:2:46            LDf                   26           18               0
255048      2:7c:6e:3a:d5:b5           LDf                 1019           17               0


The bridge table for any VRF must hold an entry for the 00:00:5e:00:01:00 VRRP MAC address with next-hop 3, as highlighted in the sample output below. 
(contrail-tools)[root@contrail-compute]$ rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index       DestMac                  Flags           Label/VNID      Nexthop           Stats
8308        2:f3:b4:d4:8f:71           LDf                   34           84               0
11348       2:3d:9e:46:2d:e0                                  -          179               0
31264       0:0:5e:0:1:0                Df                    -            3          281283
53832       2:dd:f2:45:ea:4a                                  -          163               0
70980       2:2e:73:50:c9:18           LDf                   39           84               0
71296       2:53:d2:3b:d5:18           LDf                   39           97               0
108264      2:f7:ae:a7:79:18                                  -          168               0
112924      ff:ff:ff:ff:ff:ff          LDf                  111          180             162
115240      2:0:0:0:0:2                 Df                    -           12               0
124156      2:b:48:b3:31:6a            LDf                   29           97               0
146156      2:65:b1:fa:8:1                                    -          154               0
155064      2:a5:15:86:62:78           LDf                   34           97               0
179732      3c:fd:fe:d9:a1:a8           Df                    -            3               0
199028      2:b1:5d:c3:dd:81                                  -          158               0
205564      2:0:0:0:0:1                 Df                    -           12               0
229964      2:b0:ae:f7:7c:d5                                  -          174               0
245152      2:d3:2d:a0:79:58           LDf                   29           84               0

Solution

The issue described in this article was fixed in Contrail Networking release 21.4.L1.1.5 as part of CEM-27536. 

A setup affected by it can only be recovered by restarting vRouter Agent service, which will trigger vRouter reprogramming.
vRouter Agent restart is traffic impacting and therefore the Compute node should be evacuated from any workload carrying critical traffic before restarting any service.

Modification History

.