Description

This KB article explains the procedure to increase vrouter flow table limit in kernel compute.

Symptoms

/var/log/contrail/contrail-vrouter-agent.log indicates the following: 

Error Flow entry op = Add/Update flow_handle = 508512 gen-id = -120
SANDESH: Queue Drop: VRouter [SYS_DEBUG]: VRouterError: VRouter operation failed. Error < 9 : Flow gen id Mismatch >. Object < Flow : 508512 with Source IP: X.X.X.X Source port: XXX Destination IP: X.X.X.X Destination port: XXXX Protocol 6 >. Operation < Deletion >. Message number : 110255129 controller/src/vnsw/agent/vrouter/ksync/flowtable_ksync.cc 571
Error Flow entry op = Add/Update flow_handle = 326348 gen-id = -120
SANDESH: Queue Drop: VRouter [SYS_DEBUG]: VRouterError: VRouter operation failed. Error < 9 : Flow gen id Mismatch >. Object < Flow : 326348 with Source IP: X.X.X.X Source port: XXX Destination IP: X.X.X.X Destination port: XXX Protocol 6 >. Operation < Deletion >. Message number : 110257277 controller/src/vnsw/agent/vrouter/ksync/flowtable_ksync.cc 571


Flow -s can be used to see current flow statistics.

Dropstats output in such scenarios will show increase in  Flow Table Full  counter. 

#dropstats
Flow Unusable 1697824
Flow Table Full 1886
Flow Action Drop 120
Discards 122792
Cloned Original 193

#dropstats
Flow Unusable 1713891
Flow Table Full 1894
Flow Action Drop 120
Discards 122818
Cloned Original 193

Solution

Flow table limit in vrouter can be validated using command - vrouter --info
Default value of Flow Table limit is   (512 * 1024) 

# vrouter --info
vRouter module version       3.2.11.0 (Built by contrail-builder@contrail-builder-R3_2-centos71-10-84-56-203 on 2018-06-12 13:53:53.654272)

Startup parameters
    Interfaces limit                     4352
    VRF tables limit                     4096
    NextHops limit                       65536
    MPLS Labels limit                    5120
    Bridge Table limit                   262144
    Bridge Table Overflow limit          53248
    Flow Table limit                    524288
    Flow Table overflow limit            800768
    Mirror entries limit                 255

Runtime parameters
  Performance tweaks
    GRO                                  1
    Segmentation in software             1
  TCP MSS adjust settings
    TCP MSS on packets from VM           1
    TCP MSS on packet sent to VM         1
  RPS settings
    RPS after pulling inner hdr (perfr1) 0
    RPS after GRO on pkt1 (perfr2)       1
    RPS from phys rx handler (perfr3)    1
    Pull inner header (faster version)   1
    CPU to send pkts to, if perfr1 set   0
    CPU to send pkts to, if perfr2 set   0
    CPU to send pkts to, if perfr3 set   0
  Other settings
    NIC cksum offload for outer UDP hdr  0
    Flow hold limit:                     8192
    MPLS over UDP globally               0
    Used Flow entries                    0
    Used Over Flow entries               0
    Used Bridge entries                  2
    Used Over Flow bridge entries        0


To increase flow table limit on compute, below procedure can be used: 

cd /etc/modprobe.d
Create/Edit "vrouter.conf" file and add following entry to increase the flow table support for 1M
# vi vrouter.conf
options vrouter vr_flow_entries=1048576 
( Note: flow table size upto 4 million entries are tested)
reboot compute node.
Validate using command : vrouter –info


For additional information, refer to Github's  Vrouter Module Parameters .