Description

This article describes the issue of Inline-jflow not providing the correct flow data record for BGP load balanced traffic.


 

Symptoms

  • BGP traffic is load balanced towards AS 65001 via a transit AS 65005, with multipath multiple-as being configured.
  • Inline-jflow is configured in AS 65005 to identify the traffic load balanced between AS 65002 and AS 65003, so as to reach AS 65001.
  • Traffic is load balanced; however, Inline-jflow does not provide the correct flow record data for BGP load balanced traffic.

Topology : 
                   65002       65003
                 +------+    +------+
        11.0.0.2 | IXIA |    | IXIA |10.1.0.2
                 +------+    +------+
                   C4P3|      |C1P4
                       |      |
                       |      |
           ge-4/0/0.0  |      | ge-4/0/1.0
                     +-+------+-+
                     |          |ge-2/0/3.0
                     |   65005  |---------------------Collector
                     |  MX480-5 |             
                     |          |
                     +----+-----+
                          | ge-4/0/5.0
                          |
                          |
                      C4P1|
                       +------+
                       | IXIA |10.2.0.2
                       +------+

The route that is advertised from AS 65002 and 65001 is 12.0.0.0/24 AS and the path is 65003 65001 . Multipath multiple-as is enabled in BGP for load balancing:

Flow 1 : Src addr : 15.0.0.1 Dest addr : 12.0.0.225 at 10000 pps
Flow 2 : Src addr : 15.0.0.1 Dest addr : 12.0.0.181 at 10000 pps

Configuration:

lab@MX480-5> show configuration forwarding-options
sampling {
    instance {
        sakura-ipfix {
            input {
                rate 1;
            }
            family inet {
                output {
                    flow-server 88.88.88.1 {
                        port 2055;
                        autonomous-system-type origin;
                        version-ipfix {
                            template {
                                default-temp;
                            }
                        }
                    }
                    inline-jflow {
                        source-address 88.88.88.2;
                    }
                }
            }
        }
    }                                  
}
lab@MX480-5> show configuration chassis              
redundancy {
    graceful-switchover;
}
aggregated-devices {
    ethernet {
        device-count 3;
    }
}
fpc 4 {
    sampling-instance sakura-ipfix;
}

{MASTER}
lab@MX480-5> show interfaces ge-4/0/5 | match rate
  Input rate     : 205119176 bps (20000 pps)< Inbound Interface on which sampling is enabled

  Output rate    : 360 bps (0 pps)

{MASTER}
lab@MX480-5> show interfaces ge-4/0/0 | match rate   
  Input rate     : 208 bps (0 pps)
  Output rate    : 102559736 bps (9999 pps)< Outbound Interface and the traffic is load-balanced


{MASTER}
lab@MX480-5> show interfaces ge-4/0/1 | match rate   
  Input rate     : 0 bps (0 pps)
  Output rate    : 102560152 bps (10000 pps)< Outbound Interface and the traffic is load-balanced

lab@MX480-5> show route 12.0.0.1 extensive

inet.0: 31 destinations, 32 routes (31 active, 0 holddown, 0 hidden)
12.0.0.0/24 (2 entries, 1 announced)
TSI:
KRT in-kernel 12.0.0.0/24 -> {11.0.0.2, 10.1.0.2}
Page 0 idx 0 Type 1 val 904e894
    Nexthop: 10.1.0.2
    AS path: [65005] 65003 65001 I
    Communities:
    Advertise: 00000002
Page 0 idx 1 Type 1 val 904e648
    Nexthop: 10.1.0.2
    Localpref: 100
    AS path: [65005] 65003 65001 I
    Communities:
Path 12.0.0.0 from 10.1.0.2 Vector len 4.  Val: 0 1
        *BGP    Preference: 170/-101
                Next hop type: Router, Next hop index: 1048575
                Address: 0x922c010
                Next-hop reference count: 3
                Source: 10.1.0.2
                Next hop: 11.0.0.2 via ge-4/0/0.0, selected
                Next hop: 10.1.0.2 via ge-4/0/1.0
                State: <Active Ext>     
                Local AS: 65005 Peer AS: 65003
                Age: 26:19
                Task: BGP_65003.10.1.0.2+179
                Announcement bits (4): 0-KRT 2-RT 6-BGP_RT_Background 7-Resolve tree 1
                AS path: 65003 65001 I
                AS path: Recorded
                Accepted Multipath
                Localpref: 100
                Router ID: 10.1.0.2
         BGP    Preference: 170/-101
                Next hop type: Router
                Address: 0x91f1258
                Next-hop reference count: 1
                Source: 11.0.0.2
                Next hop: 11.0.0.2 via ge-4/0/0.0, selected
                State: <Ext>
                Inactive reason: Active preferred
                Local AS: 65005 Peer AS: 65002
                Age: 26:11
                Task: BGP_65002.11.0.0.2+179
                AS path: 65002 65001 I
                Accepted MultipathContrib
                Localpref: 100
                Router ID: 11.0.0.2

{MASTER}
lab@MX480-5> show route forwarding-table destination 12.0.0.1   
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
12.0.0.0/24        user     0                    ulst 1048575     2
                              11.0.0.2           ucst   787     3 ge-4/0/0.0
                              10.1.0.2           ucst   785     3 ge-4/0/1.0

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   525     1

Routing table: TEST.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   534     1

Routing table: VRF13336.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   543     1

Routing table: default-switch.inet
Internet:                              
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   616     1

Logical system: collector
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   598     1

Logical system: collector
Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct   607     1

NPC4(MX480-5 vty)# show nhdb id  1048575
   ID      Type      Interface    Next Hop Addr    Protocol       Encap     MTU       Flags  PFE internal Flags
-----  --------  -------------  ---------------  ----------  ------------  ----  ----------  --------------------
1048575   Unilist  ge-4/0/0.0     -                      IPv4      Ethernet     0  0x00000000 0x00000000

NPC4(MX480-5 vty)# show nhdb id  1048575 extensive   
   ID      Type      Interface    Next Hop Addr    Protocol       Encap     MTU       Flags  PFE internal Flags
-----  --------  -------------  ---------------  ----------  ------------  ----  ----------  --------------------
1048575   Unilist  ge-4/0/0.0     -                      IPv4      Ethernet     0  0x00000000 0x00000000
Unilist Table (2 entries): List flags 0x0
Unilist core-facing-index: 0

  787   Unicast  ge-4/0/0.0     11.0.0.2               IPv4      Ethernet  1500  0x00000000 0x00000000
  785   Unicast  ge-4/0/1.0     10.1.0.2               IPv4      Ethernet  1500  0x00000000 0x00000000

Weight Info:  Current Weight = 0
   ID  Balance  Orig-Balance Weight  Orig-Weight  State     Install      Flags
-----  -------  --------- ------  -----------  --------  -----------  -----
  787        0        0       0            0    Active    Installed  0x00
  785        0        0       0            0    Active    Installed  0x00

  Routing-table id: 0

NPC4(MX480-5 vty)# show sample-rr route-tbl ipv4 vrf 0 prefix  12.0.0.0/24< (inline-flow case) calculates the record, in which only one record is there with the next-hop set as 11.0.0.2

======================================================================================================
    Rec#     AS-Idx    EG-Ref-Cn      State    Pre-Len             Prefix        OIF            Gateway
======================================================================================================
       1          2          1          0         24          12.0.0/24        371           11.0.0.2
NPC4(MX480-5 vty)#

NPC4(MX480-5 vty)# show ifl 371

Logical interface ge-4/0/0.0 (Index 371, Alias-Index 0 Peer-Index 0)
Channel Mode DISABLED
  Flags: (0x000000000400c000) Up SNMP-Traps Subset-iif
Addresses:
  Media address: Family: Unspecified (0), Chan: 0, Length: 0
IRB ifl BD index 65535
Reroute Ref: 0, Restore Ref: 0, LRID: 0
Residue Stats in:         0 out:         0
Protocols:
  Protocol: IPv4, MTU: 1500 bytes, Flags: 0x8000000200000000, Route table: 0
    Maximum labels: 0
    Input filter: 0, Output filter: 0, Interface class: 0, Dialer Filter: 0
    Input Simple Filter: 0, Output Simple Filter: 0
    Input implicit filters: None
    L2 Input policer: 0, L2 Output policer: 0
    Input policer: 0, Output policer: 0
    RPF fail-filter: 0, Reroute Ref: 0, Restore Ref: 0
    Service filters in: 0, 0, 0, 0, 0, 0, out: 0, 0, 0, 0, 0, 0, psf: 0
    Address(0): 11.0.0.1 (0x00) [primary] [11.0.0.0/30]
  Protocol: Multiservice, MTU: 65535 bytes, Flags: 0x0000000200000000, Route table: 0
    Maximum labels: 0
    Input filter: 0, Output filter: 0, Interface class: 0, Dialer Filter: 0
    Input Simple Filter: 0, Output Simple Filter: 0
    Input implicit filters: None
    L2 Input policer: 0, L2 Output policer: 0
    Input policer: 17000, Output policer: 0
    RPF fail-filter: 0, Reroute Ref: 0, Restore Ref: 0
Media:
  Type: Ethernet, Encapsulation: Ethernet (0x0000000E)
  MTU: 1514 bytes, Flags: 0x0000
Dependencies:
  Parent ifl index:     371
Storm control:
  BC: 0, UC: 0, Flags: 0x0

 In both cases, the nexthop and dst AS are the same; irrespective of traffic being load balanced in two interfaces.

 

Solution

  • This is expected behavior when equal-cost multipath traffic is handled.
  • This is a limitation with inline-jflow.
  • The PFE calculates some hash on each incoming packet, which is based on the selected nexthop.
  • However, when MPC (inline-flow case) calculates the record, it does not know how the packets are load-balanced.
 

NPC4(MX480-5 vty)# show sample-rr route-tbl ipv4 vrf 0 prefix  12.0.0.0/24 < (inline-flow case) calculates the record, in which only one record is there with the next-hop set as 11.0.0.2

======================================================================================================
    Rec#     AS-Idx    EG-Ref-Cn      State    Pre-Len             Prefix        OIF            Gateway
======================================================================================================
       1          2          1          0         24          12.0.0/24        371           11.0.0.2
NPC4(MX480-5 vty)#

Collector Flow Record:
SrcAddr: 15.0.0.1 (15.0.0.1)
DstAddr: 12.0.0.225 (12.0.0.225)
IP ToS: 0x08
Protocol: 17
SrcPort: 1024
DstPort: 1024
ICMP Type: 0
InputInt: 804
SrcMask: 24
DstMask: 24
SrcAS: 65005
DstAS: 65001
NextHop: 11.0.0.2<

SrcAddr: 15.0.0.1 (15.0.0.1)
DstAddr: 12.0.0.181 (12.0.0.181)
IP ToS: 0x08
Protocol: 17
SrcPort: 1024
DstPort: 1024
ICMP Type: 0
InputInt: 804
SrcMask: 24
DstMask: 24
SrcAS: 65005
DstAS: 65001
NextHop: 11.0.0.2 (11.0.0.2)<

The output, which is noticed in collector with inline-jflow configured in the above topology, is as expected with equal-cost multipath traffic. 

 

Modification History

2019-04-01: Added EX Series