When an IPv4 or IPv6 BGP FlowSpec filter that matches the icmp-type condition does not specify a protocol or next-header condition in MX Series routers, it is seen to unexpectedly drop TCP traffic.
This article provides more information about this scenario and recommends configuring the BGP FlowSpec filter with a protocol/next-header condition to prevent the TCP traffic drops.
Topology
TRAFFIC >>>>>>>>>>>>>>>>>>>>>>>>> TRAFFIC >>>>>>>>>>>>>>>>>>>>>>>>>>> TRAFFIC >>>>>>>>>>>>>>>>>>>> TRAFFIC xe-0/0/0 +-----------------------+ xe-0/0/1 +----------------------+ xe-0/3/1 IXIA ----------------| MX240 (DUT) |--------------------| ACX |---------------- IXIA +-----------------------+ xe-0/3/0 +----------------------+ | xe-0/1/0 iBGP | | +-----------------------+ | Ubuntu Server (ExaBGP)| +-----------------------+
TRAFFIC >>>>>>>>>>>>>>>>>>>>>>>>> TRAFFIC >>>>>>>>>>>>>>>>>>>>>>>>>>> TRAFFIC >>>>>>>>>>>>>>>>>>>> TRAFFIC
xe-0/0/0 +-----------------------+ xe-0/0/1 +----------------------+ xe-0/3/1 IXIA ----------------| MX240 (DUT) |--------------------| ACX |---------------- IXIA +-----------------------+ xe-0/3/0 +----------------------+ | xe-0/1/0 iBGP | | +-----------------------+ | Ubuntu Server (ExaBGP)| +-----------------------+
In the above topology:
ExaBGP is used as a FlowSpec controller to advertise flow route to MX240
The IXIA traffic generator injects traffic on MX240 via xe-0/0/0 and measures traffic egress from the ACX device
The IXIA flow 1 traffic uses IPv6 and is sent (exclusively) using TCP source port 512 and destination port 80 with the SYN flag set >>> Traffic is dropped .
The IXIA flow 2 traffic uses IPv6 and is sent (exclusively) using TCP source port 1024 and destination port 80 with the SYN flag set >>> Traffic is not dropped .
FlowSpec route is advertised by matching traffic using, for example, icmp-type 2 and destination network only. So it should NOT match the two IXIA flows being sent.
When ExaBGP Advertises Route as Shown:
announce flow route { match { icmp-type 2; destination fd01:666:2::/64; } then { rate-limit 0; } }
The following issue is seen:
TCP traffic is dropped with source-port range from 512 to 767.
TCP traffic with source-port 1024 is not affected.
IXIA before the FlowSpec route is announced
IXIA after the FlowSpec route is announced
MX240
FlowSpec route received by router
Before the FlowSpec route is advertised
labroot@jtac-mx240-r2016-re0> show bgp summary Threading mode: BGP I/O Groups: 1 Peers: 1 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inetflow.0 0 0 0 0 0 0 inet6flow.0 0 0 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 10.99.99.100 5511 8 5 0 108 2:19 Establ inet6flow.0: 0/0/0 /0 inetflow.0: 0/0/0/0
After the FlowSpec route is advertised
labroot@jtac-mx240-r2016-re0> show bgp summary Threading mode: BGP I/O Groups: 1 Peers: 1 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inetflow.0 0 0 0 0 0 0 inet6flow.0 1 1 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 10.99.99.100 5511 9 6 0 108 2:20 Establ inet6flow.0: 1/1/1 /0 inetflow.0: 0/0/0/0
FlowSpec route
BGP control plane detail showing destination prefix fd01:666:2::/64 and icmp6-type=2 as only match condition
labroot@jtac-mx240-r2016-re0> show route table inet6flow.0 detail inet6flow.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) fd01:666:2::/64,*,icmp6-type=2/term:2 (1 entry, 1 announced) *BGP Preference: 170/-101 Next hop type: Fictitious, Next hop index: 0 Address: 0x4e1681c Next-hop reference count: 1 Next hop: State: <Active Int Ext SendNhToPFE> Local AS: 5511 Peer AS: 5511 Age: 54 Validation State: unverified Task: BGP_5511.10.99.99.100 Announcement bits (1): 0-Flow AS path: I Communities: traffic-rate:0:0 Accepted Localpref: 100 Router ID: 172.16.2.3
FlowSpec filter programmed but matching TCP packets even though the intention is to match ICMP Type 2, not TCP protocol
labroot@jtac-mx240-r2016-re0> show firewall filter __flowspec_default_inet6__ Filter: __flowspec_default_inet6__ Counters: Name Bytes Packets fd01:666:2::/64,*,icmp6-type=2 4299993808 48863566
Filter as programmed in PFE
NPC0(jtac-mx240-r2016-re0 vty)# show filter index 65024 program Filter index = 65024 Optimization flag: 0xf7 Filter notify host id = 0 Pfe Mask = 0xFFFFFFFF jnh inst = 0x0 Filter properties: None Filter state = CONSISTENT term interface-group term priority 0 interface-group 134 true branch to match action in rule default-term false branch to match icmp-type in rule fd01:666:2::/64,*,icmp6-type=2 term fd01:666:2::/64,*,icmp6-type=2 term priority 0 icmp-type 2 false branch to match action in rule default-term destination-address fd01:666:2::/64 false branch to match action in rule default-term then discard count fd01:666:2::/64,*,icmp6-type=2 <<< This is the counter showing matched packets, when in fact only TCP traffic is sent. term default-term term priority 0 then accept
This behavior is documented via PR1607185 and it is functioning as designed:
When the Juniper FlowSpec implementation was done, support was added based on ietf-idr-flow-spec version-5 of the draft. If you check this version of the draft, it does not mention anything related to ICMP.
However, multiple differences exist between this implemented version and the latest RFC that is now available at the time of writing this article.
It is recommended that "icmp-type" match should always go with the next-header/protocol match for inet/inet6 with the filter. See flow (IPv6) . Otherwise, you will see the unexpected behavior as described above.
icmp6-type icmp6-type-value
ICMP6 packet type field. Normally, you specify this match in conjunction with the protocol match statement to determine which protocol is being used on the port.
In place of the numeric value, you can specify one of the following text synonyms (the field values are also listed): echo-reply (0), echo-request (8), info-reply (16), info-request (15), mask-request (17), mask-reply (18), parameter-problem (12), redirect (5), router-advertisement (9), router-solicit (10), source-quench (4), time-exceeded (11), timestamp (13), timestamp-reply (14), or unreachable (3).
Note
In the Internet Protocol version 4 (IPv4) [ RFC791 ] there is a field called "Protocol" to identify the next level protocol, which is an 8 bit field. In Internet Protocol version 6 (IPv6) [ RFC8200 ], this field is called the "Next Header" field. Protocol 58 corresponds to IPv6-ICMP - ICMP for IPv6. See IANA Protocol Numbers .
To prevent such TCP traffic drops as described earlier, it is recommended that ExaBGP should advertise routes by explicitly specifying the protocol/next-header condition (this is demonstrated below):
announce flow route { match { icmp-type 2; protocol 58 ; destination fd01:666:2::/64; } then { rate-limit 0; } }
labroot@jtac-mx240-r2016-re0> show bgp summary Threading mode: BGP I/O Groups: 1 Peers: 1 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inetflow.0 0 0 0 0 0 0 inet6flow.0 1 1 0 0 0 0 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 10.99.99.100 5511 4429 4915 0 112 1d 12:51:23 Establ inet6flow.0: 1/1/1/0 inetflow.0: 0/0/0/0
labroot@jtac-mx240-r2016-re0> show route table inet6flow.0 detail inet6flow.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) fd01:666:2::/64,*,proto=58,icmp6-type=2/term:2 (1 entry, 1 announced) *BGP Preference: 170/-101 Next hop type: Fictitious, Next hop index: 0 Address: 0x4e1681c Next-hop reference count: 1 Next hop: State: <Active Int Ext SendNhToPFE> Local AS: 5511 Peer AS: 5511 Age: 41 Validation State: unverified Task: BGP_5511.10.99.99.100 Announcement bits (1): 0-Flow AS path: I Communities: traffic-rate:0:0 Accepted Localpref: 100 Router ID: 172.16.2.3
FlowSpec filter programmed but now not matching TCP packets:
labroot@jtac-mx240-r2016-re0> show firewall filter __flowspec_default_inet6__ Filter: __flowspec_default_inet6__ Counters: Name Bytes Packets fd01:666:2::/64,*,proto=58,icmp6-type=2 0 0
Filter as programmed in PFE:
NPC0(jtac-mx240-r2016-re0 vty)# show filter index 65024 program Filter index = 65024 Optimization flag: 0xf7 Filter notify host id = 0 Pfe Mask = 0xFFFFFFFF jnh inst = 0x0 Filter properties: None Filter state = CONSISTENT term interface-group term priority 0 interface-group 134 true branch to match action in rule default-term false branch to match payload-protocol in rule fd01:666:2::/64,*,proto=58,icmp6-type=2 term fd01:666:2::/64,*,proto=58,icmp6-type=2 term priority 0 payload-protocol 58 false branch to match action in rule default-term icmp-type 2 false branch to match action in rule default-term destination-address fd01:666:2::/64 false branch to match action in rule default-term then discard count fd01:666:2::/64,*,proto=58,icmp6-type=2 term default-term term priority 0 then accept
This issue is tracked in PR1607185 . The PR modifies Junos OS behavior so that the workaround explained in this article is not needed on versions where the PR fix has been committed.