Juniper recommends that customers should not export flow record packets via the management Ethernet interface (fxp0) because this setup works differently depending on whether a user-defined default route is present or not, and may not yield expected flow monitoring results.
In this article, we demonstrate how the flow record packet is exported via fxp0 and how the user-defined default route, when configured, prevents the packet from being exported via fxp0.
Note: Our test is the sFlow basis, but the same can be seen with jFlow as well.
Test Topology
Flow collector |.112 | mgmt network 172.27.116.0/23 | fxp0|.162 [LSYS1]ge-1/0/0 ------- ge-1/0/9[default LSYS]ge-1/1/0 -------- ge-1/1/9[LSYS2] .1 .2 .1 .2 <-----192.168.10.0/24-----> <-----192.168.20.0/24----->
Configuration Without the Default Route
set logical-systems LSYS1 interfaces ge-1/0/0 unit 0 family inet address 192.168.10.1/24 set logical-systems LSYS1 routing-options static route 192.168.20.0/24 next-hop 192.168.10.2 set logical-systems LSYS2 interfaces ge-1/1/9 unit 0 family inet address 192.168.20.2/24 set logical-systems LSYS2 routing-options static route 192.168.10.0/24 next-hop 192.168.20.1 set interfaces ge-1/0/9 unit 0 family inet address 192.168.10.2/24 set interfaces ge-1/1/0 unit 0 family inet address 192.168.20.1/24 set protocols sflow sample-rate ingress 100 set protocols sflow sample-rate egress 100 set protocols sflow source-ip 172.27.116.162 set protocols sflow collector 172.27.116.112 set protocols sflow interfaces ge-1/0/9.0
Processing of sFlow Records Without a Default Route
User@MX> ping 192.168.20.2 rapid count 9999999 logical-system LSYS1 PING 192.168.20.2 (192.168.20.2): 56 data bytes !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! User@MX> show interfaces ge-1/0/9 media Physical interface: ge-1/0/9, Enabled, Physical link is Up Interface index: 160, SNMP ifIndex: 575 Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online Pad to minimum frame size: Disabled Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Schedulers : 0 Current address: 2c:21:72:b8:0a:9e, Hardware address: 2c:21:72:b8:0a:9e Last flapped : 2019-10-08 15:32:46 JST (20:59:10 ago) Input rate : 728432 bps (1083 pps) Output rate : 728432 bps (1083 pps)
Verify that the sFlow records are sent via fxp0 on the Routing Engine (RE) and they are received by the flow collector.
No. Time Source Destination Protocol Length Info 22191 35.035822 172.27.116.162 172.27.116.112 sFlow 1404 V5, agent 172.27.116.162, sub-agent ID 1, seq 12140, 8 samples Frame 22191: 1404 bytes on wire (11232 bits), 1404 bytes captured (11232 bits) Juniper Ethernet Ethernet II, Src: TeknorMi_64:68:57 (00:a0:a5:64:68:57), Dst: TeknorMi_8d:b7:8e (00:a0:a5:8d:b7:8e) Internet Protocol Version 4, Src: 172.27.116.162, Dst: 172.27.116.112 User Datagram Protocol, Src Port: 53249, Dst Port: 6343 Source Port: 53249 Destination Port: 6343 Length: 1348 Checksum: 0xbd20 [unverified] [Checksum Status: Unverified] [Stream index: 0] [Timestamps] InMon sFlow Datagram version: 5 Agent address type: IPv4 (1) Agent address: 172.27.116.162 Sub-agent ID: 1 Sequence number: 12140 SysUptime: 993 days, 14 hours, 20 minutes, 5 seconds (85846805s) NumSamples: 8 Flow sample, seq 27669 Flow sample, seq 27670 Flow sample, seq 27671 Flow sample, seq 27672 Flow sample, seq 27673 Flow sample, seq 27674 Flow sample, seq 27340 Flow sample, seq 27675
Why Are sFlow Records Sent Out via fxp0?
A route to the flow collector exists on the RE as shown.
User@MX> show route 172.27.116.112 inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.27.116.0/23 *[Direct/0] 21:50:43 > via fxp0.0
As seen, the route is in the kernel route table.
User@MX> show route forwarding-table destination 172.27.116.112 detail Routing table: default.inet Internet: Enabled protocols: Bridging, Destination Type RtRef Next hop Type Index NhRef Netif 172.27.116.112/32 dest 0 0:a0:a5:8d:b7:8e ucst 359 1 fxp0.0
On the Packet Forwarding Engine (PFE), the interface and " Nexthop prefix " is " -" .
User@MX> start shell pfe network fpc1 NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash) NPC1(MX vty)# show route ip lookup 172.27.116.112/32 Route Information (172.27.116.112/32): interface : - (0) Nexthop prefix : - Nexthop ID : 36 MTU : 0 Class ID : 0
The next-hop type in the PFE is " Reject ."
NPC1(MX vty)# show nhdb id 36 detail ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags ----- -------- ------------- --------------- ---------- ------------ ---- ------------------ ------------------ 36 Reject - - IPv4 - 0 0x0000000000000000 0x0000000000000000 BFD Session Id: 0 Topo-link: [pfe-0]: 0x10818bf490000024 ModifyNH: Subcode=SetErr JNH_BUF_MOD_DATA_ERR: reason = 0x00000028 Reason name: reject route [pfe-1]: 0x10818bf190000024 ModifyNH: Subcode=SetErr JNH_BUF_MOD_DATA_ERR: reason = 0x00000028 Reason name: reject route Dram Bytes : 412 Flags : 0x0000000000000000 Parent NHID : 0 Route Table ID : 0 PreComputed MTU: 0 Tunnel Self ID : 0x0 Tunnel Parent ID : 0x0 Parent NHID : 0 PFE:0 Encap-ptr chain: ---------------- PtrType Ptr RefCount JnhAddr Key ------- ---------- -------- ------- --- Dram Bytes: 412 NPC1(MX vty)# show route ip IPv4 Route Table 0, default.0, 0x80008: Destination NH IP Addr Type NH ID Interface --------------------------------- --------------- -------- ----- --------- default Reject 36 RT-ifl 0 0.0.0.0 Discard 34 RT-ifl 0
Given the above settings, the PFE sends the sFlow records to the default destination RE, which then sends the records to the flow collector via fxp0, because the RE knows the route to the flow collector as shown in steps 3 and 4.
Now configure a default route under the routing-option stanza. After the configuration change, no sFlow record is received by the flow collector.
routing-option
[edit] User@MX# set routing-options static route 0.0.0.0/0 next-hop 192.168.20.2 [edit] User@MX# show |compare [edit] + routing-options { + static { + route 0.0.0.0/0 next-hop 192.168.20.2; + } + } [edit] User@MX# commit commit complete [edit] User@MX# run show configuration routing-options static { route 0.0.0.0/0 next-hop 192.168.20.2; }
In this case, sFlow records are not sent out via fxp0 because a user-defined default route is present and the sFlow records can be sent toward the modified next-hop and not to the flow collector, which exists on the subnet over fxp0.
Also, the RE knows the route to the flow collector. This is the same as Step3.
User@MX> show route 172.27.116.112 inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.27.116.0/23 *[Direct/0] 22:19:05 > via fxp0.0
The RE kernel "forwarding table" also knows the route to the flow collector. This is same as Step4.
In this scenario, the PFE sends sflow records to the next hop (ge-1/1/0.0), so it does not reach the flow collector.
User@MX> start shell pfe network fpc1 NPC platform (1067Mhz MPC 8548 processor, 2048MB memory, 512KB flash) NPC1(MX vty)# show route ip lookup 172.27.116.112/32 Route Information (172.27.116.112/32): interface : ge-1/1/0.0 (343) <-------------------Next-hop is ge-1/1/0.0 Nexthop prefix : 192.168.20.2 Nexthop ID : 702 MTU : 1500 Class ID : 0 NPC1(MX vty)# show route ip NG IPv4 Route Table 0, default.0, 0x80008: Destination NH IP Addr Type NH ID Interface --------------------------------- --------------- -------- ----- --------- default 192.168.20.2 Unicast 702 RT-ifl 0 ge-1/1/0.0 ifl 343 0.0.0.0 Discard 34 RT-ifl 0
As long as a new default route is not installed (by having it defined statically or learned via dynamic routing protocols), sFlow records are sent out via fxp0 to the flow collector because it hits a "reject next-hop" on the Packet Forwarding Engine (PFE) and the packet is punted to the RE CPU.
If, however, a new default route is installed, sFlow records are not sent out via fxp0 to the flow collector. sFlow records are sent toward the modified next-hop instead.
This behavior is *not* a bug. It is a kind of product limitation for some specific use cases with the default route.
For flow-monitoring features, it is strongly recommended that customers should use network interfaces to export flow records, because fxp0 works under certain limitations as explained above.
Note: Refer to Configuring Inline Active Flow Monitoring for considerations that apply to the inline flow-monitoring instance configuration.