When configuring a firewall with policer under forwarding options to regulate ip-option packets, there is a mismatch in counters of the sum of traffic being sent and the traffic policed and passed by the router with bi-directional traffic.
FPC 2 REV 12 710-010163 HZ0649 E-FPC Type 2 CPU REV 05 710-010169 HX8560 FPC CPU-Enhanced PIC 0 REV 16 750-008155 ND5783 2x G/E IQ, 1000 BASE Xcvr 0 REV 02 740-013111 A511887 SFP-T Xcvr 1 REV 01 740-013111 6291035 SFP-T PIC 1 REV 12 750-007745 WH3545 4x OC-3 SONET, SMIR MMB 1 REV 04 710-010171 JE0317 MMB-5M3-288mbit FPC 5 REV 12 710-010163 HX8537 E-FPC Type 2 CPU REV 05 710-010169 HX8651 FPC CPU-Enhanced PIC 0 REV 16 750-008155 ND5664 2x G/E IQ, 1000 BASE Xcvr 0 REV 01 740-013111 8245783 SFP-T PIC 1 REV 24 750-001902 NE8737 4x OC-12 SONET, MM MMB 1 REV 04 710-010171 HZ4181 MMB-5M3-288mbit
Topology :
+-----------------------+ | | | Platinum | | | 34.34.34.1/24 | | +-----+----------+------+ 21.21.21.6/30 | | ge-5/0/0 | | ge-2/0/0 | | | | | | | | | | | | | | | | C4P3 | | C1P4 21.21.21.5/30 +-----+----------+-------+ 34.34.34.2/24 | | | | | IXIA | | | | | +------------------------+
Configuration :
filter deny-Router-Alert-Option { term 10 { from { ip-options any; } then { count ip-options; next term; } } term 20 { from { ip-options any; } then { policer Rate-limit-1m; count IPv4-Router-Alert; } } term 30 { then { count rest; accept; policer Rate-limit-1m { if-exceeding { bandwidth-limit 1m; burst-size-limit 250k; } then discard; }
filter deny-Router-Alert-Option { term 10 { from { ip-options any; } then { count ip-options; next term; } } term 20 { from { ip-options any; } then { policer Rate-limit-1m; count IPv4-Router-Alert; } } term 30 { then { count rest; accept;
policer Rate-limit-1m { if-exceeding { bandwidth-limit 1m; burst-size-limit 250k; } then discard; }
Traffic Statistics :
Packet Type :
Observation 1 - for uni-directional traffic :
Stream Type : When pumping traffic from IXIA, with a Single stream of 100,000 packets being sent at a rate of 5000 packets per second, the packets sent count is equal to rate limited packets + packet passed by the router.
IXIA-C4P3-> ge-5/0/0-Platinum-ge-2/0/0-> IXIA-C1P2 GE-2/0/0-IN= 100,000 IPv4-Router-Alert + Rate-limit-1m-20 = 44417+ 55583= 100,000 GE-2/0/0-IN == IPv4-Router-Alert + Rate-limit-1m-20 or GE-2/0/0-IN - IPv4-Router-Alert == Rate-limit-1m-20 lab@Platinum-RE0> show firewall Filter: __default_bpdu_filter__ Filter: GE-2/0/0-IN Counters: Name Bytes Packets GE-2/0/0-IN 4600000 100000 Filter: GE-2/0/0-OUT Counters: Name Bytes Packets GE-2/0/0-OUT 0 0 Filter: GE-5/0/0-IN Counters: Name Bytes Packets GE-5/0/0-IN 0 0 Filter: GE-5/0/0-OUT Counters: Name Bytes Packets GE-5/0/0-OUT 2495040 54240 Filter: deny-Router-Alert-Option Counters: Name Bytes Packets ip-options 4600000 100000 IPv4-Router-Alert 2556818 55583 rest 14053 25 Policers: Name Packets Rate-limit-1m-20 44417--------> == GE-2/0/0-IN - IPv4-Router-Alert
IXIA-C4P3-> ge-5/0/0-Platinum-ge-2/0/0-> IXIA-C1P2
GE-2/0/0-IN= 100,000 IPv4-Router-Alert + Rate-limit-1m-20 = 44417+ 55583= 100,000
GE-2/0/0-IN == IPv4-Router-Alert + Rate-limit-1m-20 or GE-2/0/0-IN - IPv4-Router-Alert == Rate-limit-1m-20
lab@Platinum-RE0> show firewall
Filter: __default_bpdu_filter__
Filter: GE-2/0/0-IN Counters: Name Bytes Packets GE-2/0/0-IN 4600000 100000
Filter: GE-2/0/0-OUT Counters: Name Bytes Packets GE-2/0/0-OUT 0 0
Filter: GE-5/0/0-IN Counters: Name Bytes Packets GE-5/0/0-IN 0 0
Filter: GE-5/0/0-OUT Counters: Name Bytes Packets GE-5/0/0-OUT 2495040 54240 Filter: deny-Router-Alert-Option Counters: Name Bytes Packets ip-options 4600000 100000 IPv4-Router-Alert 2556818 55583 rest 14053 25 Policers: Name Packets Rate-limit-1m-20 44417--------> == GE-2/0/0-IN - IPv4-Router-Alert
Observation 2 - For bi-directional traffic : Stream Type : When pumping traffic from IXIA Bi-directionally, with a stream of 50,000 packets being sent at a rate of 5000 packets per second from both directions, the sum of packets sent count is not equal to rate limited packets + packet passed by the router.
IXIA-C4P3-> ge-5/0/0-Platinum-ge-2/0/0 ->IXIA-C1P2 GE-5/0/0-IN+ GE-2/0/0-IN= 50,000+50,000 = 100,000 IPv4-Router-Alert + Rate-limit-1m-20 = 66775+ 56623= 123,398 GE-5/0/0-IN+ GE-2/0/0-IN< IPv4-Router-Alert + Rate-limit-1m-20 or GE-5/0/0-IN+ GE-2/0/0-IN - IPv4-Router-Alert < Rate-limit-1m-20 lab@Platinum-RE0> show firewall Filter: __default_bpdu_filter__ Filter: GE-2/0/0-IN Counters: Name Bytes Packets GE-2/0/0-IN 2300000 50000 Filter: GE-2/0/0-OUT Counters: Name Bytes Packets GE-2/0/0-OUT 773766 16821 Filter: GE-5/0/0-IN Counters: Name Bytes Packets GE-5/0/0-IN 2300000 50000 Filter: GE-5/0/0-OUT Counters: Name Bytes Packets GE-5/0/0-OUT 638296 13876 Filter: deny-Router-Alert-Option Counters: Name Bytes Packets ip-options 4600064 100002 IPv4-Router-Alert 2604630 56623 rest 29487 53 Policers: Name Packets Rate-limit-1m-20 66775----> is greater than (GE-5/0/0-IN+ GE-2/0/0-IN - IPv4-Router-Alert)
IXIA-C4P3-> ge-5/0/0-Platinum-ge-2/0/0 ->IXIA-C1P2
GE-5/0/0-IN+ GE-2/0/0-IN= 50,000+50,000 = 100,000 IPv4-Router-Alert + Rate-limit-1m-20 = 66775+ 56623= 123,398
GE-5/0/0-IN+ GE-2/0/0-IN< IPv4-Router-Alert + Rate-limit-1m-20 or GE-5/0/0-IN+ GE-2/0/0-IN - IPv4-Router-Alert < Rate-limit-1m-20
Filter: GE-2/0/0-IN Counters: Name Bytes Packets GE-2/0/0-IN 2300000 50000
Filter: GE-2/0/0-OUT Counters: Name Bytes Packets GE-2/0/0-OUT 773766 16821
Filter: GE-5/0/0-IN Counters: Name Bytes Packets GE-5/0/0-IN 2300000 50000
Filter: GE-5/0/0-OUT Counters: Name Bytes Packets GE-5/0/0-OUT 638296 13876 Filter: deny-Router-Alert-Option Counters: Name Bytes Packets ip-options 4600064 100002 IPv4-Router-Alert 2604630 56623 rest 29487 53 Policers: Name Packets Rate-limit-1m-20 66775----> is greater than (GE-5/0/0-IN+ GE-2/0/0-IN - IPv4-Router-Alert)
lab@Platinum-RE0> show system statistics ip ip: 6360568 packets with options handled without error 6360568 router alert options
There are no drops seen under show system statistics ip , when traffic is being bi-directionally pumped:
lab@Platinum-RE0> show pfe statistics traffic Packet Forwarding Engine traffic statistics: Input packets: 41870393 12436 pps Output packets: 5218235 2440 pps Packet Forwarding Engine local traffic statistics: Local packets input : 7171850 Local packets output : 5222047 Software input control plane drops : 0 Software input high drops : 0 Software input medium drops : 1 Software input low drops : 0 Software output drops : 1358751 constant Hardware input drops : 26008796 constant
There is no increment in the input and output drops in show pfe statistics traffic , when traffic is being bi-directionally pumped.
This is expected behavior. The detailed calculation is as follows:
Elaborate Calculation : The confusion arises from the sum of Rate-limit-1m-20 and the IPv4-Router-Alert being more than the total packets input. This is due to the counter Rate-limit-1m-20 being updated twice, when the packet is accepted by the PFE policer; but it is dropped by the policer within the kernel . When you configure a forwarding-options FW, the same is stored on the kernel (just like the lo0 filter, which is implemented on both the PFE and kernel.).
Here is the result after a test run. All counters are reset to 0:
lab@Platinum-RE0> show firewall Filter: GE-2/0/0-IN Counters: Name Bytes Packets GE-2/0/0-IN 2883648 62688 Filter: GE-2/0/0-OUT Counters: Name Bytes Packets GE-2/0/0-OUT 1722654 37449 Filter: GE-5/0/0-IN Counters: Name Bytes Packets GE-5/0/0-IN 4563430 99205 Filter: GE-5/0/0-OUT Counters: Name Bytes Packets GE-5/0/0-OUT 947416 20596 Filter: deny-Router-Alert-Option Counters: Name Bytes Packets IPv4-Router-Alert 4161574 90469 ip-options 7447078 161893 rest 0 0 Policers: Name Packets Rate-limit-1m-20 99519 Filter: jtac Counters: Name Bytes Packets icmp 0 0 Filter: __default_bpdu_filter__ lab@Platinum-RE0> Total Packets in = GE-2/0/0-IN + GE-5/0/0-IN = 62688 + 99205 = 161,893 Total Packets out = GE-2/0/0-OUT + GE-5/0/0-OUT = 37449 + 20596 = 58,045 Total Packets Dropped in the router = 161893 - 58045 = 103,848 Policer drop = 99519 (???)--> this count is much high Total Packet dropped in the router - Policer drop = 103848 - 99519 = 4,329 = Option queue drop = 1763 + 2566 = 4,329 IP-option = Dropped + Total out = 161893 Total in IPv4-Router-Alert - total out = 90469 - 58045 = 32,424
Filter: GE-2/0/0-IN Counters: Name Bytes Packets GE-2/0/0-IN 2883648 62688
Filter: GE-2/0/0-OUT Counters: Name Bytes Packets GE-2/0/0-OUT 1722654 37449
Filter: GE-5/0/0-IN Counters: Name Bytes Packets GE-5/0/0-IN 4563430 99205
Filter: GE-5/0/0-OUT Counters: Name Bytes Packets GE-5/0/0-OUT 947416 20596
Filter: deny-Router-Alert-Option Counters: Name Bytes Packets IPv4-Router-Alert 4161574 90469 ip-options 7447078 161893 rest 0 0 Policers: Name Packets Rate-limit-1m-20 99519
Filter: jtac Counters: Name Bytes Packets icmp 0 0
lab@Platinum-RE0>
Total Packets in = GE-2/0/0-IN + GE-5/0/0-IN = 62688 + 99205 = 161,893 Total Packets out = GE-2/0/0-OUT + GE-5/0/0-OUT = 37449 + 20596 = 58,045 Total Packets Dropped in the router = 161893 - 58045 = 103,848
Policer drop = 99519 (???)--> this count is much high
Total Packet dropped in the router - Policer drop = 103848 - 99519 = 4,329 = Option queue drop = 1763 + 2566 = 4,329
IP-option = Dropped + Total out = 161893
Total in IPv4-Router-Alert - total out = 90469 - 58045 = 32,424
If you check the FPC counter and add them up, the policer drop reported by the FPCs are smaller than the FW counter reported under CLI:
FPC2: <-- IPv4-Router-Alert + Policer drop = 35424 + 27264 = 62,688 = ge-2/0/0 in FPC5: <-- IPv4-Router-Alert + Policer drop = 55045 + 44160 = 99,205 = ge-5/0/0 in IPv4-Router-Alert = 35424 + 55045 = 90,469 (Match) Policer drop = 27264 + 44160 = 71,424 !!! BUT CLI reports 99519 => diff = 28,095 IPv4-Router-Alert - total out - Option queue drop = 32424 - 4329 = 28,095 !!!
FPC2: <-- IPv4-Router-Alert + Policer drop = 35424 + 27264 = 62,688 = ge-2/0/0 in FPC5: <-- IPv4-Router-Alert + Policer drop = 55045 + 44160 = 99,205 = ge-5/0/0 in
IPv4-Router-Alert = 35424 + 55045 = 90,469 (Match) Policer drop = 27264 + 44160 = 71,424 !!! BUT CLI reports 99519 => diff = 28,095
IPv4-Router-Alert - total out - Option queue drop = 32424 - 4329 = 28,095 !!!
Who drops these 28095 packets? The FW policer within the kernel drops the traffic.
EGFPC2(Platinum-RE0 vty)# show options statistics IP Option Values: LSRR/SSRR forwarding disabled IP Option Statistics: 0 loose source routes 0 strict source routes 0 record routes 33661 router alerts 0 other options IP Option Errors: 0 runts 0 bad versions 0 runt header lengths 0 giant header lengths 0 null frames 0 bad option lengths 0 duplicate options 0 bad option pointers 0 source route frames dropped IP Option Queue Stats: 33661 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 1763 policer drops IP option protocol queue stats: Protocol igmp (002) max qdepth 192 0 queued 0 queue drops 0 queue deletes 0 high water mark queued 0 current queued 0 policer drops Protocol Other max number tokens 025 33661 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 1763 policer drops <-- IPv4-Router-Alert + Policer drop = 62,688 = ge-2/0/0 in EGFPC2(Platinum-RE0 vty)# show filter counters Filter Counters/Policers: Index Packets Bytes Name -------- -------------------- -------------------- -------- 1 62688 2883648 GE-2/0/0-IN 2 0 0 GE-2/0/0-OUT 3 0 0 GE-5/0/0-IN 4 0 0 GE-5/0/0-OUT 5 35424 1629504 IPv4-Router-Alert 5 62688 2883648 ip-options 5 0 0 rest 5 27264 Rate-limit-1m-20 EGFPC5(Platinum-RE0 vty)# show options statistics IP Option Values: LSRR/SSRR forwarding disabled IP Option Statistics: 0 loose source routes 0 strict source routes 0 record routes 186831 router alerts 0 other options IP Option Errors: 0 runts 0 bad versions 0 runt header lengths 0 giant header lengths 0 null frames 0 bad option lengths 0 duplicate options 0 bad option pointers 0 source route frames dropped IP Option Queue Stats: 52479 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 2566 policer drops IP option protocol queue stats: Protocol igmp (002) max qdepth 192 0 queued 0 queue drops 0 queue deletes 0 high water mark queued 0 current queued 0 policer drops Protocol Other max number tokens 025 52479 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 2566 policer drops <-- IPv4-Router-Alert + Policer drop = 55045 + 44160 = 99,205 = ge-5/0/0 in EGFPC5(Platinum-RE0 vty)# show filter counters Filter Counters/Policers: Index Packets Bytes Name -------- -------------------- -------------------- -------- 1 0 0 GE-2/0/0-IN 2 0 0 GE-2/0/0-OUT 3 99205 4563430 GE-5/0/0-IN 4 0 0 GE-5/0/0-OUT 5 55045 2532070 IPv4-Router-Alert 5 99205 4563430 ip-options 5 0 0 rest 5 44160 Rate-limit-1m-20
EGFPC2(Platinum-RE0 vty)# show options statistics IP Option Values: LSRR/SSRR forwarding disabled
IP Option Statistics: 0 loose source routes 0 strict source routes 0 record routes 33661 router alerts 0 other options
IP Option Errors: 0 runts 0 bad versions 0 runt header lengths 0 giant header lengths 0 null frames 0 bad option lengths 0 duplicate options 0 bad option pointers 0 source route frames dropped
IP Option Queue Stats: 33661 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 1763 policer drops
IP option protocol queue stats: Protocol igmp (002) max qdepth 192 0 queued 0 queue drops 0 queue deletes 0 high water mark queued 0 current queued 0 policer drops
Protocol Other max number tokens 025 33661 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 1763 policer drops
<-- IPv4-Router-Alert + Policer drop = 62,688 = ge-2/0/0 in
EGFPC2(Platinum-RE0 vty)# show filter counters Filter Counters/Policers: Index Packets Bytes Name -------- -------------------- -------------------- -------- 1 62688 2883648 GE-2/0/0-IN 2 0 0 GE-2/0/0-OUT 3 0 0 GE-5/0/0-IN 4 0 0 GE-5/0/0-OUT 5 35424 1629504 IPv4-Router-Alert 5 62688 2883648 ip-options 5 0 0 rest 5 27264 Rate-limit-1m-20
EGFPC5(Platinum-RE0 vty)# show options statistics IP Option Values: LSRR/SSRR forwarding disabled
IP Option Statistics: 0 loose source routes 0 strict source routes 0 record routes 186831 router alerts 0 other options
IP Option Queue Stats: 52479 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 2566 policer drops
Protocol Other max number tokens 025 52479 queued 0 queue drops 0 queue deletes 25 high water mark queued 0 current queued 2566 policer drops
<-- IPv4-Router-Alert + Policer drop = 55045 + 44160 = 99,205 = ge-5/0/0 in
EGFPC5(Platinum-RE0 vty)# show filter counters Filter Counters/Policers: Index Packets Bytes Name -------- -------------------- -------------------- -------- 1 0 0 GE-2/0/0-IN 2 0 0 GE-2/0/0-OUT 3 99205 4563430 GE-5/0/0-IN 4 0 0 GE-5/0/0-OUT 5 55045 2532070 IPv4-Router-Alert 5 99205 4563430 ip-options 5 0 0 rest 5 44160 Rate-limit-1m-20
As the FW is configured under routing-option, each ingress PFE will policer only the traffic being received from it's interface. So, effectively, this policer is on a per PFE basis. However, from the kernel side, it is a real per-routing-instance basis, as the traffic coming from both interfaces hit the kernel under the same FW. So, if the incoming rate is X on each incoming interface, it becomes 2*X in the kernel; so, it is quite likely that the packet does not exceed the policer rate on the PFE; but does no on the kernel and gets dropped.