Description

When firewall filter reject action applies to a large number of packets per second, the PFE CPU utilization increases and hardware input drops might occur.

Symptoms

The test setup:

    +---------+                        +------+
    |  Router |                        | M7i  |
    |  Tester | FE3.1 ------- fe-1/3/1 | Pula | ge-0/0/0 -------
    |         |                        |      |
    +---------+                        +------+



Relevant interface configuration:

[edit]
georg@pula# show interfaces fe-1/3/1 
unit 0 {
    family inet {
        address 10.20.1.10/30;
    }
}

Note, that it does not matter what type of interface is being used.

Scenario 1) - No firewall filter is applied to the interface

The below output shows that approximately 127k packets are transiting the router


georg@pula> monitor interface traffic
pula                              Seconds: 0                   Time: 13:31:36

Interface    Link  Input packets        (pps)     Output packets        (pps)
 ge-0/0/0      Up              2          (0)             940844     ( 126957 )
    ...
 fe-1/3/1      Up         988038     ( 126950 )                  1          (0)


The PFE statistics don't show any drops at that point:

georg@pula> show pfe statistics traffic   
Packet Forwarding Engine traffic statistics:
    Input  packets:              1553932               126950 pps
    Output packets:              1553949               126951 pps
Packet Forwarding Engine local traffic statistics:                 
    Local packets input                 :                    6
    Local packets output                :                   22  
    Software input control plane drops  :                    0
    Software input high drops           :                    0
    Software input medium drops         :                    0
    Software input low drops            :                    0
    Software output drops               :                    0




The notificaion queues on the PFE don't show any drops

georg@pula> request pfe execute target feb command "show cchip notif" | match "disc count"
 (0xc01c) notification Q1 disc count, RO : 0x00000000
 (0xc020) notification Q2 disc count, RO : 0x00000000
 (0xc024) notification Q3 disc count, RO : 0x00000000
 (0xc028) notification Q4 disc count, RO : 0x00000000



The relevant ICMP statistics on the PFE are all at 0 

georg@pula> request pfe execute target feb command "sho icmp st" | match "request|throttle"  
           0 requests
           0 throttled
           0 tag te requests
           0 throttled
 


CPU utilization on the CFEB is at 3 percent

georg@pula> show chassis cfeb   
CFEB status:
  State                                 Online   
  Intake temperature                 23 degrees C / 73 degrees F
  Exhaust temperature                29 degrees C / 84 degrees F
  CPU utilization                     3 percent
  Interrupt utilization               8 percent
  Heap utilization                   11 percent
  Buffer utilization                 25 percent
  Total CPU DRAM                    128 MB
  Internet Processor II                 Version 2, Foundry IBM, Part number 164
  Start time:                           2007-05-16 13:30:37 CEST
  Uptime:                              9 minutes, 38 seconds






Scenario 2) - Applying the firewall filter with reject action Relevant configuration changes:

[edit]
georg@pula# show firewall
filter icmp-test {
    term 1 {
        then {
            count reject-all-packets;
            reject;
        }
    }
}

[edit]
georg@pula# set interfaces fe-1/3/1 unit 0 family inet filter input icmp-test

[edit]
georg@pula# commit
commit complete
 


Verifying that the filter rejects packets
 
georg@pula> show firewall   
Filter: __default_bpdu_filter__                               
Filter: icmp-test                                             
Counters:
Name                                                Bytes              Packets
reject-all-packets                               23493626               510731



After committing the configuration changes PFE CPU increases to 36 percent
 
georg@pula> show chassis cfeb   
CFEB status:
  State                                 Online   
  Intake temperature                 23 degrees C / 73 degrees F
  Exhaust temperature                29 degrees C / 84 degrees F
  CPU utilization                    36 percent
  Interrupt utilization               2 percent
  Heap utilization                   11 percent
  Buffer utilization                 25 percent
  Total CPU DRAM                    128 MB
  Internet Processor II                 Version 2, Foundry IBM, Part number 164
  Start time:                           2007-05-16 13:43:02 CEST
  Uptime:                              2 minutes, 21 seconds
 



Within a short period the Hardware input drops counter is heavily increasing

georg@pula> show system uptime |match current
Current time: 2007-05-16 13:45:49 CEST

georg@pula> show pfe statistics traffic | match "drops"   
    Software input control plane drops  :                    0
    Software input high drops           :                    0
    Software input medium drops         :                    0
    Software input low drops            :                    0
    Software output drops               :                    0
    Hardware input drops                :              3651769
    Info cell drops            :                    0
    Fabric drops               :                    0

georg@pula> show system uptime |match current             

Current time: 2007-05-16 13:45:52 CEST

georg@pula> show pfe statistics traffic | match "drops"   
    Software input control plane drops  :                    0
    Software input high drops           :                    0
    Software input medium drops         :                    0
    Software input low drops            :                    0
    Software output drops               :                    0
    Hardware input drops                :              4027708
    Info cell drops            :                    0
    Fabric drops               :                    0

georg@pula> 




The counters on the PFE show that drops occur in notification Q2 which is queue for all non-priority IP packets

georg@pula> show system uptime |match current                                                
Current time: 2007-05-16 13:46:11 CEST

georg@pula> request pfe execute target feb command "show cchip notif" | match "disc count"   
 (0xc01c) notification Q1 disc count, RO : 0x00000000
 (0xc020) notification Q2 disc count, RO : 0x00586af8
 (0xc024) notification Q3 disc count, RO : 0x00000000
 (0xc028) notification Q4 disc count, RO : 0x00000000

georg@pula> show system uptime |match current                                                
Current time: 2007-05-16 13:46:13 CEST

georg@pula> request pfe execute target feb command "show cchip notif" | match "disc count"   
 (0xc01c) notification Q1 disc count, RO : 0x00000000
 (0xc020) notification Q2 disc count, RO : 0x005c8893
 (0xc024) notification Q3 disc count, RO : 0x00000000
 (0xc028) notification Q4 disc count, RO : 0x00000000

georg@pula> 



It can be seen that there are many ICMP requests sent to PFE CPU for generating Destination Unreachable ICMP messages

georg@pula> show system uptime |match current                                                 
Current time: 2007-05-16 13:46:38 CEST

georg@pula> request pfe execute target feb command "show icmp st" | match "request|throttle"    
     3271189 requests
     3270574 throttled
           0 tag te requests
           0 throttled

georg@pula> show system uptime |match current                                                 
Current time: 2007-05-16 13:46:41 CEST

georg@pula> request pfe execute target feb command "show icmp st" | match "request|throttle"    
     3379164 requests
     3378527 throttled
           0 tag te requests
           0 throttled

georg@pula>

Solution

To avoid this situation, use firewall filter action discard instead of reject.