After making changes related to a firewall configuration in a scaled environment, users may observe high CPU on the Routing Engine (RE) and the Flexible PIC Concentrator (FPC) in MX Series routers.

As explained in this article, this is due to the dfwd/dfwc process handled by the single-threaded DFWD daemon, which has to update all the Packet Forwarding Engines (PFEs) with the change so that they correctly program all the FPCs.

In the following example, the firewall filter is configured with more than 15k terms, with each term having a combination of source and destination address, and a port number with/without a policer.

With the above, high CPU is seen for the mgd process followed by the dfwd/dfwc processes as shown below:

[edit firewall filter fw]
-    term term_name {
-        from {
-            source-address {
-                ip-address;
-            }
-            destination-prefix-list {
-                prefix_list;
-            }
-        }
-        then {
-            policer BW_XM;
-            accept;
-        }
-    }

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
46502 root     102    0   735M 72904K CPU3    3   0:46  99.85% mgd
   11 root     155 ki31     0K    64K CPU1    1  67.4H  75.98% idle{idle: cpu1}
   11 root     155 ki31     0K    64K CPU2    2  67.3H  75.98% idle{idle: cpu2}
   11 root     155 ki31     0K    64K RUN     0  67.2H  74.76% idle{idle: cpu0}
   11 root     155 ki31     0K    64K RUN     3  67.3H  73.68% idle{idle: cpu3}

   PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    11 root     155 ki31     0K    64K RUN     3  67.3H  71.29% idle{idle: cpu3}
    11 root     155 ki31     0K    64K RUN     0  67.2H  70.46% idle{idle: cpu0}
    11 root     155 ki31     0K    64K CPU1    1  67.4H  64.70% idle{idle: cpu1}
    11 root     155 ki31     0K    64K CPU2    2  67.3H  63.09% idle{idle: cpu2}
 46677 root      89    0   733M 51400K CPU0    0   0:01  54.49% dfwc
 46602 root      52    0   828M 97872K wait    3   0:07  40.87% dfwd
 46502 root      52    0   735M 41548K select  0   0:51  34.67% mgd

                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
  4  Online            40    100          0       53     28     23    2048       55         21
  5  Online            47     99          0       49     21     15    3584       30         25
  7  Online            44    100          0       44     25     22    2048       57         21
  8  Online            36    100          0       51     30     26    2048       59         21

                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
  4  Online            40    100          0       53     28     23    2048       55         21
  5  Online            47     99          0       49     21     15    3584       30         25
  7  Online            44    100          0       53     28     23    2048       56         21
  8  Online            36    100          0       51     30     26    2048       59         21

This is expected behavior if it occurs intermittently. DFWD is a single-threaded daemon that has to update all PFEs with the changes made so that all the FPCs are then programmed correctly. If the behavior is intermittent, it will not cause any impact on the FPC.

To avoid high CPU utilization as demonstrated above, it is recommended that you modify the firewall filters to include all common subnets/destination IP addresses/port numbers in one tuple/term.

• Use address sets where possible—Address sets simplify administration of firewall policies. They allow you to group large sets of objects so that you can address them as a single object in a security policy. The more rules you can reference to the address sets, the easier it is to make changes because most organizations have logical objects that can be grouped.

• Use single prefixes for source and destination addresses. For example, instead of using /32 addresses and adding each address separately, use a large subnet that covers most of the IP addresses that you require. Use fewer IPv6 addresses because IPv6 addresses consume more memory.

Refer to Firewall Policies Best Practices for more information.