This problem is caused traffic addressed to SRX management interface fxp0. As this interface is dedicated for management the rate limiting options are not diverse or even available
There can be quite different issues reported by SRX that can be caused because of the high traffic processing rates on fxp0 interface. All the symptoms can exist together in a device or may be some of them. The isolation of this issue is extremely difficult by just looking at the logs. We might see logs like "JTASK_SCHED_SLIP_KEVENT" but the the script used to track high CPU usage on RE may also not help to isolate and identify the issue.
Few symptoms that could be noticed are:
Unexpected Failovers.
FPC Restart on one or both nodes in a SRX Cluster
Neighbouring devices reporting pause frames from fxp0 interfaces.
Steps to confirm if there is storm on fxp0 interfaceIf the issue is reproducible, disable mgmt port and see if the issue is resolved. If the issue is resolved the issue might be due to storm on mgmt interface.
set interfaces fxp0 unit 0 disable [Note: SSH will not be available after disabling the interface]
Detection of the issue without disabling the management interfaces:
1.) Collect the following at a periodic interval of 5 sec.
show interfaces statistics fxp0
Compare the input packets delta, If there is a significant increase in the number of over any consecutive 5 sec interval there might be storm on the fxp0 interface.
2) top -sh | grep netisr #[Check the CPU utilization of netisr]
3) sysctl debug.netisr.netisr_drop_stats_dump #[Check which type of packets are being dropped]
During the time of the issue, check if there is significant increase in number of input packets on fxp0 interface. Which coincides with higher CPU utilization of netisr and increase in netisr drops.
We only have output of CPU utilization for netisr for this KB and we can see the increase
top cpu . . PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 64K RUN 2 9415.7 87.99% idle{idle: cpu2} 11 root 155 ki31 0B 64K RUN 3 9430.0 87.06% idle{idle: cpu3} 11 root 155 ki31 0B 64K RUN 0 9046.7 81.88% idle{idle: cpu0} 11 root 155 ki31 0B 64K RUN 1 9415.9 79.30% idle{idle: cpu1} 12 root -72 - 0B 432K CPU3 3 73.7H 19.87% intr{swi1: netisr 0} 0 root -76 - 0B 496K CPU0 0 44.0H 15.28% kernel{if_io_tqg_0} 12 root -72 - 0B 432K CPU2 2 290:43 9.77% intr{swi1: ipfwd} 12 root -60 - 0B 432K WAIT 1 753.6H 8.79% intr{swi4: clock (0)} top cpu . . PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 64K RUN 3 9430.0 66.06% idle{idle: cpu3} 11 root 155 ki31 0B 64K RUN 2 9415.7 65.19% idle{idle: cpu2} 11 root 155 ki31 0B 64K RUN 1 9415.9 61.38% idle{idle: cpu1} 11 root 155 ki31 0B 64K RUN 0 9046.7 57.57% idle{idle: cpu0} 12 root -72 - 0B 432K CPU2 2 73.7H 41.99% intr{swi1: netisr 0} 0 root -76 - 0B 496K CPU0 0 44.0H 40.67% kernel{if_io_tqg_0} 12 root -72 - 0B 432K RUN 0 290:45 26.95% intr{swi1: ipfwd} 12 root -60 - 0B 432K WAIT 3 753.6H 8.15% intr{swi4: clock (0)} top cpu . . PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 64K RUN 2 9415.7 76.56% idle{idle: cpu2} 11 root 155 ki31 0B 64K CPU1 1 9415.9 71.48% idle{idle: cpu1} 11 root 155 ki31 0B 64K CPU3 3 9430.0 70.90% idle{idle: cpu3} 11 root 155 ki31 0B 64K CPU0 0 9046.7 70.26% idle{idle: cpu0} 12 root -72 - 0B 432K WAIT 3 73.7H 28.96% intr{swi1: netisr 0} 0 root -76 - 0B 496K - 0 44.0H 24.85% kernel{if_io_tqg_0} 12 root -72 - 0B 432K WAIT 0 290:46 16.80% intr{swi1: ipfwd} 12 root -60 - 0B 432K WAIT 0 753.6H 9.18% intr{swi4: clock (0)}
Solution:The fix is a 2 step process:1. Upgrade to a version where the enhancement is available2. Enable policerA software enhancement was developed to introduce a rate limit for management fxp0 interface.
Software enhancement available in:junos:20.4R3-S8 junos:21.2R3-S5 junos:21.3R3-S5 junos:21.4R3-S4 junos:21.4R3-S5 junos:22.1R3-S3 junos:22.2R3-S1 junos:22.3R2-S2 junos:22.3R3 junos:22.4R2 junos:22.4R3 junos:23.1R2 junos:23.2R1 junos:23.3R1The enhancement is adding a traffic policer for fxp0 management interface to prevent bursts of SRX host inbound traffic from fxp0 interface and limit the RE CPU usage to process those packets.In regular network environment where is no overwhelming traffic towards SRX fxp0 interface this feature is not needed, in limited cases, the traffic policer can be enabled on fxp0 interface as well after upgrade to a Junos release containing enhancement, one of the release listed above or newer.How to verify the fix ?Check if the sysctl hw.mgmt_rate is present. If the sysctl is present fix is present in the release.
root@R11_re0:~ # sysctl -a | grep mgmt_rate hw.mgmt_rate: 0
The policer is not enabled by default. Policer has to be enabled using command sysctl hw.mgmt_rate
Enabling the Policer:
To enable the fxp0 managemnt interface policer from SRX CLI enter Junos shell using root user:
> start shell
Check the default sysctl hw.mgmt_rate value from Junos shell:
>sysctl -a | grep mgmt_rate
To enable the fix and set the policer rate, in the below example the maximum burst rate is set to 3000:
> sysctl hw.mgmt_rate=3000
If the value of the sysctl is set to a value greater than 0 the fix is enabled. The rate of the policer can be changed by changing the sysctl hw.mgmt_rate value. mgmt_rate should be set to a value less than 10K
To display the number of packets received by the policer and number of packets dropped,
sysctl hw.disp_mgmt_policer_stats counter is used: >sysctl hw.disp_mgmt_policer_stats hw.disp_mgmt_policer_stats: rcvd_pkts 63688648 drop_pkts 62728648