The network manager began receiving reports that the users were unable to access the Juniper Secure Connect VPN. The initial error from the client was an HTTPS error, they typically recover from by restarting the web-management daemon. However, when he attempted to do this he ran into a number of issues.

He is getting software errors when trying to SSH into the device. Console connected they are able to manage the device, but they are also seeing syslog daemon errors when they rebooted the device. After the reboot, the software errors and VPN errors are persisting.

• Several httpd core dumps:

root@SRX-01> show system core-dumps no-forwarding

...

/tftpboot/corefiles/*core*: No such file or directory

-rw-rw---- 1 root wheel   794247 Jul 21 11:35 /jail/var/tmp/httpd.core.0.gz

-rw-rw---- 1 root wheel   854297 Jul 21 13:29 /jail/var/tmp/httpd.core.1.gz

-rw-rw---- 1 root wheel   926722 Jul 21 17:13 /jail/var/tmp/httpd.core.2.gz

-rw-rw---- 1 root wheel   808004 Jul 26 03:24 /jail/var/tmp/httpd.core.3.gz

total files: 6

• The http process is consuming a high percentage of the Routing Engine's CPU:

root@SRX-01> show system processes extensive no-forwarding

 PID USERNAME PRI NICE  SIZE  RES STATE C  TIME  WCPU COMMAND

16386 nobody  123  0 23252K 10784K RUN  0 10:40 33.06% httpd

• Overall, RE CPU does not have resources available for additional tasks:

root@SRX-01> show chassis routing-engine no-forwarding

Routing Engine status:

  Temperature         34 degrees C / 93 degrees F

  CPU temperature       56 degrees C / 132 degrees F

  Total memory       4096 MB Max 1638 MB used ( 40 percent)

   Control plane memory  2320 MB Max  835 MB used ( 36 percent)

   Data plane memory    1776 MB Max  817 MB used ( 46 percent)

  5 sec CPU utilization:

   User           60 percent

   Background         0 percent

   Kernel          40 percent

   Interrupt         0 percent

   Idle            0 percent

  Model             RE-SRX340

  Serial ID           CYXXX8AFXXXX

  Start time           2024-07-08 22:10:41 CDT

  Uptime             17 days, 10 hours, 38 minutes, 17 seconds

  Last reboot reason       0x200:normal shutdown

  Load averages:         1 minute  5 minute 15 minute

                    2.12    1.93    1.96

• The "user" utilization is high most likely due to httpd; but kernel usage is also high. The following logs provide additional insight regarding the cause for high kernel usage:

Jul 26 07:16:23 SRX-01 nh_resolve_throttle:1668Next-hop resolution requests from interface 77 throttled 

Jul 26 07:17:33 SRX-01 nh_resolve_throttle:1668Next-hop resolution requests from interface 77 throttled 

Jul 26 07:23:03 SRX-01 nh_resolve_throttle:1668Next-hop resolution requests from interface 77 throttled 

• These logs report a throttle on next hop resolution requests received by the SRX. Although the log itself does not represent a problem, a high load of ARP requests can overload the FPC and cause high data plane CPU.
• The SRX is also logging CPU threshold exceeded events for the data plane. These peaks last just for a few seconds, but in instances reached 99% capacity and caused packet loss. Log examples:

Jul 26 07:16:32 SRX-01 PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value = 100

Jul 26 07:16:32 SRX-01 PERF_MON: RTPERF_CPU_UTIL_MAX: FPC 0 PIC 0 CPU Utilization greater than 99, expect packet loss

• The high CPU logs interleave with the next hop resolution throttles.
• Flow traceoptions is enabled on the device, note tracing operations contribute to increase the load on the CPU:

RT_FLOW: FLOW_TRACE_INFO: Flow trace warning: flow-trace is running, please disable it after debugging.
 

• I also see multiple login attempts on different protocols from this IP address in particular: 192.168.183.102.

Advised the network manager to restrict management and RE access by configuring web-management on an unused interface and also applying a firewall filter on the loopback interface. Also, applied protect-RE filter to limit access to the RE, and blocked J-Web access for security purposes.

They found the problem being a sensor on their environment missing a whitelist for the gateway addresses defined on the SRX. Traffic load sent to the SRX after re-applying this whitelist reduced greatly, and we are now seeing way better utilization, VPNs have been stable too.

Note that even if you don't exactly have such sensors in your network environment, if there is any other type of device sending high loads of traffic to the SRX RE, the outcome would be the same.