This article explains how to fix the "too many openfiles" error due to huge number of Jboss openfiles

We will see the below error from the server.log:

2025-02-07 11:30:02,889 ERROR [net.juniper.jnap.appstatisticsmanagement.datacollection.routine.ApplicationVolumeRESTfulReader] (EJB async-pool - 16) IO Exception Happened : java.net.SocketException: Too many open files in system

And from messages.log:

Feb 10 05:02:28 space-<hostname> crmd[3283]: warning: Couldn't read /proc/loadavg: Too many open files in system (23)

Feb 10 05:02:58 space-<hostname> crmd[3283]: warning: Couldn't read /proc/loadavg: Too many open files in system (23)

Feb 10 05:03:02 space-<hostname> vpnwebserver: runuser: failed to execute /bin/bash: Too many open files in system

Feb 10 05:03:28 space-<hostname> crmd[3283]: warning: Couldn't read /proc/loadavg: Too many open files in system (23)

Run the below command to find which process is utilizing more openfiles:

[root@space-<hostname> ~]# lsof | awk '{print $2}' | sort | uniq -c | sort -nr | head

19305783 29398

1561569 28734

232144 3428

150014 2264

 85350 28287

 70184 25442

 11045 28235

  9280 25391

   610 11023

   447 2302

As per the above output, process 29398 is utilizing lots of Openfiles which is a Jboss process.

Check if many sessions between Junos space and webcache service are opened with different source port numbers as per the command mentioned below:

lsof -p 29398 | less

java   29398 jboss *672u    IPv4         811455671      0t0      TCP space-<hostname> :webcache->space-<hostname> :57814 (ESTABLISHED)

java   29398 jboss *673u    IPv4         797681618      0t0      TCP space-<hostname> :webcache->space-<hostname> :38406 (ESTABLISHED)

java   29398 jboss *674u    IPv4         803741477      0t0      TCP space-<hostname> :webcache->space-<hostname> :39767 (ESTABLISHED)

java   29398 jboss *675u    IPv4         810487148      0t0      TCP space-<hostname> :41168->space-<hostname> :webcache (ESTABLISHED)

java   29398 jboss *676u    IPv4         809279434      0t0      TCP space-<hostname> :webcache->space-<hostname>:41168 (ESTABLISHED)

java   29398 jboss *677u    IPv4         810805541      0t0      TCP space-<hostname> :37406->space-<hostname>:webcache (ESTABLISHED)

java   29398 jboss *678u    IPv4         811485721      0t0      TCP space-<hostname> :59302->space-<hostname>:webcache (ESTABLISHED)

Confirm the same in the netstat output:

[root@space-<hostname> ~]# netstat -anep | grep :8080 | wc -l

14941

It will keep on increasing the Openfiles.

Rebooting the affected Space server should fix the issue temporarily.

This is reported in PR1854243 and fixed in SD 23.1 HPv8

2025-03-17 : Article Created

2025-09-05 : Added reboot