This article describes the execution of a shell script used to automate data collection of process, database, and system information for Junos Space. The information collected by the script is critical for debugging and providing root-cause analysis for OutOfMemory errors in Junos Space. The script is intended to be run by JTAC engineers, who will provide the collected information to Junos Space engineering.

A common result of an OutOfMemory Error condition is not being able to log into the Junos Space web user interface. Instead, the user is presented with "The username or password is incorrect." -- even for valid credentials. The error can be verified by the presence of java.lang.OutOfMemoryError: GC overhead limit exceeded entries in Space log files (for example, server.log , serviceInsight.log , and so on).

Restarting the jboss process on the affected node is usually sufficient to remedy the problem. However, important memory-state debugging information is lost due to the service restart.

Prior to rebooting any appliances or restarting jboss processes, the script should be run on all nodes in a Junos Space fabric.

From Java documentation:

"The concurrent collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line."

It is possible to increase the heap size as a temporary workaround. However, it is not a general solution and could cause the appliance to use swap memory, which adversely affects application performance.

OutOfMemory errors should be considered bugs, and should be reported to engineering via a PR.

The PR should contain the archive generated by the attached shell script.

To execute the script, perform the steps below:

1) Download the script from the following URL (log in required):

https://download.juniper.net/software/space/tools/SpaceMemCollect_102.tar.gz

2) Copy SpaceMemCollect.tar.gz to each node in the Space fabric in the /home/admin/ directory.

3) SSH into the CLI of each Space node.

4) Extract the file (the archive contains  SpaceMemCollect.sh ):

5) Change the permissions and ownership of the SpaceMemCollect.sh file:

6) Execute the shell script in the background:

7) Disown the job so that it will continue to run even if the SSH connection times out:

The script will report progress to the CLI such as:

JTAC Junos Space Memory and System Information Collection Tool
-------------------------------------------------------------------
Files will be saved at: /var/space/...
JBoss PID is xxxxx
...
[Step 1 of xx ] Collecting jmap heap dump...
...
[Step xx of xx] Archiving the files to: /var/space/xxx.tar.gz
-------------------------------------------------------------------
Script completed


After the script has completed, you can collect the archive from the /var/space/ directory. Provide this archive to Space engineering.