This article provides a work-around for correcting the error condition.

Error when loading monitor logs and widgets, with logs indicating circuit-breaking exceptions due to high memory usage in Elasticsearch.

Sample log snippet:

The logapiserver logs show 429 (Too Many Requests) errors and circuit_breaking_exception messages.

[type=circuit_breaking_exception]
elastic: Error 429 (Too Many Requests): [parent] Data too large, data for [<http_request>] 
would be [5.8gb], which exceeds the limit of [5.6gb]

Root Cause (RCA)

• Elasticsearch/OpenSearch is triggering circuit breaker protection because the data being processed exceeds the maximum allowed JVM heap memory.
• In this setup, the customer is using a Medium scale deployment, which allocates approximately 6 GB of JVM heap.
  Due to high EPS (Events Per Second) and increased memory demand, Elasticsearch rejects requests, resulting in logs not loading in the Monitor page.
• Restarting the documentdb service temporarily clears the memory and restores functionality, but this does not address the underlying capacity limitation.
  
  

Workaround: 

Restart the documentdb service to temporarily restore log visibility:
$ service documentdb restart

Verification of current Heap Usage

1) Refer to administration -> system page for system details. 

2) Use the following commands to check heap utilization and system memory:

$ show documentdb stats | heap

Sample output: 
heap_used_in_bytes: 6289389384
heap_max_in_bytes: 6442450944

# show documentdb nodes-stats | heap_used

Sample output: 
heap_used_in_bytes: 6222037648
heap_used_percent: 96
non_heap_used_in_bytes: 204352480

# show documentdb stats

Sample output: 
mem:

   total_in_bytes: 25769803776
   free_in_bytes: 1231630336
   used_in_bytes: 24538173440
   free_percent: 5
   used_percent: 95

Note: These values confirm extremely high heap usage (95–96%), triggering circuit-breaker behavior.


Permanent resolution: 

Migrate from Medium scale to Large scale deployment to increase available JVM heap memory and system resources.

Refer to the official system requirements JSD on Prem documentation. After migration, verify that heap memory has increased:

$ show service opensearch | OPENSEARCH_JAVA_OPTS

Expected Output (Large Scale Example)
OPENSEARCH_JAVA_OPTS: -Xmx15g -Xms15g

If you face any further issues, please contact JUNIPER JTAC Support for assistance.