[Security Director Insights] Elasticsearch service might crash with error "Too many open files"

From the elasticsearch.log we may see "Too many open files" error.

[2024-04-29T14:48:41,637][WARN ][o.e.i.c.IndicesClusterStateService] [osx06640] [junoslogs-2024.02.01-000002][0] marking and sending shard failed due to [failed recovery]

org.elasticsearch.indices.recovery.RecoveryFailedException: [junoslogs-2024.02.01-000002][0]: Recovery failed on {osx06640}{tkyTZ6CRTxOzn1upy6Hb7A}{8iC9DkTsRC-OdQDrmWRl7A}{127.0.0.1}{127.0.0.1:9300}{dim}

Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway

Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine

Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/nodes/0/indices/bMxPbmwuRCymO3RlO-Lzkw/0/translog/translog-1144.ckp: Too many open files

Run the below curl cmd to check current open file limit.

curl -X GET "localhost:9222/_nodes/stats/process?filter_path=**.max_file_descriptors&pretty"

If it is set to the default 65k limit, please try increasing it.

1. vi /usr/lib/systemd/system/elasticsearch.service

2. Change "LimitNOFILE" to a bigger value, example (LimitNOFILE=655350)

3. sudo systemctl daemon-reload

4. service elasticsearch restart

Confirm that the limit has been changed by running above curl command again.
Check the elasticsearch.log and see if "Too many open files" error is gone now.