Sometimes, the address object or service object search functionality in Junos Space Security Director might not return all results or correct results.

This article explains how the search index can be refreshed from the GUI as well as the Command Line Interface (CLI) in Junos Space Security Director 17.1R1 and later.

Note: From Junos Space Security Director version 17.1R1, Elasticsearch takes care of search and indexing.

Sometimes the search functionality in Junos Space Security Director does not return correct results, including that of objects that are known to exist, or were recently created.

1. Search Database not syncing between nodes
2. Database recently restored and search Database was not regenerated
3. New changes failing to update to search database.
4. Search database failing to generate properly due to other bugs.

All Space Deployments

Note : If you have a multi jboss node space deployment check the section below for multi-node space first.
 

Refresh SD Search index

1. Navigate to Network Management Platform > Administration > Applications
2. (right-click) Security Director
3. (select) Refresh Search Index. This will delete the existing data and refresh the search index data.

Make note of the time you started this job to compare to logs

Verify Search index is complete

1. Login to Junos Space CLI and use the following commands:

   Monitor the logs for completion. (You can start this log tail before pressing re-index in UI)
   Note: The following command is checking 2 log files, depending on space version the log may be found in one log or the other.

    

   tail -f /var/log/jboss/servers/server1/server.log  /var/log/jboss/servers/server1/SD.log | grep -i "full import"
   

2. You should see a log message for start and finish, it can take a long time for indexing to complete, depending on how much data you have in your environment.  For a multi-node space deployment, this message needs to be seen on all nodes.   Note: If "Start Full Import" is not seen in the logs anywhere, try again as "super" user.

   Index Start Example:

   WARN [net.juniper.jnap.sm.search.ejb.SearchAdminHandler] (Thread-1406 (HornetQ-client-global-threads-198019841)) Start Full Import

   Index Finish Example:

   WARN [net.juniper.jnap.sm.search.ejb.SearchAdminHandler] (Thread-1406 (HornetQ-client-global-threads-198019841)) Full Import Done
   

3. If you don't see any log output, the logs may have just rolled over, use the following command to search all logs:

   Note: Make sure to check timestamp and compare to current time, as this searches all logs.

   grep -i "full import"  /var/log/jboss/servers/server1/server.log*  /var/log/jboss/servers/server1/SD.log* 
   

   Search index data can be monitored for growth during indexing

   curl -XGET 'http://localhost:9200/_cat/indices?v'
   

   Example

   health status index    uuid                         pri rep docs.count docs.deleted store.size pri.store.size
   green  open   security frBj647RTjqLvJl21MaItA        2   1      23208            1      7.2mb          7.2mb
   

   Notes

   • Health should be green, if found to be yellow, attempt search re-index or contact JTAC
   • Only index named "security" should be listed.
   • Document count and size will grow periodically during re-index operation; though may pause for 5-10+ minutes between growth jumps depending on object count and system load

    

Check Disk space for /var 

Modifications to search index stop if jboss disk space reaches 90% for /var on any jboss node

From Junos Space CLI

1. Login as admin
2. Select Debug option
3. run the command "df -h"
4. Look at the line:
5. [root@space-0050569ed1d1 ~]# df -h

   Filesystem                    Size  Used Avail Use% Mounted on
   ..snip..
   /dev/mapper/jmpvgnocf-lvvar   262G   44G  206G  18% /var

/var/log/elasticsearch/log-collector.log
[2024-07-26T20:15:57,481][INFO ][o.e.c.r.a.DiskThresholdMonitor] [AAAAAAA] low disk watermark [90%] exceeded on [CCCCCCCCCCCCCCCCCCCC][AAAAAAA][/var/cache/secmgt-elastic-data/log-collector/nodes/0] free: 122.1gb[9.5%], replicas will not be assigned to this node
[2024-07-26T20:15:57,481][INFO ][o.e.c.r.a.DiskThresholdMonitor] [BBBBBBB] low disk watermark [90%] exceeded on [DDDDDDDDDDDDDDDDDDDD][BBBBBBB][/var/cache/secmgt-elastic-data/log-collector/nodes/0] free: 119.2gb[9.3%], replicas will not be assigned to this node

Important: This message will only appear on the primary elastic search node. Which may not be the VIP or Jboss master (this is ok) Check all jboss nodes for this message

Note: Identifiers above will match with "_cluster/state/nodes" output below

Looking for Errors (Exceptions)

All Search logs: (note timestamps)

grep -i SearchAdminHandler /var/log/jboss/servers/server1/server.log*  /var/log/jboss/servers/server1/SD.log*  

Listing Exceptions only:

grep -i SearchAdminHandler /var/log/jboss/servers/server1/server.log*  /var/log/jboss/servers/server1/SD.log* | grep -i Exception

Notes:
Most common error will be timeout. Search index performance has been greatly enhanced for SD 19.1 and higher.
Contact JTAC for workarounds, upgrade is recommended.

Multiple Node Space deployment

Login to Junos Space CLI, for each jboss node and use the following commands:

1. Confirm Search nodes have formed a cluster. Using the following command on each Node

   curl -XGET ' http://localhost:9200/_cluster/health?pretty '
   
   Example:

   [root@space-5254004db3f8 ~]# curl -XGET 'http://localhost:9200/_cluster/health?pretty'
   
   {
     "cluster_name" : "log-collector",
     "status" : "green",    <<< We want to see green here if all is well
     "timed_out" : false,
     "number_of_nodes" : 2,   <<<< This line should match your number of Space Jboss nodes
     "number_of_data_nodes" : 2,
     "active_primary_shards" : 2,
     "active_shards" : 4,
     "relocating_shards" : 0,
     "initializing_shards" : 0,
     "unassigned_shards" : 0,
     "delayed_unassigned_shards" : 0,
     "number_of_pending_tasks" : 0,
     "number_of_in_flight_fetch" : 0,
     "task_max_waiting_in_queue_millis" : 0,
     "active_shards_percent_as_number" : 100.0
   }
   

2. View Nodes in cluster, An entry for each node should be found

   curl -XGET 'http://localhost:9200/_cluster/state/nodes?pretty' 
   

   
   Example:

   [root@space-5254004db3f8 ~]# curl -XGET 'http://localhost:9200/_cluster/state/nodes?pretty'
   {
     "cluster_name" : "log-collector",
     "nodes" : {
       "XKgBuPLUQn-OAtPgUo45ww" : {
         "name" : "XKgBuPL",
         "ephemeral_id" : "hLSJ06AKR_yp1gOQWyIM1A",
         "transport_address" : "10.85.216.134:9300",   << Jboss Node IP
         "attributes" : { }
       },
       "QmUxBA0YQ0S4IZz_koe1RA" : {
         "name" : "QmUxBA0",
         "ephemeral_id" : "wZrEb0_mQ0SA9sBCyJwruA",
         "transport_address" : "10.85.216.132:9300",    << Jboss Node IP
         "attributes" : { }
       }
     }
   }
   

3. If any jboss nodes are missing from Step 2 output manual configuration is needed (Required for SD 19.1)  Perform these steps on all jboss nodes

   Note: If upgrade to space platform was recently performed, each node must be rebooted as part of the upgrade, if the reboot was skipped.  The required firewall rules to allow communication may not be active.
    

   1. Create Directory if it doesn't exist

      ls -l /etc/elasticsearch/discovery-file

      If not found:

      mkdir  /etc/elasticsearch/discovery-file
      

   2. Create the IP list file

      vi /etc/elasticsearch/discovery-file/unicast_hosts.txt
      

      Add 1 line per IP, containing only the IP address

      Example:

      10.85.216.132
      10.85.216.134

   3. Update elasticsearch.yml

      vi /etc/elasticsearch/elasticsearch.yml

      Add the following line to the bottom of the file only if it is not present (Check the complete content of the file)

      discovery.zen.hosts_provider: file

   4. Update Permissions

      chown -R elasticsearch:space   /etc/elasticsearch/discovery-file
      chmod 750 /etc/elasticsearch/discovery-file
      chmod 755 /etc/elasticsearch/discovery-file/unicast_hosts.txt

   5. Restart elasticsearch on all nodes.  The system will take a few minutes to restart and find each node.

      service elasticsearch restart

   6. Check the command output from Step 2, All nodes should now be seen in the output. (This may take 1-10 minutes before processes completely start and find each other depending on environment)

      curl -XGET 'http://localhost:9200/_cluster/state

   7. Refresh search index from web UI following steps above

      The Refresh Search Index CLI procedure for Junos Space 16.1R3 and earlier is detailed here:  KB30468 - Junos Space Security Director re-index object search [juniper.net] .

2023-08-24: Added a clarification: "Add the following line to the bottom of the file only if it is not present (Check the complete content of the file)"

2019-07-17: ​Clarified verification steps, added how to check for errors, and added Cluster formation check and correction

• PR 1312104 - [SD] Search for Address Objects in 17.1R1 gives error "no data available" when Integrated LC is installed