The JSA Console is responsible for replicating its database and also pushing deployment configuration to all managed hosts in a deployment. Occasionally, however, one or more hosts might time out during the Deploy Changes process. The Console and all managed hosts in the deployment must have matching tokens in the /opt/qradar/conf/host_tokens.masterlist and /opt/qradar/conf/host.token files to avoid deployment issues.
After deploying changes, the Console or the managed hosts time out. In the /var/log/qradar.log file, the following messages appear. Note the IP address of the appliance that timed out from the user interface or the logs.
Host token issues may be reported with the following log messages:
Unable to retrieve authentication token for RPC call
Host token invalid. Unable to download database updates
Unable to decrypt the host token from: /opt/qradar/conf/host.token
Failed Read Host Token File: host.token
Example on 7.5.0+ Console
[ConfigChangeObserver Timer[1]] com.q1labs.core.shared.jsonrpc.RPC: [INFO] [NOT:0000006000][X.X.X.X/- -] [-/- -]Following message suppressed 39 times in 300000 milliseconds [ConfigChangeObserver Timer[1]] com.q1labs.core.shared.jsonrpc.RPC: [ERROR] [NOT:0000003000][X.X.X.X/- -] [-/- -]Unable to retrieve authentication token for RPC call [ConfigChangeObserver Timer[1]] com.q1labs.frameworks.crypto.DecryptException: com.ibm.si.mks.CryptoException: Failed to decrypt data -- 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' [ConfigChangeObserver Timer[1]] at com.q1labs.frameworks.crypto.CryptoUtils.decrypt(CryptoUtils.java:56) [ConfigChangeObserver Timer[1]] at com.q1labs.core.shared.jsonrpc.RPC.readAuthenticationToken(RPC.java:291) [ConfigChangeObserver Timer[1]] at com.q1labs.core.shared.jsonrpc.RPC.executeMethodWithTimeout(RPC.java:213) [ConfigChangeObserver Timer[1]] at com.q1labs.hostcontext.configuration.ConfigChangeObserver$CheckDeployRequestTimer.getActionRequest(ConfigChangeObserver.java:426) [ConfigChangeObserver Timer[1]] at com.q1labs.hostcontext.configuration.ConfigChangeObserver$CheckDeployRequestTimer.timeExpired(ConfigChangeObserver.java:401) [ConfigChangeObserver Timer[1]] at com.q1labs.hostcontext.configuration.ConfigChangeObserver$ConfigChangeObserverTask.run(ConfigChangeObserver.java:662) [ConfigChangeObserver Timer[1]] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) [ConfigChangeObserver Timer[1]] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319) [ConfigChangeObserver Timer[1]] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191) [ConfigChangeObserver Timer[1]] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [ConfigChangeObserver Timer[1]] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [ConfigChangeObserver Timer[1]] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [ConfigChangeObserver Timer[1]] at java.lang.Thread.run(Thread.java:822) [ConfigChangeObserver Timer[1]] Caused by: [ConfigChangeObserver Timer[1]] com.ibm.si.mks.CryptoException: Failed to decrypt data -- 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' [ConfigChangeObserver Timer[1]] at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:390) [ConfigChangeObserver Timer[1]] at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:373) [ConfigChangeObserver Timer[1]] at com.ibm.si.mks.Crypto.decrypt(Crypto.java:73) [ConfigChangeObserver Timer[1]] at com.q1labs.frameworks.crypto.CryptoUtils.decrypt(CryptoUtils.java:53) [ConfigChangeObserver Timer[1]] ... 12 more [ConfigChangeObserver Timer[1]] Caused by: [ConfigChangeObserver Timer[1]] javax.crypto.BadPaddingException: Given final block not properly padded [ConfigChangeObserver Timer[1]] at com.ibm.crypto.provider.AbstractBufferingCipher.a(Unknown Source) [ConfigChangeObserver Timer[1]] at com.ibm.crypto.provider.AbstractBufferingCipher.engineDoFinal(Unknown Source) [ConfigChangeObserver Timer[1]] at javax.crypto.Cipher.doFinal(Unknown Source) [ConfigChangeObserver Timer[1]] at com.ibm.si.mks.KeyStoreCrypto.decrypt_old(KeyStoreCrypto.java:387) [ConfigChangeObserver Timer[1]] ... 15 more
Example on 7.5.0+ managed host
managed-host.local systemd[1]: Starting hostcontext daemon... managed-host.local systemd[1]: Started hostcontext daemon. managed-host.local python[12887]: detected unhandled Python exception in '/opt/qradar/lib/python/qradar/mks.py' managed-host.local replication[12496]: Host token invalid. Unable to download database updates. managed-host.local hostcontext[11107]: com.q1labs.hostcontext.lifecycle.LifeCycleException: Unable to reset running lock managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.start(BackupRecoveryEngine.java:5349) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext.start0(HostContext.java:733) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext.access$700(HostContext.java:98) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.HostContext$5.run(HostContext.java:915) managed-host.local hostcontext[11107]: Caused by: com.q1labs.configservices.hostcontext.exception.BackupException: unable to release running lock, future actions will not run until this lock is released managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.releaseRunningLock(BackupRecoveryEngine.java:1726) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.start(BackupRecoveryEngine.java:5337) managed-host.local hostcontext[11107]: ... 3 more managed-host.local hostcontext[11107]: Caused by: com.q1labs.configservices.hostcontext.exception.BackupException: Unable to determine if backup already running managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.setBackupRunning(BackupRecoveryEngine.java:556) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.releaseRunningLock(BackupRecoveryEngine.java:1722) managed-host.local hostcontext[11107]: ... 4 more managed-host.local hostcontext[11107]: Caused by: java.lang.Exception: Tomcat is not running. Unable to update a backup running lock (key:BACKUP_RUNNING_105, jsonObject:null) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.core.BackupUtils.setBackupRunningLock(BackupUtils.java:2221) managed-host.local hostcontext[11107]: at com.q1labs.hostcontext.backup.BackupRecoveryEngine.setBackupRunning(BackupRecoveryEngine.java:552) managed-host.local hostcontext[11107]: ... 5 more managed-host.local systemd[1]: hostcontext.service: main process exited, code=exited, status=1/FAILURE managed-host.local systemd[1]: Unit hostcontext.service entered failed state. managed-host.local systemd[1]: hostcontext.service failed.
Example on 7.4.3 and older on the Console or managed host:
[hostcontext.hostcontext] [ConfigChangeObserver Timer[1]] com.q1labs.configservices.hostcontext.exception.HostContextException: Failed to execute url https://127.0.0.1/console/fetchConfig/globalset_list.xml HTTP/1.1 400 Bad Request [hostcontext.hostcontext] [main] java.lang.Exception: Unable to decrypt the host token from: /opt/qradar/conf/host.token [hostcontext.hostcontext] [main] com.q1labs.configservices.common.ConfigServicesException: Failed
Perform the following steps to diagnose the issue and if the tokens on the managed host and the Console do not match, apply the solution:
Use SSH to log in to the JSA Console as the root user.
Check the sums of the tokens on the Console by using md5sum and review the output to confirm whether the MD5 sums match each other.
updatedb;md5sum $(locate host_tokens.masterlist)
Example of matching output:
9c42f5150dd2a3e3923a9bfb35d15a22 /opt/qradar/conf/host_tokens.masterlist 9c42f5150dd2a3e3923a9bfb35d15a22 /store/configservices/host_tokens.masterlist 9c42f5150dd2a3e3923a9bfb35d15a22 /store/configservices/backup/deployed/GLOBALSET/host_tokens.masterlist 9c42f5150dd2a3e3923a9bfb35d15a22 /store/configservices/backup/deployed/LOCALSET/host_tokens.masterlist 9c42f5150dd2a3e3923a9bfb35d15a22 /store/configservices/deployed/GLOBALSET/host_tokens.masterlist 9c42f5150dd2a3e3923a9bfb35d15a22 /store/configservices/deployed/LOCALSET/host_tokens.masterlist
From the Console, record the host token entry for the managed host that failed to deploy by using the managed host's IP address.
grep x.x.x.x /opt/qradar/conf/host_tokens.masterlist
Note: On JSA 7.4.3, a new hash was implemented to reinforce security. The entries in the file are expected to be longer than in previous versions.
Example token on 7.4.2 and earlier versions
x.x.x.x=55fe8879-f67a-422f-9dec-d8061e62dab0
Example token on 7.4.3+
x.x.x.x=AQAAAAAAAAABBcctA0NsEeX96c01QFrwId0F8xSr7LHMwvDLEbzdEjEeB2gpzERLuCjGIMHwyN6sASAR8ZNBtRUJbeR+BY6prw==
Open an SSH session to the managed host that is experiencing a deployment issue.
Confirm that the host token on the managed host in /opt/qradar/conf/host.token matches the token recorded in /opt/qradar/conf/host_tokens.masterlist from the Console.
cat /opt/qradar/conf/host.token
Note: There is no newline after the CLI output on a valid token.
Compare the hash between the files with the md5sum command on the managed host.
[root@hostname-managed_host ~]# updatedb; md5sum $(locate host.token) 829268b46f16e4c4ca3bcf7d53e05aae /opt/qradar/conf/host.token 829268b46f16e4c4ca3bcf7d53e05aae /store/configservices/deployed/LOCALSET/host.token
Administrators must manually update the /opt/qradar/conf/host.token on the affected managed host and match it with the entry in /opt/qradar/conf/host_tokens.masterlist.
Log in to the Console as the root user.
Gather the token associated with the affected managed host.
grep '<Managed host IP>' /opt/qradar/conf/host_tokens.masterlist x.x.x.x=arOnjSosaAtTqFgx1111111111111111111111111111111111111V2
Log in to the affected managed host with a mismatched token as the root user.
Back up the original token.
mkdir -pv /store/JTAC/ cp -fv /opt/qradar/conf/host.token /store/JTAC/host.token.backup-$(date +%F)
With the echo command, update the /opt/qradar/conf/host.token file and do not insert a carriage return.
echo -n "arOnjSosaAtTqFgx1111111111111111111111111111111111111V2" > /opt/qradar/conf/host.token
Example output on 7.4.2 and earlier versions:
echo -n '55fe8879-f67a-422f-9dec-d8061e62dab0' > /opt/qradar/conf/host.token
Example output on 7.4.3 and later versions:
echo -n 'AQAAAAAAAAABBcctA0NsEeX96c01QFrwId0F8xSr7LHMwvDLEbzdEjEeB2gpzERLuCjGIMHwyN6sASAR8ZNBtRUJbeR+BY6prw==' > /opt/qradar/conf/host.token
Confirm the correct value:
The output must display a managed host token that matches the primary token from the Console. In our example, it was: arOnjSosaAtTqFgx1111111111111111111111111111111111111V2.
Restart the hostcontext service and wait at least 2 minutes.
Note: The restart of the hostcontext service affects other functions such as correlations, searches, offenses creation, and other functions. Administrators are advised to run the following steps during a scheduled maintenance to avoid undesired results.
systemctl restart hostcontext
systemctl status hostcontext
Log in to the JSA Console as an administrator.
Click the Admin tab.
Click Deploy Changes.
Wait for the deployment to replicate changes to all managed hosts. The host with the updated token is expected to deploy successfully. If other managed hosts in the deployment fail to deploy successfully, you can repeat this procedure to confirm whether the tokens match.
Contact Juniper support if the issue still persists.
2023-07-28: Initial version