This article provide details on how to use "jsqlsyncd_collect_data.sh" script which can be used to collect data, memory stats and auto restart jsqlsyncd when its memory usage is really high.

• JSQLSYNCD process memory usage is really high. In the below example memory size is ~4G ( 4021M)

UserA@FW-SITE1> show system processes extensive | match jsqlsync

PID USERNAME PRI NICE  SIZE  RES STATE  C  TIME  WCPU COMMAND

21672 root     28  0 4021M 1169M select  1 11:20  0.00% jsqlsyncd{jsqlsyncd}

21672 root     20  0 4021M 1169M sbwait  0  0:01  0.00% jsqlsyncd{jsqlsyncd}

21672 root     20  0 4021M 1169M sigwai  1  0:01  0.00% jsqlsyncd{jsqlsyncd}

• Following error messages observed in /var/log/messages

Jul 16 18:45:52 FW-SITE1 jsqlsyncd[21672]: failed to dump logs

Jul 16 18:45:52 FW-SITE1 jsqlsyncd[21672]: SQL error: out of memory

• Following error messages observed in debug log file /var/log/jsqlsyncd_log.txt

Jul 16 23:40:11 jsqlsyncd[21672]:ERROR:js_ReadRepl:2365 failed to dump logs

Jul 16 23:40:11 jsqlsyncd[21672]:ERROR:js_ReadRepl:2366 SQL error: out of memory

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_check_db:3135 /var/db/secintel_db/sql/manifest.db: sync_id=0 state set HOT

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_BackupOk:2704 time=15s up=1 filterup=1

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_BackupOk:2704 time=15s up=1 filterup=1

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_check_db:3062 /var/db/secintel_db/sql/CC.db: sync_id=0 wait=0 status=0 (HOT)

Jul 16 23:40:11 jsqlsyncd[21672]:ERROR:js_ReadRepl:2333 can't open db /var/db/secintel_db/sql/CC.db rc=7

• Secintel replication db file size is higher than regular db files. In the below example manifest.db.repl.sdb and CC.db.repl.sdb size is higher than respective db files i,e manifest.db and CC.db.

root@FW-SITE1 :/var/db/secintel_db/sql # ls -ltr

total 2143656

-rw-r--r-- 1 root wheel    24576 Apr 19 05:25 Infected-Hosts.db

-rw-r--r-- 1 root wheel    40960 Apr 19 05:25 Whitelist.db

-rw-r--r-- 1 root wheel    24576 Apr 19 05:30 Infected-Hosts.db.repl.sdb

-rw-r--r-- 1 root wheel    24576 Jul 15 08:35 Whitelist.db.repl.sdb

-rw-r--r-- 1 root wheel   9592832 Jul 16 21:26 CC.db

-rw-r--r-- 1 root wheel 2180726784 Jul 16 22:59 CC.db.repl.sdb

-rw-r--r-- 1 root wheel    24576 Jul 16 22:59 manifest.db

-rw-r--r-- 1 root wheel   3944448 Jul 16 23:28 manifest.db.repl.sdb

Shell script jsqlsyncd_collect_data.sh can be  used to auto restart jsqlsyncd when the memory usage goes above the expected values and this script can also be used to collect logs automatically. This shell script can be executed with in SRX device. 

The script can be downloaded from here .

jsqlsyncd_collect_data.sh script should be added to cron job to run periodically as suitable for customer with `--restart[=<mem_size_in_KB>]` option to restart the jsqlsyncd process automatically when the memory footprint goes over the given memory size.


Script can also dump jsqlsynd memory footprint periodically (`--memstat[=<mem_log_file>]`), enable and collect jsqlsyncd log for specified duration (`--collect[=<duration_in_sec>]`) periodically. when it is used for collecting data, the script can also gather the collected logs in a consolidated archive (`--gather[=<dire_for_collection>]`).

Here are the guidelines to add the script to cronjob. 

• Login in to the SRX as root access shell using "start shell" and follow below instructions.

$ crontab -e #this will open vi editor
#add the below line (<i>):
0 23 * * * <absolute_path_to_the_script> --restart >> /var/log/jsqlsyncd_script_logs.txt
# save and quit (<esc>:wq)
verify it is added -
$ crontab -l
note: `0 23 * * *` signifies that the job will run everyday at 23:00 hr, this can changed according to customers requirement, downtime etc.

All the arguments for the script are optional, however at least one option has to be specified. the script simply exits if no option is given.

• Multiple jobs for the script can be added, e.g. one job for checking memory limit every hour, and another to collect logs once every day etc

0 23 * * * <absolute_path_to_the_script> --restart >> /var/log/jsqlsyncd_script_logs.txt # runs everyday at 23:00hr to restart jsqlsyncd if memory limit exceeded

0 0 * * * <absolute_path_to_the_script> --collect >> /var/log/jsqlsyncd_script_logs.txt # runs everyday at 00:00hr to collect jsqlsyncd logs for 5 min(default)

0 * * * * <absolute_path_to_the_script> --memstat >> /var/log/jsqlsyncd_script_logs.txt # runs every hour at 00min to dump jsqlsynd memstat