This article explains what steps can be followed to troubleshoot high memory utilization by jsqlsyncd process in SRX devices.

• JSQLSYNCD process memory usage is really high. In the below example memory size is ~4G ( 4021M)

UserA@FW-SITE1> show system processes extensive | match jsqlsync

PID USERNAME PRI NICE  SIZE  RES STATE  C  TIME  WCPU COMMAND

21672 root     28  0 4021M 1169M select  1 11:20  0.00% jsqlsyncd{jsqlsyncd}

21672 root     20  0 4021M 1169M sbwait  0  0:01  0.00% jsqlsyncd{jsqlsyncd}

21672 root     20  0 4021M 1169M sigwai  1  0:01  0.00% jsqlsyncd{jsqlsyncd}

• Following error messages observed in /var/log/messages

Jul 16 18:45:52 FW-SITE1 jsqlsyncd[21672]: failed to dump logs

Jul 16 18:45:52 FW-SITE1 jsqlsyncd[21672]: SQL error: out of memory

• Following error messages observed in debug log file /var/log/jsqlsyncd_log.txt

Jul 16 23:40:11 jsqlsyncd[21672]:ERROR:js_ReadRepl:2365 failed to dump logs

Jul 16 23:40:11 jsqlsyncd[21672]:ERROR:js_ReadRepl:2366 SQL error: out of memory

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_check_db:3135 /var/db/secintel_db/sql/manifest.db: sync_id=0 state set HOT

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_BackupOk:2704 time=15s up=1 filterup=1

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_BackupOk:2704 time=15s up=1 filterup=1

Jul 16 23:40:11 jsqlsyncd[21672]:DBUG:js_check_db:3062 /var/db/secintel_db/sql/CC.db: sync_id=0 wait=0 status=0 (HOT)

Jul 16 23:40:11 jsqlsyncd[21672]:ERROR:js_ReadRepl:2333 can't open db /var/db/secintel_db/sql/CC.db rc=7

• Secintel replication db file size is higher than regular db files. In the below example manifest.db.repl.sdb and CC.db.repl.sdb size is higher than respective db files i,e manifest.db and CC.db.

root@FW-SITE1 :/var/db/secintel_db/sql # ls -ltr

total 2143656

-rw-r--r-- 1 root wheel    24576 Apr 19 05:25 Infected-Hosts.db

-rw-r--r-- 1 root wheel    40960 Apr 19 05:25 Whitelist.db

-rw-r--r-- 1 root wheel    24576 Apr 19 05:30 Infected-Hosts.db.repl.sdb

-rw-r--r-- 1 root wheel    24576 Jul 15 08:35 Whitelist.db.repl.sdb

-rw-r--r-- 1 root wheel   9592832 Jul 16 21:26 CC.db

-rw-r--r-- 1 root wheel 2180726784 Jul 16 22:59 CC.db.repl.sdb

-rw-r--r-- 1 root wheel    24576 Jul 16 22:59 manifest.db

-rw-r--r-- 1 root wheel   3944448 Jul 16 23:28 manifest.db.repl.sdb

Following log collection guidelines will help to investigate the problem further. 

• Check the process usage periodically ( around 6 hr interval) using below command and collect the outputs. 

• Login to shell as root and verify the db files and repl.sdb files sizes under /var/db/secintel_db/sql path. Collect the output of "ls -ltr  /var/db/secintel_db/sql/* "
•  If the jsqlsyncd process usage is consistenly above 980MB and repl.sdb files also start to grow then enable the jsqlsyncd debug traces and collect it for 3min.
  •   Enable Debug traces: Login to shell as root user and execute below command      

•  Disable Debug traces: Login to shell as root user and execute below command 
       echo 0 > /var/tmp/JSDBDebug
•   Collect all the logs under /var/log/jsqlsyncd_log.txt*

• Collect all secintel db files  from below mentioned paths.  

             /var/db/secintel_db/sql 

              /var/db/ipid

Recovery : 

Following steps can help to recover the memory usage of jsqlsyncd. 

• Login to shell as root and the get the pid of jsqlsyncd process. 

• Generate coredump of the process which will also restart the process using below command