Customer reported "Alarm VMHost RE 0 Disk 1 Write Rate Threshold Cross"

We made some cleanup to clear the alarm & Currently all alarm is clear.

But we need to find what cause the space is consuming. Its still increasing now.

Commit is failing due to this.

root@JTAC# commit check

Apr 11 03:40:11

re0:

error: Could not open configuration database (juniper.data+). Please check space available in /var/. To clear up space run the command 'request system storage cleanup'

root@JTAC> show system storage 

Apr 10 11:32:33

Filesystem       Size    Used   Avail Capacity  Mounted on

/dev/gpt/junos     10.0G    2.3G    6.9G    25% /.mount

/dev/gpt/config     446M    248K    410M    0% /.mount/config

/dev/gpt/var       14G    13G  -164.3M   101% /.mount/var

tmpfs          30G    88K    30G    0% /.mount/tmp

/var/jails/rest-api    14G    13G  -164.3M  101% /.mount/packages/mnt/junos-runtime/web-api/var

tmpfs          8.0G    2.8M    8.0G    0% /.mount/mfs

We went on call & checked this in live.

• We checked the user folders & found no huge volume files.
• Upon checking process stats, we could see rtsockmon generating a huge file with 13.9 G size.

root@JTAC:/var/home # procstat -af

 PID COMM        FD T V FLAGS  REF OFFSET PRO NAME    

<snip>

89634 rtsockmon      1 v r -w------ 2 14934605633 - -        

• We could see rtsockmon process running since 2^nd April, for almost 188 hours

root@JTAC:/var/home # ps -aux | grep 89634

root   89634 9.6 0.0 10856  5748 2- R  2Apr26  188:22.63 rtsockmon -nrt

• This is possible if someone started rtsockmon cli in RE shell & let the session run.
• This basically captures the rpd transactions when routes/nh gets add/deleted/flushed.
• Hence we proceeded to kill the process & thereby stopping the file.

root@JTAC:/var/home # kill 89634

root@JTAC:/var/home # ps -aux | grep 89634

root   9358 0.0 0.0 12676  2384 3 S+ 15:10    0:00.00 grep 89634

root@JTAC:/var/home # ps -aux | grep rtsockmon

root   9402 0.0 0.0 12676  2392 3 S+ 15:10    0:00.00 grep rtsockmon

• Now the storage is also free.

manaf@JTAC> show system storage              

Apr 11 15:10:56

Filesystem       Size   Used   Avail Capacity Mounted on

/dev/gpt/junos    10.0G   2.3G   6.9G   25% /.mount

/dev/gpt/config    446M   296K   410M    0% /.mount/config

/dev/gpt/var      14G   182M    13G    1% /.mount/var

tmpfs          44G    88K    44G    0% /.mount/tmp

/var/jails/rest-api    14G   182M    13G   1% /.mount/packages/mnt/junos-runtime/web-api/var

tmpfs         8.0G   2.9M   8.0G    0% /.mount/mfs

• Since RE0 commit is not in sync now, you will enable & check the commit later.
• Also the RE switchover will be planned in a MW.