Description

Lower end of SRX platforms do not come with a hard disk, they only come with compact flash; which could run out of storage space. This article provides a workaround to storage space problems.
 

Symptoms

While trying to copy Junos image to SRX device via FTP, it is complaining of shortage in storage.   It reports ' write failed, filesystem is full ' and ' No space left on device ', as shown below:

lab@srx240_a> start shell 
% cd /var/tmp
% ftp 192.168.100.2
Connected to 192.168.100.2.
220 srx240_b FTP server (Version 6.00LS) ready.
Name (192.168.100.2:lab): 
331 Password required for lab.
Password:
230 User lab logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> bin
200 Type set to I.
ftp> cd /var/tmp
250 CWD command successful.
ftp> get junos-srxsme-9.6R2.1-domestic.tgz
local: junos-srxsme-9.6R2.1-domestic.tgz remote: junos-srxsme-9.6R2.1-domestic.tgz
200 PORT command successful.
150 Opening BINARY mode data connection for 'junos-srxsme-9.6R2.1-domestic.tgz' (174059005 bytes).
 95% |********************************************************************        |   158 MB    00:0
2 ETA
/: write failed, filesystem is full
 95% |********************************************************************        |   159 MB    00:0
2 ETA
ftp: local: junos-srxsme-9.6R2.1-domestic.tgz: No space left on device
 95% |********************************************************************        |   159 MB    00:0
2 ETA
426 Data connection: Broken pipe.
167000736 bytes received in 60.19 seconds (2.65 MB/s)

We can see that all 1GB of the compact flash (/cf) is used.
lab@srx240_a> show system storage 
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/da0s1a             898M       827M       4.0K      100%  /
devfs                   1.0K       1.0K         0B      100%  /dev
devfs                   1.0K       1.0K         0B      100%  /dev/
/dev/md0                478M       478M         0B      100%  /junos
/cf                     898M       827M       4.0K      100%  /junos/cf
devfs                   1.0K       1.0K         0B      100%  /junos/dev/
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/bo0s1e              24M        22K        22M        0%  /config
/dev/md1                168M        13M       142M        8%  /mfs
/dev/da0s1f              61M       542K        55M        1%  /cf/var/log
/cf/var/jail            898M       827M       4.0K      100%  /jail/var
devfs                   1.0K       1.0K         0B      100%  /jail/dev
/dev/md2                 39M       4.0K        36M        0%  /mfs/var/run/utm
/dev/md3                1.8M       4.0K       1.7M        0%  /jail/mfs

lab@srx240_a> 

Solution

One solution is to run the command, request system storage cleanup .   Note:- This option does not allow to selectively delete files, but instead deletes all files in one shot.

lab@srx240_a> request system storage cleanup 

List of files to delete:

         Size Date         Name
  6979B Dec 15 06:53 /cf/var/log/ike.0.gz
  6991B Dec 15 04:50 /cf/var/log/ike.1.gz
   890B Dec  4 12:36 /cf/var/log/install.0.gz
   143B Dec 15 07:49 /cf/var/log/interactive-commands.0.gz
   120B Sep 24 19:08 /cf/var/log/interactive-commands.1.gz
  6170B Dec 15 07:49 /cf/var/log/messages.0.gz
  4230B Sep 24 19:08 /cf/var/log/messages.1.gz
 165.7M Dec  4 12:36 /cf/var/sw/pkg/junos-9.6R1.13.tgz
 166.1M Dec  8 13:54 /cf/var/sw/pkg/junos-9.6R2.11.tgz
1841.7K Dec  4 12:32 /cf/var/sw/pkg/junos-boot-srxsme-9.6R1.13.tgz
1843.7K Dec  8 13:47 /cf/var/sw/pkg/junos-boot-srxsme-9.6R2.11.tgz
  4010B Dec  8 14:02 /cf/var/tmp/cleanup-pkgs.log
 124.0K Dec  8 13:59 /cf/var/tmp/gres-tp/env.dat
     0B Aug  4 08:29 /cf/var/tmp/gres-tp/lock
     4B Dec  8 14:02 /cf/var/tmp/idp_license_info
    76B Dec  8 14:03 /cf/var/tmp/krt_gencfg_filter.txt
   712B Dec  8 14:03 /cf/var/tmp/sampled.pkts
     0B Aug  4 08:27 /cf/var/tmp/spu_kmd_init
Delete these files ? [yes,no] (no) yes <<<<

Another method to find large files, for example bigger than 10MB, and to selectively delete files is to use the following command.  When the command is run in shell mode, it will show the files consuming the largest amount up space in the CF:

>start shell
%sh -c 'find / -size +10485760c 2> /dev/null' | xargs du -h | sort -nr

228M    /cf/packages/junos-vsrx-12.1X47-D20.7-domestic
 24M    /usr/sbin/flowd_vsrx_hm
 24M    /usr/sbin/flowd_vsrx
 16M    /cf/var/tmp/flowd_vsrx.core-tarball.1.tgz
 16M    /cf/var/tmp/flowd_vsrx.core-tarball.0.tgz
 15M    /usr/sbin/rpd
 13M    /mfs/var/run/db/schema.db

As in the above example, /cf/var/tmp/ has two core file saved which is consuming much of the space.  Deleting these files can free up the necessary space.

Once deleted, confirm the status of the available space in /cf by using the following command:

lab@srx240_a> show system storage 
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/da0s1a             898M       491M       336M       59%  /
devfs                   1.0K       1.0K         0B      100%  /dev
devfs                   1.0K       1.0K         0B      100%  /dev/
/dev/md0                478M       478M         0B      100%  /junos
/cf                     898M       491M       336M       59%  /junos/cf
devfs                   1.0K       1.0K         0B      100%  /junos/dev/
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/bo0s1e              24M        22K        22M        0%  /config
/dev/md1                168M        13M       142M        8%  /mfs
/dev/da0s1f              61M       470K        56M        1%  /cf/var/log
/cf/var/jail            898M       491M       336M       59%  /jail/var
devfs                   1.0K       1.0K         0B      100%  /jail/dev
/dev/md2                 39M       4.0K        36M        0%  /mfs/var/run/utm
/dev/md3                1.8M       4.0K       1.7M        0%  /jail/mfs

lab@srx240_a>

In this SRX device example, 336MB of storage was freed up in the compact flash. Now there is no problem copying the Junos image to the SRX:

ftp> get junos-srxsme-9.6R2.1-domestic.tgz
local: junos-srxsme-9.6R2.1-domestic.tgz remote: junos-srxsme-9.6R2.1-domestic.tgz
200 PORT command successful.
150 Opening BINARY mode data connection for 'junos-srxsme-9.6R2.1-domestic.tgz' (174059005 bytes).
100% |**********************************************************************************|   165 MB    00:00 ETA
226 Transfer complete.
174059005 bytes received in 61.45 seconds (2.70 MB/s)
ftp>

If you still are getting the error, refer to the 'Software Installation Preparation' section of KB16652 - SRX Getting Started - Junos Software Installation [juniper.net]  for more suggestions.

Modification History

2019-07-01: Method to find large files and selectively delete files was added.