Description

The /tmp directory on a JUNOS router contains a large number of small files which names are all in the format cprodxxxxxx , such as cprod0SHkpH. In some case, where the number of these files gets too big, configuration commits might fail. Attempting to remove the Advanced Insight script bundle (AI-Scripts) might also result in an error.

Symptoms

The cprodxxxxxx files are usually between 10 and 30 bytes in size. They are located in the /tmp directory on the JUNOS router. 

user@router> file list /tmp detail | match cprod    
----------  1 root  wheel         10 Apr 28 15:19 cprod0SHkpH
----------  1 root  wheel         12 Apr 28 15:20 cprod4BXQ2r
----------  1 root  wheel         10 Apr 28 15:19 cprod579gw2
----------  1 root  wheel         20 Apr 28 15:19 cprod58yecV
----------  1 root  wheel         24 Apr 28 15:19 cprodF5QQ4s
----------  1 root  wheel         24 Apr 28 15:19 cprodHAoqhU
----------  1 root  wheel         20 Apr 28 15:19 cprodHvpEY2
----------  1 root  wheel         20 Apr 28 15:20 cprodIatwF6
----------  1 root  wheel         24 Apr 28 15:19 cprodQNvJC4
----------  1 root  wheel         12 Apr 28 15:20 cprodRy5b9z
----------  1 root  wheel         16 Apr 28 15:19 cprodSAFktD
----------  1 root  wheel         12 Apr 28 15:20 cprodUlcgIC
----------  1 root  wheel         20 Apr 28 15:20 cprodV224BR
----------  1 root  wheel         20 Apr 28 15:19 cprodYJaDp7
----------  1 root  wheel         20 Apr 28 15:19 cprodbpU6xO
----------  1 root  wheel         12 Apr 28 15:20 cprodfS2QzT
----------  1 root  wheel         24 Apr 28 15:19 cprodfr0Cqv
----------  1 root  wheel         20 Apr 28 15:19 cprodhbNXyf
----------  1 root  wheel         16 Apr 28 15:19 cprodkggNCm
----------  1 root  wheel         20 Apr 28 15:19 cprodkhMv3t
----------  1 root  wheel         20 Apr 28 15:20 cprodlYGyS5
----------  1 root  wheel         16 Apr 28 15:19 cprodo4KG3Q
----------  1 root  wheel         20 Apr 28 15:19 cprodpiLFjG
----------  1 root  wheel         10 Apr 28 15:19 cprodqENlNo
----------  1 root  wheel         10 Apr 28 15:19 cprodsGhXfl
----------  1 root  wheel         20 Apr 28 15:20 cprodsNswR6
----------  1 root  wheel         16 Apr 28 15:19 cprodz31LfI
----------  1 root  wheel         20 Apr 28 15:19 cprodzKcIRD

user@router>

In some cases, there can be a very large number of these files.
In one instance seen on a production router, the number of these files got big to the point where configuration commits were no longer possible.
When this happens, the following error messages are displayed: 
user@router# commit check
re0:
commit-script
Cannot open handle for output file
error: Reading the configuration from event scripts failed
error: configuration check-out failed
Trying to delete the AI-Scripts bundle also results in an error: 
user@router> request system scripts delete
/usr/libexec/ui/package: cannot create /tmp/.pkg.LCK.50502: No space left 
on device

Solution

These are temporary files that each contain a CLI command in text format: 

root@router% cat /tmp/cprodV0Y8NU
show jtree 1 memory
root@router%

These files appear on routers running Op Scripts. They should normally be deleted, but due to a software bug, they remain in /tmp on the router. Most notably these files will begin appearing after the Advanced Insight Scripts (AI-Scripts) package is added on the router.  In this case, the cprodxxxxxx files contain the commands the router executes when it is compiling the Juniper Message Bundles (JMBs).

In the situation where commits were no longer possible, Investigation showed that there were more than 280 000 cprodXXXXXX files in /tmp . Contrary to what the error message suggests ( No space left on device ), there was enough space left. However there was no remaining free inode in the /tmp directory ( Cannot open handle for output file ). New files could not be created. After deleting the cprodXXXXXX files, configuration commits were proceeding successfully as expected.

These files are deleted from /tmp upon router reboot.
This issue is described in PR/304750 .
PR/304750 is fixed in 9.4R1, and later releases.