This article explains the behaviour of memory allocation after the snapshot creation.

VMhost memory is not freed after executing "request vmhost snapshot" command on SRX. The customer had faced upgrade failure because of the memory issue.

Below you can see the vmhost memory allocation printouts before and after a vmhost snapshot, the amount of free memory decreased from 26G to 14G.

admin@srx> show vmhost resource-usage
Sep 25 15:35:24
Compute cluster: rainier-re-cc

Compute node: rainier-re-cn
CPU Usage
=========
15:35:24 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
15:35:24 all 1.91 0.00 2.64 0.02 0.00 0.01 0.00 66.37 0.00 29.05
15:35:24 0 5.71 0.00 7.02 0.06 0.00 0.03 0.00 0.00 0.00 87.18
15:35:24 1 5.73 0.00 7.06 0.06 0.00 0.04 0.00 0.00 0.00 87.11
15:35:24 2 0.00 0.00 0.60 0.00 0.00 0.00 0.00 99.40 0.00 0.00
15:35:24 3 0.01 0.00 0.37 0.00 0.00 0.00 0.00 99.63 0.00 0.00
15:35:24 4 0.01 0.00 0.38 0.00 0.00 0.00 0.00 99.61 0.00 0.00
15:35:24 5 0.01 0.00 0.43 0.00 0.00 0.00 0.00 99.56 0.00 0.00


Memory Usage
============
total used free shared buffers cache available
Mem: 128097 100542 26799 2 272 481 26413
Swap: 0 0 0

admin@srx> request vmhost snapshot
Sep 25 15:36:42
warning: Existing data on the target may be lost
Proceed ? [yes,no] (no) yes


admin@srx> show vmhost resource-usage
Sep 25 17:27:35
Compute cluster: rainier-re-cc

Compute node: rainier-re-cn
CPU Usage
=========
17:27:35 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
17:27:35 all 1.91 0.00 2.64 0.02 0.00 0.01 0.00 66.37 0.00 29.05
17:27:35 0 5.71 0.00 7.02 0.06 0.00 0.03 0.00 0.00 0.00 87.17
17:27:35 1 5.73 0.00 7.06 0.06 0.00 0.04 0.00 0.00 0.00 87.11
17:27:35 2 0.00 0.00 0.60 0.00 0.00 0.00 0.00 99.40 0.00 0.00
17:27:35 3 0.01 0.00 0.37 0.00 0.00 0.00 0.00 99.63 0.00 0.00
17:27:35 4 0.01 0.00 0.38 0.00 0.00 0.00 0.00 99.61 0.00 0.00
17:27:35 5 0.01 0.00 0.43 0.00 0.00 0.00 0.00 99.56 0.00 0.00


Memory Usage
============
total used free shared buffers cache available
Mem: 128097 100585 14252 2 122 13136 26353
Swap: 0 0 0

What the customer is observing in the SRX is an expected behaviour for now. If we look at the available memory before and after the snapshot creation it's not reduced.

-- Before --


Memory Usage
============
total used free shared buffers cache available
Mem: 128097 100542 26799 2 272 481 26413  


-- After --

Memory Usage
============
total used free shared buffers cache available
Mem: 128097 100585 14252 2 122 13136 26353  << Cache is increased but available is still almost the same amount.
Swap: 0 0 0

This caching happens only once, If you run multiple snapshots we will not see the cache increasing. Memory will be released by the kernel automatically when there is a memory demand. Upgrade failures were seen because of a memory leak. This has nothing to do with the snapshot creation and memory caching.