This KB explains the meaning of log message "/kernel: %KERN-2: swapping out sshd[73743] target: 12954 free: 1847" seen on Junos plataforms and also lists down steps which can be used to find the cause and mitigate the same.
Below logs will be seen on the device :-
Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out sshd[73781] target: 12954 free: 1847 Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out sshd[73754] target: 12954 free: 1847 Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out sshd[73751] target: 12954 free: 1847 Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out sshd[73748] target: 12954 free: 1847 Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out cli[73747] target: 12954 free: 1847 Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out sshd[73745] target: 12954 free: 1847 Feb 27 00:36:03.304 device /kernel: %KERN-2: swapping out sshd[73743] target: 12954 free: 1847
In rare scenarios, you may see events like protocol flaps and process crashes as well. Below logs might also be seen along with the above logs:-
Feb 27 00:36:00.217 device /kernel: %KERN-3: rt_pfe_veto: Severe memory pressure, low-free-pages is below threshold. Op 4 err 55, rtsm_id 0:-1, msg type 5 Feb 27 00:36:00.217 device /kernel: %KERN-3: rt_pfe_veto: number of pages free = (697) number of pages in cache = (1136) free page threshold = (1881) error = No Buffer curproc = chassisd
The logs are related to swap memory usage on the device and these logs inform us that swap memory is in use on the device.
Swap memory is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory.
Please follow the below steps to troubleshoot:-
Step-1:- Please check if there were any changes performed on the device or in the network around the time when these logs started generating. Analyze the event as it will help to understand the trigger for swap memory usage.
Step 2:- Collect Multiple Iterations of the below Outputs as this will help to understand which process is utilizing the maximum memory
show system virtual memory (Execute in a gap of 5-10secs and at least 4-5times)
show system processes extensively | except 0.00 ((Execute in a gap of 5-10secs and at least 4-5times)
You may see Output like below if Swap memory is still in use.
user@device> show system processes extensive no-forwarding <Truncated output> Swap: 614M Total, 13M Used, 601M Free, 2% Inuse <<<<<<<<<<<<<<<<<<<<< 2% Inuse (Swap memory)
However, if this is not incrementing then it means that the Memory on the system is stabilized however swap memory usage will not be free on its own and the system needs to be rebooted in a planned maintenance window to clear the usage of swap memory.
Multiple iterations of the command "show system virtual-memory" can help to identify if swap memory usage is increasing on the device or not. Below are sample outputs which can help in the analysis:-
SWAP 2 141K - 2 64 25 swap pager pageins 25 swap pager pages paged in 310 swap pager pageouts 3559 swap pager pages paged out 3328 swap pages used 3472 peak swap pages used SWAP 2 141K - 2 64 25 swap pager pageins 25 swap pager pages paged in 310 swap pager pageouts 3559 swap pager pages paged out 3328 swap pages used 3472 peak swap pages used SWAP 2 141K - 2 64 25 swap pager pageins 25 swap pager pages paged in 310 swap pager pageouts 3559 swap pager pages paged out 3328 swap pages used 3472 peak swap pages used
Above are three iterations collected in 10-second intervals and we don’t see "swap pageins" incrementing which tells us that swap memory usage is not increasing.
Step-3:- If Swap memory usage is continuously increasing then this can impact the device and in such a scenario it is recommended to open a JTAC case.
Step-4:- If Swap memory usage is not increasing then to free the usage, reboot the device in a planned maintenance window. Reboot will also help to clear inactive memory being used by Processes which can aid to increase the overall free memory on the RAM.
Step-5:- Monitor the system for 48 hours and if the event repeats please contact JTAC for further support.