What is the meaning of syslog Aug 2 15:56:41 qfx-switch /kernel: %KERN-2: swapping out sshd[76614] target: 12954 free: 7754 and when does it occur.
Below logs indicate that there were too many ssh sessions running in parallel causing swap memory trigger.
Example LOGS:-
Aug 2 15:56:41 qfx-switch mgd[8889]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8889], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8891]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8891], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8924]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8924], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8926]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8926], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8932]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8932], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8934]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8934], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8940]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8940], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8942]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8942], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8952]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8952], ssh-connection '', client-mode 'junoscript'
Aug 2 15:56:41 qfx-switch mgd[8954]: %INTERACT-6-UI_LOGIN_EVENT: User 'root' login, class 'super-user' [8954], ssh-connection '', client-mode 'junoscript'
Swap memory trigger:-
Aug 2 15:56:41 qfx-switch /kernel: %KERN-2: swapping out sshd[76622] target: 12954 free: 7754
Aug 2 15:56:41 qfx-switch /kernel: %KERN-2: swapping out sshd[76614] target: 12954 free: 7754
root@qfx-switch> show system processes extensive no-forwarding
last pid: 27131; load averages: 1.18, 0.72, 0.61 up 139+17:55:03 23:34:39
174 processes: 2 running, 151 sleeping, 21 waiting
Mem: 973M Active, 143M Inact, 493M Wired, 59M Cache, 69M Buf, 180M Free
Swap: 614M Total, 13M Used, 601M Free, 2% Inuse <<<<<<<<<<<<<<<<<<<<< 2% Inuse (Swap memory)
Check for below outputs from the impacted device to confirm swap memory is currently not being used:-
show system virtual memory (Execute this in a gap of 5-10secs and run it atleast 4-5times to check if swap memory usage is incrementing)
show system processes extensive | except 0.00 (Execute this in a gap of 5-10secs and run it atleast 4-5times to check if swap memory usage is incrementing)
For example :-
Currently the swap memory is not in use and can be confirmed by checking the show system virtual-memory output:-
I have cropped below logs from output of show system virtual-memory to check if the swap pages usage is incrementing, but from the below output we can confirm that it is not incrementing and hence not in use anymore.
LOGS:-
SWAP 2 141K - 2 64
25 swap pager pageins
25 swap pager pages paged in
310 swap pager pageouts
3559 swap pager pages paged out
3328 swap pages used
3472 peak swap pages used
Swap: 614M Total, 13M Used, 601M Free, 2% Inuse <<<<<<<<<<<<<<<<<<<<< 2% Inuse and has not incremented
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
10 root 155 52 0K 12K RUN 2133.7 54.79% idle
1807 root 43 -1 1255M 592M select 740.6H 16.02% fxpc
27055 root 50 0 65980K 52416K select 0:00 5.25% cli
27047 remote 41 0 66008K 52496K select 0:00 4.31% cli
2008 root 45 0 67688K 54016K select 173.5H 3.76% mib2d
1807 root -52 -52 1255M 592M select 740.6H 2.29% fxpc
27048 root 41 0 72224K 15720K select 0:00 2.15% mgd
27056 root 65 0 72060K 14464K select 0:00 1.40% mgd
2009 root 4 0 200M 95884K kqread 69.4H 1.27% rpd
2028 root 42 0 33088K 26100K select 71.7H 1.12% snmpd
27043 root 42 0 10560K 5792K select 0:00 1.10% sshd
Workaround/Solution:-
As per above example:- If you want to recover from 2% Inuse to 0% Inuse, reboot is the only option, if the swap memory is not in use it will still show the memory being utilized as it was being used at some point.
If you are wanting to recover the swap memory to 0% Inuse then reboot the device during Maintenance Window and monitor the device after reboot to check if the swap usage logs reappear.
Also, try not to open too many ssh sessions in parallel to avoid triggering of swap memory usage.