We noticed mastership switchover from master to backup routing engine while creating vmhost snapshot.

Commands used for creating vmhost snapshot:

request vmhost snapshot routing-engine both

request vmhost snapshot routing-engine 

From interactive commands, we identified the multiple attempts for "request-vmhost-snapshot" in a short time interval.

"request vmhost snapshot routing-engine both" command was used.

Interactive commands logs:

Nov 25 04:32:26 2024 re0.LabRouter mgd[50956]: UI_CMDLINE_READ_LINE: User 'P3073170', command 'request vmhost snapshot routing-engine both '

Nov 25 04:32:26 2024 re0.LabRouter mgd[51029]: UI_CMDLINE_READ_LINE: User 'root', command 'command rpc rpc command request vmhost snapshot scc-dont-forward '

Nov 25 04:32:26 2024 re0.LabRouter mgd[51029]: UI_JUNOSCRIPT_CMD: User 'root' used JUNOScript client to run command 'request-vmhost-snapshot'

Nov 25 04:42:50 2024 re0.LabRouter mgd[53446]: UI_CMDLINE_READ_LINE: User 'root', command 'command rpc rpc command request vmhost snapshot scc-dont-forward '

Nov 25 04:42:50 2024 re0.LabRouter mgd[53446]: UI_JUNOSCRIPT_CMD: User 'root' used JUNOScript client to run command 'request-vmhost-snapshot'

At the same time we see that mastership was failed over to backup routing engine.

The reason for mastership switchover shows as "keepalive timer expired"

Log Messages :

Nov 25 04:32:43 2024 re0.LabRouter kernel: mastership: Lost mastership, time since last refresh: 11

Nov 25 04:32:43 2024 re0.LabRouter kernel: mastership: routing engine 0 relinquishing as master: keepalive timer expired

Nov 25 04:32:43 2024 re0.LabRouter kernel: vksid: 0 VKS old/cur: 1/0 RE old/cur: 1/0

Nov 25 04:32:43 2024 re0.LabRouter kernel: vksid: 1 VKS old/cur: 0/0 RE old/cur: 1/0

Nov 25 04:32:43 2024 re0.LabRouter kernel: jlock (R) hog reported for 10712 msecs: time_uptime (acq=75335235, rel=75335246), ticks (acq=172712244, rel=172722956), net_jlock_uptime (acq=75335040, rel=75335040) thread 0xfffff800178a2000 (ppt_11_80000017): CPU 2, td_lastruntime 172722955, ticks 172722956: offending fn peer_master_work() (../../../../../../../../../src/junos/bsd/sys/netpfe/pfeman_peer.c, line 5491): { 0xffffffff817efc25 0xffffffff817f09c0 0xffffffff81bf3714 0xffffffff8040cf81 0xffffffff8056ea4e }. nvcsw 0, nivcsw 1.

Chassisd logs:

Nov 25 04:33:21 CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Online (jnxFruContentsIndex 9, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName Routing Engine 1, jnxFruType 6, jnxFruSlot 1)

Nov 25 04:33:21 notify_fru_power_on: NULL kvpairs for Routing Engine 1

Please initiate vmhost snapshot for one RE at a time and wait for the execution of command before attempting it for another routing engine.