Commit job IDs are stuck between 6760 and 6761:
Mar 15 15:31:04 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6760,job-status: Successfully committed 6760,job-comment: ,schedule-name:
Mar 15 15:31:04 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6761,job-status: Successfully committed 6761,job-comment: ,schedule-name:
Mar 15 15:32:04 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6760,job-status: Successfully committed 6760,job-comment: ,schedule-name:
Mar 15 15:32:04 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6761,job-status: Successfully committed 6761,job-comment: ,schedule-name:
Mar 15 15:32:54 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6760,job-status: Successfully committed 6760,job-comment: ,schedule-name:
Mar 15 15:32:54 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6761,job-status: Successfully committed 6761,job-comment: ,schedule-name:
user@mx960> show system commit server queue job-status all | no-more
Pending commits:
Id: 6761
Last Modified: Mon Mar 16 21:49:15 2026
Status: Waiting in queue
...
{master}
user@mx960> show system commit server queue id 6760
Completed commits:
Id: 6760
Last Modified: Mon Mar 16 21:50:05 2026
Status: Successfully loaded file 6760 <==not completed
user@mx960> show system commit server queue id 6761
Last Modified: Mon Mar 16 21:49:11 2026
Status: Successfully loaded file 6761 <==not completed
user@mx960>
console@mx960> configure batch
warning: uncommitted changes will be discarded on exit
Entering configuration mode
{master}[edit]
console@mx960# commit
Added to commit queue request-id: 6761 <<<==this job-id 6761 or 6760 always stuck here
re1:
configuration check succeeds
re0:
commit complete
console@mx960#
user@mx960> file show /var/db/commit-queue/.job_id
6761
user@mx960> f
delete files under /var/db/commit-queue/ no help.
From the logs, we observe a continuous RCP failure on the new master RE (RE0) attempting to pull commit-queue contents from RE1.
This RCP is part of the commit-queue GRES synchronization (commitd -s) and only occurs when a GRES switchover has taken place,
so it is possible that a GRES event triggered the issue.
Mar 15 15:30:07 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:31:07 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:32:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:33:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:33:54 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6762,job-status: Successfully committed 6762,job-comment: ,schedule-name:
Mar 15 15:33:54 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6763,job-status: Successfully committed 6763,job-comment: ,schedule-name:
Mar 15 15:34:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:34:58 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6760,job-status: Successfully committed 6760,job-comment: ,schedule-name:
Mar 15 15:34:58 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6761,job-status: Successfully committed 6761,job-comment: ,schedule-name:
Mar 15 15:35:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:36:03 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6760,job-status: Successfully committed 6760,job-comment: ,schedule-name:
Mar 15 15:36:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:36:53 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6760,job-status: Successfully committed 6760,job-comment: ,schedule-name:
Mar 15 15:37:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
Mar 15 15:37:35 mx960 commitd: UI_COMMIT_BATCH_COMPLETED: job-id: 6761,job-status: Successfully committed 6761,job-comment: ,schedule-name:
Mar 15 15:38:08 mx960 commitd: LIBJNX_REPLICATE_RCP_ERROR: rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue :
The RCP command "rcp -r -T re1:/var/db/commit-queue/ /var/db/commit-queue : " copies the entire commit-queue directory from RE1 to RE0, including the .job_id file.
The RCP was in progress, and .job_id (containing value 6761) was copied from RE1 to RE0, but the RCP ultimately times out and is reported as a failure.
Since the RCP is considered failed, the cleanup step (deletion of commit-queue contents on RE0) never runs, and RE0 retains its .job_id = 6761.
Meanwhile on RE0, new batch commits increment the local .job_id (6761 -> 6762), but when jlaunchd respawns commitd -s ~60 seconds later,
the RCP again fails but copies .job_id = 6761 from RE1, overwriting RE0's locally incremented value.
This creates the cyclic pattern we observe in the logs.
Identifying the root cause of the RCP failure will be key to addressing this issue permanently.