This article depicts the process of troubleshooting of GRES enabled routers.

As an example, let’s consider a scenario in which the entire configuration is in place and the kernel replication states look fine, but during a lab RE switchover test, you observe traffic loss. Because GRES is enabled, no packets should be dropped. This KB describes how to analyze the situation and how to check what is happening.

To understand more about GRES and kernel replication, you may refer the KB KB81548 [juniper.net] .
https://juniper.lightning.force.com/lightning/s/article/MXPTX-How-Graceful-Routing-Engine-Switchover-works-and-Kernel-Replication

Steps to Troubleshoot:

(1) Check if both the RE are having the same Junos version.

(2) If yes, then verify the configuration.

[edit]
lab@r1# set chassis redundancy graceful-switchover

[edit]
lab@r1# set chassis redundancy failover on-disk-failure

[edit]
lab@r1# set chassis redundancy failover on-loss-of-keepalives

[edit]
lab@r1# set chassis redundancy routing-engine 0 master

[edit]
lab@r1# set chassis redundancy routing-engine 1 backup

[edit]
lab@r1# show chassis redundancy
routing-engine 0 master;
routing-engine 1 backup;
failover {
   on-loss-of-keepalives;
   on-disk-failure;
}
graceful-switchover;

(3) Verify the status of GRES by executing the show system switchover command from the backup RE:

{backup}
lab@r1> show system switchover
Graceful switchover: On
Configuration database: Ready
Kernel database: Version incompatible
Peer state: Out of transition

(4) Enabling traceoptions on all the protocols and GRES knobs helps us get to the root of the problem:

lab@r1# show protocols ospf
traceoptions {
   file ospf.trace;
   flag all detail;
}

[edit]
lab@r1# show protocols bgp
traceoptions {
   file bgp.trace;
   flag all detail;
}

[edit]
lab@r1# show chassis
redundancy {
   routing-engine 0 master;
   routing-engine 1 backup;
   failover {
       on-loss-of-keepalives;
       on-disk-failure;
   }
   graceful-switchover {
       traceoptions {
           flag all;
       }
       enable;
   }
}

[edit]
lab@r1# show routing-options
static {
   route 66.129.243.0/24 next-hop 172.18.66.1;
}
forwarding-table {
   traceoptions {
       flag all detail;
   }
}

Note: It is not recommended to have flag all turned on under any protocol traceoptions in the production network. Logging all protocol events takes a toll on RPD processing, which could jeopardize real-time protocol managements such as BGP keepalives or OSPF Hellos.

(5) Then check the traceoptions output with the show log command. If you see from the logs that the protocols are working correctly and that it was not a protocol error that caused the issue, the next step is to debug the actual GRES communication.

(6) When you enable traceoptions in the GRES portion of the chassis redundancy configuration and on the forwarding table, JUNOS software begins placing debugging information about redundancy and the forwarding engine into log files, which you can then review.

-> Because ksyncd is the daemon responsible for kernel route replication, its logs usually reveal a lot of information about state, potential errors, misconfigurations, and software bugs. Check the logs with the following command:

lab@r1> show log ksyncd

Sep 15 21:57:52 KSYNCD release 9.1R2.10 built by builder on 2008-07-01
05:06:40 UTC starting, pid 4566
Sep 15 21:57:52 Not runnable attempt 0 reason: undefined mode (errors: none)
Sep 15 21:58:01 Terminated: 15 signal received, posting signal
Sep 15 21:58:01 inspecting pending signals
Sep 15 21:58:01 SIGTERM posted, exiting
Sep 18 02:17:16 KSYNCD release 9.1R2.10 built by builder on 2008-07-01 05:06:40
UTC starting, pid 8998
Sep 18 02:17:16 Commit sync knob is NOT configured on master RE			<<<<<<<<<<<<<
Sep 18 02:17:16 Stop attempting to perform initial sync
Sep 18 02:17:16 config state: ready

In above, we see that the reason for GRES not working properly was because the master RE wasn't configured with Commit Sync.

Another possible issue is actually the KA (Keep Alive) packets are dropped or missing between Master and Backup RE, leading to GRES sync issue. This error can be found by checking for "Loss of Communications" lines in mastership or syslogs.

Similarly, you may find other reasons like version mismatch etc.