This issue seen with WAN Assurance MIST HUB and Spoke Cert based IPSEC tunnel deployment and the tunnel authentication during node failover and the new Primary Node with old certification key information stored in the memory of the new Primary Node.

The following logs and symptoms are seen from the box 

From the Spoke side RSI counters

user@spoke-firewall> show security ipsec inactive-tunnels no-forwarding

  Total inactive tunnels: 6
  Total inactive tunnels with establish immediately: 6
  ID           Port   Gateway          Pending SAs   Tunnel Down Reason
  131079       4500   [A.B.C.D]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (100 times)
  131080       4500   [A.B.C.D]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (100 times)
  131081       4500   [V.X.Y.Z]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (45 times)
  131082       4500   [V.X.Y.Z]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (45 times)
  131082       4500   [E.F.G.H]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (45 times)
  131082       4500   [E.F.G.H]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (45 times)
  131083       4500   [I.J.K.L]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (100 times)
  131084       4500   [I.J.K.L]        1             Negotiation failed with error code AUTHENTICATION_FAILED received from peer (104 times)
  

From the Spoke message side:
  
kmd[4634]: IKE negotiation failed with error: Authentication failed. IKE Version: 2, VPN: [vpn-name] Gateway: [Gateway-name], Local: [VPN LOCAL IP-SPOKE]/4500, Remote:[REMOTE-IP-HUB]/4500, Local IKE-ID: Not-Available, Remote IKE-ID: Not-Available, VR-ID: 11: Role: Initiator

From the Hub Side:

kmd[11196]: IKE negotiation failed with error: Authentication failed. IKE Version: 2, VPN: [Hub-VPN-NAME] Gateway: [Hub-Gateway-Name], Local:[REMOTE-IP-HUB]/4500, Remote: [VPN LOCAL IP-SPOKE]/38086, Local IKE-ID: Not-Available, Remote IKE-ID: Not-Available, VR-ID: 5: Role: Responder

The cert related file checksums are all good between Primary and Secondary Node side and the issue is with the Memory side of the PKID process

RCA:

If PKID running on backup node is busy when the key file delete and new key file addition happens on that node, then PKID gets kqueue event after new key file addition is done. Because of this updating new key to PKID memory fails as per current design. This leads to mismatch of cert & key-pair in memory thereby causing tunnel establishment failure.

The following steps can be considered as mitigation steps

1.Failover RG0 back to the working Node

2.Restarting the pki services on both nodes will clear the memory and this step can be considered as proactive steps as well 

Fix:

This bug is tracked on PR1839090 

+ Adding product list 

RCA:
Following are the sequence of HA sync operations received on backup for certificate renewal:
1. Old certificate file deletion
2. Old key-pair file deletion
3. New key-pair file addition
4. New certificate file addition.
 
On backup node the kqueue service detects each of these operations and informs PKID to update accordingly in memory. In this particular customer scenario, the issue of mismatching cert&key-pair in memory(but matching in file-system and are same as the ones in active node) is possible either because of backup node not receiving operation 1/2 because of HA sync failure.

DEV team works to make kqueue event handling more robust in PKID so that it will work as expected even if HA sync fails for some events.