Description

Customer faced KMD Core on the device-

root@R1> show system core-dumps no-forwarding
/var/crash/*core*: No such file or directory
-rw-rw---- 1 root field 3113426 Jun 4 17:22 /var/tmp/kmd.core-tarball.0.tgz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
total files: 1


 

KMD Core happened at Jun 4 17:22. 

Solution

KMD

  • KMD stands for Key Management Daemon which is responsible for managing the cert keys for IPSec tunnels. 
  • This is the control plane daemon responsible for IPsec related operations.

 

KMD Core

  • KMD was trying to fetch a local certificate with ID 'R1-xxxxxxxxxxxx' which was not available with PKID.
  • KMD core is seen, when the local certificate keypair is missed repeatedly.
  • The kmd process was reporting missing keypair for local certificate, which may be due to wrong or missing certificate file on the device. As a result, the kmd process reinitiates the SA's frequently, and subsequently, kmd process crashed. 
  • It was trying to access a memory which is already being freed or overwritten. As a result KMD crash happened.
Jun 4 18:30:03 R1 pkid[2315]: PKID_NO_LOCAL_CERT: Local certficate R1-xxxxxxxxxxxx is not enrolled            <<<<<<<
Jun 4 18:30:03 R1 kmd[9080]: KMD_INTERNAL_ERROR: kmd_policy_get_cert_keypair_recv_ipc: KMD-PKID-IPC Invalid inputs buffer 0x0 policy 0x1e2d100
Jun 4 18:30:03 R1 pkid[2315]: PKID_NO_LOCAL_CERT: Local certficate R1-xxxxxxxxxxxx is not enrolled            <<<<<<<
Jun 4 18:30:03 R1 kmd[9080]: KMD_INTERNAL_ERROR: kmd_policy_get_cert_keypair_recv_ipc: KMD-PKID-IPC Invalid inputs buffer 0x0 policy 0x1e2d100
Jun 4 18:30:12 R1 pkid[2315]: PKID_NO_LOCAL_CERT: Local certficate R1-xxxxxxxxxxxx is not enrolled            <<<<<<<
Jun 4 18:30:12 R1 kmd[9080]: KMD_INTERNAL_ERROR: kmd_policy_get_cert_keypair_recv_ipc: KMD-PKID-IPC Invalid inputs buffer 0x0 policy 0x1e2d000
Jun 4 18:30:12 R1 pkid[2315]: PKID_NO_LOCAL_CERT: Local certficate R1-xxxxxxxxxxxx is not enrolled            <<<<<<<
Jun 4 18:30:12 R1 kmd[9080]: KMD_INTERNAL_ERROR: kmd_policy_get_cert_keypair_recv_ipc: KMD-PKID-IPC Invalid inputs buffer 0x0 policy 0x1e2d000

 

 

 

 

  • After this, customer corrected the local-certificate ID.
  • At Jun 5 00:15 customer changed it to R1-yyyyyyyyyy.
  • But they forgot to apply it in IPSec config.
  • Tunnel will not come up until its applied to the IPSec config.
  • At Jun 5 00:23 the IPSec config was modified and committed. After that the Tunnel came up.

 

Jun 5 00:15:54 R1 mgd[12254]: UI_CMDLINE_READ_LINE: User 'abc', command 'set services ipsec-vpn ike policy R1 local-certificate R1-yyyyyyyyyy '            <<<<<<<
Jun 5 00:20:07 R1 mgd[12254]: UI_CMDLINE_READ_LINE: User 'abc', command 'commit comment "BGP sessions down. cert change" and-quit '
Jun 5 00:20:07 R1 mgd[12254]: UI_COMMIT: User 'abc' requested 'commit' operation (comment: BGP sessions down. cert change)
Jun 5 00:23:31 R1 mgd[12254]: UI_COMMIT_PROGRESS: Commit operation in progress: commit complete
Jun 5 00:23:31 R1 mgd[12254]: UI_COMMIT_COMPLETED: : commit complete
Jun 5 00:23:33 R1 kmd[9080]: KMD_PM_SA_ESTABLISHED: Tunnel cfg name: __ss.term-R1__, Local gateway: 10.x.x.x, Remote gateway: 10.x.x.x, Local ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Remote ID: ipv4_subnet(any:0,[0..7]=0.0.0.0/0), Direction: inbound, SPI: 0xfe776f8, AUX-SPI: 0, Mode: Tunnel, Type: dynamic ,SA op: Added

 

 

Steps to avoid KMD Core

In future you may use the following commands to check the validity of the certificates and keys.
 

How to check Cert IDs validity

root@test_router> show security pki local-certificate
LSYS: root-logical-system
Certificate identifier: localXXXXXXXXX           <<<<<<<<<<<<<<<<<
Issued to: ASC-0, Issued by: CN = AS-ROOT
Validity:
Not before: 04-15-2023 18:28 01C            <<<<<<<<<<<<<<<<<
Not after: 04-16-2033 18:28 UTC            <<<<<<<<<<<<<<<<<
Public key algorithm: IsaEncryption (1024 bits)
Keypair Location: Keypair generated locally
LSYS: root-logical-system
Certificate identifier: localXXXXXXXX
Issued to: DUT, Issued by: C - RO, L - Bucharest, O - Ixia, 00 = IxLoad, CN = RootCA
Validity:
Not before: 09-13-2023 17:26 UTC            <<<<<<<<<<<<<<<<<
Not after: 09-13-2024 17:26 UTC            <<<<<<<<<<<<<<<<<
Public key algorithn: ecdsaincryption (384 bits)
Keypair Location: Keypair generated locally

 

 

 

 

 

root@test_router> show security pki ca-certificate
LSYS: root-logical-system
CA profile: XXXXXXXXXX
Certificate identifier: XXXXXXXXXX                     <<<<<<<<<<<
Issued to: AS-ROOT, Issued by: CN = AS-ROOT
Validity:
Not before: 04-15-2023 18:28 UTC                     <<<<<<<<<<<
Not after: 04-16-2033 18:28 UTC                     <<<<<<<<<<<
Public key algorithm: IsaEncryption (1024 bits)
Keypair Location: Keypair generated locally
ISYS: root-logical-system
CA profile: ecdsa384
Certificate identifier: XXXXXXXXXX
Issued to: RootCA, Issued by: C." RO, L " Bucharest, 0 = Ixia, OU = IxLoad, CN = RootCA
Validity:
Not before: 09-13-2023 17:26 UTC                     <<<<<<<<<<<
Not after: 09-13-2024 17:26 UTC                     <<<<<<<<<<<
Public key algorithm: ecdsaEncryption (384 bits)
Keypair Location: Keypair generated locally

 

 

 

root@test_router> request security pki local-certificate verify certificate-id XXXXXXXXXX
local certificate XXXXXXXXXX verification success

 

Modification History

2024-09-13 : Article Created