In rare case scenario if  upgrade of PHUB (Provider HUB) is initiated through CSO and PHUB failed to boot post upgrade then PHUB need to be recovered through USB install. Once PHUB is restored and loaded with previous backed up configuration, PKI certificates need to be updated manually in order to establish IPSEC tunnels with spoke sites. This article explains how to manually load PKI certificates on PHUB device.

• Customer using CSO SaaS 6.3 version.
• PHUB  device upgrade initiated from 20.4X15.1 to 22.2R3-S4.11 junos version through CSO.
• PHUB device failed to boot after issuing upgrade command on the device.  
• PHUB device restored through USB installation and PHUB loaded with backed up running configuration.  
• All IPSEC tunnels between the spoke sites of respective tenants and PHUB went down due to lack of right PKI certificates.

Below mentioned steps need to be executed to  load PKI certificates manually on PHUB device. 

• Ensure PHUB is loaded to the last backed up running configuration.Login to CSO portal and  Navigate to Resources -> Provider Hub Devices -> PHUB device and select the configuration tab. Here select the running configuration tab to extract last configuration synced between CSO and the device.
• Once the configuration is committed successfully ensure PHUB device can establish outbound ssh connection with CSO server.
• Navigate to CSO tenant -> Administration > Certificate Management > VPN Authentication and select the PHUB device and  select renew certificate. This will create  key-pair and local certificate on the PHUB device.
• Engage Juniper SaaS Operations team to fetch the CA certificate chain from the PKI server for particular tenant.

              Note:  In case multi tier PKI used then complete CA-certificate chain is  required.

•    Juniper operations team   who will have access to CSO backend infrastructure , can  execute below command from tssm-core microservice to fetch CA certificate for respective tenant. 
• root@csp:/usr/local/lib/python2.7/dist-packages/tssm_services/script# sscep_static getca -u http://PKISERVERIPADDRESS:8030/csrscep/sdwan -c /tmp/ca.crt -vvvvvv

     << Sample Output >>

scep_static: starting sscep, version 0.6.1
sscep_static: new transaction
sscep_static: transaction id: SSCEP transactionId
sscep_static: hostname: 88.82.4.98
sscep_static: directory: csrscep/sdwan
sscep_static: port: 8030
sscep_static: SCEP_OPERATION_GETCA
sscep_static: server returned status code 200
sscep_static: MIME header: application/x-x509-ca-ra-cert
sscep_static: valid response from server

sscep_static: found certificate with
  subject: /C=GB/O=CFTSJTAC Limited/CN=CFTSJTAC Mobile Issuing Authority/CN=CSRES RA (AW)
  issuer: /C=GB/O=CFTSJTAC Limited/CN=CFTSJTAC Mobile Issuing Authority
  usage: Key Encipherment
  MD5 fingerprint: 71:72:3D:93:98:00:DA:F4:9C:CE:4F:19:B3:9E:9A:5F
sscep_static: certificate written as /tmp/ca.crt-0

sscep_static: found certificate with
  subject: /C=GB/O=CFTSJTAC Limited/CN=CFTSJTAC Mobile Issuing Authority/CN=CSRES RA (AW)
  issuer: /C=GB/O=CFTSJTAC Limited/CN=CFTSJTAC Mobile Issuing Authority
  usage: Digital Signature
  MD5 fingerprint: B7:8C:D1:2F:1C:CA:A9:42:C1:E6:1D:1D:51:49:0C:5F
sscep_static: certificate written as /tmp/ca.crt-1

sscep_static: found certificate with
  subject: /C=GB/O=CFTSJTAC  Limited/CN=CFTSJTAC Mobile Issuing Authority
  issuer: /C=GB/O=CFTSJTAC  Limited/CN=CFTSJTAC Root CA
  usage: Certificate Sign, CRL Sign
  MD5 fingerprint: 08:F3:95:4F:3C:39:1C:8A:A5:7E:84:5C:7F:AB:A4:F3
sscep_static: certificate written as /tmp/ca.crt-2. ==========================> Intermediate certificate

sscep_static: found certificate with
  subject: /C=GB/O=CFTSJTAC  Limited/CN=CFTSJTAC  Root CA
  issuer: /C=GB/O=CFTSJTAC  Limited/CN=CFTSJTAC  Root CA
  usage: Certificate Sign, CRL Sign
  MD5 fingerprint: 88:7F:A4:30:83:F5:43:9F:68:41:AB:8D:AA:0D:70:72
sscep_static: certificate written as /tmp/ca.crt-3.  ============================ > Root Certificate

In the above sample output multi tier CA certificates are used where there is root certificate ( ca.crt-3) and intermediate certificate ( ca.crt-2) . The other entries ( ca.crt-0 and ca.crt.1) corresponding to RA ( registration authority) can be ignored.

• Generate MD5 for each ca certificate which will be used as ca profile name in the device configuration.

For example :

 cat ca.crt-3 | md5

4f53f3e5f0329cb03f05295adb064f1a

 cat ca.crt-2 | md5

848c9acb109829533bb022683cf2eacc

•  As PHUB already loaded with last backed up running configuration , ideally ca profile configuration would have been already present matching to md5 checksum as shown below.

pki {

                ca-profile 848c9acb109829533bb022683cf2eacc{

                    ca-identity 848c9acb109829533bb022683cf2eacc;

                    revocation-check {

                        crl {

                            disable on-download-failure;

                        }

                    }

                }

                ca-profile 4f53f3e5f0329cb03f05295adb064f1a{

                    ca-identity 4f53f3e5f0329cb03f05295adb064f1a;

                    revocation-check {

                        disable;

                    }

                }

Note :  If ca profile configuration is missing, then add the respective configuration and commit the same.

• Copy the root and intermediate  CA certificates to the PHUB device in /var/tmp path.

root@default-project:~ # ls -ltr /var/tmp/ | grep *.crt

-rw-r--r--  1 root  wheel       0 Nov 1 16:49 ca-root.crt

-rw-r--r--  1 root  wheel       0 Nov 1 16:49 ca-interim.crt

 Note : Certificates file names are updated with proper names for better readability .

• Load the root and intermediate CA certificates on the PHUB device using below commands

request security pki ca-certificate load ca-profile 4f53f3e5f0329cb03f05295adb064f1a filename /var/tmp/ca-root.crt

request security pki ca-certificate load ca-profile 848c9acb109829533bb022683cf2eaccfilename var/tmp/ca-interim.crt

• Verify the CA certificate details using below commands in the cli. 

show security pki ca-certificate detail | no-more 

show security pki local-certificate | no-more

• Finally Verify the IPSEC tunnels comes up between Spoke sites and PHUB device. 

Draft -> Publish