Description

This article explains how a user can resolve the 'Site Edit' Job failure due to 'update_nat_policy'   task error .

  • User edit the site to change WAN link properties from active to backup.
  • Edit Site Job failed in below mentioned task.

Task: update_nat_policy_phase0

  •  Dec 4, 2023, 1:32:32 PMupdate_nat_policy_phase0 failed
  • Dec 4, 2023, 1:32:32 PMTask complete

Symptoms

Kibana logs has below pattern for Secmgt-SM microservice

Dec 4, 2023 @ 13:32:19.482 csp-secmgt-sm Unable to acquire JDBC Connection. Retrying. Retry count: 3 WARN -

Dec 4, 2023 @ 13:32:17.473 csp-secmgt-sm Unable to acquire JDBC Connection. Retrying. Retry count: 2 WARN -

Dec 4, 2023 @ 13:32:15.463 csp-secmgt-sm Unable to acquire JDBC Connection. Retrying. Retry count: 1 WARN -

Dec 4, 2023 @ 13:32:13.431 csp-secmgt-sm Unable to acquire JDBC Connection. Retrying. Retry count: 0 WARN -

 

Solution

The above mentioned symptoms related to failure with respect to AUTO NAT policy failure for the specific site. 

Here are the high level steps involved in  ‘AUTONAT POLICY’ workflow 

  • Convert objects in snapshot
  •  delete snapshot version in DB . 
  •  save the policy snapshot files to the secmgt microservice pod.
  •  save snapshot version in DB
  •  Deploy policy snapshot to the device.
  •  Delete the policy snapshot version from DB.
There are 2 Delete snapshot calls to the DB. In the failure case if for some reason the delete api call was not successful then mysql ( smDb → ServiceVersionMetaDataEntity ) will still have an entry for the respective AUTO NAT policy snapshot. So  the deploy thread will exit if it finds the entry in smDb → ServiceVersionMetaDataEntity table.

Following steps need to be implemented through API Client (Postman) to get around this problem. 
  • Based on error logs identify the AUTO NAT policy snapshot version as shown below 


        Dec 4, 2023 @ 13:32:13.398 csp-secmgt-sm save snapshot url is http://testcso.com:8080/api/juniper/sd/policy-management/nat/save-snapshot?service-id=5473070&version=0 

    In the above logs message 5473070 is the snapshot version. 

  • Through Postman API  client check if there is any stale entry in the DB for AUTO NAT policy snapshot for respective device using below api.

       Method:  GET

       URL : https://testcso.com/api/juniper/sd/policy-management/nat/policies/5473070/versions

      Output :

{
    "version-meta-datas": {
        "version-meta-data": [
            {
                "id": 8351376,
                "version": 0,
                "tooltip-string": "1",
                "service-moId": "x:5473070",
                "data-format-version": "v7",
                "created-by-user-name": "admin",
                "name": "1",
                "service-id": "5473070",
                "definition-type": "CUSTOM",
                "snapshot-version": 1,
                "service-type": "NAT",
                "service-version": "0",
                "created-time": 1696849828649,
                "created-time-in-millis": 1696849828649,
                "edit-version": 0,
                "predefined": false,
                "moId": "net.juniper.space.sd.versioning.jpa.VersionMetaDataEntity:8351376"
            }
        ],
        "uri": "/api/juniper/sd/policy-management/nat/policies/5473070/versions",
        "total": 1
    }
}

  • If there is entry then the delete the respective snapshot entry using below api. 

      Method: DELETE

   URL: https://adminui.prod.cso-portal.vodafone.com/api/juniper/sd/policy-management/nat/policies/5473070/serviceVersion/0
 

  • Now cross verify if the snapshot version is deleted successfully using below api.

         Method:  GET

       URL : https://testcso.com/api/juniper/sd/policy-management/nat/policies/5473070/versions

      Output: 
                  
{ "version-meta-datas": { "version-meta-data": [], "uri": "/api/juniper/sd/policy-management/nat/policies/5473070/versions", "total": 0 } }

 

 

  • Once the above workaround is implemented, user can retry the the last failed "AUTO NAT Policy' deployment for the respective site. 
  • Once the 'AUTO NAT' policy deployment successful then use can proceed with 'Edit Site' workflow again.  

Modification History

2023-12-14 : Article Created