Description

Highway process can get stuck on startup due to DNS related race conditions for IPSec tunnels. 

The IPSec client plugin (3.6.1) attempted to correct a run time race condition between IPSec tunnel starting before DNS was fully operational in the IPSec namespace. Two changes were made to resolve this issue:

1. A verification step was added to ensure successful DNS resolution before starting IPsec tunnel.
2. A watch dog service was added to monitor IPSec tunnel services and restart them every 30 seconds if not.

Release Notes (IPSec Plugin Release Notes)

Resolved the startup race condition by enforcing stricter default timeouts for DNS operations.

Severity

Major

Status

Resolved

Resolved In

IPSEC Client Plugin Release 3.6.2

Product

SSR

Functional Area

IPSec Client Plugin

Symptoms

During the initial boot process of a router with the IPSec plugin enabled, the WAN interfaces can take some time to fully come up, creating two interactions with the IPSec controller:

  • Each call to verify DNS resolution can take up to 6 mins 40 seconds to timeout since the network is unreachable.

During normal operation this verification process ranges from a few hundred milliseconds to a few seconds.

  • The watchdog, in an attempt to start the failed IPsec tunnel service, initiates another verify DNS resolution API call.

Due to the delayed initialization of the WAN interfaces, the IPSec controller would get backed up with health-check tasks. This backlog prevented the forwarding plane from initializing the device interfaces. In the instance of the reproduction, there was a backlog of 9 hours before the API call from the forwarding plane could be processed. This was verified by letting the reproduction system stay in the stuck state for more than 9 hours. The device interfaces eventually updated the operational status of the Broadband interfaces.

Solution

Disable IPSec client

While the issue does not happen in every environment, one option is to temporarily disable the IPSec plugin for planned reboot of devices.

Monitor and Restart the IPSec controller

Once the system is in the problem state, the following steps can be performed from the linux shell to unblock the highway process:

  1. Run: systemctl stop ipsec-controller
  2. Wait for the interfaces and peers over WAN interfaces to come up.
  3. Restart the IPSec controller and other necessary services. Run: touch /var/lib/128technology/plugins/ipsec/config.json  

 

Modification History

Imported from public SSR documentation 4/1/25.