Description

  1. In certain cases there may be a need to stop Apstra from trying to push an already committed change to the network. This document describes:
  • A high level overview of Apstra's deployment process
  • The use-cases where a Pending deployment may need to be blocked
  • The method that can be applied to temporary stop the Pending deployment

Symptoms

Context:

When committing a Blueprint, Apstra will push the staged changes to the device(s). When this is successful the config change is immediately followed by the collection of the running configuration. This config is stored in Apstra and is called the Golden Config. It is used by Apstra's IBA modules to validate the device is running the correct config: When a device's running config is found to deviate from the golden config an anomaly is raised on the Dashboard, which points to a diff showing the exact differences.

Only once the golden config is collected is the Deployment considered successful. It wil go from 'Pending' state to 'Success' on the Dashboard. If golden can not be collected, the config remains in Pending state and Apstra will continue to try to complete the cycle.

Use Cases:

There are two known use cases:
  1. Connectivity to the Apstra VM transverses over the managed fabric. The Pending change breaks Apstra <> Device connectivity. Apstra configures the devices but the break means Golden can not be collected, so the cycle is not completed.
  2. The user has taken a backup, wants to make changes offline, and restore the backup with changes in production. However, he does NOT want to have those changes pushed immediately after restoring the modified backup. Or, he wants to control how and when devices are updated.
NOTE re. use-case 1: Management traffic MUST be Out of Band at all times. it is strongly recommended to alter / update the mgmt connectivity ASAP to ensure continued and stable network management. 

Solution

IMPORTANT: the solution means the cycle is halted. It is not broken. Meaning, the moment the 'halt' is removed Apstra will again push the changes to the devices. To completely remove a Pending deployment from Apstra complex maintenance is required. Please contact JTAC for support.

The solution involves a modification of the Linux routing table on the Apstra VM and varies slightly for each use-case:

1. Log on to the Apstra VM CLI

2. Add a null0 route for the device mgmt IP. This is required for each device where a config deployment needs to be stopped. Note: these are NOT persistent across a reboot

$ sudo ip route add blackhole <ip_address>

3. Confirm the blackhole route exists correctly in the kernel routing table:

$ admin@aos-server:~$ netstat -nr | grep <ip_address>

The output should have an asterisk, “*”, rather than an interface name. For example:


admin@aos-server:~$ netstat -nr
Kernel IP routing table
Destination    Gateway Genmask         Flags MSS Window irtt Iface
<ip_address>   0.0.0.0 255.255.255.255 UH.     0 0         0 *
admin@aos-server:~$
 

4. Depending on the use-case, either: 
        - Log on to the device and manually remove the offending config from CLI - OR:
        - Restore the previously updated backup with pending changes (see Apstra documentation).

Make sure ALL devices with pending changes have a corresponding blackhole route.

5. Apstra is not able to update the device. Any manual changes remain in place. At this point it is possible the pending change needs to be removed permanently: contact JTAC for support.

6. As required, the blackhole route can be removed on a per-device basis as follows:

admin@aos-server:~$ sudo ip route del blackhole <ip_address>


 

Modification History

24/08/2023 Initial version