Whenever we upgrade multiple T1 layer device in a Data center enviornment, we may face issue with traffic blackholing for some time due to delay in ECMP re-calculation. This KB explains the best practice to follow during such upgrades.

Delay in ECMP re-calculation, resulting traffic blackholing.

Draining multiple nodes in multiple blocks can create multiple failure scenarios in terms of traffic convergence.

It is not advised to drain devices in multiple blocks around the same time.

We also discussed adding a delay in each drain operation can help the rest of the devices to finish the ECMP re-calculation without churn.

General recommendation is to give at least 60 seconds, however exact number can be determined by looking at evo-pfemand traces on all devices that are connected a device undergoing drain.