Customer has a PTX10003 whose ae0 was converted from 12x100GE to 4x400GE bundle-members. Traffic was gracefully migrated away via IGP costs prior to change to traverse router421:ae1/router411:ae1 cross-link. Once bundle converted to 400GE members and link restored traffic was gracefully migrated back to ae0. After that, the traffic got black-holed.

Traffic black-hole on PTX10003 after moving 100g links bundle to 400g links bundle

From the PTX,

op@router421> show platform dependency-state    

Dependency state summary :
  Application           Node    Context     Status
  <...> 
  ehmd                  re0     all         OK     
  evo-aftmand-zx        re0     all         ERROR  
  evo-cda-zx            re0     all         OK     
<...>
  msvcsd                re0     all         ERROR 

op@router421> show platform dependency-state detail                                                   
evo-aftmand-zx.all (re0) Root Cause Incomplete Object Info :
  GUID                  : 824657920949 <<<<<<<<<<<<<<<<<<<<<<<<<< GUID is the same 
  OLC                   : 26740123035158988
  Producer ID           : re0::rpdagent::50331966
  Class                 : first class object
  Type                  : net::juniper::rtnh::Nexthop
  Is del                : true
  Contents :
   af_fixup             :
   ifl_idx_fixup        :
   nh_bitflags          : HW indirection
   nh_family            : inet
   nh_id                : {index:41211}
   nh_type              : indirect
   pfetokens            :
   tbl_nh_info          :
   ucast_nh_info        :
  AMM details :
   Ref Counts
     Publish            : 0
     BQ                 : 2
     App-Controller     : 1
     Evl Tables         : 0
     Total              : 4
   Is Facade live       : true
   Relation count       : 0
   In-use in Publish    : false
   In-use in BQ         : false
  Potential Reason      : Delete is gated because it's dependent(guid: 824657920949) has not been deleted <<<<<<<<<<<<<<<<<<< GUID is the same 
  Potential Remedy      : request system application restart app rpdagent node re0

The root-cause is the GUID is the same as the GUID whose DEL is getting gated, i.e. it’s pointing to itself as dependent.

This can be identified as PR1752267:

A traffic blackhole is observed when the next-hop gets stuck in the pending-delete state waiting for the associated routes to get deleted, but never gets an opportunity to process the deletion further and remains stuck in the system due to the device does not communicate deletion of the associated routes.

This issue happens when a route (R1) is initially linked to a next-hop (Nh1) after an event triggers (for example: interface goes down) R1 to realign with a different next-hop (Nh2), the previous next-hop (Nh1) is deleted, but the delete is not notified, because the Route (R1) is still marked having link with next-hop (Nh1)