Customer reported an issue as below.

We have recently migrated the subscribers to a new backhaul provider, but some subscribers are not coming online on the new CVC.

We are learning the Mac on the CORE switch, but subscribers are not coming online on BNG.

Also, it comes online when we move the same subscriber back to the old CVC.

some customer not able to login with the new SVLAN number, but can be back online with the old SVLAN number. show interface-set can display the old interface-set stuck there.

This type of problem is caused by the stale ACI-based(Agent-Circuit-Identifier) interface-set is stuck in the system. RCA is unknown yet, the investigation is still on going.
At this stage, there has no way to clear those stuck interface-set, customer has to reboot the router or restart bbe-smgd daemon to clear problem.


Below are some steps for troubleshooting this issue.


The new (svlan,cvlan) = (3665, 2376), the old one = (1694,2376).


op@BNG> show subscribers agent-circuit-identifier AVC000189718314
Total subscribers: 0, Active Subscribers: 0


In smgd traceoptions:
Jun 20 12:37:05.964696 PppoeBbePlugin::bbeFlowNotifyFilterHandler: demux0.3221267679 Created iflIndex = 536914907 bbeIndex = 43995 changes = 0x12
 
Jun 20 12:37:06.121251 bbe_autoconf_packet_received: lookup string = , len = 161 vif_status = 1
Jun 20 12:37:06.121284 bbe_autoconf_parse_packet_for_aci: DHCPv4 aci string: 41 56 43 30 30 30 31 38 39 37 31 38 33 31 34 ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jun 20 12:37:06.121294 bbe_get_aci_lookup_string: configured aci-vlan options  aci_plus_ari = 0  aci = 1 , ari = 0  accept_no_ids = 0
Jun 20 12:37:06.121302 bbe_get_aci_lookup_string: ACI set lookup AVC000189718314 
<<<<<<<< ACI string is correct
Jun 20 12:37:06.121308 bbe_autoconf_create_aci_vlan: configured acR961i-vlan options  aci_plus_ari = 0  aci = 1 , ari = 0  accept_no_ids = 0
Jun 20 12:37:06.121320 bbe_get_aci_lookup_string: configured aci-vlan options  aci_plus_ari = 0  aci = 1 , ari = 0  accept_no_ids = 0
Jun 20 12:37:06.121356 bbe_get_aci_lookup_string: configured aci-vlan options  aci_plus_ari = 0  aci is NULL, ari is NULL  accept_no_ids = 0
Jun 20 12:37:06.121363 bbe_autoconf_vlan_create_failure No lookup string
Jun 20 12:37:06.121371 bbe_ifd_vif_get_next: ifd ae4 vtag 0x02e51bb7, aci-string : freeing vpkt: ifl = 0x7507010, packet = 0x60da62400
Jun 20 12:37:06.121379 bbe_ifd_vif_get_next: ifd ae4 vtag 0x02e51bb7, aci-string , is_aci_over_static_ifl false: done with vif, freeing object and returning NULL
Jun 20 12:37:06.121391 bbe_autoconf_vlan_create_failure : Garbage collect aci uifl, index=536914907
Jun 20 12:37:06.121401 bbe_autoconf_sdb_update_client_session_state: updating session 28520970 state to 3
<<<<<<<< creation failed




Then we found the old interface-set and the d-vlan interface still there
% cli -c "show interfaces interface-set" | grep -B5 AVC000189718314
      Max Sessions: 32000, Max Sessions VSA Ignore: Off
Interface set: aci-15460-demux0.3221239418
  Interface set index: 536885370
  Interface set snmp index: 603994234
    ACI VLAN:
      Agent Circuit ID: AVC000189718314
%


{master}
op@BNG> show subscribers interface demux0.3221239418
Jun 25 12:33:25
Interface                       IP Address/VLAN ID                      User Name                      LS:RI
demux0.3221239418               0x8100.1694 0x8100.2376                                           default:default


{master}
op@BNG> show subscribers interface demux0.3221239418 extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221239418
Interface type: Dynamic
Underlying Interface: ae4
Dynamic Profile Name: AUTO-STACKED-VLAN-DEMUX
Dynamic Profile Version: 1
State: Active
Session ID: 19210
PFE Flow ID: 14014
Stacked VLAN Id: 0x8100.1694
VLAN Id: 0x8100.2376
Login Time: 2024-06-04 00:34:27 AEST <<<<<<<< as can be seen this d-vlan interface was created weeks ago.




We tried all kinds of way to remove them, but failed.


{master}
op@BNG> clear auto-configuration interfaces demux0.3221239418
VLAN demux0.3221239418 has 1 iflsets


{master}
op@BNG> clear auto-configuration interfaces demux0.3221239418
VLAN demux0.3221239418 has 1 iflsets
<<<<<< demux0.3221239418 still there


op@BNG> clear auto-configuration interfaces interface-set aci-15460-demux0.3221239418
Interface-set aci-15460-demux0.3221239418 deleted
<<<<<<<<< Although it shows ifl-set is deleted, but as can be seen below, it is still there


{master}
op@BNG> show subscribers interface demux0.3221239418 extensive
Type: VLAN
Logical System: default
Routing Instance: default
Interface: demux0.3221239418
Interface type: Dynamic
Underlying Interface: ae4
Dynamic Profile Name: AUTO-STACKED-VLAN-DEMUX
Dynamic Profile Version: 1
State: Active
Session ID: 19210
PFE Flow ID: 14014
Stacked VLAN Id: 0x8100.1694
VLAN Id: 0x8100.2376
Login Time: 2024-06-04 00:34:27 AEST






{master}
op@BNG> request system subscriber-management release-session id 19210
Session will be forcibly released.
Do you want to continue ? [yes,no] (no) yes
Jun 25 12:54:42


Dependent sessions (1) present, Request release of all dependent sessions first