The issue happened after upgrading from 20.4R3-S3.4 to 23.4R2-S3.9
Downstream clients were note able to ping each other across EVPN fabric until one linkin MC-LAG was disabled

DMAC programming in Broadcom on one switch
====================================================================================================
 
This is T2 encap in TD2+
 
--------------------------------------------------
First LEAF
--------------------------------------------------
 
EGR_VFI.epipe0[21]: <STG=0x1ff,OVID=0x3ed,MEMBERSHIP_PROFILE_PTR=1,FLEX_CTR_POOL_NUMBER=1,FLEX_CTR_OFFSET_MODE=1,FLEX_CTR_BASE_COUNTER_IDX=0x19,EN_EFILTER=1>
 
VFI is 21 ---> 0x15
 
The MAC is present in this VFI and pointing to VPLAG 0x3ffa
-----------------------------------------------------------
L2_ENTRY.ipipe0[54420]: <VPG_TYPE=1,VPG=0x3ffa,VLAN_ID=0x15,VIF:L2MC_PTR=0x3ffa,VFI=0x15,VALID=1,TGID=0x3fa,PORT_NUM=0x7a,OVID=0x15,MODULE_ID=0x7f,MAC_BLOCK_INDEX=1,MAC_ADDR=0x00000c07ac0a,L2MC_PTR=0x3ffa,L2:VPG_TYPE=1,L2:VPG=0x3ffa,L2:VLAN_ID=0x15,L2:VFI=0x15,L2:TGID=0x3fa,L2:PORT_NUM=0x7a,L2:MODULE_ID=0x7f,L2:MAC_BLOCK_INDEX=1,L2:MAC_ADDR=0x00000c07ac0a,L2:L3MC_PTR=0x3ffa,L2:L2MC_PTR=0x3ffa,L2:KEY=0x000000301eb0280153,L2:HASH_LSB=0xac0a,L2:DEST_TYPE=2,L2:DESTINATION=0x3ffa,L2:DATA=0x0000067ff4,L2:CLASS_ID=1,L2:ASSOCIATED_DATA=0x0000067ff4,KEY_TYPE=3,KEY=0x000000301eb0280153,IVID=0x28,HITDA=1,DEST_TYPE=2,DESTINATION=0x3ffa,DATA=0x0000067ff4,CLASS_ID=1,ASSOCIATED_DATA=0x0000067ff4>
 
 
    The VPLAG is present in PFE sw and is pointing to both remote switches
    ==========================================================
    lbnh = 2094 esi.2094 vplag_id = 1078 vplag_port_id = 0x3ffa
      VTEP additional tokens 1
      Additional VTEP token[0] 36
      child_nh = 1833 type = 1 dvp = 33 intf = vtep.32776(877) tunnel = 4c000009 port = 0
         uc_egress_if = 108224 dvp = 36 tunnel = 4c000009 port = 1
      VTEP additional tokens 2
      Additional VTEP token[0] 31
      Additional VTEP token[1] 39
      child_nh = 1807 type = 2 dvp = 24 intf = vtep.32773(874) tunnel = 4c000006
    dvp = 24 31 egress_if mis-match hw 0 != sw 108224  <<<<<<<<<<<<<<<--- SW/HW mis-match pointed in the output itself
         uc_egress_if = 108235 dvp = 39 tunnel = 4c000006 port = 2
    lbnh = 2094 vplag = 1078 total_dvp num of intf = 0
 
!!!!!! The VPLAG group is broken here: Count is 64 and the pointer is 4095, which is impassible
 
ING_DVP_TABLE.ipipe0[16378]: <NEXT_HOP_INDEX=6,ENABLE_VPLAG_RESOLUTION=1,ECMP_PTR=6,DVP_GROUP_PTR=6,>
 
EGR_VPLAG_GROUP.epipe0[6]: <DVP_LAG_ID=0x3ffa,COUNT=0x3f,BASE_PTR=0xfff>  <--- Wrong!
 
--------------------------------------------------
Let's check the same on another LEAF:
--------------------------------------------------
 
EGR_VFI.epipe0[21]: <STG=0x1ff,OVID=0x3ed,MEMBERSHIP_PROFILE_PTR=1,FLEX_CTR_POOL_NUMBER=1,FLEX_CTR_OFFSET_MODE=1,FLEX_CTR_BASE_COUNTER_IDX=0x19,EN_EFILTER=1>
VFI is also 21
 
Same programming for the DMAC:
------------------------------
 
L2_ENTRY.ipipe0[54420]: <VPG_TYPE=1,VPG=0x3ffa,VLAN_ID=0x15,VIF:L2MC_PTR=0x3ffa,VFI=0x15,VALID=1,TGID=0x3fa,PORT_NUM=0x7a,OVID=0x15,MODULE_ID=0x7f,MAC_BLOCK_INDEX=1,MAC_ADDR=0x00000c07ac0a,L2MC_PTR=0x3ffa,L2:VPG_TYPE=1,L2:VPG=0x3ffa,L2:VLAN_ID=0x15,L2:VFI=0x15,L2:TGID=0x3fa,L2:PORT_NUM=0x7a,L2:MODULE_ID=0x7f,L2:MAC_BLOCK_INDEX=1,L2:MAC_ADDR=0x00000c07ac0a,L2:L3MC_PTR=0x3ffa,L2:L2MC_PTR=0x3ffa,L2:KEY=0x000000301eb0280153,L2:HASH_LSB=0xac0a,L2:DEST_TYPE=2,L2:DESTINATION=0x3ffa,L2:DATA=0x0000067ff4,L2:CLASS_ID=1,L2:ASSOCIATED_DATA=0x0000067ff4,KEY_TYPE=3,KEY=0x000000301eb0280153,IVID=0x28,HITDA=1,DEST_TYPE=2,DESTINATION=0x3ffa,DATA=0x0000067ff4,CLASS_ID=1,ASSOCIATED_DATA=0x0000067ff4>
 
 
==========================================================
lbnh = 2094 esi.2094 vplag_id = 1078 vplag_port_id = 0x3ffa
  VTEP additional tokens 1
  Additional VTEP token[0] 35
  child_nh = 1803 type = 1 dvp = 26 intf = vtep.32774(875) tunnel = 4c000007 port = 0
     uc_egress_if = 108225 dvp = 35 tunnel = 4c000007 port = 1
  VTEP additional tokens 2
  Additional VTEP token[0] 34
  Additional VTEP token[1] 40
  child_nh = 1802 type = 2 dvp = 25 intf = vtep.32773(874) tunnel = 4c000006
dvp = 25 34 egress_if mis-match hw 0 != sw 108225  <<<<<<<<<--- Same mismatch is mentioned in virtual output
     uc_egress_if = 108237 dvp = 40 tunnel = 4c000006 port = 2
lbnh = 2094 vplag = 1078 total_dvp num of intf = 0
 
 
The same issue as on this LEAF
 
ING_DVP_2_TABLE.ipipe0[16378]: <NEXT_HOP_INDEX=6,ENABLE_VPLAG_RESOLUTION=1,ECMP_PTR=6,DVP_GROUP_PTR=6>
 
EGR_VPLAG_GROUP.epipe0[6]: <DVP_LAG_ID=0x3ffa,COUNT=0x3f,BASE_PTR=0xfff>  <--- Wrong!

For a normal IPC order/dependence, those venhs should come and be installed before lbnh. In issue case, the order was reversed and vplag install failed.

Resolved in - 23.4R2-S6