After interface configuration change, acx7100 stops forwarding traffic with error message "Failed to program the hardware with an error - table is full "

Error messages related to TPID programming are caught:

May 32 14:31:47 2325 router evo-pfemand[15394]: EVO_PFEMAND_ASIC_PROGRAM_FAILED: Failed to program the hardware with an error - table is full

May 32 14:31:47 2325 router  evo-pfemand[15394]: [t:15394] [Error] compName = "Broadcom", tpName = "BrcmErrorMsg", msg = " Failed to program the hardware with an error - = table is full, during = jbcm_port_tpid_class_set"

May 32 14:31:47 2325 router  evo-pfemand[15394]: [t:15394] [Error] BrcmPlusIf: BrcmDnxForwardingPort::tpidClassSetbcm_port_tpid_class_set failed for ifd: et-0/0/3 IfdIndex :1286 port:1811939332 tpid1:0xfffffffe tpid2:0xfffffffe tag_format:8 flags:0x12Table full ret: -6 unit:0

May 32 14:31:47 2325 router evo-pfemand[15394]: [t:15394] [Error] BrcmPlusIf: BrcmDnxForwardingPort::initForwardingInfoFailed to set tpid for port 1811939332 IfdIdx:1286 IfdName:et-0/0/3 rv-6

Related configuration:

           show configuration interfaces et-0/0/3 | display inheritance no-comments

flexible-vlan-tagging;

native-vlan-id 99;

mtu 9172;

encapsulation flexible-ethernet-services;

ether-options {

ethernet-switch-profile {

tag-protocol-id [ 0x88a8 0x8100 ];

}

}

unit 10 {

vlan-id 200;

family inet {

address 10.10.10.1/24;

}

}

Traffic impact:

Traffic was not received on et-0/0/3.10, multiple VLAN-ID's have been attempted, but no traffic was received/forwarded.

After analyzing BRCM inputs, it seems like the issue is running out of LLVP profiles for this particular tag structure combinations.

LLVP means Link Layer Vlan Processing and the issue is right on ingress LLVP profile  and this resource in question here is tag structures related to per port.

It can dump the outputs below from router:

-------------------

start shell pfe network fpc0

show evo-pfemand ifd details

show evo-pfemand ifd details | match "IfdName|Installed Ingress Tag Formats,  any_tag_in" 

exit

-------------------

If you get any interfaces with this symptom as below, which means that interface cannot forward/receive traffic at all.

 -------------------

IfdIdx:1286 IfdName:et-0/0/3 isLocal:true
Unit:0 IfdUnitMask:0x1 Core:0
PortNum:4 IfdPpPort(Local/Anchor):103
brcmGport:0x8000004 brcmSystemPort:0x6c000004
TmPort:4 ModuleIdPerCore:1 FpcSlot:0
Aggregate:0 AggregateMember:0 AeIfdIdx:0
Irb:0 Lsi:0 SystemLocalIfd:0 Vtep:0 Lt:0
PseudoInterface:0
EncapsulationType:52
FlexTagging:1 StackedTagging:0
VlanTagging:0 InnerVlanRangeExist:0
HierarchicalScheduler:0
SxePhysical:0 SxeVirtual:0
NativeVlanId:99 NativeIflIdx:16316
AccessVlanId:0 MacLearnEn:0
MaxAllowedTpid:4 ConfiguredTpids:2
Tpid1:0x8100 Tpid2:0x88a8
Tpid3:0x0 Tpid4:0x0
lacpTrapInstalled:0 lldpTrapInstalled:0 MacSecTrapInstalled:0
StpEnable:0 BpduProtectEnable:0 HwSTPTrapEnable:0 vstpIfbdCount:0 HwVstpTrapEnable:0 StpIndex:70
Installed Ingress Tag Formats,  any_tag_in  <<<<<< wrong sign
Installed Egress Tag Formats,
Ingress Tag Structure Configured to drop traffic,  any_tag_in  
pfe_portE's pdl info:2052
IFL list
16304 16316 16315
Installed inner-lists:refCount

-------------------

It can also dump the following outputs below :

-------------------

start shell pfe network fpc0

set evo-pfemand bcmshell cmd "swstate algo_port_pp port_pp_tpid_profile_ingress dump" 

set evo-pfemand bcmshell cmd "swstate algo_port_pp dump port_pp_tpid_profile_ingress" | match "keys ="  

exit

-------------------

The example display as below:

root@lab-router:pfe> set evo-pfemand bcmshell cmd "swstate algo_port_pp port_pp_tpid_profile_ingress dump"    
========================================================================================
|  General Information Template Manager: algo_port_pp_db.port_pp_tpid_profile_ingress  |
========================================================================================
| Name                                         | Advanced | First   | Nof       | Nof      | Nof free |
|                                                   | Algorithm | profile | profiles | profiles | profiles |
|                                                   |                  |             |              | in use   |              |
========================================================================================
| Ingress LLVP Profile Template | No              | 1          | 7           | 6           | 1        |        --------> profile in use is 6 vs max profile 7
========================================================================================

labroot@lab-router:pfe> set evo-pfemand bcmshell cmd "swstate algo_port_pp dump port_pp_tpid_profile_ingress" | match "keys ="    
| 1       | 35         | keys ={is_prio, outer_tpid, inner_tpid}, data = {the LLVP entry data} |
| 2       | 1          | keys ={is_prio, outer_tpid, inner_tpid}, data = {the LLVP entry data} |
| 3       | 11         | keys ={is_prio, outer_tpid, inner_tpid}, data = {the LLVP entry data} |
| 4       | 27         | keys ={is_prio, outer_tpid, inner_tpid}, data = {the LLVP entry data} |
| 5       | 6          | keys ={is_prio, outer_tpid, inner_tpid}, data = {the LLVP entry data} |
| 6       | 1          | keys ={is_prio, outer_tpid, inner_tpid}, data = {the LLVP entry data} |

Root Cause:

Please refer to TSB99914 [juniper.net]:

https://supportportal.juniper.net/s/article/tag-protocol-id-or-TPID-limitation-on-ACX7k-routers

This issue was caused by some TPID limitation on ACX7K.