Description

Dynamic Tunnel route resolution over another tunneled route is not supported. Once RPD tries to resolve the tunneled route over another tunneled route, then RPD stops responding. RPD will remain in this state until we remove this tunnel over tunnel resolution. In this case, even RPD/router restart does not help.

Symptoms

In an example scenario, route 20.20.20.20/32 is being resolved over 10.10.10.10/32. 10.10.10.10/32 must be a non-tunneled route. It disappears from the routing table and another bigger subnet (10.10.10.0/24 or 0/0) becomes active. This bigger subnet is tunneled on another prefix. Then RPD goes in a problematic state where it stops responding. Rebooting/restarting the RPD/router does not work and it stays in this condition. We need to remove the tunneled (10.10.10.0/24 or 0/0) NH from the 20.20.20.20/32.

RPD after the error condition occurs: 

root@ptx1000-r2006> show system processes extensive | grep rpd
14361 root 103 0 4897M 3948M CPU0 0 22:55 100.00% rpd{RPD.EN_US}
14361 root 20 0 4897M 3948M kqread 0 1:04 0.00% rpd{krtio-th}
14361 root 20 0 4897M 3948M kqread 1 0:40 0.00% rpd{TRACETHREAD.EN_US}
14361 root 20 0 4897M 3948M kqread 1 0:01 0.00% rpd{bgpio-0-th}

root@ptx1000-r2006> show route 20.20.20.20 
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]
^C[abort]

The following log messages are seen: 

Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_unilist_vxlan_ehandles(),11342:Creating egress handles failed: fe 0 for indr nh 2097154, idxd 1
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [0] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 0 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 0 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [1] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 1 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 1 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [2] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 2 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 2 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [3] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 3 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 3 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [4] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 4 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 4 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [5] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 5 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 5 for indr nh 2097154
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_unilist_vxlan_ehandles(),11342:Creating egress handles failed: fe 0 for indr nh 2097159, idxd 1
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [0] df count is zero
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 expr_create_indr_vxlan_ehandles_per_inst: Error 7 while adding egress desc on fe 0 for indr nh 2097159
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 EXPR-VIRTUAL,expr_create_indr_vxlan_ehandles(),10055:Error 7, egress handles failed on fe 0 for indr nh 2097159
Aug 19 18:21:23  ptx1000-r2006 : %PFE-3: fpc0 JENCAP:ERR:jprds_encap_desc_add(),1288: [1] df count is zero

Solution

One possible workaround is to define a policy in tunnel forwarding options to let the tunnel know the prefixes it can resolve. 

​routing-options {
    dynamic-tunnels {
        forwarding-rib inet.0 {
            inet-import Restrict_NH_Resolution;
        }
        DYN_TEST {
            source-address 10.1.0.1;
            bgp-signal;
            destination-networks {
                10.0.0.0/26 preference 5;
                10.0.0.41/32 preference 5;
                10.0.0.42/32 preference 5;
            }
        }
    }
}
policy-options {
    policy-statement Restrict_NH_Resolution {
        term Accept_Direct_Routes {
            from protocol direct;
            then accept;
        }
        term Accept_Allowed_Prefixes {
            from {
                route-filter 10.10.10.10/32 exact;
            }
            then accept;
        }
        term REJECT-ALL {
            then reject;
        }
    }
}