Description

On MX platforms, it might be observed that, krt high priority add queue stuck with multiple entries pointing to error ' Operation not supported'.

Symptoms

Example output when running the CLI command  'show krt queue':
 
{master}
user@mx_RE0> show krt queue    
Oct 11 23:09:52
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 2 queued
                ADD nhtype Router index 0 (412)
                    error 'Operation not supported'        <<<
                    kqp '0x55ae540'
                ADD nhtype Router index 0 (412)
                    error 'Operation not supported'.      <<<
                    kqp '0x55ae680'
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 0 queued  
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued

Solution

Checking the configuration, we noticed an IP prefix is routed to more than one demux0 IFLs, which is not supported in JUNOS.
 
[edit routing-options]
        route 10.140.0.20/32 {
            qualified-next-hop demux0.10682512;
            qualified-next-hop demux0.12602516;
            no-readvertise;
        }

We can also check the active static routing table using the below command
 
{master}
lab@MX960_RE0> show route protocol static |match "/32|demux0" |except ">|/"
Oct 11 23:12:21 
                      to #0 0.0.0.0.0.0 via demux0.10682512
 
Once we delete the routing on this demux0 IFL then the issue is resolved

Oct 11 23:31:06  MX960_RE0 mgd[81855]: UI_CMDLINE_READ_LINE: User 'lab', command 'delete routing-instances abc routing-options static route 10.140.0.20/32 qualified-next-hop demux0.10682512 '

krt queue is clean now

{master}
user@MX960_RE0> show krt queue 
Oct 12 17:09:01
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued             
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued
 
The below message logs were also observed during the krt stuck queue time, but it is cleared and these messages also stopped.

Oct 11 23:30:17  MX960_RE0 rpd[5148]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable
Oct 11 23:31:12  MX960_RE0 rpd[5148]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable
Oct 11 23:31:12  MX960_RE0 rpd[5148]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable
 

The issue happens only when there are two or more unnumbered IFLs. In this case, we push the route with two next-hops, where the next-hop IP is the same, but outgoing IFLs are different.

Modification History

.

Related Information

PR860580 is a related one.