Description

" RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled" is flooding in the log messages as below:

Dec  5 03:54:43 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled

Symptoms

After some configuration changes committed, the customer found the RPD_KRT_Q error messages flooding on the box.
They didn't know the impact, but they thought it should be potential impact to their end customer. 
---------
Dec  5 03:59:31 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:00:35 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:00:35 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:02:48 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:02:48 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:03:40 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
---------

Solution

To troubleshoot the reason for "RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled".

Please at least get these krt info at first , then collect KRT traceoptions if you can, of course,  RSI/var-log should also be collected.

> show krt queue
> show krt async queue detail

KRT traceoptions (please remove this after data collection)

set routing-options forwarding-table traceoptions file <FILENAME>
set routing-options forwarding-table traceoptions file size 50m
set routing-options forwarding-table traceoptions flag all

Troubleshoot steps from the collected information:

One. configuration change made at the very beginning of this flooding KRT_Q log.
---------
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 bandwidth 100m '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 vlan-id 1138 ' 
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 family inet filter input FIA-Filter '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 layer2-policer output-policer 100m '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 family inet6 filter input FIA-Filter-v6 '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 family inet6 address 2001:6000:0:8::F:21E6/64 '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 family inet6 nd6-max-cache 10 ' 
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 family inet6 nd6-new-hold-limit 10 
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'set interfaces ae17 unit 1138 family inet policer arp FIA_ARP_Policer '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_CMDLINE_READ_LINE: User 'ironman', command 'commit check | no-more '
Dec  5 03:48:18 2023  re0.NONAME mgd[96894]: UI_COMMIT_PROGRESS: Commit operation in progress: Obtaining lock for commit
---------
Two. suspected error log.
---------
Dec  5 03:49:05 2023  re0.NONAME kernel: New hold soft (80%) ND nh cache limit for intf idx 969 reached. <<<<<<
Dec  5 03:54:05 2023  re0.NONAME kernel: Max cache hard ND nh cache limit for intf idx 969 reached.<<<<<< what's the interface for index 969 ?
Dec  5 03:54:43 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
--------
Dec  5 03:54:43 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 03:59:31 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:00:35 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:00:35 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:02:48 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:02:48 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:03:40 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:04:44 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:05:44 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:06:05 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:06:05 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:06:43 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:07:25 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:08:17 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:08:17 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
Dec  5 04:08:42 2023  re0.NONAME rpd[8362]: RPD_KRT_Q_RETRIES: nexthop ADD: Operation canceled
<snipped>
---------

Three. KRT info  as below.
------
re0.NONAME>show krt queue
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
high priority V4oV6 tcnh delete queue: 0 queued
high prioriy anchor gencfg delete queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
high prioriy anchor gencfg add-change queue: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 42 queued
ADD nhtype Router index 0 (1025)
error 'Operation canceled'
kqp '0x66b36280'
ADD nhtype Router index 0 (1025)
error 'Operation canceled'
kqp '0x66b29f80'
ADD nhtype Router index 0 (1025)
error 'Operation canceled'
kqp '0x66b1b100'
<snipped>
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued
------
re0.NONAME>show krt async queue detail
<snipped>
                ADD gf 6 inst id 0 2001:6000:0:8::/64 nexthop
         2001:6000:0:8::f:883 , ae17.1138  <<<<<<still ae17.1138 related to this NH
                       kqp '0x66b36280'
<snipped>

------
Four. the interface statistics and ipv6 neighbor information:
-----
Logical interface ae17.1138 (Index 969) (SNMP ifIndex 3737) (Generation 6535)
    Flags: Up SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.1138 ]  Encapsulation: ENET2
    Layer 2 output policer: 100m-ae17.1138-ifl-o
<snipped>
    Protocol inet6, MTU: 9150
    Max nh cache: 10, New hold nh limit: 10, Curr nh cnt: 10, Curr new hold cnt: 10, NH drop cnt: 7952136  <<<<<this is the KRT Q log root source.
    Generation: 6750, Route table: 0
      Input Filters: FIA-Filter-v6
      Addresses, Flags: Is-Preferred Is-Primary
        Destination: 2001:6000:0:8::/64, Local: 2001:6000:0:8::f:21e6
    Generation: 1277
      Addresses, Flags: Is-Preferred
        Destination: fe80::/64, Local: fe80::219:e204:72bd:5fc2
    Protocol multiservice, MTU: Unlimited, Generation: 1279
    Generation: 6751, Route table: 0
      Policer: Input: FIA_ARP_Policer-ae17.1138-inet-arp
-----
show ipv6 neighbors | match |1138     <<<<<ipv6 neighbor unreachable under the same interface ae17.1138

2001:6000:0:8::f:50d                     none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:737                     none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:883                     none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:b65                     none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:1741                    none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:1a25                    none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:1bbf                    none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:21e7                    none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:24f7                    none               unreachable  3    no   no      ae17.1138
2001:6000:0:8::f:2611                    none               unreachable  3    no   no      ae17.1138
<snipped>
-----

The findings:
NH drop cnt on the interface refers to the number of neighbor discovery requests not serviced because the interface maximum queue size limits have been reached.
Then the uninstalled ipv6 nexthop triggers the KRT_Q retry flooding log.

Reference link:
https://www.juniper.net/documentation/us/en/software/junos/neighbor-discovery/topics/topic-map/neighbor-discovery-cache-protection.html#d101e156__d101e201

The final solution should increase this max nh cache locally on the interface in order to stop these flooding messages:

[ edit interfaces interface-name unit unit number family inet6]
user@router# set nd6-max-cache limit  50

#Configure the maximum size of the neighbor discovery cache for the interface

Modification History

Created on March 23,2024
Modified ipv6 address on March 25,2024

Related Information

KB19071 [juniper.net] Syslog message: RPD_KRT_Q_RETRIES