Description

This article describes the data that can be collected for the KRT queue "ENOENT -- Item not found" on QFX 5K devices  
 

 

Symptoms

root@QFX5120-32C> show krt queue

Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
high priority V4oV6 tcnh delete queue: 0 queued
high prioriy anchor gencfg delete queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
high prioriy anchor gencfg add-change queue: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 1 queued
        CHANGE FROM ifd index 4 ifl subunit 258, iff family 0, instance id 0, state (null), (53298)
                    error 'ENOENT -- Item not found'
                    kqp '0xf5d5e00'
                 TO ifd index 4 ifl subunit 258, iff family 0, instance id 0, state (null), (53298)
                    error 'ENOENT -- Item not found'
                    kqp '0xf5d5e00'
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued

Log messages: 
Jun  31 23:22:43  QFX5120-32C rpd[7919]: RPD_KRT_Q_RETRIES: ifl iff CHANGE: No such file or directory
Jun  31 23:24:24  QFX5120-32C rpd[7919]: RPD_KRT_Q_RETRIES: ifl iff CHANGE: No such file or directory
Jun  31 23:26:04  QFX5120-32C rpd[7919]: RPD_KRT_Q_RETRIES: ifl iff CHANGE: No such file or directory
Jun  31 23:29:24  QFX5120-32C rpd[7919]: RPD_KRT_Q_RETRIES: ifl iff CHANGE: No such file or directory
Jun  31 23:31:29  QFX5120-32C rpd[7919]: RPD_KRT_Q_RETRIES: ifl iff CHANGE: No such file or directory
Jun  31 23:34:11  QFX5120-32C rpd[7919]: RPD_KRT_Q_RETRIES: ifl iff CHANGE: No such file or directory
 

Solution

There isn't a know trigger for this issue but if there is a suspected configuration that caused this, we can collect the following for analysis:

Configure the following traces before the replication attempts:

  • L2ald
set protocols l2-learning traceoptions file l2ald.log files 10 size 100m
set protocols l2-learning traceoptions file size 1g
set protocols l2-learning traceoptions level all
set protocols l2-learning traceoptions flag all
  •  RPD
set protocols evpn traceoptions file evpn.log files 10 size 100m
set protocols evpn traceoptions file size 1g
set protocols evpn traceoptions flag all
  •  routing-options
set routing-options traceoptions file routing-options-trace
set routing-options traceoptions file size 100m
set routing-options traceoptions file files 10
set routing-options traceoptions flag all
  • Collect rtsockmon
rtsockmon -rt > /var/tmp/rtsockmon.txt &
  • Collect pfe live core:
root@QFX5120-32C:RE:0% ps -aux | grep pfe
root   8164  2.3 0.0 87912  944 - S   29Oct22  2724:51.80 /usr/sbin/dc-pfe -N (fxpc)
root     40  0.0 0.0     0   16 - DL  29Oct22     0:00.00 [if_pfe_listen]
root   8898  0.0 0.9 523232 34936 - S   29Oct22    85:43.97 /usr/libexec32/pfed -N
root  99748  0.0 0.1 16604 2252 0 S+  17:03       0:00.00 grep pfe
root@QFX5120-32C:RE:0% gcore -s -c /var/tmp/gcore.live.1 8164

 root@QFX5120-32C:RE:0% ls -ltr | grep gcore
-rw-r--r-- 1 root    wheel    36253696 Dec 29 17:04 gcore.live.1

  •  Collect kernel live core
{master:0}
root@QFX5120-32C> request system live-core 
{master:0}
root@QFX5120-32C> file list /var/tmp/ detail | grep vmcore 
-rw-rw-rw- 1 root wheel 501735424 Dec 29 17:06 vmcore.live.0
  •  Collect rpd live core
root@QFX5120-32C> start shell
root@QFX5120-32C:RE:0% ps -aux | grep rpd
root   8897  0.0 2.5 632320 98448 - S   29Oct22    96:12.50 /usr/sbin/rpd -N
root  99786  0.0 0.1 16604 2252 0 S+  17:09       0:00.00 grep rpd
root@QFX5120-32C:RE:0% gcore -s -c /var/tmp/rpd.core.1 8897
root@QFX5120-32C:RE:0% ls -ltr | grep rpd
-rw-r--r-- 1 root    wheel    158810112 Dec 29 17:09 rpd.core.1

If the issue is replicated, collect the following:

  • Collect pfe live core:

root@QFX5120-32C:RE:0% ps -aux | grep pfe
root   8164  2.3 0.0 87912  944 - S   29Oct22  2724:51.80 /usr/sbin/dc-pfe -N (fxpc)
root     40  0.0 0.0     0   16 - DL  29Oct22     0:00.00 [if_pfe_listen]
root   8898  0.0 0.9 523232 34936 - S   29Oct22    85:43.97 /usr/libexec32/pfed -N
root  99748  0.0 0.1 16604 2252 0 S+  17:03       0:00.00 grep pfe
root@CQFX5120-32C:RE:0% gcore -s -c /var/tmp/gcore.live.2 8164
root@QFX5120-32C:RE:0% ls -ltr | grep gcore
-rw-r--r-- 1 root    wheel    36253696 Dec 29 17:04 gcore.live.2

  •  Collect kernel live core

{master:0}
root@QFX5120-32C> request system live-core  
{master:0}
root@QFX5120-32C> file list /var/tmp/ detail | match vmcore    
-rw-rw-rw- 1 root wheel 501735424 Dec 29 17:06 vmcore.live.0
-rw-rw-rw- 1 root wheel 501744402 Dec 29 18:21 vmcore.live.1

  •  Collect rpd live core 

root@QFX5120-32C> start shell
root@QFX5120-32C:RE:0% ps -aux | grep rpd
root   8897  0.0 2.5 632320 98448 - S   29Oct22    96:12.50 /usr/sbin/rpd -N
root  99786  0.0 0.1 16604 2252 0 S+  17:09       0:00.00 grep rpd
 
root@QFX5120-32C:RE:0% gcore -s -c /var/tmp/rpd.core.2 8897
root@QFX5120-32C:RE:0% ls -ltr | grep rpd
-rw-r--r-- 1 root    wheel    158810112 Dec 29 17:09 rpd.core.2 

  • Collect RSI, Var logs, and host logs

request support information | save /var/log/rsi_YYYY-MM-DD.txt
file archive compress source /var/log/* destination /var/tmp/re0.tgz
https://supportportal.juniper.net/s/article/QFX5100-How-to-collect-logs-from-the-host-shell

Upload the following from /var/tmp/ along with the rsi and logs collected from the above step

rtsockmon.txt
gcore.live.1
vmcore.live.0
rpd.core.1
gcore.live.2
vmcore.live.1
rpd.core.2

Modification History

.