Kernel Routing Table (KRT) Queue continuously showing ADD and DELETE operations.

There are scenarios in networks when we see that whenever there is network event it follows with the removal/addition of route from RE, Kernel.

This can be viewed via the command show krt queue

• Ideal Output when there is no network event or churn.

lab@mx> show krt queue

Routing table add queue: 0 queued

Interface add/delete/change queue: 0 queued

Top-priority deletion queue: 0 queued

Top-priority change queue: 0 queued

Top-priority add queue: 0 queued

high priority V4oV6 tcnh delete queue: 0 queued

high prioriy anchor gencfg delete queue: 0 queued

High-priority multicast add/change: 0 queued

Indirect next hop top priority add/change: 0 queued

Indirect next hop add/change: 0 queued

high prioriy anchor gencfg add-change queue: 0 queued

MPLS add queue: 0 queued

Indirect next hop delete: 0 queued

High-priority deletion queue: 0 queued

MPLS change queue: 0 queued

High-priority change queue: 0 queued

High-priority add queue: 0 queued

Normal-priority indirect next hop queue: 0 queued

Normal-priority deletion queue: 0 queued

Normal-priority composite next hop deletion queue: 0 queued

Low prioriy Statistics-id-group deletion queue: 0 queued

Normal-priority change queue: 0 queued

Normal-priority add queue: 0 queued

Least-priority delete queue: 0 queued

Least-priority change queue: 0 queued

Least-priority add queue: 0 queued

Normal-priority pfe table nexthop queue: 0 queued

EVPN gencfg queue: 0 queued

Normal-priority gmp queue: 0 queued

Routing table delete queue: 0 queued

Low priority route retry queue: 0 queued

• And when there is an addition or deletion of route we can see the below output.

lab@mx> show krt queue

Routing table add queue: 3 queued

               ADD table index 8, gf 1

                   kqp '0x6b86f340'

               ADD table index 8, gf 2

                   kqp '0x6b86ee00'

               ADD table index 8, gf 6

                   kqp '0x6b86de40'

Interface add/delete/change queue: 0 queued

Top-priority deletion queue: 0 queued

Top-priority change queue: 0 queued

Top-priority add queue: 0 queued

high priority V4oV6 tcnh delete queue: 0 queued

high prioriy anchor gencfg delete queue: 0 queued

High-priority multicast add/change: 0 queued

Indirect next hop top priority add/change: 0 queued

Indirect next hop add/change: 0 queued

high prioriy anchor gencfg add-change queue: 0 queued

MPLS add queue: 0 queued

Indirect next hop delete: 0 queued

High-priority deletion queue: 0 queued

MPLS change queue: 0 queued

High-priority change queue: 0 queued

High-priority add queue: 0 queued

Normal-priority indirect next hop queue: 0 queued

Normal-priority deletion queue: 3390 queued

            DELETE gf 6 inst id 0 x.x.x.x type 3

                   kqp '0x6a3950c0'

• As long as there is only ADD and DELETE then it is fine as there are bound to happen route changes in the network but it should also get clear, if not then follow the below procedure to find out the routes which are continuously flapping and try to find out the source/originator of that route.
• show route summary <<< Find how much is the scale if it is below 100K we can run show route | match 00:0 and try to find out which route is flapping it can be in any table ie inet.0, inet6.0, inet.3, inet6.3,bgp.l3vpn.0,bgp.l2vpn.0,bgp.evpn.0

lab@mx> show route summary 
Highwater Mark (All time / Time averaged watermark)
inet.0: 924112 destinations, 1844609 routes (921050 active, 3058 holddown, 48 hidden)
              Direct:     29 routes,     29 active
               Local:    106 routes,    106 active
                 BGP: 1844473 routes, 920914 active
              Static:      1 routes,      1 active

inet6.0: 182201 destinations, 182201 routes (180219 active, 1982 holddown, 0 hidden)
              Direct:     13 routes,     13 active
               Local:    102 routes,    102 active
                 BGP: 182085 routes, 180103 active
               INET6:      1 routes,      1 active

• Parallely better way is you can run the below command for maximum 5mins and minimum 1min.

Another way to check the rtsockmon output is as follows: 

> start shell

% rtsockmon -t > /var/tmp/rtsockmon.txt

(wait 1 minute)

Press CTRL+C

Then in a Unix-like OS which is not Junos OS, issue:

% cat rtsockmon.txt | grep inet | grep add | grep route | cut -c 50- | awk '{print $1 " " $2}' | sort | uniq -c | rev |cut -b 7-| rev |sort

The output will look something like this:

3   y.y.y.y

151 x.x.x.x

2   z.z.z.z

The first column will be the number of times the route was added. The second column is the prefix. So in the above example, x.x.x.x flapped: 151 times!

• After identifying the flapped route we can do the show route x.x.x.x detail/extensive and find the source/Originator of the route and understand why it is flapping and we have to stop that.