[M/MX/T-Series] Next-hop change failure in Packet Forwarding Engine (PFE)

Description

This article describes the issue of Next-hop change failure in the PFE.

Symptoms

Generally, when the number of routes exceed the capacity, the router stops adding new routes and next-hops or changing next hops. The following messages are logged in the log messages file:

Apr 2 14:00:50.449 LOG: Err] RT: Failed prefix add IPv4:0 - 17.7.189.85 (add rt entry into jtree failed) Apr 2 14:00:50.449 LOG: Err] RT: Failed prefix add IPv4:0 - 17.7.189.85, nh 564, ifidx 0, nhifl 0, flag 0x8000, cos 0 Apr 2 14:00:50.534 LOG: Err] RT: Failed prefix add IPv4 - 17.7.193.255 (No memory) on FE 0 Apr 2 14:00:50.764 LOG: Err] RT: Failed prefix add IPv4 - 17.7.194.5 (No memory) on FE 0 Apr 2 14:00:50.764 LOG: Err] RT: Failed prefix add IPv4:0 - 17.7.194.5 (add rt entry into jtree failed) Apr 2 14:00:50.841 LOG: Err] RT: Failed prefix add IPv4 - 17.7.198.91 (No memory) on FE 0 Apr 3 08:28:09 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.75.47 (unknown prefix) Apr 3 08:28:09 einste-re1 fpc4 RT: Failed prefix change IPv4:0 - 17.11.127.70 (unknown prefix) Apr 3 08:28:09 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.79.79 (unknown prefix) Apr 3 08:28:09 einste-re1 fpc4 RT: Failed prefix change IPv4:0 - 17.11.130.226 (unknown prefix) Apr 3 08:28:09 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.83.243 (unknown prefix)

The solution would be to reduce the number of routes on the device. However, even after reducing the number of routes to the following threshold, the prefixes still fail to change next-hops. The following messages are logged in the messages file:

Apr 3 08:28:09 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.86.201 (unknown prefix)
Apr 3 08:28:09 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.91.208 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.96.117 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.100.182 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.105.222 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.110.97 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.114.96 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.118.128 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.122.193 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.126.225 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.130.224 (unknown prefix

Solution

The next-hop change failures are due to the following reasons (the X.X.X.X prefix is used as an example):

The router's PFE reached its limit and cannot add any more routes.

The router's RPD process learns a new prefix - x x.x.x.x and RPD sends this prefix to PFE.

As PFE does not have enough memory, the addition of this prefix fails.

Now the user reduces the number of routes to below the threshold.

RPD receives a next-hop change for the x.x.x.x prefix.

RPD sends this change to PFE.

PFE fails to change this next-hop, as it installed the prefix and the following errors are generated:

Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.118.128 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.122.193 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.126.225 (unknown prefix)
Apr 3 08:28:10 einste-re1 fpc6 RT: Failed prefix change IPv4:0 - 17.11.130.224 (unknown prefix)

This is expected behavior. To avoid this issue, it is mandatory to restart the device's PFE. This will ensure that the PFE adds all the prefixes learned by RPD.