Description

This Article describes the issue of the %PFE-3: invalid nh (4241) error message being generated for bulk statistics.

Symptoms

When automatically polling LSP statistics by using set protocols mpls statistics file <filename> or manually by using SNMP, the following error messages are generated:

May 12 03:30:03.709 lab fpc11 NH: invalid nh (1656) for bulk statistics
May 12 03:30:03.708 lab fpc11 NH: invalid nh (1653) for bulk statistics
These error messages are accompanied by the following messages in the syslog file:
lab /kernel: RT_PFE: NH IPC op 30 (GET NH BULK STATS) failed, err 1 (Unknown) peer_class 0, peer_index 11 peer_type 17
lab /kernel: RT_PFE: NH IPC op 30 (GET NH BULK STATS) failed, err 1 (Unknown) peer_class 0, peer_index 11 peer_type 17
lab /kernel: RT_PFE: NH IPC op 30 (GET NH BULK STATS) failed, err 1 (Unknown) peer_class 0, peer_index 11 peer_type 17

Solution

The above error messages occur due to the polling statistics for a specific next hop not being available or not in the right state as installed in the PFE. For example, as per the above error message, the state of the next hop (nh) in the PFE is as follows:

ADPC11(lab vty)# show nhdb id 1653
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- --
1653 Discard - - MPLS - 0 0x00000000 0x00000000


ADPC11(lab vty)# show nhdb id 1656
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- --
1656 Discard - - MPLS - 0 0x00000000 0x00000000
The next hop is installed in the discard state. While this may be normal, polling to this next hop will result in the above error message being generated.

To confirm the cause, the following outputs were examined:

The forwarding table of the next hops is as follows:
lab@R1> show route forwarding-table family mpls | match 1656
304241 user 0 ulst 1050644 2
200.195.106.238 Swap 325888 1656 1 xe-2/3/2.201

{MASTER}
lab@R1> show route forwarding-table family mpls | match 1653
200.195.106.238 Swap 323312 1653 1 xe-2/3/2.201
544772 user 0 ulst 1050812 2
The forwarding table indicates that the next hops are installed correctly. Next, the status of the next hops was checked on another FPC. The issue was reported in FPC11; FPC8 was checked:
NPC8(lab vty)# show nhdb id 1653
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- --
1653 Unicast xe-2/3/2.201 - MPLS Ethernet 9174 0x00000001 0x00000002

NPC8(lab vty)# show nhdb id 1656
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- --
1656 Unicast xe-2/3/2.201 - MPLS Ethernet 9174 0x00000001 0x00000002

NPC8(lab vty)# exit
FPC 8 indicates that this next hop is installed correctly. All of the PFE's should reflect the same state; so it is not normal for FPC11 to be out of sync with respect to the next hops (nh).

As the nh id's are not in sync and are pointing to the same interface, the least disruptive way to get them back in sync is to just flap the interface during a scheduled window. This will cause the nh to be removed or changed. Then, when the interface comes back up, there is another nh change or add operation that should install the correct nh on all of the PFE's.

Alternatively, another more disruptive way to fix this out of sync behavior is to perform an RE switchover with the GRES configuration being disabled. The outage time here could be more and is similar to a router reboot.