Description

This article explains the cause and recovery steps when a QFX10002-60c device exhibits below log messages

fpc0 expr_nh_set_platform_tokens: For nh 652, num of fabric tokens passed is 6
fpc0 expr_nh_set_platform_tokens: Expecting <= 2 tokens, FPC in enhanced mode, kernel in inter-op mode 
fpc0 PFE_ERROR_FAIL_OPERATION: Failed to Build Encap Params in nh_unilist_add
fpc0 PFE_ERROR_FAIL_OPERATION: Type specific Add failed generic failure nh-id:652
fpc0 PFE_ERROR_FAIL_OPERATION: Failed to instantiate new copy for NH(652)
fpc0 PFE_ERROR_FAIL_OPERATION: Failed to change state for NH (652)
 

Symptoms

 
  • QFX10002-60c platform only supports enhanced-mode. It is enabled by default regardless of the configuration present or not.
  • If PFE is reporting “FPC in enhanced mode, kernel in inter-op mode” meaning network-services mode is out of sync between kernel and PFE.
  • Hence, if kernel is in any other mode apart from Enhanced-mode, host routes will not be programmed on PFE and could lead to Traffic blackholing
The same can be confirmed using following commands
 
> show chassis network-services 
Network Services Mode: Enhanced-Mode

> show configuration chassis network-services | display inheritance

> show configuration groups junos-defaults chassis 
##
## protect: groups junos-defaults
##
network-services enhanced-mode;

> start shell
% sysctl -a | grep netsvc
net.netsvc: 6      >>>>>>>>>>>>>>>>>>>>>>>>>> If any value seen other than 6, it is out of sync.

% vty fpc0
vty)# show shim fpc enh-mode
ENH-MODE: paradise mode


The value of the net.netsvc variables are:
0 : IP
1 : ethernet
2 : enhanced IP
3 : enhanced Ethernet
4 : all IP
5 : all Ethernet
6 : enhanced mode
7 : lan

Solution

It is advised to reboot the chassis to reprogram the network-services mode.
After the reboot the logging should stop and verify the network-services value should be 6.

> start shell
% sysctl -a | grep netsvc
net.netsvc: 6 

Modification History

2024-03-04 : Article Created