After upgrade from 18.4X16.9 to 18.4X29.3 using "request system software add no-validate" knob, customer noticed a change in the configuration syntax on his QFX5200 device when the device came up in the newer version.

lab@qfx> show system rollback compare 1 0
/config/juniper.conf:902:(32) value 5 is not equal to 3: 5
[edit interfaces ae100 unit 0 family mpls maximum-labels]
'maximum-labels 5;'
value 5 is not equal to 3


QFX5k doesn't support more than 3 labels. Meaning that the fact that it was 5 as per the CLI configuration in earlier versions, it was really never in effect since the hardware was only capable of supporting 3. It was a bug and was resolved in later codes. Now, the configuration doesn't allow you to configure any other value other than 3 on QFX devices.

After the device is upgraded, the customer changed the configuration of maximum-label under the interfaces.


Due to this configuration change, the packet discards seemed to be increasing on the device. Also, there were some forwarding next-hops were getting discarded. The issue recovered after a device reboot.

 

Below is an example of a QFX device where packet drops can be checked from Broadcom shell prompt.

lab@qfx>request pfe execute command "set dcbcm bcmshell \" l3 egress show\""  target fpc0

SENT: Ukern command: set dcbcm bcmshell " l3 egress show"

GOT: HW (unit 0)

GOT: Entry Mac         Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC

GOT: 100002 00:00:00:01:02:03 2 1 0t 0 -1 yes no 1 no

GOT: 100003 00:00:00:01:02:03 2 1 255 0 -1 yes no 24 no

GOT: 100004 00:00:00:01:02:03 2 1 0 0 -1 no yes 409 no <<<<<< Here the RefCount is high which means that packets are being discarded. 

While trying to make the changes in the below hierarchy, we may encounter the issue. 
 

lab@qfx# set interfaces xe-0/0/0.0 family mpls maximum-labels ?

Possible completions:

 <maximum-labels>  Protocol family maximum number of labels (3..5)


The issue is seen across all Junos and EVO platforms.

When the maximum-labels are changed, few of the routes are pointing to "100004" and are discarded/dropped and not punted to CPU. 

The difference between "family mpls deactivate" and "mpls maxium-label configuration change" is described below:

1. family mpls toggling:
 

-- When we are toggling mpls, two commits are involved. This generates a separate MPLS-DOWN and MPLS-UP events that are spread apart because of two manual commits.

-- On MPLS-DOWN event BGP marks the peer with PeerMplsDown flag. And launches import-reevaluation, during which it marks the BGP.l3vpn.0 routes hidden. This deletes the mpls.0 route as-well, when the hidden-route is flashed to BGP groups for withdrawal.

-- On MPLS-UP event BGP resets the PeerMplsDown flag on the BGP peer. And launches import-reevaluation, during which it resets the hidden flag on BGP.l3vpn.0 routes, and when they are flashed to BGP groups, the new mpls-label routes are created.

2. In mpls maximum-label case, we see two behaviors: one working and one non-working. (because of race condition described below).
 

-- Here in this case only one commit is involved. Generates both MPLS-DOWN and MPLS-UP events which sometimes arrive back to back and are processed in a background job in BGP.

-- In the Non-working case: before the BGP import-re-evaluation job visits the routes, both MPLS-DOWN and MPLS-UP events are processed by the ifa-change job, and it state-compresses. i.e. the BGP import-reevaluation job does not see PeerMplsDown state. So no change happens to the BGP.l3vpn.0 and mpls.0 routes

-- In the working case: the BGP import-reevaluation job gets to visit the route before the MPLS-UP event is processed by the ifa-change job. In this case, we see the BGP.l3vpn.0 route went to hidden and mpls.0 route was also deleted.

This is more of a timing issue caused by kernel's behavior altering nexthop state produced by rpd. It is unclear why kernel does this and what is the downside if this behavior in kernel is avoided/altered. This is a product limitation and a design extension is required to deal with the situation.
 

Workaround 1:
 

To avoid this, we need do a two step commit, first deactivate 'family mpls' and commit, before doing the 'maximum-label' config change. 

Step 1:
deactivate interfaces <interface_id> family mpls
commit

Step 2:
set interfaces <interface_id> family mpls maximum-labels 3
activate interfaces <interface_id> family mpls
commit

Workaround 2:

You can alternatively reboot the device to recover from the issue.

Note - The issue can be seen when the maximum mpls-label is changed in presence of any BGP Labeled-families (e.g. inet labeled-unicast) or transit functionality (export policy with nexthop-self).

2023-07-13 - Initial publication