Description

On SRX Series devices starting from "junos:21.4R3-S5", In Chassis Cluster, when the Tenant system configuration enabled, then NSD will remain high on primary device, if one of the below mentioned trigger is hit:
Trigger: 
* When one of the devices are rebooted
* When one of the devices are powered on
* When RG0 is failed over

Symptoms

>> When the below mentioned Tenant configuration is present and any one of the description mentioned triggers are hit, then the code gets stuck and run into the loop which causes nsd to keep on running in the background and stays nearly 100% all the time.
 
>> Tenant system configuration:

set tenants Juniper routing-instances CUST_Juniper instance-type virtual-router
set tenants Juniper routing-instances CUST_Juniper routing-options static route 0.0.0.0/0 next-hop 10.10.1.1
set tenants Juniper routing-instances CUST_Juniper interface lt-0/0/0.1
set tenants Juniper routing-instances CUST_Juniper interface reth1.123
set tenants Juniper security policies from-zone Juniper_A to-zone Juniper_B policy ANY_IN match source-address any
set tenants Juniper security policies from-zone Juniper_A to-zone Juniper_B policy ANY_IN match destination-address any
set tenants Juniper security policies from-zone Juniper_A to-zone Juniper_B policy ANY_IN match application any
set tenants Juniper security policies from-zone Juniper_A to-zone Juniper_B policy ANY_IN then permit
set tenants Juniper security policies from-zone Juniper_A to-zone Juniper_B policy ANY_IN then log session-init
set tenants Juniper security zones security-zone Juniper_A host-inbound-traffic system-services all
set tenants Juniper security zones security-zone Juniper_A host-inbound-traffic protocols all
set tenants Juniper security zones security-zone Juniper_A interfaces reth1.123
set tenants Juniper security zones security-zone Juniper_B host-inbound-traffic system-services all
set tenants Juniper security zones security-zone Juniper_B host-inbound-traffic protocols all
set tenants Juniper security zones security-zone Juniper_B interfaces lt-0/0/0.1

>>RE CPU Idle will go to 0 on the primary device as shown below: 

{primary:node0}
root@SRX> show chassis routing-engine
 Nov 30 19:36:33
 node0:
 --------------------------------------------------------------------------
 Routing Engine status:
     Temperature                 38 degrees C / 100 degrees F
     CPU temperature             38 degrees C / 100 degrees F
     Total memory              1900 MB Max   475 MB used ( 25 percent)
     Memory utilization          24 percent
     5 sec CPU utilization:
       User                      31 percent
       Background                 0 percent
       Kernel                    68 percent
       Interrupt                  1 percent
       Idle                       0 percent

 
>> NSD process will be showing 100% utilized all the time on primary device as shown below:

{primary:node0}
 root@SRX> show system processes extensive
 Nov 30 19:36:56
 node0:
 --------------------------------------------------------------------------
 last pid: 65345;  load averages:  1.50,  1.82,  1.78  up 8+08:24:48    19:36:56
 373 threads:   3 running, 310 sleeping, 1 zombie, 59 waiting
 CPU: 32.2% user,  0.0% nice, 66.7% system,  0.5% interrupt,  0.6% idle
 Mem: 40M Active, 1307M Inact, 188K Laundry, 397M Wired, 190M Buf, 123M Free
 Swap: 1639M Total, 1639M Free
 
   PID USERNAME    PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
 23829 root        102    0   758M    73M RUN    187.0H 100.00% nsd
 

 

Solution

Solution:
It has been addressed with PR1776480, and Engineering has committed fix in versions: junos:21.4R3-S6 junos:23.2R2 junos:23.4R2
 
Workaround:
Please remove the tenant system from the configuration or follow the below steps, if tenant system is required for the device.
If the tenant system is required then, failover of RG0 or reboot or power outage of the device can cause this issue to reoccur.
Please follow the below steps as a work around to overcome the issue, if it reoccurs:
Step:1, Remove Tenant system configuration and commit
Step:2, Restart network-security using command "restart network-security immediately"
Step:3, Add the Tenant system configuration back and commit

Note: The above WA is only applicable till the next failover of RG0 or reboot or power outage.

Modification History

2023-12-22 : Article Created
2024-01-02 : Article Updated and Published