Description


The cluster after working for a while lost connectivity and created NSD core dumps on both nodes. The connection works now after disconnecting on of the two upstream switches. 

Symptoms

Customer upgraded the device from 20.4 to 21.4R3-S4 using ISSU process.
After upgrading customer started to see NSD core, traffic outage and also alarm on the device.


Alarm:
2023-09-06 06:13:39 CEST Major NSD fails to restart because subcomponents fail



Core dump:
-rw-rw---- 1 root wheel  9537768 Sep 6 06:14 /var/tmp/nsd.core-tarball.0.tgz
-rw-rw---- 1 root wheel  2302612 Sep 7 00:01 /var/tmp/nsd.core-tarball.1.tgz
-rw-rw---- 1 root wheel  9496127 Sep 7 00:02 /var/tmp/nsd.core-tarball.2.tgz

Customer also had destination nat related configuration on the cluster 

nat {
    destination {
      pool smtp-dmz-1 {
        address 2001:638:700:1004::1:a9/128;
      }
      pool smtp-dmz-2 {
        address 2001:638:700:1004::1:aa/128;
      }
      pool smtp-dmz-3 {
        address 2001:638:700:1004::1:ab/128;
      }
      rule-set dst-nat {
        from zone outside;




 

Solution

Platforms configured in a cluster, when ISSU upgrade is performed from a Junos version lower than 21.4R3 to 21.4R3 or higher, the nsd process might crash, core files are generated and alarms are triggered.

This is a Software defect in the junos and is being tracked using the PR mentioned below:

https://prsearch.juniper.net/problemreport/PR1724777


The other workaround for this issue is to use minimal downtime method.
If the ISSU is already performed, then customer needs to reboot both the nodes to recover from this issue.

Modification History

Article created
2024-09-24 : Article modified and published