Customer running Junos 23.4R2-S7,

Customer reported multiple core crash across SPC3 PIC's when they added the NAT pool and NAT rule config.
They followed correct procedure of deactivating/ activating the service-set.

Issue will be seen when more than 81 NAT pools are configured and SNMP polling is enabled on NAT services.
The SPC3 PIC may encounter a crash resulting in a core dump.

re0> show system core-dumps no-forwarding

-rw-r--r--  1 root  wheel  621128076 Oct 15 04:44 /var/crash/core--fpc3.pic1-flowd_spc3.elf.4.tgz
-rw-r--r--  1 root  wheel  647447498 Oct 15 04:46 /var/crash/core--fpc4.pic0-flowd_spc3.elf.3.tgz
-rw-r--r--  1 root  wheel  707329077 Oct 15 04:47 /var/crash/core--fpc7.pic0-flowd_spc3.elf.3.tgz
-rw-r--r--  1 root  wheel  708702920 Oct 15 04:44 /var/crash/core--fpc7.pic1-flowd_spc3.elf.3.tgz
-rw-r--r--  1 root  wheel  720360884 Oct 15 04:44 /var/crash/core--fpc8.pic1-flowd_spc3.elf.3.tgz

 

Workaround:

This behaviour is caused by a software bug (PR1913556). Enabling SNMP polling with more than 81 NAT pools triggers the crash and impacts service stability.

As a temporary workaround, customers can configure additional NAT pools without enabling SNMP until the fix is applied.

Root cause:

We have identified that the issue occurs due to a memory handling problem when NAT statistics are fetched (SNMP) for more than 81 pools. 

During this process, the system continues to collect data without properly resetting certain counters. As a result, the data stored in memory becomes corrupted, causing these crashes. 

Dev fixed this bug to ensure memory is handled correctly when larger numbers of pools with SNMP involved, as customer requested fix will be committed in 23.4R2-S7

We can confirm this issue is specific to MX-SPC3. This will not happen in MS-MPC.