Firewall SPC3 got crash and created core dump when gtpu-distribution is enabled on the device

customer had upgraded the device from 20.4 to 21.4 code and after that they have witnessed multiple core dumps on the device whenever they enable the gtpu-distribution.

Looking at the backtrace, it is definitely related to GTP traffic.

We should be seeing core being saved like below:

root@SRX> show system core-dumps no-forwarding

-rw-r--r-- 1 root wheel 2630606008 Mar 8 11:56 /var/crash/core-SRX-FW1-fpc1.pic0-flowd_spc3.elf.0.tgz

-rw-r--r-- 1 root wheel 3317956663 Mar 8 06:08 /var/crash/core-SRX-FW1-fpc1.pic1-flowd_spc3.elf.0.tgz

-rw-r--r-- 1 root wheel 2976363158 Mar 8 07:43 /var/crash/core-SRX-FW1-fpc1.pic1-flowd_spc3.elf.1.tgz

Backtrace of the Core related to GTP traffic

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at /usr/src/debug/glibc/2.24-r0/git/sysdeps/unix/sysv/linux/raise.c:58
#1  0x0000003db503440a in __GI_abort () at /usr/src/debug/glibc/2.24-r0/git/stdlib/abort.c:89
#2  0x00000000013c2f89 in junos_abort () at ../../../../../src/pfe/platform/flowd-spc3/platform/tvp_junos_glue.c:550
#3  0x000000000133b834 in panic (format_string=0x2f6cb20 "ukern malloc called from non-ukern thread") at ../../../../../src/pfe/platform/flowd-mt/flowd_mt_tvp_panic.c:91
#4  0x0000000000454a99 in heap_block_get (heap=0x7b709c0 <kernel_heap>, size=<optimized out>, callers_pc=<optimized out>) at ../../../../../../../src/pfe/ukern/common/heap.c:2573
#5  0x0000000000459fe2 in pile_malloc (header=header@entry=0x40c5f0c0) at ../../../../../../../src/pfe/ukern/common/pile.c:233
#6  0x000000000045a2ff in pile_zalloc (header=0x40c5f0c0) at ../../../../../../../src/pfe/ukern/common/pile.c:353
#7  0x00000000004b82e5 in radix_attach_alloc (root=0x11aefa40 <apn_tree_root>) at ../../../../../../../../src/pfe/common/toolkits/radix2/radix.c:187
#8  radix_add (root=root@entry=0x11aefa40 <apn_tree_root>, new=new@entry=0xccb39400) at ../../../../../../../../src/pfe/common/toolkits/radix2/radix.c:696
#9  0x0000000001805c37 in gtp_apn_tree_add_node (apn_tree=0x11aefa40 <apn_tree_root>, apn_rnode=apn_rnode@entry=0xccb39400, pfxlen_bit=pfxlen_bit@entry=40, r_inst=r_inst@entry=0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_apn_ctrl.c:82
#10 0x00000000018061c6 in gtp_apn_entry_add (apn_entry=apn_entry@entry=0xccb39400) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_apn_ctrl.c:418
#11 0x000000000180628e in gtp_apn_entry_create (apn_str=apn_str@entry=0x1c68a3360 "ctnet") at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_apn_ctrl.c:519
#12 0x0000000001806ec9 in gtp_apn_entry_get (apn_str=apn_str@entry=0x1c68a3360 "ctnet") at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_apn_ctrl.c:541
#13 0x000000000183e51f in gtp_tnl_attach_apn (ctnl=ctnl@entry=0x1c9c01110, apn_string=apn_string@entry=0x1c68a3360 "ctnet") at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tunnel.c:705
#14 0x00000000018410c3 in gtp_tnl_create_ctnl (cti=cti@entry=0x1c68a3258) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tunnel.c:785
#15 0x000000000183aab9 in gtp_tproc_create_ctnl (tact=tact@entry=0x1c68a3240, pctx=pctx@entry=0x183ff38de0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tnl_proc.c:270
#16 0x000000000183bc85 in gtp_tproc_crt_sess (ctnl_in=ctnl_in@entry=0x0, tact=tact@entry=0x1c68a3240, pctx=pctx@entry=0x183ff38de0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tnl_proc.c:520
#17 0x000000000183b27c in gtp_tproc_local (proc_fn=proc_fn@entry=0x183bc5a <gtp_tproc_crt_sess>, tact=tact@entry=0x1c68a3240, pctx=pctx@entry=0x183ff38de0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tnl_proc.c:714
#18 0x000000000183ccdb in gtp_tproc_start (tact=tact@entry=0x1c68a3240, pctx=pctx@entry=0x183ff38de0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tnl_proc.c:815
#19 0x000000000183d0b6 in gtp_msg_tproc_do_next (pkt=pkt@entry=0x1c6004cc0, req=req@entry=0x1c68677c0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tnl_proc.c:1247
#20 0x000000000183d857 in gtp_msg_tnl_proc (tact=tact@entry=0x1c68a3240, pkt=pkt@entry=0x1c6004cc0, req=req@entry=0x1c68677c0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_tnl_proc.c:1273
#21 0x00000000018300b3 in vec_create_pdp_rsp (pkt=0x1c6004cc0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_msg_v1.c:571
#22 0x0000000001850a86 in gtpv1_sig_vector (pkt=0x1c6004cc0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp_v1.c:188
#23 0x000000000180108a in gtp_enforce_sig (pkt=pkt@entry=0x1c6004cc0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp.c:1224
#24 0x000000000180248a in gtp_enforce_msg1 (pkt=pkt@entry=0x1c6004cc0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp.c:1293
#25 0x000000000180262b in gtp_enforce_pkt (pkt=0x1c6004cc0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp.c:1344
#26 0x0000000001802c05 in gtp_pkt_handler (sid=sid@entry=5635003703991, alg_id=<optimized out>, dir=dir@entry=1 '\001', jbuf=jbuf@entry=0x1621c61a80, sess_exflag=0) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp.c:1459
#27 0x00000000018048bd in jsf_gtp_data_handler (sctx=<optimized out>, ev=<optimized out>) at ../../../../../../../../../src/junos/jsf/plugin/gprs/application/gtp/gtp.c:724
#28 0x0000000001883106 in jsf_gprs_mgr_data_handler (sctx=0x183ff391f0, ev=<optimized out>) at ../../../../../../../../../src/junos/jsf/plugin/gprs/jsf_gprs_plugin.c:384
#29 0x0000000000b89908 in flow_jsf_plugins_vector_ (pak_ptr=pak_ptr@entry=0x183ff39520) at ../../../../../src/pfe/usp/rt/applications/flow/jsf_flow_utils.c:2943
#30 0x0000000002b14301 in flow_jsf_c2s_vector (pak_ptr=0x183ff39520) at ../../../../../src/pfe/usp/rt/applications/flow/flow_fast.c:4686
#31 0x0000000002b1125d in flow_process_fast_path (pak_ptr=0x183ff39520) at ../../../../../src/pfe/usp/rt/applications/flow/flow_fast.c:1009
#32 0x0000000000adda5b in flow_process_pkt_exception (mbuf=mbuf@entry=0xa3533000, uarg=uarg@entry=0x50488af8, thread_id=<optimized out>) at ../../../../../src/pfe/usp/rt/applications/flow/flow_fast.c:9132
#33 0x0000000002b127b0 in flow_process_pkt (mbuf=mbuf@entry=0xa3533000, size=size@entry=1 '\001', uarg=0x50488af8) at ../../../../../src/pfe/usp/rt/applications/flow/flow_fast.c:9735
#34 0x0000000002af6db3 in jexec_eval_flow_v4func (jexec_thr_ctx=0x183fbff7c0, call_fn_nh=<optimized out>, m=0xa3533000) at ../../../../../src/pfe/fwdd-rt/ft/jexec/jexec_fwdd.c:2000
#35 jexec_eval_nh (jexec_thr_ctx=<optimized out>, m=<optimized out>) at ../../../../../src/pfe/fwdd-rt/ft/jexec/jexec_fwdd.c:2722
#36 jexec_run (jexec_thr_ctx=<optimized out>, m=<optimized out>) at ../../../../../src/pfe/fwdd-rt/ft/jexec/jexec_fwdd.c:2807
#37 jexec_lookup (m=m@entry=0xa3533000, starting_nh=starting_nh@entry=0x183ff39ecc) at ../../../../../src/pfe/fwdd-rt/ft/jexec/jexec_fwdd.c:2914
#38 0x0000000002afc815 in ft_pktproc_loop (pkt=pkt@entry=0xa3533000, stage=<optimized out>, stage@entry=FT_PKTPROC_STAGE_LOOKUP) at ../../../../../src/pfe/usp/rt/fwdd-rt/ft/pktproc/ft_pktproc.c:283
#39 0x00000000013475ba in ft_cp_to_spu_local_data_handler (m=m@entry=0xa3533000) at ../../../../../src/pfe/usp/rt/fwdd-rt/ft/sched/platform/ft_dpq_handler_srx.c:1899
#40 0x0000000000f39d25 in dcp_flow_fwd_pak_to_spu_fast (sessp=sessp@entry=0xe2238ac88, mbuf=mbuf@entry=0xa3533000, msg_type=msg_type@entry=80 'P', direct_call=<optimized out>) at ../../../../../src/pfe/usp/rt/applications/flow/central-point/cp_flow_event.c:3356
#41 0x0000000002b1658e in dcp_flow_appcp_query_ack_handler (event=0x170a85cf3c) at ../../../../../src/pfe/usp/rt/applications/flow/central-point/cp_flow_event.c:3261
#42 dcp_flow_appcp_query_reply_batch_handler (mbuf=0xa1cf7800, mbuf=0xa1cf7800) at ../../../../../src/pfe/usp/rt/applications/flow/central-point/cp_flow_event.c:3119
#43 cp_flow_process_event (mbuf=0xa1cf7800) at ../../../../../src/pfe/usp/rt/applications/flow/central-point/cp_flow_event.c:460
#44 0x00000000004b2e67 in fast_jmpi_msg_processing (m=m@entry=0xa1cf7800) at ../../../../../src/pfe/usp/control/toolkits/jmpi/fast_jmpi.c:411
#45 0x0000000002aff4c8 in ft_mt_thd_fwd_func (hwt_id=0x7) at ../../../../../src/pfe/platform/flowd-mt/flowd_mt_ft.c:1129
#46 0x0000000002267db1 in eal_thread_loop (arg=<optimized out>) at /volume/build/vsdk/21.4/spc3_JUNOS_214_R3_S6_BRANCH_20240208_1250/sb-vsdk/obj-x86-wrl9pmb-srx-spc3/BUILD/dpdk_18_11/tmp_dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:153
#47 0x00007fa13d76c3f4 in start_thread (arg=0x183ff3f580) at /usr/src/debug/glibc/2.24-r0/git/nptl/pthread_create.c:456
#48 0x0000003db50e7e7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

This issue has been introduced because of an RLI where GTP-C APN based rate limiting was enabled.
Workaround is to disable APN-learning feature on the device using the hidden command 
 "set security gtp apn-learning-disable"

Also we need to reboot the device for the command to take effect.
Once the above command is added and device rebooted, core will stop from happening.
This has been fixed using the PR 1736985

https://prsearch.juniper.net/problemreport/PR1736985