Article Detail

Description

This article explains about how NH-exhaustion leads to ping failure and also how to resolve this issue.

Symptoms

Checked the routing-information in the RE and PFE, and the control plane works ok & make sure no routes programming issue in the PFE.

Check for below log messages:

Jan 22 11:50:07.149 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New FIB highwatermark for routes: 1593455 [2026-01-22 11:50:06]
Jan 22 11:50:07.366 2026  PE-D6-ULN-HSI smid: SMID_RSMON_FPC_HEAP_MEMORY_STATUS: Resource Monitor: FPC 9 HEAP memory has exceeded set threshold 70
Jan 22 11:50:10.009 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New RIB highwatermark for unique destinations: 2651394 [2026-01-22 11:50:09]
Jan 22 11:50:22.667 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 5538794 [2026-01-22 11:50:22]
Jan 22 11:50:36.546 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New FIB highwatermark for routes: 1682196 [2026-01-22 11:50:35]
Jan 22 11:50:40.146 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New RIB highwatermark for unique destinations: 2751606 [2026-01-22 11:50:39]
Jan 22 11:50:52.862 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 5697344 [2026-01-22 11:50:52]
Jan 22 11:50:57.360 2026  PE-D6-ULN-HSI smid: SMID_RSMON_FPC_HEAP_MEMORY_EXCEEDED: FPC Resource Monitor: FPC 1 Heap Memory has crossed free memory watermark of 20
Jan 22 11:51:04.385 2026  PE-D6-ULN-HSI smid: SMID_RSMON_FPC_HEAP_MEMORY_EXCEEDED: FPC Resource Monitor: FPC 10 Heap Memory has crossed free memory watermark of 20
<..>
Jan 22 11:55:02.691 2026  PE-D6-ULN-HSI chassisd[11359]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 1
Jan 22 11:55:02.691 2026  PE-D6-ULN-HSI chassisd[11359]: CHASSISD_IPC_WRITE_ERR_NULL_ARGS: FRU has no connection arguments fru_send_msg FPC 1
Jan 22 11:55:02.703 2026  PE-D6-ULN-HSI kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
<..>
Jan 22 11:55:06.546 2026  PE-D6-ULN-HSI rpd[24902]: RPD_RT_HWM_NOTICE: New FIB highwatermark for routes: 2002006 [2026-01-22 11:55:06]
Jan 22 11:55:07.115 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: radix node add failed 
Jan 22 11:55:07.115 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: rt prefix add failed 
Jan 22 11:55:07.115 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: proto ipv4,len 16 prefix 132.107/16 nh 1651309 
Jan 22 11:55:07.308 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: route process failed 
Jan 22 11:55:07.562 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: radix node add failed 
Jan 22 11:55:07.607 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: rt prefix add failed 
Jan 22 11:55:07.722 2026  PE-D6-ULN-HSI fpc10 PFE_ERROR_FAIL_OPERATION: proto ipv4,len 24 prefix 132.107.198/24 nh 1651310

Need to check below PFE shell level logs for impacted FPC's and capture live core-dump files

Exceptions dump for "sw error" confirmed the ICMP packets are dropped due to "sw error" exceptions.

NPC10(PE-D6-ULN-HSI vty)# show jnh 2 exceptions terse

Reason Type Packets Bytes

==================================================================

PFE State Invalid

----------------------

sw error DISC( 64) 4849 412204 <<<<<<<<<<<<<<<<this entry was increasing in both FPC1 and FPC10

Collect live core-dumps and decode see if its matching below BT:

core-NPC1.gz.core.0:
#0  vpanic (format_string=0x446c28e4 "abort 0x%lx", ap=0x491ab440) at ../../../../../../../src/pfe/ukern/cpu-ppc/ppc603e_panic.c:66
#1  0x4004932c in panic (format_string=<optimized out>) at ../../../../../../../src/pfe/ukern/cpu-ppc/ppc603e_panic.c:82
#2  0x426353e8 in abort () at ../../../../../../../../src/pfe/common/toolkits/libstdc++/embedded/libstdc++-ukern.c:55
#3  0x42eae784 in __gnu_cxx::_6::__verbose_terminate_handler () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/vterminate.cc:53
#4  0x42eae614 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:43
#5  0x42eae650 in std::terminate () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:53
#6  0x42eae4b8 in __cxxabiv1::__cxa_rethrow () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_throw.cc:109
#7  0x42eae770 in __gnu_cxx::_6::__verbose_terminate_handler () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/vterminate.cc:83
#8  0x42eae614 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:43
#9  0x42eae650 in std::terminate () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:53
#10 0x42eae568 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=<optimized out>, dest=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_throw.cc:76
#11 0x42eab7ac in operator new (sz=512) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/new_op.cc:63
#12 0x430c25b8 in allocate (__n=<optimized out>, this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/ext/new_allocator.h:91
#13 _M_allocate_node (this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:430
#14 std::_6::deque<KtreeSpace::Node*, std::_6::allocator<KtreeSpace::Node*> >::_M_push_front_aux (this=0x6b62f014, __t=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/bits/deque.tcc:332
#15 0x409db140 in push_front (__x=<optimized out>, this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:1040
#16 KtreeSpace::Ktree::trimMe (this=0x6b62f010, a_trimMe=0x7ca78ad0) at ../../../../../../../../src/pfe/common/toolkits/ktree/ktree.cc:3184
#17 0x409f675c in KtreeSpace::FourWayRightAttachedNode::insertOrModify (this=0x7ca78ad0, key=0x491ab6a8, keyLength=<optimized out>, a_bitToMatch=<optimized out>, a_nextHop=..., parentNode=0xa16a96e0, grandParent=<optimized out>, nextNode=<optimized out>, status=@0x491ab5f0: KtreeSpace::statusSuccess, cleanParent=0x491ab678, ktree=...) at ../../../../../../../../src/pfe/common/toolkits/ktree/ktreeInsert.cc:3225
#18 0x409f4194 in KtreeSpace::Ktree::iterativeInsertOrModify (this=0x6b62f010, key=<optimized out>, keyLength=16, a_nextHop=..., cleanParent=0x491ab678) at ../../../../../../../../src/pfe/common/toolkits/ktree/ktreeInsert.cc:251
#19 0x409f475c in KtreeSpace::Ktree::insertOrModify (this=0x6b62f010, key=0x491ab6a8, keyLength=16, pointer=0x491ab688, nextHopCookie=195948557, cleanParent=0x491ab678, keyString=<optimized out>) at ../../../../../../../../src/pfe/common/toolkits/ktree/ktreeInsert.cc:95
#20 0x409d18d0 in Trinity_Ktree::insert (this=<optimized out>, key=<optimized out>, keyLength=<optimized out>, prefixStr=<optimized out>, nextHop=0x6ba9cc28, noHwFlush=TRUE) at ../../../../../../../../../../src/pfe/common/pfe-arch/trinity/toolkits/ktree/ktree_trinity.cc:229
#21 0x409d1a60 in ktree_insert (ktreep=0x6b62f010, prefix=0x6b0f524d "\\201?", prefixlen=16, prefix_str=0x491abc9c "52.129.63/24", nh=0x6ba9cc28, noHwFlush=TRUE) at ../../../../../../../../../../src/pfe/common/pfe-arch/trinity/toolkits/ktree/ktree_trinity.cc:3423
#22 0x40724c8c in jnh_add_prefix_table (table=<optimized out>, prefix=0x6b0f524d "\\201?", prefixlen=16, prefix_str=0x491abc9c "52.129.63/24", nh_cookie=0x6ba9cc28, noHwFlush=TRUE) at ../../../../../src/pfe/common/pfe-arch/trinity/toolkits/jnh_core/jnh_table.c:497
#23 0x41babb38 in trinity_rt_create_layer3 (p_rtt=0x6a312fb8, prefix=0x6b0f524c "4\\201?", prefixlen=24, prefixstr=0x491abc9c "52.129.63/24", rnh=0x6ba9cc28, isBulk=TRUE) at ../../../../../src/pfe/common/pfe-arch/trinity/applications/route/rt.c:1062
#24 0x41baf5e4 in trinity_rt_create (args=0x491ab8a8) at ../../../../../src/pfe/common/pfe-arch/trinity/applications/route/rt.c:238
#25 0x41b6dc98 in rt_entry_add_msg_proc (rt_params=0x491abc70) at ../../../../../../../../../src/pfe/common/applications/route/hal/rt_entry.c:4596
#26 0x41b7e288 in rt_msg_handler (pipe=0x47101b60, msg=0xb470e598) at ../../../../../../../../../src/pfe/common/applications/route/hal/rt_msg.c:736
#27 0x418d3738 in pfeman_session_manager (pipe=0x47101b60, msg=0xb470e598) at ../../../../../src/pfe/common/applications/pfeman/pfeman_session.c:620
#28 0x418c90d8 in pfeman_rt_msg_handler (pipe=0x47101b60) at ../../../../../src/pfe/common/applications/pfeman/pfeman_rt_xdpc.c:453
#29 pfeman_rt_thread () at ../../../../../src/pfe/common/applications/pfeman/pfeman_rt_xdpc.c:1233
#30 0x40033b8c in thread_detach_tty (thread=<optimized out>) at ../../../../../src/pfe/ukern/common/thread.c:141
#31 0x40029608 in add_small_nodes (index=<optimized out>, heap=<optimized out>) at ../../../../../src/pfe/ukern/common/heap.c:5656
#32 heap_block_get_small_node (heap=0xbadc0ffe, size=1400136043, callers_pc=0x51202020) at ../../../../../src/pfe/ukern/common/heap.c:5100
#33 0x00000000 in ?? ()
(gdb)  q


core-NPC10.gz.core.2:

#0  vpanic (format_string=0x446c28e4 "abort 0x%lx", ap=0x4917b108) at ../../../../../../../src/pfe/ukern/cpu-ppc/ppc603e_panic.c:66
#1  0x4004932c in panic (format_string=<optimized out>) at ../../../../../../../src/pfe/ukern/cpu-ppc/ppc603e_panic.c:82
#2  0x426353e8 in abort () at ../../../../../../../../src/pfe/common/toolkits/libstdc++/embedded/libstdc++-ukern.c:55
#3  0x42eae784 in __gnu_cxx::_6::__verbose_terminate_handler () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/vterminate.cc:53
#4  0x42eae614 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:43
#5  0x42eae650 in std::terminate () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:53
#6  0x42eae4b8 in __cxxabiv1::__cxa_rethrow () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_throw.cc:109
#7  0x42eae770 in __gnu_cxx::_6::__verbose_terminate_handler () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/vterminate.cc:83
#8  0x42eae614 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:43
#9  0x42eae650 in std::terminate () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:53
#10 0x42eae568 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=<optimized out>, dest=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_throw.cc:76
#11 0x42eab7ac in operator new (sz=512) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/new_op.cc:63
#12 0x430c26bc in allocate (__n=<optimized out>, this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/ext/new_allocator.h:91
#13 _M_allocate_node (this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:430
#14 std::_6::_Deque_base<KtreeSpace::Node*, std::_6::allocator<KtreeSpace::Node*> >::_M_create_nodes (this=0x630c6784, __nstart=0xb3dfc1dc, __nfinish=0xb3dfc1e0) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:526
#15 0x430c27b4 in std::_6::_Deque_base<KtreeSpace::Node*, std::_6::allocator<KtreeSpace::Node*> >::_M_initialize_map (this=0x630c6784, __num_elements=0) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:500
#16 0x409dd05c in _Deque_base (__num_elements=<optimized out>, __a=..., this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:385
#17 deque (__a=..., this=<optimized out>) at ../../../../../../../../obj/common/include/pfe/c++/bits/stl_deque.h:684
#18 KtreeSpace::Ktree::Ktree (this=0x630c6780, n=...) at ../../../../../../../../src/pfe/common/toolkits/ktree/ktree.cc:129
#19 0x409d2884 in Trinity_Ktree::Trinity_Ktree (this=<optimized out>, inst=1, name=..., root_addr=<optimized out>, nh_type=<optimized out>, hw_handle=<optimized out>, jnh_app=<optimized out>, jnh_app_root=<optimized out>) at ../../../../../../../../../../src/pfe/common/pfe-arch/trinity/toolkits/ktree/ktree_trinity.cc:121
#20 0x409d2d68 in ktree_alloc (ktreep=0xb3df6f30, inst=1, name=<optimized out>, root_addr=0x4917b310, nh_type=1 '\\001', hw_handle=0x595bd8c0, jnh_app=33566748, jnh_app_root=33566748) at ../../../../../../../../../../src/pfe/common/pfe-arch/trinity/toolkits/ktree/ktree_trinity.cc:3276
#21 0x40726974 in jnh_create_table_internal (table_inst=<optimized out>, name=0x43e6608c "IIF-TREE", nresults=1, n_indirect_results=0, skip_bits=8, ptrp=0x4917b3a0, ptrp_indirect=0x0, nh_type=<optimized out>, default_nh=0x45e86ad8 <JNH_DISCARD_IIF_LOOKUP>, jnh_app=33566748, tablep=0x4821e634) at ../../../../../src/pfe/common/pfe-arch/trinity/toolkits/jnh_core/jnh_table.c:2514
#22 0x40726e64 in jnh_create_table (inst=<optimized out>, name=<optimized out>, skip_bits=<optimized out>, ptr=0x449fbe48 <ucode_context+568>, nh_type=<optimized out>, default_nh=<optimized out>, jnh_app=<optimized out>, tablep=<optimized out>) at ../../../../../src/pfe/common/pfe-arch/trinity/toolkits/jnh_core/jnh_table.c:248
#23 0x40546f9c in jnh_iftable_init (inst=1, encap_type=0 '\\000', iftablep=0x5a03bff8, encap_flags=<optimized out>) at ../../../../../src/pfe/common/pfe-arch/trinity/toolkits/jnh/jnh_iftable.c:554
#24 0x405e6d14 in jnh_stream_create_features (parent_stream_id=<optimized out>, ifd_pm=<optimized out>, ifd_mpls_pm=<optimized out>, entryp=<optimized out>, inst=<optimized out>) at ../../../../../src/pfe/common/pfe-arch/trinity/toolkits/jnh/jnh_stream.c:3572
#25 jnh_stream_attach (inst=1, stream_id=0, etype=<optimized out>, count=<optimized out>, ifd_mpls_pm=0, ifd_pm=0, tbl_id=0, parent_stream=65535, comp_table_id=65535, parent_stream_id=0x0) at ../../../../../src/pfe/common/pfe-arch/trinity/toolkits/jnh/jnh_stream.c:888
#26 0x40133b68 in pfe_ifd_chip_setup (ifd=0xb3df0bf0, multi_ifd=<optimized out>) at ../../../../../src/pfe/common/toolkits/pfe/pfe_trinity.c:1535
#27 0x40107ad8 in pfe_ifd_attach (porttype=138, ifd=0xb3df0bf0) at ../../../../../src/pfe/common/toolkits/pfe/pfe_if.c:6818
#28 0x41913114 in ifrt_ifd_add_vector (mh=0x473ad688) at ../../../../../src/pfe/common/toolkits/if/ifrt_ifd.c:616
#29 0x4190df84 in ifrt_command_handler (ipcmsg=<optimized out>, pipe=0x47101b18) at ../../../../../src/pfe/common/toolkits/if/ifrt.c:535
#30 0x4190e2e4 in ifrt_msg_handler (pipe=0x47101b18, ipcmsg=0x4b4faa30) at ../../../../../src/pfe/common/toolkits/if/ifrt.c:620
#31 0x418d31c4 in pfeman_session_end (session=0x4b4fa8e0, cmd=<optimized out>) at ../../../../../src/pfe/common/applications/pfeman/pfeman_session.c:939
#32 0x418d3ad4 in pfeman_session_control (msg=<optimized out>, pipe=<optimized out>) at ../../../../../src/pfe/common/applications/pfeman/pfeman_session.c:712
#33 pfeman_session_msg_handler (msg=<optimized out>, pipe=<optimized out>) at ../../../../../src/pfe/common/applications/pfeman/pfeman_session.c:654
#34 pfeman_session_manager (pipe=0x47101b18, msg=0xb4854758) at ../../../../../src/pfe/common/applications/pfeman/pfeman_session.c:563
#35 0x418c90d8 in pfeman_rt_msg_handler (pipe=0x47101b18) at ../../../../../src/pfe/common/applications/pfeman/pfeman_rt_xdpc.c:453
#36 pfeman_rt_thread () at ../../../../../src/pfe/common/applications/pfeman/pfeman_rt_xdpc.c:1233
#37 0x40033b8c in thread_detach_tty (thread=<optimized out>) at ../../../../../src/pfe/ukern/common/thread.c:141
#38 0x40029608 in add_small_nodes (index=<optimized out>, heap=<optimized out>) at ../../../../../src/pfe/ukern/common/heap.c:5656
#39 heap_block_get_small_node (heap=0xbadc0ffe, size=1400136043, callers_pc=0x51202020) at ../../../../../src/pfe/ukern/common/heap.c:5100
#40 0x00000000 in ?? ()
(gdb)  



core-NPC10.gz.core.0:
#0  vpanic (format_string=0x446c28e4 "abort 0x%lx", ap=0x4b6ea4e0) at ../../../../../../../src/pfe/ukern/cpu-ppc/ppc603e_panic.c:66
#1  0x4004932c in panic (format_string=<optimized out>) at ../../../../../../../src/pfe/ukern/cpu-ppc/ppc603e_panic.c:82
#2  0x426353e8 in abort () at ../../../../../../../../src/pfe/common/toolkits/libstdc++/embedded/libstdc++-ukern.c:55
#3  0x42eae784 in __gnu_cxx::_6::__verbose_terminate_handler () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/vterminate.cc:53
#4  0x42eae614 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:43
#5  0x42eae650 in std::terminate () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:53
#6  0x42eae4b8 in __cxxabiv1::__cxa_rethrow () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_throw.cc:109
#7  0x42eae770 in __gnu_cxx::_6::__verbose_terminate_handler () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/vterminate.cc:83
#8  0x42eae614 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:43
#9  0x42eae650 in std::terminate () at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_terminate.cc:53
#10 0x42eae568 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=<optimized out>, dest=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/eh_throw.cc:76
#11 0x42eab7ac in operator new (sz=512) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/new_op.cc:63
#12 0x42eab2f8 in operator new[] (sz=<optimized out>) at ../../../../../../../../src/gpl-dist/gcc/jnpr/4.2.1/libstdc++-v3/libsupc++/new_opv.cc:37
#13 0x430a9a08 in google::protobuf::internal::RepeatedPtrFieldBase::Reserve (this=0xb2ffd2b0, new_size=65) at ../../../../../../../../src/external/bsd/protobuf-old/dist/src/google/protobuf//repeated_field.cc:51
#14 0x4171b034 in Add<google::protobuf::RepeatedPtrField<edge_stats>::TypeHandler> (this=<optimized out>) at ../../../../../../../../src/external/bsd/protobuf-old/dist/src/google/protobuf/repeated_field.h:841
#15 Add (this=<optimized out>) at ../../../../../../../../src/external/bsd/protobuf-old/dist/src/google/protobuf/repeated_field.h:1048
#16 add_edges (this=<optimized out>) at ../../../../../../../../src/pfe/common/applications/agent/fabric.pb.h:703
#17 alloc_pfe_to_pfe_edge_stats_handle (handle=<optimized out>) at ../../../../../../../../src/pfe/common/applications/agent/agent_fabric_wrapper.cc:59
#18 0x4246d9a4 in cmt_agent_fabric_queue_reap (size=<optimized out>, fabric_message_handle=<optimized out>, sensorp=<optimized out>) at ../../../../../src/pfe/common/applications/cmt/cmt_agent_fabric.c:273
#19 cmt_agent_fabric_reap (sensorp=0x58442d50, inst_handle=<optimized out>, TelemetryStream_handle=<optimized out>, buffer_size=5000) at ../../../../../src/pfe/common/applications/cmt/cmt_agent_fabric.c:418
#20 0x4176c98c in sensor_driver_reap (sensorp=<optimized out>, handlep=<optimized out>, instance=<optimized out>, TelemetryStream_handle=<optimized out>, reap_buffer=<optimized out>, reap_buffer_size=<optimized out>) at ../../../../../../../../src/pfe/common/applications/agent/sensor_driver.c:142
#21 0x41776d5c in sensor_tap_form_telemetry_msg (sensorp=0x58442d50, buffer=0x584f49e0 "\
\\rPE-D6-ULN-HSI\\020\
\\030", sizep=0x4b6eb700) at ../../../../../../../../src/pfe/common/applications/agent/sensor_tap.c:309
#22 0x41777134 in sensor_tap_read (sensorp=0x58442d50, args=<optimized out>) at ../../../../../../../../src/pfe/common/applications/agent/sensor_tap.c:667
#23 0x41777730 in sensor_read_and_export (sensorp=0x58442d50, args=0x0) at ../../../../../../../../src/pfe/common/applications/agent/sensor_tap.c:464
#24 0x41719304 in agent_db_iterate (fp=0x41777594 <sensor_read_and_export>, arg=0x0, is_jv=TRUE) at ../../../../../../../../src/pfe/common/applications/agent/agent_db.c:496
#25 0x4171475c in agent_periodic_update () at ../../../../../../../../src/pfe/common/applications/agent/agent.c:160
#26 0x4011f474 in pfe_stats_background () at ../../../../../src/pfe/common/toolkits/pfe/pfe_stats.c:773
#27 0x40033b8c in thread_detach_tty (thread=<optimized out>) at ../../../../../src/pfe/ukern/common/thread.c:141
#28 0x400efab0 in pfe_index_init (pfe_index_context=0xbadc0ffe, max_index=<optimized out>) at ../../../../../src/pfe/common/toolkits/pfe/pfe_if.c:12900
#29 0x88047ff9 in ?? ()

Solution

Analysis of the PFE cores collected during the problematic period indicates that the heap is fully exhausted.

This condition is caused by Next-Hop (NH) overscaling, with NH usage exceeding 1 million entries.

A review of the configuration and live core shows that VRF routes are installing 1:1 unicast next hops, which is driving the NH explosion.

This typically occurs when the chained composite next hop feature is not enabled.

Recommendation:

Enable the chained composite next-hop option to remove redundant unicast NHs and significantly reduce overall NH scale.

Use the following configuration in PE nodes:

regress@mts-bng-master# set routing-options forwarding-table chained-composite-next-hop ingress l3vpn

Logs:

NPC10(PE-D6-ULN-HSI jrca uart)# show heap

show heap

ID Base Total(b) Free(b) Used(b) % Name

-- ---------- ----------- ----------- ----------- --- -----------

0 470fb320 1825589348 4164872 1821424476 99 Kernel >>>>>>>>>>>>>>

1 b3dffb88 67108860 53539420 13569440 20 LAN buffer

2 bcdfffe0 52428784 52428784 0 0 Blob

3 b7dffb88 83886076 83886076 0 0 ISSU scratch

4 48f74190 458752 256192 202560 44 gzip stream

NPC10(PE-D6-ULN-HSI jrca uart)# ^[ sh nhdb summary

sh nhdb summary

Total number of NH = 1151719 >>>>>>>>>>>>>>

NPC10(PE-D6-ULN-HSI jrca uart)# sh nhdb summary detail

sh nhdb summary detail

Flushing 5001 cache lines

Type Count

--------- ---------

Discard 93

Reject 84

Unicast 573701 >>>>>>>>>>>>>>

Unilist 1

Indexed 0

Indirect 577140 >>>>>>>>>>>>>>

Hold 0

...

NPC10(PE-D6-ULN-HSI jrca uart)# sh route summary

sh route summary

IPv4 Route Tables:

Index Routes Size(b)

-------- ---------- ----------

Default 4076 570660

1 29 4056

2 17 2376

3 9 1256

4 5 696

6 5 696

7 582119 81496664 >>>>>>

8 197 27576

9 39 5456

10 6 836

11 6 836

12 331 46336

13 37 5176

14 5 696

15 37 5176

16 37 5176

17 1900 265996

18 3066 429236

19 228 31916

20 66 9236

21 183 25616

22 53 7416

23 60 8396

24 1077 150776

25 2095 293296

26 907 126976

28 482267 67517376

29 131 18336

36736 5 696

36737 5 696

Multiple routes having same protocol nexthop having separate copies of ucast NH.

IPv4 Route Table 7, BRAS-SPEEDY-ULN.7, 0x41000:

Destination NH IP Addr Type NH ID Interface

--------------------------------- --------------- -------- ----- ---------

223.255.231/24 118.98.52.77 Indirect 1054869 RT-ifl 0 ae0.0 ifl 323

IPv4 Route Table 7, BRAS-SPEEDY-ULN.7, 0x41000:

Destination NH IP Addr Type NH ID Interface

--------------------------------- --------------- -------- ----- ---------

223.255.240/22 118.98.52.77 Indirect 1358543 RT-ifl 0 ae0.0 ifl 323

223.255.240/24 118.98.52.77 Indirect 1358544 RT-ifl 0 ae0.0 ifl 323

223.255.241/24 118.98.52.77 Indirect 1358545 RT-ifl 0 ae0.0 ifl 323

223.255.242/24 118.98.52.77 Indirect 1358546 RT-ifl 0 ae0.0 ifl 323

223.255.243/24 118.98.52.77 Indirect 1358547 RT-ifl 0 ae0.0 ifl 323

NPC10(PE-D6-ULN-HSI jrca uart)# sh nhdb id 1054869 recursive

sh nhdb id 1054869 recursive

1054869(Indirect, IPv4, ifl:323:ae0.0, pfe-id:0, i-ifl:0:-)

8157(Unicast, IPv4->MPLS, ifl:323:ae0.0, pfe-id:0)

NPC10(PE-D6-ULN-HSI jrca uart)# sh nhdb id 1358543 recursive

sh nhdb id 1358543 recursive

1358543(Indirect, IPv4, ifl:323:ae0.0, pfe-id:0, i-ifl:0:-)

316312(Unicast, IPv4->MPLS, ifl:323:ae0.0, pfe-id:0)

NPC10(PE-D6-ULN-HSI jrca uart)# s^H

NPC10(PE-D6-ULN-HSI jrca uart)# sh nhdb id 1358544 recursive

sh nhdb id 1358544 recursive

1358544(Indirect, IPv4, ifl:323:ae0.0, pfe-id:0, i-ifl:0:-)

316313(Unicast, IPv4->MPLS, ifl:323:ae0.0, pfe-id:0)

Modification History

2026-02-17 : Article Created