There was many abnormal log related to PFE/Next-hop, then FPC reboot, had core-dump.
fpc1 jnh_partition_ok_per_inst(5016): jnh_expand_partition failed, inst 1, jnh_app unilist-selector, dwords 65536
fpc1 trinity_nh_unilist_resource_check(3251): Insufficient JNH_APP_UNILIST_SELECTOR memory
fpc1 PFE_ERROR_NOT_FOUND: NH: nh_id:1048711, proto:ipv4, type:Unilist not installed in H/W(resource low)
fpc1 nh_indirect_initialize:962Failed to find Target 1048711 for indirect 1048779 creating a discard
fpc1 nh_indirect_get_dependency:1035Failed to get nh 1048779 dependency to target NH 1048711
fpc1 jnh_partition_ok_per_inst(5016): jnh_expand_partition failed, inst 1, jnh_app ktree, dwords 65536
fpc1 jnh_add_prefix_table(474): PFE 255 Table Partition, ktree does not have enough room
fpc1 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_change failed
fpc1 PFE_ERROR_FAIL_OPERATION: route process failed
fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_change failed
fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
fpc0 jnh_partition_ok_per_inst(5016): jnh_expand_partition failed, inst 0, jnh_app unilist-selector, dwords 65536
fpc0 trinity_nh_unilist_resource_check(3251): Insufficient JNH_APP_UNILIST_SELECTOR memory
This issue is related to internal PR 1703798, which was fixed via PR 1706680.
We can see multiple LSP flaps happening before the PFE memory errors are seen, which induces high churns.
These logs are a result of higher scale testing in newer platforms and changes to memory allocation algorithm.
The fixes are basically enhancements to help debugging certain failures related to out-of memory conditions.
RCA:
++++++++++++++++++++++++++++++++++++
The problem seems to be around how the memory and the pages are getting allocated and freed during the churn. Since, counters are allocated and freed at the same time, we are allocating and freeing new page(s) often. This leads to frequent allocation and freeing of dma buffers which make the ager list longer. If the churn is very high, and the system is busy, it may take a while to age/free these buffers.
One potential solution is to reuse the dma buffer from the freelist prior to getting from the heap. Since DMA may be in progress, we can reuse the buffers that are sitting for that time period on the freelist.
The other problem that needs to be fixed is preventing the core. With l3 config, we don't seem to core but only with the mpls config that is being tested. We need to put some checks in ifl/iff create/attach to make sure we don't assert if there was a memory allocation failure.
In summary this is a scale/performance related issue attributed to ifl churn, and optimization in memory management and graceful handing of memory allocation failures are the fixes that we need.