Description

root@re0> show system core-dumps no-forwarding
re0:
--------------------------------------------------------------------------
-rw-r--r-- 1 root root  2269234387 May 5 17:23 /var/core/re0/vmcore.2023-05-05_17-15-57.tar.gz
-rw-r--r-- 1 root root  2289540249 May 6 06:38 /var/core/re0/vmcore.2023-05-06_06-31-39.tar.gz <<<<<<<<<

Symptoms

vmcore is generated 

Solution

Root Cause:

From Junos 22.2 - 22.4 codes, a series of changes were made to the JTD (Junos Tunnel Driver) kernel module to address non-scalable log-based debugging, and to provide as much context information as possible when an error occurs; One of the PRs (internal) is associated with these changes.

In the Linux kernel, all incoming/outgoing packets are passed around in SKBs (Socket Kernel Buffers). SKBs have a reference count associated with them; when an SKB is freed, it’s reference count is decremented by 1, and only if it is now 0 is the SKB freed. In the EVO networking stack, all incoming & outgoing WAN i/f packets have to go through JTD. For outgoing packets, the JTD driver as part of its processing may make Linux calls which can result in the SKB being freed. To prevent this from happening, that internal PR was raised to effectively bump up the reference count on the SKB prior to making such calls so that if such calls were not successful, the SKB would still be valid for the purposes of providing detailed qualifying information about the error which had occurred. Unfortunately, there are places in the Linux kernel that are not tolerant of SKBs having a reference count >1 (cloned SKBs) and make calls to BUG/BUG_ON causing the kernel to panic in such cases; this is what we are seeing in our case.

 The resolution to this issue is to undo the SKB reference increment logic associated with making these Linux calls in JTD; this is at the expense of having very much reduced context information should these Linux calls detect an error and free the SKB proper.

Fixed releases:
evo:22.3R2-EVO evo:22.3R3-EVO evo:22.3X80-D30-EVO evo:22.4R2-EVO evo:22.4R2-S1-EVO evo:23.1R1-EVO

 

Modification History

Hello Christine, 

Thank you for the follow-up. It seems the information did not save when I created it last time. I have updated it once again. Hopefully it is visible now. I have also changed the status to validated now.