Description

This article will describe tools that are used to collect data and analyze Contrail vrouter memory issues.

Symptoms

In order to speed up problem resolution, the following information should be collected and attached when opening a support case.

Customers can run GDB (GNU Debugger) against already generated core dumps or capture them manually when memory leaks are suspected (for example, when you see disproportional vrouter memory consumption compared with number of flows).

Solution

Get the backtrace of an existing core dump

A unhandled exception during code execution can result in a core dump being generated. In Contrail Networking setups core dump files will be stored in /var/crashes directory.

A core dump file has the following naming format:
core.<process_name>.<process_pid>.<hostname>.<epoch_timestamp>

Example: core.contrail-vroute.215.compute1.1674161631

In general, when submitting a support ticket, it is recommended to run gdb against each existing core dump and provide the backtrace for faster resolution.

Run the below commands to collect the backtrace of a vRouter Agent process core dump on the Compute node it was generated.

1. Access the running vRouter Agent container and navigate into /var/crashes directory

[root@compute1]# docker exec -it contrail_vrouter_agent bash
(vrouter-agent)[root@compute1 /]$ cd /var/crashes/
(vrouter-agent)[root@compute1 /var/crashes]$ ls
core.contrail-vroute.215.compute1.1674161631


2. Open core file using 'gdb' utility, which requires two positional arguments.

(vrouter-agent)[root@compute1 /var/crashes]$ gdb /usr/bin/contrail-vrouter-agent core.contrail-vroute.215.compute1.1674161631


3. Once inside the gdb console, as denoted by the (gdb) prompt, issue the command bt or bt full to retrieve the core dump file backtrace.

[... output cropped ...]
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0  0x00007f8d085e33d7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install contrail-vrouter-agent-1912.L4.1-13.el7.x86_64
(gdb) bt
#0  0x00007f8d085e33d7 in raise () from /lib64/libc.so.6
#1  0x00007f8d085e4ac8 in abort () from /lib64/libc.so.6
#2  0x00007f8d085dc1a6 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007f8d085dc252 in __assert_fail () from /lib64/libc.so.6
#4  0x000000000160520c in KSyncBulkMsgContext::~KSyncBulkMsgContext() ()
#5  0x000000000162583a in KSyncSockUds::Init(boost::asio::io_service&, std::string const&, std::string const&) ()
#6  0x000000000795c000 in ?? ()
#7  0x000000000793e3c0 in ?? ()
#8  0x000000000793e3e0 in ?? ()
#9  0x0000000001eef220 in ?? ()
#10 0x00007f8d091b0edc in tbb::internal::padded_allocate(unsigned long, unsigned long) () from /lib/libtbb.so.2
#11 0x00007f8d091b10f1 in tbb::internal::NFS_Allocate(unsigned long, unsigned long, void*) () from /lib/libtbb.so.2
#12 0x0000000000ed9573 in SandeshTaskGroup::~SandeshTaskGroup() ()
#13 0x0000000000000000 in ?? ()


 

Manually generate vRouter Agent process core dump


It is possible to manually capture a core dump of the vRouter Agent running process, if there is suspected memory leak (but no core dump) using the gcore utility.

Note: This operation can potentially cause a temporary hiccup in traffic. Use it with caution, during a maintenance window.
 

1. Access the running vRouter Agent container and retrieve the PID of the contrail-vrouter-agent process. 

[root@compute1]# docker exec -it contrail_vrouter_agent bash
(vrouter-agent)[root@compute1 /]$
(vrouter-agent)[root@compute1 /]$ pidof contrail-vrouter-agent
215

2. Using the PID, generate a core file using the gcore utility. 
(vrouter-agent)[root@compute1 /]$ gcore 215

Note: The core file produced by gcore utility can have considerable size, exceeding few gigabytes. Ensure there is enough disk space to accommodate it.

Modification History

2019-08-29: Updated back trace steps.
2019-05-09: Added 'bt full' to list of information to collect.
2023-03-16: Article rewritten and improved to reflect procedure applicable on containerised Contrail Networking setups