vrouter coredump files contain the address space (memory) of a vrouter agent process when it terminates unexpectedly, and are usually triggered by the kernel in response to vrouter agent crashes. From the coredump files, experienced engineers/developers can find important clues to the problem's cause because these files retain a post-mortem snapshot of the vrouter agent's state at the time of the crash, and come up with fixes.

One problem that is faced is reproducing the exact coredump file in the lab environment so that developers/engineers can look at the running setup and explore possible triggers for the problem. Unfortunately, it is usually difficult to trigger the same coredump.

In this article, we introduce a method that uses a script to dump a backtrace of all threads of a running vrouter agent process with gdb, so that the backtrace can be monitored and collected in real time.

Even without a coredump generated in the lab test, we can still run gdb on the running vrouter agent process to collect the backtrace in real time:

gdb --batch -ex "thread apply all bt" -p `pidof contrail-vrouter-agent` >> logfile.log

• The --batch option instructs gdb to run in "batch mode."

• The -ex option gives commands that you typically run under gdb, for example "bt," "bt all," "thread 2," and so on. Here we give thread apply all bt to retrieve the backtrace of all threads simultaneously.

• The -p option gives the process ID. Here we use pidof to retrieve the pid from the process name, which is contrail-vrouter-agent in Contrail compute.

With this " batch " mode gdb command, real-time backtraces of all threads of the running vrouter agent process can be collected in a file. With the following one-liner shell script, we can keep collecting data when the test is ongoing:

while :; do date; echo "pass 40s"; \
gdb --batch -ex "thread apply all bt" -p `pidof contrail-vrouter-agent` >> logfile.log\
sleep 30; done

Developers/engineers can then check the logfile.log anytime for any suspicious data.