Description

This KB describes decoding a vrouter agent crash using a stand-alone agent container.

Symptoms

In a vRouter agent crash, it may be necessary to decode the crash file using GDB (GNU Debugger) within the matching container. This will help identify the root cause of the crash and enable remediation to prevent similar occurrences in the future.

Solution

The solution to decode the crash file of the vrouter agent involves bringing up a container that matches the crashed agent and downloading the contrail-debug container. Once downloaded, the contrail-vrouter-agent.debug file needs to be extracted, and the container needs to be brought up without the controller. Finally, GDB can be run to extract the backtrace from the crash file. This process can help diagnose the issue and provide insights into how to fix it.

Pre-requisites
  1. Docker host which can be a VM or a bare-metal server installed with any Linux-based OS
  2. Core file from the node where the crash or manual core has been collected.
  3. Access to enterprise-hub.juniper.net 

Procedure:

  1. On the docker host, login to the enterprise-hub.juniper.net using the command

    # docker login [email protected]

    enter your login credentials when prompted 

  2. Download the debug container image from enterprise-hub.juniper.net

    For example:  If the crash file was generated by vrouter-agent with the version 2011.L4.395, then the command would be:

    You may have to request the customer to share the exact version and build ID. This info can be found from the output of contrail-status

    docker pull enterprise-hub.jnpr.net/contrail-container-prod/contrail-debug:2011.L4.395

  3. Similarly, download the agent container with the same version tag:

    docker pull enterprise-hub.jnpr.net/contrail-container-prod/contrail-vrouter-agent:2011.L4.395

  4. Find the debug binary for the agent in the host's file system:

    find / -name contrail-vrouter-*
    /var/lib/docker/overlay2/15729dd8fa37327983b4ceb448b2232b332c9b0bf4c487352468482c5ec9f17a/diff/usr/lib/debug/usr/bin/contrail-vrouter-agent.debug

  5. Copy the crash file and debug file from the above-identified path to the /var/crashes/

    cp <crash file> /var/crashes
    cp var/lib/docker/overlay2/15729dd8fa37327983b4ceb448b2232b332c9b0bf4c487352468482c5ec9f17a/diff/usr/lib/debug/usr/bin/contrail-vrouter-agent.debug /var/crashes/

  6. Start the agent container without using default entrypoint which requires the vrouter-agent to find vhost0 interface:

    docker run -v /var/crashes:/var/crashes --entrypoint /bin/bash -it enterprise-hub.jnpr.net/contrail-container-prod/contrail-vrouter-agent:2011.L4.395

  7. Copy the debug binary from the /var/crashes/ to /usr/bin/

    cp /var/crashes/contrail-vrouter-agent.debug /usr/bin/

  8. Run the gdb on the crash file:

    gdb contrail-vrouter-agent /var/crashes/core.contrail-vroute.1704667.wesnc01node066.1676183260 

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/contrail-vrouter-agent...Reading symbols from /usr/bin/contrail-vrouter-agent...(no debugging symbols found)...done.
(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 1704667]
[New LWP 2357133]
[New LWP 1711039]
[New LWP 2357137]
[New LWP 2357130]
[New LWP 1704680]
[New LWP 2198873]
[New LWP 2357138]
[New LWP 2198872]
[New LWP 1704684]
[New LWP 1704685]
[New LWP 2357132]
[New LWP 1704686]
[New LWP 1704679]
[New LWP 1711040]
[New LWP 2357134]
[New LWP 1704677]
[New LWP 1704683]
[New LWP 2965676]
[New LWP 2357135]
[New LWP 2965677]
[New LWP 2357131]
[New LWP 2933586]
[New LWP 1704678]
[New LWP 2933587]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007ffa7ea7f4f3 in std::_Rb_tree_increment(std::_Rb_tree_node_base const*) () from /lib64/libstdc++.so.6
Missing separate debuginfos, use: debuginfo-install contrail-vrouter-agent-2011-395.el7_9.x86_64
(gdb) file /var/crashes/contrail-vrouter-agent.debug 
warning: core file may not match specified executable file.
Reading symbols from /var/crashes/contrail-vrouter-agent.debug...done.
(gdb) bt
#0  0x00007ffa7ea7f4f3 in std::_Rb_tree_increment(std::_Rb_tree_node_base const*) () from /lib64/libstdc++.so.6
#1  0x00000000016e9f02 in BFD::Session::CallStateChangeCallbacks (this=0xeb76e10, key=..., new_state=@0xec0c0: <error reading variable>) at controller/src/bfd/bfd_session.cc:297
#2  0x0000000002a501e0 in ?? ()
#3  0x0000000000e16d84 in VrouterControlInterface::Process (this=0x116a5e00, pkt=...) at controller/src/vnsw/agent/pkt/vrouter_interface.h:157
#4  0x00007fffef05bae0 in ?? ()
#5  0x00007fffef05bbe0 in ?? ()
#6  0x00007fffef05bb60 in ?? ()
#7  0x0000000002a500a0 in ?? ()
#8  0x00007fffef05bd20 in ?? ()
#9  0x00000000016f6dbe in address_v6 (other=<error reading variable: can't compute CFA for this frame>, this=<error reading variable: can't compute CFA for this frame>)
    at /usr/include/boost/asio/ip/impl/address_v6.ipp:61
#10 address (other=<error reading variable: can't compute CFA for this frame>, this=<error reading variable: can't compute CFA for this frame>) at /usr/include/boost/asio/ip/impl/address.ipp:56
#11 SessionKey (this=<error reading variable: can't compute CFA for this frame>) at controller/src/bfd/bfd_common.h:85
#12 value (this=<error reading variable: can't compute CFA for this frame>) at /usr/include/boost/bind/bind.hpp:112
#13 storage1 (this=<error reading variable: can't compute CFA for this frame>) at /usr/include/boost/bind/storage.hpp:40
#14 storage2 (this=<error reading variable: can't compute CFA for this frame>) at /usr/include/boost/bind/storage.hpp:76
#15 list2 (this=<error reading variable: can't compute CFA for this frame>) at /usr/include/boost/bind/bind.hpp:275
#16 bind_t (this=0xeb76e10) at /usr/include/boost/bind/bind.hpp:864
#17 invoke<boost::_bi::bind_t<boost::_bi::unspecified, boost::function<void(const BFD::SessionKey&, const BFD::BFDState&)>, boost::_bi::list2<boost::_bi::value<BFD::SessionKey>, boost::_bi::value<BFD::BFDState> > >, boost::_bi::bind_t<boost::_bi::unspecified, boost::function<void(const BFD::SessionKey&, const BFD::BFDState&)>, boost::_bi::list2<boost::_bi::value<BFD::SessionKey>, boost::_bi::value<BFD::BFDState> > > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:39
#18 boost::asio::detail::completion_handler<boost::_bi::bind_t<boost::_bi::unspecified, boost::function<void (BFD::SessionKey const&, BFD::BFDState const&)>, boost::_bi::list2<boost::_bi::value<BFD::SessionKey>, boost::_bi::value<BFD::BFDState> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (
    owner=0x7fffef05bc00, base=<optimized out>) at /usr/include/boost/asio/detail/completion_handler.hpp:67
#19 0x0000000002a501e0 in ?? ()
#20 0x000000000000006e in ?? ()
#21 0x00007fffef05bb60 in ?? ()
#22 0x0000000000000000 in ?? ()

Modification History

2023-04-07: Initial publication

Related Information