[Contrail] How to perform packet capture on vif0/0 and filter it with encapsulated IP addresses

Description

This article describes how to capture packets on vif0/0 by using encapsulated IP addresses instead of generic routing encapsulation (GRE) header IP addresses.

Symptoms

For troubleshooting end-to-end flows between any given pair of IP addresses or networks wherein the packets are encapsulated within either MPLSoUDP or MPLSoGRE tunnels, users might want to filter the packets in a user packet dump based on internal IP addresses rather than the IP addresses of the MPLSoUDP or MPLSoGRE tunnels.

In the example illustration used for demonstrating how this can be done, there are two compute nodes that are either connected on the same switch or different switches within a Data Center or across Data Centers.

While troubleshooting communication between VM1 and VM3, the engineer has already been able to capture traffic on the TAP interface associated with VM1.

(vrouter-agent)[root@overcloud-contraildpdk-0 /]$ vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, Ig=Igmp Trap Enabled

vif0/0      PCI: 0000:3b:00.0 (Speed 10000, Duplex 1)
            Type:Physical HWaddr:f8:f2:1e:79:5b:d0 IPaddr:0.0.0.0
            Vrf:0 Mcast Vrf:65535 Flags:TcL3L2VpEr QOS:-1 Ref:14
            RX device packets:7441695  bytes:578288843 errors:0
            RX port   packets:5279607 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:5279607  bytes:446009683 errors:0
            TX packets:1362941  bytes:109114260 errors:0
            Drops:3788982
            TX port   packets:1362941 errors:0
            TX device packets:1362950  bytes:115692738 errors:0

vif0/1      PMD: vhost0
            Type:Host HWaddr:f8:f2:1e:79:5b:d0 IPaddr:10.0.0.26
            Vrf:0 Mcast Vrf:65535 Flags:L3DEr QOS:-1 Ref:13
            RX device packets:614715  bytes:39492604 errors:0
            RX queue  packets:614715 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:614715  bytes:39492604 errors:0
            TX packets:4075972  bytes:267655351 errors:0
            Drops:3
            TX queue  packets:4075972 errors:0
            TX device packets:4075972  bytes:267655351 errors:0

vif0/2      Socket: unix
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
            Vrf:65535 Mcast Vrf:65535 Flags:L3Er QOS:-1 Ref:3
            RX port   packets:603999 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:603999  bytes:52302282 errors:0
            TX packets:1439030  bytes:145567255 errors:0
            Drops:0
--SNIP--
vif0/8      PMD: tap4c15c815-82
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.1.3
            Vrf:6 Mcast Vrf:6 Flags:PL3L2DEr QOS:-1 Ref:12
            RX port   packets:156 errors:0 syscalls:156
            RX queue  packets:2 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:156  bytes:7152 errors:0
            TX packets:270  bytes:11938 errors:0
            ISID: 0 Bmac: 02:4c:15:c8:15:82
            Drops:0
            TX port   packets:164 errors:106 syscalls:163

(vrouter-agent)[root@overcloud-contraildpdk-0 /]$ vifdump vif0/8      
vif0/8      PMD: tap4c15c815-82
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mon8, link-type EN10MB (Ethernet), capture size 262144 bytes
06:55:57.922985 ARP, Request who-has 10.10.1.3 tell bd_core-b.west.unispherenetworks.com, length 28
06:55:57.922998 ARP, Reply 10.10.1.3 is-at 02:4c:15:c8:15:82 (oui Unknown), length 28
06:55:58.649049 IP 10.10.1.3 > 10.10.1.4: ICMP echo request, id 49156, seq 1658, length 64
06:55:59.649734 IP 10.10.1.3 > 10.10.1.4: ICMP echo request, id 49156, seq 1659, length 64
06:56:00.649782 IP 10.10.1.3 > 10.10.1.4: ICMP echo request, id 49156, seq 1660, length 64
06:56:01.650304 IP 10.10.1.3 > 10.10.1.4: ICMP echo request, id 49156, seq 1661, length 64
06:56:02.650855 IP 10.10.1.3 > 10.10.1.4: ICMP echo request, id 49156, seq 1662, length 64
06:56:03.651222 IP 10.10.1.3 > 10.10.1.4: ICMP echo request, id 49156, seq 1663, length 64

Note: Instead of the vifdump command, you can use tcpdump if the vRouter type is not dpdk.

In the above output, you can observe that the source machine is sending ping packets. However, the same is not visible on the TAP interface on VM3. Hence, it was decided to capture traffic on the BMS1 physical interface and BMS2 physical interface, that is, vif0 or P1P1.

Here is a sample packet capture between compute nodes 10.0.0.173 and 10.0.0.26. Since these packets are between compute nodes, Contrail vRouter is using MPLSoUDP as the default.

(vrouter-agent)[root@overcloud-contraildpdk-0 /]$ vifdump -i vif0/0
08:59:27.896003 f8:f2:1e:79:66:90 > f8:f2:1e:79:5b:d0, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 18523, offset 0, flags [none], proto UDP (17), length 130)
    10.0.0.173.61441 > 10.0.0.26.6635: UDP, length 102
        0x0000:  f8f2 1e79 5bd0 f8f2 1e79 6690 0800 4500
        0x0010:  0082 485b 0000 4011 1d4a 0a00 00ad 0a00
        0x0020:  001a f001 19eb 006e 0000 0003 b140 024c
        0x0030:  15c8 1582 02d8 d7c9 de0a 0800 4500 0054
        0x0040:  599c 0000 4001 0af3 0a0a 0104 0a0a 0103
        0x0050:  0000 5611 c004 2369 6067 6619 0000 0000
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000
08:59:27.896013 f8:f2:1e:79:5b:d0 > f8:f2:1e:79:66:90, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 59373, offset 0, flags [none], proto UDP (17), length 130, bad cksum 0 (->7db7)!)
    10.0.0.26.61553 > 10.0.0.173.6635: UDP, length 102
        0x0000:  f8f2 1e79 6690 f8f2 1e79 5bd0 0800 4500
        0x0010:  0082 e7ed 0000 4011 0000 0a00 001a 0a00
        0x0020:  00ad f071 19eb 006e 0000 0003 d140 02d8
        0x0030:  d7c9 de0a 024c 15c8 1582 0800 4500 0054
        0x0040:  9663 4000 4001 8e2b 0a0a 0103 0a0a 0104
        0x0050:  0800 4e11 c004 2369 6067 6619 0000 0000
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000

Note: If there is a large amount of traffic across the compute node, then it would be difficult to read through the hex dump and identify the packet of interest.

Solution

The Linux utility tcpdump or vifdump in Contrail has advanced filters that you can use to filter exact hex values within the packet by providing their relative location within the captured packets.

Example: vifdump vif0/0 -ennvl "proto UDP && ip[58:4] == 0x0a0a0104 || ip[62:4] == 0x0a0a0104" -xxx

This example pertains to capturing packets either destined to or originating from 10.10.1.4.

Decoding the Advanced Filters

How did we arrive at the filter in the above example?

In the example, proto UDP indicates that you want to filter UDP packets on vif0/0.

ip[x:y] indicates that the filtering should start from byte x for y bytes. For example, ip[58:4] would filter bytes 58 to 61 (first byte begins at 58).

How did we select the 58th byte as the IP address?

To be able to filter packets based on the source or destination IP address of the inner packet, you need to identify its relative location within the encapsulation header. In the case of MPLSoUDP, the inner packet source IP address starts from the 58th byte onward when counted from the outer IP header's starting byte. Similarly, the inner destination IP address field starts from the 62nd byte when counted from the outer IP header's starting byte.

Similarly, the MPLSoGRE packets will have encapsulated source and destination addresses starting from 40 and 44 bytes, respectively.

How to convert an IP address to a hexadecimal value?

Use any online IP to hex converter tool.

To filter MPLSoGRE packets with the inner source and destination IP addresses:

(vrouter-agent)[root@overcloud-contraildpdk-0 /]$ vifdump vif0/0 -ennvl "proto GRE && ip[40:4] == 0x0a0a0104 || ip[44:4] == 0x0a0a0104" -xx

To filter MPLSoUDP packets with the inner source and destination IP addresses:

(vrouter-agent)[root@overcloud-contraildpdk-0 /]$ vifdump vif0/0 -ennvl "proto UDP && ip[58:4] == 0x0a0a0104 || ip[62:4] == 0x0a0a0104" -xx

vif0/0      PCI: 0000:3b:00.0 (Speed 10000, Duplex 1)
tcpdump: listening on mon0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:59:27.896003 f8:f2:1e:79:66:90 > f8:f2:1e:79:5b:d0, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 18523, offset 0, flags [none], proto UDP (17), length 130)
    10.0.0.173.61441 > 10.0.0.26.6635: UDP, length 102
        0x0000:  f8f2 1e79 5bd0 f8f2 1e79 6690 0800 4500
        0x0010:  0082 485b 0000 4011 1d4a 0a00 00ad 0a00
        0x0020:  001a f001 19eb 006e 0000 0003 b140 024c
        0x0030:  15c8 1582 02d8 d7c9 de0a 0800 4500 0054
        0x0040:  599c 0000 4001 0af3 0a0a 0104 0a0a 0103
        0x0050:  0000 5611 c004 2369 6067 6619 0000 0000
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000
08:59:27.896013 f8:f2:1e:79:5b:d0 > f8:f2:1e:79:66:90, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 59373, offset 0, flags [none], proto UDP (17), length 130, bad cksum 0 (->7db7)!)
    10.0.0.26.61553 > 10.0.0.173.6635: UDP, length 102
        0x0000:  f8f2 1e79 6690 f8f2 1e79 5bd0 0800 4500
        0x0010:  0082 e7ed 0000 4011 0000 0a00 001a 0a00
        0x0020:  00ad f071 19eb 006e 0000 0003 d140 02d8
        0x0030:  d7c9 de0a 024c 15c8 1582 0800 4500 0054
        0x0040:  9663 4000 4001 8e2b 0a0a 0103 0a0a 0104
        0x0050:  0800 4e11 c004 2369 6067 6619 0000 0000
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000
08:59:28.895734 f8:f2:1e:79:5b:d0 > f8:f2:1e:79:66:90, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 59376, offset 0, flags [none], proto UDP (17), length 130, bad cksum 0 (->7db4)!)
    10.0.0.26.61553 > 10.0.0.173.6635: UDP, length 102
        0x0000:  f8f2 1e79 6690 f8f2 1e79 5bd0 0800 4500
        0x0010:  0082 e7f0 0000 4011 0000 0a00 001a 0a00
        0x0020:  00ad f071 19eb 006e 0000 0003 d140 02d8
        0x0030:  d7c9 de0a 024c 15c8 1582 0800 4500 0054
        0x0040:  9718 4000 4001 8d76 0a0a 0103 0a0a 0104
        0x0050:  0800 77cd c004 236a 27aa 7519 0000 0000
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000
08:59:28.895751 f8:f2:1e:79:66:90 > f8:f2:1e:79:5b:d0, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 18528, offset 0, flags [none], proto UDP (17), length 130)
    10.0.0.173.61441 > 10.0.0.26.6635: UDP, length 102
        0x0000:  f8f2 1e79 5bd0 f8f2 1e79 6690 0800 4500
        0x0010:  0082 4860 0000 4011 1d45 0a00 00ad 0a00
        0x0020:  001a f001 19eb 006e 0000 0003 b140 024c
        0x0030:  15c8 1582 02d8 d7c9 de0a 0800 4500 0054
        0x0040:  59b5 0000 4001 0ada 0a0a 0104 0a0a 0103
        0x0050:  0000 7fcd c004 236a 27aa 7519 0000 0000
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
vifdump: deleting vif 4351...

Note: Please refer KB109979 [juniper.net] for the behaviour in the latest contrail-tools versions.

Modification History

Added behaviour for latest contrail-tools