Description

After upgrading Contrail Cloud (referred to as CC henceforth in the article), users may notice that some of the VMIs are missing in the Horizon UI but still exist in the Contrail WebUI or Neutron port list.

This scenario may happen when the Neutron server flaps between ML2 and Contrail for example, or when race conditions exist during the update of a VM's instance_info_cache , both of which would corrupt the instance_info_cache .

OpenStack has a periodic monitoring task called _heal_instance_info_cache , whose job is to detect such errors in the ​instance_info_cache and recover the cache as appropriate. However, this task does not work as designed, which is why some of the VMIs are seen to be missing.

This article describes the OpenStack defect and the fix expected in forthcoming OpenStack releases.

 

Symptoms

The issue is demonstrated as follows:

  • Initially four (4) ports are seen in a particular VM instance. 

$ nova interface-list a64ed18d-9868-4bf0-90d3-d710d278922d
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+
| ACTIVE | 6c230305-43f8-42ec-9936-61fe67551168 | 96343d33-5dd2-4289-b0cc-e6c664c2ddd9 | 10.0.0.3,fdda:5d77:e18e:0:f816:3eff:fee8:3333 | fa:16:3e:e8:33:33 |
| ACTIVE | 71e6c6ad-8016-450f-93f2-75e7e014084d | 9e702a96-2744-40a2-a649-33f935d83ad3 | 172.24.4.16,2001:db8::c | fa:16:3e:6d:dc:85 |
| ACTIVE | a74c9ee8-c426-48ef-890f-3988ecbe95ff | 9e702a96-2744-40a2-a649-33f935d83ad3 | 172.24.4.15,2001:db8::e | fa:16:3e:cf:0c:e0 |
| ACTIVE | b89d6863-fb4c-405c-89f9-698bd9773ad6 | 96343d33-5dd2-4289-b0cc-e6c664c2ddd9 | 10.0.0.5,fdda:5d77:e18e:0:f816:3eff:fe53:231c | fa:16:3e:53:23:1c |
+------------+--------------------------------------+----------------------------

  • These four ports exist in the metadata file of that VM. 

$ sudo virsh dumpxml instance-00000002 | grep -i tap
      <target dev='tap6c230305-43'/>
      <target dev='tapb89d6863-fb'/>
      <target dev='tapa74c9ee8-c4'/>
      <target dev='tap71e6c6ad-80'/>​

  • ​After the instance_info_cache is corrupted, one VMI, tap6c230305-43 , is reported to be missing from the VM's XML file. 

​$ sudo virsh dumpxml instance-00000002 | grep -i tap
      <target dev='tapb89d6863-fb'/>
      <target dev='tapa74c9ee8-c4'/>
      <target dev='tap71e6c6ad-80'/>​

  • However, Neutron/Contrail still shows that four interfaces are available. Note that the nova interface-list command calls Neutron to request for information. 

$nova interface-list a64ed18d-9868-4bf0-90d3-d710d278922d
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+
| ACTIVE | 6c230305-43f8-42ec-9936-61fe67551168 | 96343d33-5dd2-4289-b0cc-e6c664c2ddd9 | 10.0.0.3,fdda:5d77:e18e:0:f816:3eff:fee8:3333 | fa:16:3e:e8:33:33 |
| ACTIVE | 71e6c6ad-8016-450f-93f2-75e7e014084d | 9e702a96-2744-40a2-a649-33f935d83ad3 | 172.24.4.16,2001:db8::c | fa:16:3e:6d:dc:85 |
| ACTIVE | a74c9ee8-c426-48ef-890f-3988ecbe95ff | 9e702a96-2744-40a2-a649-33f935d83ad3 | 172.24.4.15,2001:db8::e | fa:16:3e:cf:0c:e0 |
| ACTIVE | b89d6863-fb4c-405c-89f9-698bd9773ad6 | 96343d33-5dd2-4289-b0cc-e6c664c2ddd9 | 10.0.0.5,fdda:5d77:e18e:0:f816:3eff:fe53:231c | fa:16:3e:53:23:1c |
+------------+--------------------------------------+--------------------------------------+-----------------------------------------------+-------------------+

 

Solution

If the instance info_cache is corrupted, the _heal_instance_info_cache task is meant to fix it via the information provided by Neutron because Neutron is the source of truth for all ports attached to VMs. But it is only refreshing the cache based on its current contents—that is, it is doing nothing. This is the reason for the VMI to be missing.

The _heal_instance_info_cache periodic monitoring task of Nova is implemented based on the port list from active memory (Nova DB), and not from the Neutron server's port list. If the instance_info_cache is not correct, the Neutron server should be the actual trust anchor to correct the instance_info_cache via the _heal_instance_info_cache .

This issue has been reported by different parties/vendors and an OpenStack defect has been recorded in https://bugs.launchpad.net/nova/+bug/1751923.  ​It has been fixed in OpenStack/Nova 19.0.0.0rc1, and may be adopted in a future CC release.

 

Meanwhile, to resolve the problem of missing VMIs in the VM's XML file, you need to rebuild the instance_info_cache when it is found to be inconsistent with the information in the Neutron server.