Description

This KB article indicates VRR (Virtual Route Reflector) issue arises due to reboot of server in case of CSO HA 5.1.1 environment and a way to recover VRR.

Symptoms

Tenant addition failure :

Task: 
Mar 5, 2020, 1:49:57 AMadding customer admin to tenant test4
Mar 5, 2020, 1:49:58 AMCreate tenant admin success.
Mar 5, 2020, 1:49:59 AMIAM granted group access to project test4 ccb4fcfb-32ab-41ff-9efd-b4008fbb9625
Mar 5, 2020, 1:50:02 AMCheck for customer object existence default-domain:test4:test4
Mar 5, 2020, 1:50:02 AMCreating customer default-domain:test4:test4
Mar 5, 2020, 1:50:02 AMDisable Root CA to customer ['default-domain', u'test4', u'test4']
Mar 5, 2020, 1:50:03 AMCreated customer default-domain:test4:test4 9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e
Mar 5, 2020, 1:50:04 AMOnboarding tenant in vims [] with tenant_type small
Mar 5, 2020, 1:50:07 AMadding properties to customer [u'default-domain', u'test4', u'test4']
Mar 5, 2020, 1:50:10 AMstarting Tenant creation in vrr
Mar 5, 2020, 1:55:39 AMTenant creation in VRR returned a failure status. Error message {u'status': u'FAILURE', u'output': u'. Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 280, in vrr_tenant_op_aux\n create_vrr_tenant_task.tenant_vrr_operation(op_flag=create)\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 150, in tenant_vrr_operation\n raise Exception\nException\n', u'X-Request-Id': u'UI_e1f44dea-3729-8143-3471-9363a61f783d,4YOI,t0mZ', u'rpc_type': u'CREATE_VRR_TENANT', u'request_id': u'3fff5d16-b142-40ef-856d-585b02fb0bb1'}
Mar 5, 2020, 2:00:59 AMTask Fail Tenant creation in VRR returned a failure status. Error message {u'status': u'FAILURE', u'output': u'. Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 280, in vrr_tenant_op_aux\n create_vrr_tenant_task.tenant_vrr_operation(op_flag=create)\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 150, in tenant_vrr_operation\n raise Exception\nException\n', u'X-Request-Id': u'UI_e1f44dea-3729-8143-3471-9363a61f783d,4YOI,t0mZ', u'rpc_type': u'CREATE_VRR_TENANT', u'request_id': u'3fff5d16-b142-40ef-856d-585b02fb0bb1'}
Mar 5, 2020, 2:00:59 AMTask complete
Mar 5, 2020, 2:00:59 AMTask Fail Tenant creation in VRR returned a failure status. Error message {u'status': u'FAILURE', u'output': u'. Traceback (most recent call last):\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 280, in vrr_tenant_op_aux\n create_vrr_tenant_task.tenant_vrr_operation(op_flag=create)\n File "/usr/local/lib/python2.7/site-packages/routing_manager_services/tasks/vrr_tenant_task.py", line 150, in tenant_vrr_operation\n raise Exception\nException\n', u'X-Request-Id': u'UI_e1f44dea-3729-8143-3471-9363a61f783d,4YOI,t0mZ', u'rpc_type': u'CREATE_VRR_TENANT', u'request_id': u'3fff5d16-b142-40ef-856d-585b02fb0bb1'}
Mar 5, 2020, 2:00:59 AMJob Fail Unknown id: Error: oper 3 url /tssm/customer/9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e body {"customer":{"state": "failed", "uuid": "9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e"}} response {"status_code": "404", "error_tag": "No ID Found", "error_message": "The resource with id 9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e of type customer does not exist", "error_app_message": "Unknown id: 9a7fefdb-ab68-4dbc-994e-b6b6ef79dd9e", "error_diag": "This error occurs when the resource with the specified id is not found", "error_code": "40027"}

Solution

  1. Check the status of all vm using command - virsh list --all and vrr should be in running status
  2. Run  ./components_health.sh - to check health status of rest of components.
  3. In certain scenarios listed in  KB35551 [juniper.net]  rabbitmq can be in unhealthy state post reboot of server
  4. Login to startup server and navigate to CSO install directory
  5. Run recovery of rabbitmq first ( if rabbitmq is in unhealthy state)
  6. Recovery of rabbitmq should restart central and regional msvm ( Refer  KB35551 [juniper.net] )
  7. Run recovery for vrr
root@startupserver1:/Contrail_Service_Orchestration_5.1.1# ./recovery.sh
***************
This tool assists you recover your CSO setup.
***************


Following components can be recovered

1: mariadb
2: vrr
3: icinga
4: rabbitmq

Specify one of the component to recover (In Number) : 2
INFO     Started recovering vrr component at 2020-03-05 11:54:42.336216 ...
INFO     VRR recovery is initiated...
ERROR    Vrr - X.X.X.X is unhealthy
ERROR    Vrr - X.X.X.X is unhealthy
INFO     Recovery takes time, please be patient
INFO     VRR recovery started. Please wait...
INFO     VRR config sync job has been initiated, please monitor from CSO UI for the status of the job...
INFO     Completed recovering vrr component at 2020-03-05 11:58:01.668640 .
INFO     Time taken to recover 0:03:19.332424