API calls slow responses were observed while

++ While fetching the instances from horizon UI 

++ Executing the OpenStack list commands for ports/network-trunk list etc. 

++ The issue causes the problem with regards to accessing the instances from Horizon UI which has recursive calls to contrail-apis like (VMIs, instance-ips etc.)

++ User listing the tenants from horizon UI is delayed. 

Troubleshooting Steps :-

++ Execute the openstack CLI commands with time option.

# time openstack port list

e.g. output

real 0m10.291s  => Time span of 10 seconds

user 0m3.969s

sys   0m0.180s

Note : real is the actual amount of time it took to run the command

++ Look into the log "neutron-api.server" and check the call "neutron.wsgi" 

++ grep for the pattern with one request-id e.g. 

req-2758ae81-5062-40c5-b03d-33e80cb69e7b 

++ Check the trace for the request-id in neutron-api

++ The trace will suggest the time span of the entire call request time

e.g. from neutron-api.server

INFO neutron.wsgi [req-2758ae81-5062-40c5-b03d-33e80cb69e7b "GET /v2.0/ports?tenant_id=<> HTTP/1.1" status: 200 len: 1035588 time: 10.1136715. => (Entire time responds)

++ Check the same request-id in contrail-api logs to see the entire sub-calls requests time.

Note :- The purpose of this troubleshooting steps is to help determine and break the requests time between neutron-api and contrail-api