The time it takes from the start of an HTTP request until the time the last byte arrives at the client is the full request response. If you eliminate the client entirely it is the time after the client sends the HTTP request (ending in \n\n) until the time the server responds with the first byte of the response.
In practice, rarely is the network bandwidth or the client bandwidth the problem. The performance culprit usually involves the word "database". i.e. under-provisioned database hardware, low database IOPS (HD vs SSD vs NVME), a poor performing query, or many fast queries that add up to be slow. Rarely is it an inefficient python algorithm. Sometimes it is row-level database lock contention.
Every API response contains the headers X-API-Time and X-API-Total-Time where X-API-Time <= X-API-Total-Time. X-API-Time is set by the middleware function TimingMiddleware.process_response(). As of writing this TimingMiddleware appears very close to the top of the MIDDLEWARE settings list. Middleware process_response are processed in the reverse order, thus, TimingMiddleware.process_response() runs very close to the end of the middleware stack and therefore should result in X-API-Time =~ X-API-Total-Time. If X-API-Total-Time - X-API-Time is large then their is a performance problem in a small slice of code in between the middleware response processing and the DRF response processing. If X-API-Total-Time and X-API-Time are larger, then there is likely a performance problem and you need to enable SQL_DEBUG to determine if it is database related.
When SQL_DEBUG = True the headers X-API-Query-Time and X-API-Query-Count will be included in the response. A high X-API-Query-Time and X-API-Query-Count indicates there are many small queries that add up. A high X-API-Query-Time and low X-API-Query-Count indicates there are 1 or more expensive queries. A low X-API-Query-Time indicates you should look elsewhere (i.e. python algorithm or slow network).
TL;DR Look at headers X-API-Time and X-API-Total-Time. If large then enable SQL_DEBUG = True and look at X-API-Query-Time and X-API-Query-Count. If X-API-Query-Time or X-API-Query-Count are large then it is likely a database issue.
This is a useful tool for examining per-api endpoint SQL queries, performance, headers, requests, signals, cache, logging, and more.
To enable DDT, you need to set your INTERNAL_IPS to the IP address of your load balancer. This can be overridden by creating a new settings file beginning with local_ in awx/settings/ (e.g. local_overrides.py).
This IP address can be found by making a GET to any page on the browsable API and looking for information like this in the standard output:
awx_1 | 14:42:08 uwsgi.1 | 172.18.0.1 GET /api/v2/tokens/ - HTTP/1.1 200
Allow this IP address by adding it to the INTERNAL_IPS variable in your new override local settings file, then navigate to the API and you should see DDT on the
right side. If you don't see it, make sure to set DEBUG=True.
Note that enabling DDT is detrimental to the performance of AWX and adds overhead to every API request. It is recommended to keep this turned off when you are not using it.