nginx: Increase the number and size of proxy buffers (PROJQUAY-6950) (q…

…uay#3303)

* nginx: Increase the number and size of proxy buffers (PROJQUAY-6950)
From [nginx documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering):

> When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives.
> If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk.
> Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.
>
> When buffering is disabled, the response is passed to a client synchronously, immediately as it is received. nginx will not try to read the whole response from the proxied server.
> The maximum size of the data that nginx can receive from the server at a time is set by the proxy_buffer_size directive.

By default, the value of `proxy_buffer_size` directive, if not set in the nginx configuration, is equal to one memory page which on most platforms equals 4 KiB of memory. When `FEATURE_PROXY_STORAGE` is turned on and STS driver is used, the size of headers reaches that limit of 4 KiB and, if it surpasses it, nginx will error out and will not process the request. With this PR we set the buffers to an adequate size so that proxy requests are properly processed by nginx.

* Add additional nginx directive to make buffers work

Makefile

@@ -1,6 +1,6 @@
 SHELL := /bin/bash
 DOCKER ?= docker
-DOCKER_COMPOSE ?= $(DOCKER)-compose
+DOCKER_COMPOSE ?= $(DOCKER) compose
 
 export PATH := ./venv/bin:$(PATH)
 

conf/nginx/server-base.conf.jnj

@@ -98,6 +98,8 @@ location ~ ^/_storage_proxy/([^/]+)/([^/]+)/([^/]+)/(.+) {
 
     proxy_buffering off;
     proxy_request_buffering off;
+    proxy_buffer_size 32k;
+    proxy_buffers 4 32k;
 
     proxy_read_timeout 60s;
 }
@@ -171,6 +173,8 @@ location ~ /v2/([^/]+)(/[^/]+)+/blobs/ {
 
     proxy_buffering off;
     proxy_request_buffering off;
+    proxy_buffer_size 32k;
+    proxy_buffers 4 32k;
     proxy_read_timeout 2000;
     proxy_send_timeout 2000;
     proxy_temp_path /tmp 1 2;