feat: Pre-Fetch Streamer Messages (#269)

Historical streamer messages are not fetched in coordinator prior to the
IndexerRunner call. As a result, we cannot apply the latency saving
benefits of having coordinator cache the streamer message for use by
runner. Instead, we want to pre fetch from S3 so that runner invocations
don't have to wait for the streamer message to be retrieved from S3.

In addition, it's possible for real-time messages to backup temporarily
preventing the cached message from being used. So, we also want to
prefetch any messages which aren't found in the cache.

The new workflow works by having two loops for each worker thread: a
producer and a consumer. The producer loads a promise for fetching the
block (either from cache or S3) into an array. The consumer then removes
the first element from the array and processes it, deleting the streamer
message upon success. While one block is being processed, the other
blocks are being fetched. This ensures that wait time is minimal. The
producer loop attempts to keep the array as close to full as possible.

docker-compose.yml

@@ -88,7 +88,23 @@ services:
       HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
       HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
       HASURA_GRAPHQL_AUTH_HOOK: http://hasura-auth:4000/auth
+  grafana:
+    image: grafana/grafana
+    volumes:
+      - grafana:/var/lib/grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=secret
+
+  prometheus:
+    image: prom/prometheus
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+    ports:
+      - "9090:9090"
 
 volumes:
   postgres:
   redis:
+  grafana:

prometheus.yml

@@ -0,0 +1,7 @@
+global:
+  scrape_interval: 1s
+
+scrape_configs:
+  - job_name: 'queryapi-runner'
+    static_configs:
+      - targets: ['host.docker.internal:9180']