Requests to load balancer not sent to the same application server

[siebrand]

We have the following set up:

• AWS auto-scaling group of image recognition servers with HTTP and the latest patches submitted here (2 instances)
• HTTPS termination at an AWS Application Load Balancer with two health instances in a target group.

When I interactively request "Auto tag images" in the Elvis pro client for an image, I see the following happening:

Server 1 logs:

Mar 22 12:35:52 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:35:52.651] [INFO]  HTTP Server started at port: 9090
Mar 22 12:35:52 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:35:52.654] [INFO]  Recognize imported files on import: true
Mar 22 12:35:52 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:35:52.654] [INFO]  REST API enabled: true
Mar 22 12:35:52 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:35:52.654] [INFO]  Ping endpoint enabled at: /SupaSekretEndpoint
Mar 22 12:35:52 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:35:52.654] [INFO]  Requests are logged at: /var/log/elvis-image-recognition/access.log
Mar 22 12:36:53 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:53.993] [INFO]  API call received: POST "/api/recognize" with body: {"q":"id:EMOWFYh1KyfA3nzLgJvohR","models":["aaa03c23b3724a16a56b629203edc62c","bd367be194cf45149e75f01d59f77ba7","eee28c313d69466f836ab83287a54ed9","c386b7a870114f4a87477c0824499348","e0be3b9d6a454f0493ac3a30784001ff","e466caa0619f444ab97497640cefc4dc"]}
Mar 22 12:36:54 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:54.026] [INFO]  Not logged in, logging in...
Mar 22 12:36:54 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:54.038] [INFO]  Login successful!
Mar 22 12:36:54 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:54.050] [INFO]  Processing in-progress: 06139051-bbb7-4139-bb22-6f282273eb82, total hits: 1, startIndex: 0, batchSize: 5, hits found: 1, query: {"QueryStringQuery":{"queryString":"(id:EMOWFYh1KyfA3nzLgJvohR) AND assetDomain:image"}}
Mar 22 12:36:54 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:54.051] [INFO]  Image recognition started for asset: EMOWFYh1KyfA3nzLgJvohR
Mar 22 12:36:55 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:55.738] [INFO]  Image recognition finshed for asset: EMOWFYh1KyfA3nzLgJvohR (LighthouseLighthouseLighthouseLighthouseLighthouseLighthouseLighthouseLighthouseLighthouseLighthouse.jpg)
Mar 22 12:36:55 t-clarifai-i-024743f09d32fae97 elvis-image-recognition: [22-03-2018 12:36:55.738] [INFO]  Processing completed: 06139051-bbb7-4139-bb22-6f282273eb82, successCount: 1, failedCount: 0

Server 2 logs:

Mar 22 12:35:44 t-clarifai-i-00b1309d05e82164b elvis-image-recognition: [22-03-2018 12:35:44.072] [INFO]  HTTP Server started at port: 9090
Mar 22 12:35:44 t-clarifai-i-00b1309d05e82164b elvis-image-recognition: [22-03-2018 12:35:44.075] [INFO]  Recognize imported files on import: true
Mar 22 12:35:44 t-clarifai-i-00b1309d05e82164b elvis-image-recognition: [22-03-2018 12:35:44.076] [INFO]  REST API enabled: true
Mar 22 12:35:44 t-clarifai-i-00b1309d05e82164b elvis-image-recognition: [22-03-2018 12:35:44.076] [INFO]  Ping endpoint enabled at: /SupaSekretEndpoint
Mar 22 12:35:44 t-clarifai-i-00b1309d05e82164b elvis-image-recognition: [22-03-2018 12:35:44.076] [INFO]  Requests are logged at: /var/log/elvis-image-recognition/access.log
Mar 22 12:36:53 t-clarifai-i-00b1309d05e82164b elvis-image-recognition: [22-03-2018 12:36:53.936] [INFO]  API call received: OPTIONS "/api/recognize" with body: {}

It seems as if the requests do not end up at the same application server. I think this may be because the OPTIONS request gets a set-cookie in the response header, but this this not used in the request header for the follow-up POST request to /api/recognize, so the load balancer sends another set-cookie header in the response. The pro client should set this cookie.

I have checked the auto_tag_images Elvis plugin code, and from what I can tell, something might fail in index.html to set the headers correctly.