Enable reference to AWSEBWorkerQueue and AWSEBV2LoadBalancer in a CloudFormation template

[dennisvang]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave "+1" or "me too" comments. They generate extra noise for issue followers and do not help prioritize the request.

Tell us about your request

It would be great if there were a simple way to reference the AWSEBWorkerQueue resource for a Worker environment created in a CloudFormation template.

For example, something like !GetAtt MyWorkerEnvironment.AWSEBWorkerQueue would be nice.

The docs for AWS::ElasticBeanstalk::Environment only mention EndpoinURL as a return value, so I was silently hoping that !GetAtt MyWorkerEnvironment.EndpointURL would return the SQS queue url. However, unfortunately, stack creation just fails altogether with an error:

Attribute 'EndpointURL' does not exist

Similarly, being able to access the AWSEBV2LoadBalancer for a WebServer environment would be of great help in setting up HTTPS configuration.

Is this request specific to an Elastic Beanstalk platform?

no

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

In a single CloudFormation template that creates both a WebServer and a Worker environment, I am trying to get the worker's SQS queue url (or name), and assign it to an environment property for the webserver.

Ideally, I would like to be able to use something like Fn::GetAtt to get the SQS url from the worker resource, for example:

Resources:

  MyWorkerEnvironment:
    Type: AWS::ElasticBeanstalk::Environment
    # ...

  MyWebServerEnvironment:
    Type: AWS::ElasticBeanstalk::Environment
    DependsOn: MyWorkerEnvironment
    # ...
    Properties:
      # ...
      OptionSettings:
        - Namespace: aws:elasticbeanstalk:application:environment
          OptionName: WORKER_QUEUE
          Value: !GetAtt MyWorkerEnvironment.AWSEBWorkerQueue  # <-- this would be great, but does not work

This seems like a very basic issue that everyone using webservers and workers with cloudformation will run into at some point.

Are you currently working around this issue?

I've found, and tested, several alternatives:

• the manual approach: create the stack, copy the SQS queue url, and paste it into an environment property (or hardcode it)

• use an environment manifest file, i.e. env.yaml, to link the environments, as in the example:

  AWSConfigurationTemplateVersion: 1.1.0.0
  EnvironmentLinks:
    "WORKER_QUEUE": "<name of my worker environment>"

• use ElasticBeanstalk.Client.describe_environment_resources() from boto3 in the app (or use the aws cli equivalent), for example (simplified):

  worker_queue = None
  client = boto3.client('elasticbeanstalk', region_name='eu-west-3')
  response = client.describe_environment_resources(EnvironmentName='<name of my worker env>')
  for queue in response['EnvironmentResources']['Queues']:
      if queue['Name'] == 'WorkerQueue':
          worker_queue = queue['URL']

• use .ebextensions to export the value of AWSEBWorkerQueue to CloudFormation, then use Fn::ImportValue in the CloudFormation template, as in:

  .ebextensions/resources.config

  Outputs:
    WorkerQueue:
      Description: 'Worker SQS queue'
      Value: { "Ref" : "AWSEBWorkerQueue" }
      Export:
        Name: MyWorkerQueue

  and in my-cloudformation-template.yml

  # ...
          - Namespace: aws:elasticbeanstalk:application:environment
            OptionName: WORKER_QUEUE
            Value: !ImportValue MyWorkerQueue
  # ...

These all work, but all suffer from various drawbacks.

I also noticed that the stack created by AWS::ElasticBeanstalk::Environment for the worker environment already has an output called AWSEBWorkerQueueURL. However, I could not figure out how to get that into my template.

Additional context

n/a

Attachments

n/a

[dennisvang]

Actually, it would be great if other resources could also be accessed from the template that defines the AWS::ElasticBeanstalk::Environment. For example, AWSEBV2LoadBalancer would be very convenient.

[dennisvang]

As a workaround, it is relatively easy to create a custom worker queue and dead-letter queue, with configurations matching those of the default worker queue/dlq, and then specifying the WorkerQueueURL in the worker environment configuration.

In CDK this could look something like:

// custom queues with config like default worker queue
const commonWorkerQueueProps: sqs.QueueProps = {
  deliveryDelay: cdk.Duration.seconds(0),
  maxMessageSizeBytes: 256000, // 256KB
  receiveMessageWaitTime: cdk.Duration.seconds(0),
  retentionPeriod: cdk.Duration.days(14),
};
const workerDeadLetterQueue = new sqs.Queue(
  this,
  'WorkerDeadLetterQueue',
  {
    ...commonWorkerQueueProps,
    visibilityTimeout: cdk.Duration.seconds(30),
  },
);
const workerQueue = new sqs.Queue(this, 'WorkerQueue', {
  ...commonWorkerQueueProps,
  deadLetterQueue: { maxReceiveCount: 10, queue: workerDeadLetterQueue },
  visibilityTimeout: cdk.Duration.hours(12),
});

and in the CDK worker environment definition:

...
optionSettings: [
  ...,
  {
    namespace: 'aws:elasticbeanstalk:sqsd',
    optionName: 'WorkerQueueURL',
    value: workerQueue.queueUrl,
  },
  ...
],
...