Host Dispatch on AWS ECS

[rilutham]

Hi everyone, we are planning to host Dispatch on AWS ECS environment. Here are some questions:

• Which Dockerfile that we should use?
• How to define the task definition? Is it will be a container running a web server, or need an additional container running scheduler?

[kevgliss]

I'm not sure if you've already figured this out. But in general, we are fairly unopinionated about deployment and provide docker files as references. If you do have some insight into ECS we'd be happy to have it documented here or in our docs.

[rilutham]

Hi @kevgliss , I've put the implementation detail here #789 (reply in thread) . Is this something we can put in this doc or somewhere else?

[Ant0wan]

@rilutham I would suggest you build the image using the original Dockerfile in the repo. Also deploying it using any container solution (podman, Docker, cri-o...). You'll save time. And it works great.

[rilutham]

Thanks, @Ant0wan ! I was successfully building the image using the original Dockerfile. Here is the implementation detail #789 (reply in thread)

[csetarun2]

@rilutham were u able to build this application in ECS ? is it working ? Can you share the steps if its possible, since dispatch requires 3 container (dispatch-web, scheduler, database) , how to do it in task definition

[rilutham]

Hi @csetarun2!
Yes, it's working. Here is some detail of my implementation:

1. We're using a dedicated Postgres DB in AWS RDS
2. We store all the environment variables in AWS SSM Parameter Store
3. We use the original Dockerfile and store the image on AWS ECR
4. There are two containers in the task definition (server and scheduler)

{
  "containerDefinitions": [
    {
      "logConfiguration": {
        "logDriver": "json-file",
        "options": {
          "max-size": "10m",
          "max-file": "3",
          "labels": "app,env,image"
        }
      },
      "portMappings": [
        {
          "hostPort": 0,
          "protocol": "tcp",
          "containerPort": 8000
        }
      ],
      "cpu": 0,
      "secrets": [
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/INCIDENT_STORAGE_FOLDER_ID",
          "name": "INCIDENT_STORAGE_FOLDER_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_PROJECT_ID",
          "name": "GOOGLE_SERVICE_ACCOUNT_PROJECT_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY_ID",
          "name": "GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY",
          "name": "GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_DELEGATED_ACCOUNT",
          "name": "GOOGLE_SERVICE_ACCOUNT_DELEGATED_ACCOUNT"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_CLIENT_ID",
          "name": "GOOGLE_SERVICE_ACCOUNT_CLIENT_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_CLIENT_EMAIL",
          "name": "GOOGLE_SERVICE_ACCOUNT_CLIENT_EMAIL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_DEVELOPER_KEY",
          "name": "GOOGLE_DEVELOPER_KEY"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_DOMAIN",
          "name": "GOOGLE_DOMAIN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/INCIDENT_NOTIFICATION_DISTRIBUTION_LISTS",
          "name": "INCIDENT_NOTIFICATION_DISTRIBUTION_LISTS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/INCIDENT_NOTIFICATION_CONVERSATIONS",
          "name": "INCIDENT_NOTIFICATION_CONVERSATIONS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_TIMELINE_EVENT_REACTION",
          "name": "SLACK_TIMELINE_EVENT_REACTION"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_BAN_THREADS",
          "name": "SLACK_BAN_THREADS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_APP_USER_SLUG",
          "name": "SLACK_APP_USER_SLUG"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_SIGNING_SECRET",
          "name": "SLACK_SIGNING_SECRET"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_API_BOT_TOKEN",
          "name": "SLACK_API_BOT_TOKEN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_WORKSPACE_NAME",
          "name": "SLACK_WORKSPACE_NAME"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/BUSINESS_HOURS_YEAR",
          "name": "BUSINESS_HOURS_YEAR"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/ANNUAL_COST_EMPLOYEE",
          "name": "ANNUAL_COST_EMPLOYEE"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_UI_URL",
          "name": "DISPATCH_UI_URL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_HELP_SLACK_CHANNEL",
          "name": "DISPATCH_HELP_SLACK_CHANNEL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_HELP_EMAIL",
          "name": "DISPATCH_HELP_EMAIL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/VUE_APP_SENTRY_DSN",
          "name": "VUE_APP_SENTRY_DSN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SENTRY_DSN",
          "name": "SENTRY_DSN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SECRET_KEY",
          "name": "SECRET_KEY"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/COMPOSE_PROJECT_NAME",
          "name": "COMPOSE_PROJECT_NAME"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/POSTGRES_PASSWORD",
          "name": "POSTGRES_PASSWORD"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/POSTGRES_USER",
          "name": "POSTGRES_USER"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/POSTGRES_DB",
          "name": "POSTGRES_DB"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_AUTHENTICATION_DEFAULT_USER",
          "name": "DISPATCH_AUTHENTICATION_DEFAULT_USER"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_JWT_SECRET",
          "name": "DISPATCH_JWT_SECRET"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_AUTHENTICATION_PROVIDER_SLUG",
          "name": "DISPATCH_AUTHENTICATION_PROVIDER_SLUG"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_PORT",
          "name": "DATABASE_PORT"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_NAME",
          "name": "DATABASE_NAME"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_CREDENTIALS",
          "name": "DATABASE_CREDENTIALS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_HOSTNAME",
          "name": "DATABASE_HOSTNAME"
        }
      ],
      "memory": null,
      "memoryReservation": null,
      "image": "account-id.dkr.ecr.region-id.amazonaws.com/dispatch",
      "dependsOn": null,
      "healthCheck": {
        "retries": 3,
        "command": [
          "CMD-SHELL",
          "curl -f localhost:8000 || exit 1"
        ],
        "timeout": 5,
        "interval": 15,
        "startPeriod": null
      },
      "essential": true,
      "links": null,
      "dockerLabels": {
        "app": "Dispatch",
        "image": "account-id.dkr.ecr.region-id.amazonaws.com/dispatch",
        "env": "production"
      },
      "name": "app"
    },
    {
      "logConfiguration": {
        "logDriver": "json-file",
        "secretOptions": null,
        "options": {
          "max-size": "10m",
          "max-file": "3",
          "labels": "app,env,image"
        }
      },
      "portMappings": [],
      "command": [
        "scheduler",
        "start"
      ],
      "cpu": 0,
      "secrets": [
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/INCIDENT_STORAGE_FOLDER_ID",
          "name": "INCIDENT_STORAGE_FOLDER_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_PROJECT_ID",
          "name": "GOOGLE_SERVICE_ACCOUNT_PROJECT_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY_ID",
          "name": "GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY",
          "name": "GOOGLE_SERVICE_ACCOUNT_PRIVATE_KEY"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_DELEGATED_ACCOUNT",
          "name": "GOOGLE_SERVICE_ACCOUNT_DELEGATED_ACCOUNT"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_CLIENT_ID",
          "name": "GOOGLE_SERVICE_ACCOUNT_CLIENT_ID"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_SERVICE_ACCOUNT_CLIENT_EMAIL",
          "name": "GOOGLE_SERVICE_ACCOUNT_CLIENT_EMAIL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_DEVELOPER_KEY",
          "name": "GOOGLE_DEVELOPER_KEY"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/GOOGLE_DOMAIN",
          "name": "GOOGLE_DOMAIN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/INCIDENT_NOTIFICATION_DISTRIBUTION_LISTS",
          "name": "INCIDENT_NOTIFICATION_DISTRIBUTION_LISTS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/INCIDENT_NOTIFICATION_CONVERSATIONS",
          "name": "INCIDENT_NOTIFICATION_CONVERSATIONS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_TIMELINE_EVENT_REACTION",
          "name": "SLACK_TIMELINE_EVENT_REACTION"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_BAN_THREADS",
          "name": "SLACK_BAN_THREADS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_APP_USER_SLUG",
          "name": "SLACK_APP_USER_SLUG"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_SIGNING_SECRET",
          "name": "SLACK_SIGNING_SECRET"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_API_BOT_TOKEN",
          "name": "SLACK_API_BOT_TOKEN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SLACK_WORKSPACE_NAME",
          "name": "SLACK_WORKSPACE_NAME"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/BUSINESS_HOURS_YEAR",
          "name": "BUSINESS_HOURS_YEAR"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/ANNUAL_COST_EMPLOYEE",
          "name": "ANNUAL_COST_EMPLOYEE"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_UI_URL",
          "name": "DISPATCH_UI_URL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_HELP_SLACK_CHANNEL",
          "name": "DISPATCH_HELP_SLACK_CHANNEL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_HELP_EMAIL",
          "name": "DISPATCH_HELP_EMAIL"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/VUE_APP_SENTRY_DSN",
          "name": "VUE_APP_SENTRY_DSN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SENTRY_DSN",
          "name": "SENTRY_DSN"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/SECRET_KEY",
          "name": "SECRET_KEY"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/COMPOSE_PROJECT_NAME",
          "name": "COMPOSE_PROJECT_NAME"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/POSTGRES_PASSWORD",
          "name": "POSTGRES_PASSWORD"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/POSTGRES_USER",
          "name": "POSTGRES_USER"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/POSTGRES_DB",
          "name": "POSTGRES_DB"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_AUTHENTICATION_DEFAULT_USER",
          "name": "DISPATCH_AUTHENTICATION_DEFAULT_USER"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_JWT_SECRET",
          "name": "DISPATCH_JWT_SECRET"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DISPATCH_AUTHENTICATION_PROVIDER_SLUG",
          "name": "DISPATCH_AUTHENTICATION_PROVIDER_SLUG"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_PORT",
          "name": "DATABASE_PORT"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_NAME",
          "name": "DATABASE_NAME"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_CREDENTIALS",
          "name": "DATABASE_CREDENTIALS"
        },
        {
          "valueFrom": "arn:aws:ssm:region-id:account-id:parameter/ecs/Dispatch/DATABASE_HOSTNAME",
          "name": "DATABASE_HOSTNAME"
        }
      ],
      "memory": null,
      "memoryReservation": null,
      "image": "account-id.dkr.ecr.region-id.amazonaws.com/dispatch",
      "dockerLabels": {
        "app": "Dispatch",
        "image": "account-id.dkr.ecr.region-id.amazonaws.com/dispatch",
        "env": "production"
      },
      "name": "scheduler"
    }
  ],
  "placementConstraints": [],
  "memory": "2500",
  "taskRoleArn": null,
  "compatibilities": [
    "EC2"
  ],
  "taskDefinitionArn": "arn:aws:ecs:region-id:account-id:task-definition/Dispatch:1",
  "family": "Dispatch",
  "networkMode": null,
  "runtimePlatform": null,
  "cpu": "2048",
  "revision": 1,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

[csetarun2]

Hi @rilutham

I have tried same steps.

1. created aws rds (postgres) , can able to load data with local docker netflix app setup
2. after running dispatch application, tagged the image and pushed it into the private ecr registry
3. stored all the parameters inside parameter store
4. using above task definition which u have shared , I have modified and ran as a service, getting these error

any idea about this error
even though I have stored in system parameter store, and referred in the task definition..

my tasks are always stopping due to this error,

[Ant0wan]

Hi @csetarun2 , you have not set any DISPATCH_ENCRYPTION_KEY as env variable or in your dispatch configuration. Set anything you want and it should be working fine. Try to base64, it could help.

[csetarun2]

Thanks @Ant0wan , looks there is issue while access those ssm parameters, I have stored all those parameters inside s3 object, now I can able to build image

[csetarun2]

@rilutham I have noticed there is issue after deploying container in ECS, since u have used healthcheck curl -f localhost:8000 , by default in our container there is no curl package was there, anyone before pushing into container registry login into the running container then install curl pkg, then commit the container into image, then push it,

[Ant0wan]

You can use wget instead

[csetarun2]

Hi @rilutham
in the task definition you have mentioned 2 containers to run using the same image (one for the app other one for scheduler) , Do we require both , as @mvilanova mentioned (#2581) inside one container all these (The core, web, and scheduler should be running on the same container) will run, so we can remove the scheduler container info from taskdefinition right / is there any necessity to run scheduler container separately ? Please let me know

[parachill]

@rilutham Hi, I just create dispatch ECS service but, when task starts it gets error "dispatch.auth.service:Unable to determine user email based on configured auth provider or no default auth user email defined"

Do you know how to solve this?

[csetarun2]

try with default configurtion values, have u used any okta/gsuite ?