Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECS ServicesStable Waiter fails if DescribeServices response does not include deployments #3314

Open
1 task
TimMullender opened this issue Nov 29, 2024 · 2 comments
Labels
bug This issue is a confirmed bug. p2 This is a standard priority issue waiter

Comments

@TimMullender
Copy link

Describe the bug

waiter = ecs_client.get_waiter("services_stable")
waiter.wait(cluster=cluster, services=[service])

The ServicesStable waiter in Boto3 fails for some services with the following error message

"In function length(), invalid type for value: None, expected one of: ['string', 'array', 'object'], received: \"null\""

This appears to happen when the services[*].deployments element is absent from the DescribeServices response

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The waiter should wait until the services are stable, and not throw an error

Current Behavior

"errorMessage": "In function length(), invalid type for value: None, expected one of: ['string', 'array', 'object'], received: \"null\"",
    "errorType": "JMESPathTypeError",
    "requestId": "94d7b720-8490-4577-9ce7-93b19726c814",
    "stackTrace": [
       REDACTED
        "  File \"/var/lang/lib/python3.11/site-packages/botocore/waiter.py\", line 55, in wait\n    Waiter.wait(self, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/botocore/waiter.py\", line 348, in wait\n    if acceptor.matcher_func(response):\n",
        "  File \"/var/lang/lib/python3.11/site-packages/botocore/waiter.py\", line 229, in acceptor_matches\n    return expression.search(response) == expected\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/parser.py\", line 509, in search\n    result = interpreter.visit(self.parsed, value)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 145, in visit_comparator\n    self.visit(node['children'][0], value),\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 169, in visit_function_expression\n    current = self.visit(child, value)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 180, in visit_filter_projection\n    if self._is_true(self.visit(comparator_node, element)):\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 259, in visit_not_expression\n    original_result = self.visit(node['children'][0], value)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 253, in visit_and_expression\n    matched = self.visit(node['children'][0], value)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 145, in visit_comparator\n    self.visit(node['children'][0], value),\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 94, in visit\n    return method(node, *args, **kwargs)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/visitor.py\", line 171, in visit_function_expression\n    return self._functions.call_function(node['value'], resolved_args)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/functions.py\", line 80, in call_function\n    self._validate_arguments(resolved_args, signature, function_name)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/functions.py\", line 91, in _validate_arguments\n    return self._type_check(args, signature, function_name)\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/functions.py\", line 97, in _type_check\n    self._type_check_single(actual[i], allowed_types,\n",
        "  File \"/var/lang/lib/python3.11/site-packages/jmespath/functions.py\", line 111, in _type_check_single\n    raise exceptions.JMESPathTypeError(\n"
    ]

Reproduction Steps

I'm unsure what causes a service to not have the deployments element, the service is deployed using CodeDeploy's Blue/Green deployment

Possible Solution

No response

Additional Information/Context

I believe the line causing the issue is here but I'm unsure what it's trying to achieve by checking deployments

SDK version used

1.35.45

Environment details (OS name and version, etc.)

AWS Lambda, Python 3.11 Runtime

@TimMullender TimMullender added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Nov 29, 2024
@TimMullender
Copy link
Author

Also reproduced with the AWS CLI

aws ecs wait services-stable --cluster ClusterName --services ServiceName

In function length(), invalid type for value: None, expected one of: ['string', 'array', 'object'], received: "null"

@tim-finnigan tim-finnigan self-assigned this Nov 29, 2024
@tim-finnigan tim-finnigan added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Nov 29, 2024
@tim-finnigan
Copy link
Contributor

tim-finnigan commented Dec 3, 2024

Thanks for reaching out. The line you linked in the waiter model hasn't changed for ~6 years, so I'm not sure if something changed on the with the DescribeServices API that would cause this to no longer work. But per the API documentation it looks like there's no guarantee that deployments would be in the response.

Generally service teams own and maintain waiter models like this, but if there is an issue with the original implementation then this may be a bug to address here — and should also maybe be reviewed at a cross-SDK level. I'll mark this issue for further review.

@tim-finnigan tim-finnigan added waiter p2 This is a standard priority issue and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. needs-triage This issue or PR still needs to be triaged. labels Dec 3, 2024
@tim-finnigan tim-finnigan removed their assignment Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a confirmed bug. p2 This is a standard priority issue waiter
Projects
None yet
Development

No branches or pull requests

2 participants