pulumi up fails on with 2.9.0 but works in 2.8.0 due to breaking CustomDomain.verification field name change

[BryceLohr]

What happened?

#141 introduced a breaking change where the verification field of the CustomDomain resource was renamed to verifications. Our Pulumi program uses that field to create a Route53 record, but after upgrading it crashes with the following error:

Diagnostics:
  pulumi:pulumi:Stack (authentication-dev):
    error: Program failed with an unhandled exception:
    error: Traceback (most recent call last):
      File "/usr/local/bin/pulumi-language-python-exec", line 107, in <module>
        loop.run_until_complete(coro)
      File "/usr/local/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
        return future.result()
      File "/app/infra/.venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 126, in run_in_stack
        await run_pulumi_func(lambda: Stack(func))
      File "/app/infra/.venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
        await wait_for_rpcs()
      File "/app/infra/.venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 110, in wait_for_rpcs
        raise exception
      File "/app/infra/.venv/lib/python3.10/site-packages/pulumi/output.py", line 169, in run
        value = await self._future
      File "/app/infra/.venv/lib/python3.10/site-packages/pulumi/output.py", line 194, in run
        transformed: Input[U] = func(value)
      File "/app/infra/src/authentication/./auth0.py", line 195, in <lambda>
        .apply(lambda o: print("custom_domain.verification[0]:", o) or o["methods"])
    TypeError: 'NoneType' object is not subscriptable
    error: an unhandled error occurred: Program exited with non-zero exit code: 1

    custom_domain.verification: [None]
    custom_domain.verification[0]: None

    exception calling callback for <Future at 0xffff8d606b90 state=finished returned tuple>
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 330, in _invoke_callbacks
        callback(self)
      File "/usr/local/lib/python3.10/asyncio/futures.py", line 398, in _call_set_state
        dest_loop.call_soon_threadsafe(_set_state, destination, source)
      File "/usr/local/lib/python3.10/asyncio/base_events.py", line 790, in call_soon_threadsafe
        self._check_closed()
      File "/usr/local/lib/python3.10/asyncio/base_events.py", line 510, in _check_closed
        raise RuntimeError('Event loop is closed')
    RuntimeError: Event loop is closed

Here's the relevant piece of code for the above:

def extract_record(custom_domain: auth0.CustomDomain) -> pulumi.Output[str]:
    """Extract the actual CNAME record value needed for custom domain verification"""
    record = (
        pulumi.Output.all(custom_domain.verifications)
        .apply(lambda o: print("custom_domain.verification:", o) or o[0])
        .apply(lambda o: print("custom_domain.verification[0]:", o) or o["methods"])
        .apply(lambda o: print("custom_domain.verification[0]['methods']:", o) or o[0])
        .apply(lambda o: print("custom_domain.verification[0]['methods'][0]:", o) or o["record"])
    )
    record.apply(lambda o: print("custom_domain.verification[0]['methods'][0]['record']:", o))
    return record

The resource in the state file looks like this:

      {
        "urn": "urn:pulumi:dev::authentication::auth0:index/customDomain:CustomDomain::dev-auth0-custom-domain-name",
        "custom": true,
        "id": "cd_<redacted>",
        "type": "auth0:index/customDomain:CustomDomain",
        "inputs": {
          "__defaults": [],
          "domain": "<redacted>",
          "type": "auth0_managed_certs"
        },
        "outputs": {
          "domain": "<redacted>",
          "id": "cd_<redacted>",
          "primary": true,
          "status": "ready",
          "type": "auth0_managed_certs",
          "verification": {
            "methods": [
              {
                "name": "cname",
                "record": "dev-<redacted>.edge.tenants.us.auth0.com"
              }
            ]
          }
        },
        "parent": "urn:pulumi:dev::authentication::pulumi:pulumi:Stack::authentication-dev",
        "provider": "urn:pulumi:dev::authentication::pulumi:providers:auth0::default_2_9_0::c37cef8c-3c95-4e12-b365-4f18c49bb9e6",
        "propertyDependencies": {
          "domain": null,
          "type": null
        },
        "sequenceNumber": 1
      },

Steps to reproduce

I unfortunately don't have time to create a stand-alone minimal reproduce case to link to right now, but here's the general outline:

1. Create a Pulumi program that uses pulumi-auth0 2.8.0.
2. Use it to create an auth0.CustomDoamin resource, and another resource that depends on the verification field.
3. Deploy the stack and check that the resources were created properly.
4. Upgrade to pulumi-auth0 2.9.0.
5. Run pulumi preview. It should throw an error about verification not being a valid field.
6. Change verification to verifications in your code and try again.
7. This time the preview will succeed, but it will fail when actually applying, because there's nothing in verifications.

Trying to manually edit the state file to change the name of the field there to verifications doesn't work either.

Expected Behavior

The verification field should exist and have the same schema as it did in version 2.8.0. The upstream Terraform provider 0.30.2 does not contain this change.

Actual Behavior

Upgrading to 2.9.0 causes the issue described above instead of succeeding as it does in 2.8.0. Downgrading to 2.8.0 restores the functionality.

Versions used

CLI
Version      3.33.2
Go Version   go1.17.10
Go Compiler  gc

Plugins
NAME    VERSION
auth0   2.9.0
aws     5.6.0
python  unknown
random  4.7.0

Host
OS       debian
Version  11.2
Arch     aarch64

This project is written in python (/app/infra/.venv/bin/python3 v3.10.2)

Current Stack: modern-life/dev

TYPE                                                           URN
pulumi:pulumi:Stack                                            urn:pulumi:dev::authentication::pulumi:pulumi:Stack::authentication-dev
pulumi:providers:aws                                           urn:pulumi:dev::authentication::pulumi:providers:aws::default
pulumi:providers:auth0                                         urn:pulumi:dev::authentication::pulumi:providers:auth0::default_2_9_0
auth0:index/resourceServer:ResourceServer                      urn:pulumi:dev::authentication::auth0:index/resourceServer:ResourceServer::dev-auth0-hasura-api
auth0:index/client:Client                                      urn:pulumi:dev::authentication::auth0:index/client:Client::dev-auth0-modern-life-system
auth0:index/action:Action                                      urn:pulumi:dev::authentication::auth0:index/action:Action::dev-auth0-add-hasura-jwt-claims
auth0:index/customDomain:CustomDomain                          urn:pulumi:dev::authentication::auth0:index/customDomain:CustomDomain::dev-auth0-custom-domain-name
pulumi:providers:aws                                           urn:pulumi:dev::authentication::pulumi:providers:aws::default_5_6_0
aws:cloudfront/originAccessIdentity:OriginAccessIdentity       urn:pulumi:dev::authentication::aws:cloudfront/originAccessIdentity:OriginAccessIdentity::dev-auth0-cdn-assets-bucket-access-id
aws:s3/bucket:Bucket                                           urn:pulumi:dev::authentication::aws:s3/bucket:Bucket::dev-auth0-cdn-assets
aws:cloudfront/cachePolicy:CachePolicy                         urn:pulumi:dev::authentication::aws:cloudfront/cachePolicy:CachePolicy::dev-auth0-cdn-cache-policy
auth0:index/email:Email                                        urn:pulumi:dev::authentication::auth0:index/email:Email::dev-auth0-sendgrid-integration
auth0:index/triggerBinding:TriggerBinding                      urn:pulumi:dev::authentication::auth0:index/triggerBinding:TriggerBinding::dev-auth0-login-flow
pulumi:providers:auth0                                         urn:pulumi:dev::authentication::pulumi:providers:auth0::default_2_8_0
auth0:index/connection:Connection                              urn:pulumi:dev::authentication::auth0:index/connection:Connection::dev-auth0-modern-life-users
aws:route53/record:Record                                      urn:pulumi:dev::authentication::aws:route53/record:Record::dev-auth0-account-cname
aws:s3/bucketPolicy:BucketPolicy                               urn:pulumi:dev::authentication::aws:s3/bucketPolicy:BucketPolicy::dev-auth0-cdn-cloudfront-access-policy
auth0:index/customDomainVerification:CustomDomainVerification  urn:pulumi:dev::authentication::auth0:index/customDomainVerification:CustomDomainVerification::dev-auth0-domain-verification
aws:cloudfront/responseHeadersPolicy:ResponseHeadersPolicy     urn:pulumi:dev::authentication::aws:cloudfront/responseHeadersPolicy:ResponseHeadersPolicy::dev-auth0-cdn-cors-and-security-headers
auth0:index/tenant:Tenant                                      urn:pulumi:dev::authentication::auth0:index/tenant:Tenant::dev-auth0-tenant
aws:cloudfront/distribution:Distribution                       urn:pulumi:dev::authentication::aws:cloudfront/distribution:Distribution::dev-auth0-cdn-assets
auth0:index/emailTemplate:EmailTemplate                        urn:pulumi:dev::authentication::auth0:index/emailTemplate:EmailTemplate::dev-auth0-password-reset-and-welcome-email


Found no pending operations associated with modern-life/dev

Backend
Name           pulumi.com
URL            https://app.pulumi.com/brycelohr
User           brycelohr
Organizations  brycelohr, modern-life

NAME           VERSION
black          22.3.0
mypy           0.960
pip            22.0.4
pulumi-auth0   2.9.0
pulumi-aws     5.6.0
pulumi-random  4.7.0
pylint         2.13.9
tldextract     3.3.0
toml           0.10.2
wheel          0.37.1

Pulumi locates its logs in /tmp by default

Additional context

We're unable to deploy our stack that uses pulumi-auth0 on version 2.9.0 because of this issue: we're stuck with either the code error about the invalid field name, or the invalid state, which means we cannot use this version at all. We have to stay on the older version in order to keep the team moving forward.

Since this is breaking change, I would have expected a major version bump and a migration guide on how to update our user-land code to the new changes.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[stack72]

Hi @BryceLohr

I apologise for the issue here - unfortunately the underlying TF provider introduced a change that meant that verification can now be more than 1 - if you run a pulumi refresh, I believe this may work as expected?

Paul

[mikhailshilkov]

The issue is more than a year old and is still awaiting feedback. I'll take the liberty to close it assuming it's not relevant anymore, but please open a new one if you still need assistance.