Sort Key for GSI out of sync with Cloudformation Template

[matt-at-allera]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

GraphQL API

Amplify Version

v5

Amplify Categories

No response

Backend

Amplify CLI

Environment information

# Put output below this line
  System:
    OS: macOS 13.2.1
    CPU: (8) arm64 Apple M1
    Memory: 56.33 MB / 8.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 20.7.0 - ~/.nvm/versions/node/v20.7.0/bin/node
    npm: 10.1.0 - ~/.nvm/versions/node/v20.7.0/bin/npm
  Browsers:
    Chrome: 126.0.6478.127
    Safari: 16.3
  npmPackages:
    @aws-amplify/core: ^5.8.2 => 5.8.12 
    @aws-amplify/core/internals/aws-client-utils:  undefined ()
    @aws-amplify/core/internals/aws-client-utils/composers:  undefined ()
    @aws-amplify/core/internals/aws-clients/pinpoint:  undefined ()
    @aws-amplify/ui-react: ^5.1.1 => 5.3.3 
    @aws-amplify/ui-react-internal:  undefined ()
    @aws-amplify/ui-react-storage: ^2.1.4 => 2.3.3 
    @aws-sdk/client-cognito-identity-provider: ^3.515.0 => 3.569.0 
    @aws-sdk/client-ses: ^3.515.0 => 3.569.0 
    @aws-sdk/lib-dynamodb: ^3.515.0 => 3.569.0 
    @aws-sdk/util-utf8-browser: ^3.259.0 => 3.259.0 (3.6.1, 3.186.0)
    @babel/cli: ^7.22.10 => 7.24.5 
    @babel/core: ^7.22.10 => 7.24.5 
    @babel/plugin-proposal-private-property-in-object: ^7.21.11 => 7.21.11 (7.21.0-placeholder-for-preset-env.2)
    @babel/preset-env: ^7.22.10 => 7.24.5 
    @babel/preset-typescript: ^7.24.1 => 7.24.1 
    @cyntler/react-doc-viewer: ^1.14.0 => 1.14.1 
    @cypress/angular:  0.0.0-development 
    @cypress/mount-utils:  0.0.0-development 
    @cypress/react:  0.0.0-development 
    @cypress/react18:  0.0.0-development 
    @cypress/svelte:  0.0.0-development 
    @cypress/vue:  0.0.0-development 
    @cypress/vue2:  0.0.0-development 
    @emotion/react: ^11.10.6 => 11.11.4 
    @emotion/styled: ^11.10.6 => 11.11.5 
    @mui/lab: ^5.0.0-alpha.124 => 5.0.0-alpha.170 
    @mui/material: ^5.11.15 => 5.15.16 
    @newrelic/browser-agent: ^1.260.0 => 1.260.0 
    @testing-library/jest-dom: ^5.16.5 => 5.17.0 
    @testing-library/react: ^13.4.0 => 13.4.0 
    @testing-library/user-event: ^13.5.0 => 13.5.0 
    @types/jest: ^27.5.2 => 27.5.2 
    @types/node: ^17.0.45 => 17.0.45 (18.19.32)
    @types/react: ^18.2.46 => 18.3.1 
    @types/react-dom: ^18.2.18 => 18.3.0 
    @types/react-router-dom: ^5.3.3 => 5.3.3 
    amazon-quicksight-embedding-sdk: ^2.6.0 => 2.7.0 
    aws-amplify: ^5.3.12 => 5.3.18 
    aws-sdk: ^2.1473.0 => 2.1615.0 
    aws-sdk-client-mock: ^3.0.1 => 3.1.0 
    axios: ^1.6.7 => 1.6.8 
    buffer: ^6.0.3 => 6.0.3 (4.9.2, 5.7.1)
    chart.js: ^4.4.3 => 4.4.3 
    chart.js-auto:  undefined ()
    chart.js-helpers:  undefined ()
    chartjs-plugin-datalabels: ^2.2.0 => 2.2.0 
    cypress: ^13.6.0 => 13.8.1 
    eslint-plugin-cypress: ^3.3.0 => 3.3.0 
    graphql: ^16.8.1 => 16.8.1 (15.8.0)
    html2canvas: ^1.4.1 => 1.4.1 
    idb: ^8.0.0 => 8.0.0 (5.0.6, 7.1.1)
    jest: ^29.0.0 => 29.7.0 (27.5.1)
    jspdf: ^2.5.1 => 2.5.1 
    jszip: ^3.10.1 => 3.10.1 
    mammoth: ^1.6.0 => 1.7.2 
    path-browserify: ^1.0.1 => 1.0.1 
    prettier: ^3.2.5 => 3.2.5 
    process: ^0.11.10 => 0.11.10 
    qrcode.react: ^3.1.0 => 3.1.0 
    react: ^18.2.0 => 18.3.1 (18.2.0)
    react-app-rewired: ^2.2.1 => 2.2.1 
    react-beautiful-dnd: ^13.1.1 => 13.1.1 
    react-chartjs-2: ^5.2.0 => 5.2.0 
    react-csv: ^2.2.2 => 2.2.2 
    react-dom: ^18.2.0 => 18.3.1 
    react-router-dom: ^6.24.0 => 6.24.0 
    react-scripts: ^5.0.1 => 5.0.1 
    react-select: ^5.8.0 => 5.8.0 
    react-signature-canvas: ^1.0.6 => 1.0.6 
    sass: ^1.71.1 => 1.77.0 
    source-map-loader: ^4.0.1 => 4.0.2 (3.0.2)
    swr: ^2.0.4 => 2.2.5 
    ts-jest: ^29.1.1 => 29.1.2 
    ts-loader: ^9.5.1 => 9.5.1 
    tsconfig-paths-webpack-plugin: ^4.1.0 => 4.1.0 
    typescript: ^4.9.5 => 4.9.5 
    web-vitals: ^2.1.4 => 2.1.4 (3.5.2)
    workbox-background-sync: ^6.6.0 => 6.6.0 (6.6.1)
    workbox-broadcast-update: ^6.6.0 => 6.6.0 
    workbox-cacheable-response: ^6.6.0 => 6.6.0 
    workbox-core: ^6.6.0 => 6.6.0 (6.6.1)
    workbox-expiration: ^6.6.0 => 6.6.0 
    workbox-google-analytics: ^6.6.1 => 6.6.1 (6.6.0)
    workbox-navigation-preload: ^6.6.0 => 6.6.0 
    workbox-precaching: ^6.6.0 => 6.6.0 
    workbox-range-requests: ^6.6.0 => 6.6.0 
    workbox-routing: ^6.6.0 => 6.6.0 (6.6.1)
    workbox-strategies: ^6.6.0 => 6.6.0 (6.6.1)
    workbox-streams: ^6.6.0 => 6.6.0 
    xlsx: ^0.18.5 => 0.18.5 
  npmGlobalPackages:
    @aws-amplify/cli: 12.11.1
    @newrelic/publish-sourcemap: 5.1.0
    aws-cdk: 2.127.0
    corepack: 0.19.0
    newrelic: 11.19.0
    npm: 10.1.0
    serve: 14.2.3
    snyk: 1.1241.0
    typescript: 5.2.2

Describe the bug

A schema.graphql change was made where the sortKeyFields of two @indexs were updated:

Before:

type ValidSupplierFormTypes @model {
    ...
    formTypeId: ID! @index(name: "byFormType", sortKeyFields: ["supplierId"])
}

type ValidSupplierTypes @model {
    ...
    supplierTypeId: ID! @index(name: "bySupplierType", sortKeyFields: ["supplierId"])
}

After:

type ValidSupplierFormTypes @model {
    ...
    formTypeId: ID! @index(name: "byFormType", sortKeyFields: ["supplierSiteId"])
}

type ValidSupplierTypes @model {
    ...
    supplierTypeId: ID! @index(name: "bySupplierType", sortKeyFields: ["supplierSiteId"])
}

I understand that this is not best practice, as an index with a new name should be created to support the new sort key field. However, during deployment, it appears that the underlying GSIs for the DynamoDB tables were updated to have the new sort key field. The update to the nested api stack failed, presumably because the app sync resolvers did not like the change. However, upon rollback the updated indices were NOT successfully updated, and the respective CloudFormation templates still have the supplierSiteId as the range key.

If I do an amplify pull on the environment, the schema contains the original @index configurations.

I attempted to resolve this manually by deleting the indices and creating them with the old sort key fields. However, all that did was create drift in the CloudFormation stack.

I need a mechanism synchronize what exists in the cloudformation template with what Amplify has stored for these two indices.

Expected behavior

I would expect initial push to fail as Amplify should not support only changing the sortKeyFields on indices.

Reproduction steps

1. Add a @model to the schema.graphql file with an index with a sortKeyField
2. Push the changes
3. Update the sortKeyField to a new field
4. Push again (should fail due to AppSync resolver updates failing)
5. Observe DynamoDB table with updated indices but Amplify environment still stating old index configurations

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[matt-at-allera]

It's worth noting that I also have an AWS Support ticket (#172062706600026) open with CloudFormation. They have advised for me to open a ticket here.

[chrisbonifacio]

Hi @matt-at-allera thanks for raising this. It doesn't seem to be related to the amplify-js library so I will transfer it to the amplify-category-api repo for better support.

[palpatim]

Hi @matt-at-allera,

Can you confirm you're using version 12.11.1 of the Amplify CLI you're using? (i.e., What is the output of amplify version?)

I would expect initial push to fail as Amplify should not support only changing the sortKeyFields on indices.

Amplify does support updates to GSIs: if a GSI has changed, the deployment deletes the index and recreates it. In addition, customers can batch more than one GSI update in a single schema push. That said, you're right that our recommendation for large schemas, or for tables with significant amounts of data, is to do it incrementally by deleting the index, pushing, then re-adding a new index with the updated sort key fields. The reason for this is that the time required to perform multiple GSI updates can exceed the Amplify CLI timeout or the expiration of the local credentials. In that case, we recommend doing an amplify push --iterative-rollback to roll back to the last known good state.

In a test I ran locally with a small schema and very short-lived credentials:

• I initiated the amplify push to update the GSI
• The push timed out
• The DDB table showed to have deleted the GSI
• There was no stack drift
• Amplify status still showed an update operation for the API
• I was then able to perform an amplify push --iterative-rollback to restore the previous GSI

Based on your report...

I attempted to resolve this manually by deleting the indices and creating them with the old sort key fields. However, all that did was create drift in the CloudFormation stack.

...it sounds like an iterative rollback isn't an option?

I'm curious to know what CloudFormation actually thinks about the state of the current resources. Can you provide the results of a stack drift detection on the model stack that actually owns the DynamoDB table and GSI? To find that:

• Go to CloudFormation
• Find the root Amplify stack, which should have a description like "Root Stack for AWS Amplify CLI"
• In the Resources tab of the Root Stack, click the Physical ID link for the API stack. It will have a Type of AWS::CloudFormation::Stack, and a Logical ID that begins with api
• In the Resources tab of the API Stack, click the Physical ID link for the ValidSupplierFormTypes nested stack. It will have a Type of AWS::CloudFormation::Stack, and a Logical ID of ValidSupplierFormTypes
• In the ValidSupplierFormTypes nested stack, click Stack actions then Detect drift
• In the ValidSupplierFormTypes nested stack, click Stack actions then View drift results. Refresh until the page shows results for the table (that is, the resource with the Type of AWS::DynamoDB::Table)

[matt-at-allera]

Hi @palpatim thanks for the response.

Yes, I am running 12.11.1:

$ amplify version
12.11.1

...it sounds like an iterative rollback isn't an option?

I've tried the iterative rollback, and it fails with the same error noted in my support ticket:

Resource handler returned message: "Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name."

I have performed the drift detection several times. It currently states that there is no drift for either table which is what led me to the assumption that Amplify is de-synced from the CloudFormation template. Results are attached. It had said there was drift when I manually reverted the table indices back to the last successful Amplify push (supplierId as sort key instead of supplierSiteId).

Should we commence with communicating here and not in the AWS support ticket?

[palpatim]

Thank you for sharing the drift detection results. Also, I apologize for the forked communication -- I default to GitHub but am happy to communicate either here or via the support ticket! I'll post some questions here and duplicate them in the ticket, so please feel free to respond in whichever channel you prefer.

Since I haven't been able to reproduce this locally, I want to summarize the current state:

• DynamoDB - NEW index names (as reported by inspecting the table in the console)
• CloudFormation - NEW index names (as evidenced by CloudFormation showing no stack drift on the DDB tables)
• Amplify Local project - OLD index names (as reported in backend/api/.../schema.graphql)

I'm assuming amplify pull doesn't complain about overwriting any local changes, which implies that your local workspace is in sync with the deployment state. To verify this, can you report what amplify pull ; amplify status shows? Assuming amplify status shows no changes, that means Amplify state is internally consistent.

Next step would be to do as you mentioned above: make Amplify reflect the current state of CloudFormation. We'll do that by editing the deployment state of the S3 bucket, which is a bit tedious, but relatively straightforward to do. I'll prepare some instructions for making those edits.

[matt-at-allera]

@palpatim of course and no worries, I'll keep responding here.

Since I haven't been able to reproduce this locally, I want to summarize the current state:

DynamoDB - NEW index names (as reported by inspecting the table in the console)
CloudFormation - NEW index names (as evidenced by CloudFormation showing no stack drift on the DDB tables)
Amplify Local project - OLD index names (as reported in backend/api/.../schema.graphql)

Your summary of the current state appears to be correct. I did not take perfect notes as to what caused us to get into this state, unfortunately. Knock on wood it doesn't happen again, but if it does, I'll be more methodical in tracking the steps that resulted in this de-synced state.

I'm assuming amplify pull doesn't complain about overwriting any local changes, which implies that your local workspace is in sync with the deployment state. To verify this, can you report what amplify pull ; amplify status shows? Assuming amplify status shows no changes, that means Amplify state is internally consistent.

That's correct. amplify pull; amplify status reports No Change for each category & resource.

Next step would be to do as you mentioned above: make Amplify reflect the current state of CloudFormation. We'll do that by editing the deployment state of the S3 bucket, which is a bit tedious, but relatively straightforward to do. I'll prepare some instructions for making those edits.

Perfect, I am unfamiliar with that process but had conjectured that would be necessary for resolution. I'll await your instructions!

[matt-at-allera]

@palpatim if you could send those instructions today, if possible, that would be very much appreciated. We've been blocked on making production pushes for 3 weeks now, and the pressure is mounting. I'm sure you understand the situation we're in. Having this resolved over the weekend is our priority.

Please let me know if there is anything I can do to speed this up or provide additional, necessary information

[palpatim]

@matt-at-allera

Sorry for the delay on this -- I am putting together instructions by using a local environment that attempts to recreate your starting conditions (DDB & CFN reflecting the new index name; Amplify state reflecting the old index name), but I'm finding that as long as I've created the index names properly in DDB/CFN, then I am able to update the name in my Amplify schema and let amplify push do the work.

The error message, Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name. indicates that there are other GSI attributes being updated --that is, the proposed change from Amplify does not match the current state in DDB.

Let's try comparing the CFN and Amplify stack definitions to see where the discrepancy is:

• Go to CloudFormation
• Find the root Amplify stack, which should have a description like "Root Stack for AWS Amplify CLI"
• In the Resources tab of the Root Stack, click the Physical ID link for the API stack. It will have a Type of AWS::CloudFormation::Stack, and a Logical ID that begins with api
• In the Resources tab of the API Stack, click the Physical ID link for the ValidSupplierFormTypes nested stack. It will have a Type of AWS::CloudFormation::Stack, and a Logical ID of ValidSupplierFormTypes
• In the ValidSupplierFormTypes nested stack, click Template then Copy
• Paste the template JSON into a file, let's call it ValidSupplierFormTypes-cfn.json

Next, let's get Amplify's view of the stack:

• In your local workspace, find the file amplify/backend/api/<api name>/build/stacks/ValidSupplierFormTypes.json

What I'm expecting to find is only differences related to the GSI's sortKeyField:

• The Resources.ValidSupplierFormTypesTable.Properties.AttributeDefinitions array should have only attributes that are part of the model's key schema or GSIs. Both the CFN and Amplify templates should show these fields, although the order may be different:
  • id
  • supplierId
  • formTypeId
  • supplierSiteId
  • tenant
  • (Explanation: the fields listed above are present in keys and GSIs in both versions of the table, so we'd expect to see them called out in the AttributeDefinitions)
• The Resources.ValidSupplierFormTypesTable.GlobalSecondaryIndexes array should show only a difference in the byFormType GSI:
  • The CFN version's KeySchema array should have a HASH key of formTypeId, and a RANGE key of supplierSiteId
  • The Amplify version's KeySchema array should have a HASH key of formTypeId, and a RANGE key of supplierId

My speculation is that there are other differences in either the AttributeDefinitions or GlobalSecondaryIndexes that are conflicting.

Once we narrow down the difference in the ValidSupplierFormTypes table, we'll do the same for ValidSupplierTypes

[matt-at-allera]

@palpatim

Thanks for responding and understanding the urgency. This approach makes sense to me.

ValidSupplierFormTypes.json
Amplify's view of the stack is on the left, and the actual CFN template is on the right:

This could be the main issue - that current Amplify environment does not even have an understanding of what supplierSiteId is.

As expected, the bySupplierSite index is missing from the Amplify side as well as the RANGE key being supplierId.

---

Let's do the same for ValidSupplierTypes.json:

The results appear to match what we see for the other table.

---

Does this mean the best path forward is to:

1. amplify pull the environment to be in sync with Amplify's state.
2. Add only the supplierSiteId as a field to both the ValidSupplierFormTypes and ValidSupplierTypes model.
3. amplify push the change
4. observe added supplierSiteId field
5. observe removed bySupplierSite index and RANGE key of byFormType / bySupplierType switched from supplierSiteId -> supplierId in both DDB and CFN
6. push again with latest changes in master Git branch ?

Let me know if you would take a different approach.

Additionally, I am still not convinced that the error message Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name. is occurring due to a property other than the RANGE key switching. Unless I am missing something, that configuration change seems like the likely culprit.

[palpatim]

Sorry, it appears the version of the schema uploaded to the support ticket is different from the one you're working from--the schema version I am working from doesn't even have a formTypeId field on the ValidSupplierTypes model. Apologies for the oversight. :(

In any case, the bottom line is: We need to make your local schema.graphql look like the CloudFormation template before we do an amplify push. That ensures that CloudFormation won't try to apply any index changes during the push, even though Amplify will detect a change and attempt to do an incremental GSI update for the changes it detects.

The steps I propose:

1. amplify pull the environment to be in sync with Amplify's state.
2. In the ValidSupplierFormTypes model:
   a. Add the supplierSiteId as an indexed field, referencing the previously-existing formTypeId in the sortKeyField.
   b. Add supplierTypeId to the sortKeyFields of the byFormType index
   c. The relevant fields of ValidSupplierFormTypes should look like:

   supplierSiteId: String @index(name: "bySupplierSite", sortKeyFields: ["formTypeId"])
   formTypeId: String @index(name: "byFormType", sortKeyFields: ["supplierSiteId"])

3. In the ValidSupplierTypes model:
   a. Add the supplierSiteId as an indexed field, referencing the previously-existing supplierTypeId in the sortKeyField.
   b. Add supplierSiteId to the sortKeyFields of the bySupplierType index
   c. The relevant fields of ValidSupplierFormTypes should look like:

   supplierSiteId: String @index(name: "bySupplierSite", sortKeyFields: ["supplierTypeId"])
   supplierTypeId: String @index(name: "bySupplierType", sortKeyFields: ["supplierSiteId"])

4. At this point, the locally generated CFN template should be the same as the template downloaded from CFN itself. amplify push the local changes. Amplify will create an iterative GSI deployment to add the "new" indexes, but those will not actually result in a changeset in CFN.

After the push succeeds, you should be in sync, and can make changes to indexes. Given the initial problems, I'd recommend making changes incrementally, with a push in between each one.

Additionally, I am still not convinced that the error message Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name. is occurring due to a property other than the RANGE key switching. Unless I am missing something, that configuration change seems like the likely culprit.

Based on the fact that my schema version is not the same as yours, I agree with this assessment -- it's likely the range key changes that are causing the issue.

[matt-at-allera]

@palpatim

Got it. This all makes sense. I have definitely tried a deployment with these fields specified as you state, but also with a few other changes that may have gotten in the way. It's good to isolate it to the smallest set of changes we can perform at once.

In a few hours when our production traffic dies down, I will perform the steps to both the ValidSupplierFormTypes and ValidSupplierTypes models and report back 🤞

[palpatim]

I should have noted above -- you can verify the CFN template that Amplify generates before the push by issuing amplify api gql-compile. That command runs the transformers locally, which regenerates files in the build folder. That means you can issue the command and then compare amplify/backend/api/<api name>/build/stacks/ValidSupplierFormTypes.json and amplify/backend/api/<api name>/build/stacks/ValidSupplierTypes.json to the appropriate CFN templates and ensure no diffs before pushing.

[matt-at-allera]

@palpatim

I've attempted the proposed solution and am encountering a strange behavior that makes me think we may need to modify something in the s3 deployment bucket?

Upon running the amplify api gql-compile the configuration of ValidSupplierTypes.json appears to match that of the deployed CFN template. However, during the CFN update / UPDATE_ROLLBACK_IN_PROGRESS state, I noticed that the template shown in the CFN console had the index configured as:

{
            "IndexName": "bySupplierType",
            "KeySchema": [
              {
                "AttributeName": "supplierTypeId",
                "KeyType": "HASH"
              },
              {
                "AttributeName": "supplierId",
                "KeyType": "RANGE"
              }
            ],
            "Projection": {
              "ProjectionType": "ALL"
            },
            "ProvisionedThroughput": {
              "Fn::If": [
                "ShouldUsePayPerRequestBilling",
                {
                  "Ref": "AWS::NoValue"
                },
                {
                  "ReadCapacityUnits": {
                    "Ref": "DynamoDBModelTableReadIOPS"
                  },
                  "WriteCapacityUnits": {
                    "Ref": "DynamoDBModelTableWriteIOPS"
                  }
                }
              ]
            }
          },

Notice the RANGE key is supplierId, even though the schema and the built file have it listed as supplierSiteId. These pushes are failing with that same Cannot update GSI's properties ....

Any idea why the deployment isn't using the updated configuration?

EDIT: I've tried with amplify push --force with the same result

[matt-at-allera]

Okay I was able to get an amplify push to go through. After updating the schema.graphql to reflect the actual DDB/CFN state, I ended up having to copy ValidSupplierTypes.json and ValidSupplierFormTypes.json from the local build directory into the #current-cloud-build.zip file in the S3 Deployment Bucket. Once that step was complete, I ran amplify pull, updated the schema.graphql file once more, and performed a final amplify push --force.

In a nutshell, this enabled the state of Amplify to "sync" up with the state of CFN. The question remains as to what caused this de-sync, but at least I know how this process can be solved in the future.

Leaving this ticket open to perform further investigation and to make sure I can make the necessary forward-progress updates.

[matt-at-allera]

The rest of the updates to the application succeeded. I've read up that sometimes making schema changes during a local amplify push can cause the #current-cloud-backend in S3 to desync from the local cli. However, it's strange to me that the remote versions of Amplify and CFN/DDB could be out of sync from each other. @palpatim I don't have concrete reproduction steps on this, but I am wondering, from your experience, if you have any clue how that could've happened?

[palpatim]

@matt-at-allera I'm glad you were able to complete the updates.

Unfortunately we've not been able to identify repro steps for this either. We have seen intermittent reports of this happening, but have been unable to repro using the affected schemas and various methods to interrupt the deployment (e.g., severing network connections, using expired credentials). In the cases where we do encounter a deployment error, an amplify push --iterative-rollback was enough to bring local and cloud deployment states back into sync.

I haven't tried doing a schema modification alongside a push, but in theory it might cause backend->#current-cloud-backend drift. But that shouldn't cause #current-cloud-backend to get out of sync with CFN. That said, Amplify Gen 1 deployment state is distributed amongst two sources of truth: local and cloud, and a race condition or other non/semi-deterministic bug could be accounting for this. But unfortunately without a reliable repro, we aren't able to make meaningful progress on a root cause analysis.

[matt-at-allera]

@palpatim thanks for the help on this one. I'm going to go ahead and close out the issue.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[EricDunaway]

@palpatim Something simialar just happened to me.

I had a failed push. I had added three indexes to a table and made some other changes, but the push errored out because a lambda did not have a needed security group value in the team provider.

Amplify did not roll back the table changes, and amplify pull returned the schema in the old state without the new indexes.

Looking in Cloudformation it showed the stack as having the added indexes and no drift
but a push would not let me add or remove the indexes or push any other changes

One thing to note is that this was an iterative deployment, and it did not fail until step 4/4