Integration of new StatusCreatedAt GSI in web application

[srtalbot]

closes #3051

Context on why we had to create a new GSI: #4205 (comment)

Deployment plan (for both Staging and Production):

1. Review, update, merge and release (feat: add CreatedAt field to Vault CONF# items forms-terraform#810)
2. Write an run migration script to update all existing Vault CONF# items
   • Add CreatedAt property (equal to the one that already exists in NAME# items) in CONF# items (chore: add migration script to add CreatedAt property to Vault CONF# items #4678)
3. Review, update, merge and release (feat: update new Status#CreatedAt field when changing Status in DynamoDB Vault entries #4257)
4. Write and run migration script to update existing Vault NAME# items. This script should be executed in a period where there is the lowest traffic possible on GC Forms (e.g. Saturday at noon) as there is a very very very minimal risk of race condition where both the script and a user of GC Forms would try to update the exact same item in the Vault.
   • Add or update Status#CreatedAt property (using the existing Status and CreatedAt properties from the same item) in NAME# items (chore: add migration script to add StatusCreatedAt property to Vault NAME# items #4679)
5. Review, merge and release:
   • Remove our WAF POST request limit (infra side) (chore: remove POST request limit forms-terraform#909)
6. Review, merge and release:
   • Switch use of the Status GSI by StatusCreatedAt in the web application. New index does not return as many properties as the old one so we need to change how the web app loads all the data needed to render a complete response page (Reapply "feat: replace old Status index with new StatusCreatedAt one (#4691)" (#4761) #4763)
7. Review, merge and release:
   • Remove any reference to the old Status property in the infra (Nagware). If applicable, use Status#CreatedAt instead of Status (refactor: replace use of Status with new Status#CreatedAt attribute when requesting Vault items forms-terraform#904)
   • Remove any reference to the old Status property in the API. If applicable, use Status#CreatedAt instead of Status (refactor: replace use of Status with new Status#CreatedAt attribute when requesting Vault items forms-api#116)
8. Review, merge and release:
   • Remove any reference to the old Status property in the web application. If applicable, use Status#CreatedAt instead of Status (refactor: replace use of Status with new Status#CreatedAt attribute when requesting Vault items #4707)
9. Review, merge and release (make sure all changes above have been running for a couple of days in Production):
   • Delete the Status GSI in DynamoDB Vault table as well as Status property in Vault table (chore: remove unused Vault Status global secondary index as well as Status property in Vault table forms-terraform#903)

[anikbrazeau]

Related issue: #3051

[craigzour]

Deployment plan has been updated after I discovered a scenario that was not originally taken into account. @srtalbot @Abi-Nada

[srtalbot]

Part 2 is now in prod!

On part 3 of the deployment plan. Testing before merging to staging.

[craigzour]

Part 3 is now released in Production.

Will now be testing part 4 in Staging.

[craigzour]

Part 4 migration script was successfully run in Production on Saturday (Nov 30th)

[craigzour]

Added new step 5 that will need to be released before the old step 5 (which is now step 6).

[craigzour]

Part 6 has been deployed to Production

[craigzour]

Part 7 is currently running in Staging and should be ready to be released in Production.

[craigzour]

Part 7 > API is now running in Production
Next Step is Part 7 > Infra

[srtalbot]

Infra release for today so that we can release the app tomorrow!

[craigzour]

Part 7 > Infra is now running in Production

[craigzour]

Part 8 is now running in Staging and awaiting Production release

[craigzour]

Part 8 is now running in Production.

The plan is now to wait for at least a week to see if everything works fine with the full integration of the new Index. After that we can proceed with Part 9 which will result in the old Index being completely deleted from DynamoDB.