Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#2903: Target Stable/Staging for Clone Workflow #3061

Merged
merged 7 commits into from
Nov 14, 2024
Merged

#2903: Target Stable/Staging for Clone Workflow #3061

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
af23b7d
Update clone-staging.yaml
Matt-Spence Nov 8, 2024
321052d
Update and rename clone-staging.yaml to clone-db.yaml
Matt-Spence Nov 8, 2024
5e96cec
Apply suggestions from code review
Matt-Spence Nov 13, 2024
dc2ade2
Additional review comments.
Matt-Spence Nov 13, 2024
c3fb2dd
Update clone-db.yaml
Matt-Spence Nov 14, 2024
a3ffae2
Update clone-db.yaml
Matt-Spence Nov 14, 2024
40b9b4b
Merge remote-tracking branch 'origin' into ms/2903-clone-stable-to-st…
Matt-Spence Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions .github/workflows/clone-staging.yaml → .github/workflows/clone-db.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
name: Clone Staging Database
# This workflow runs at the top of every hour and can be manually run as needed
# The workflow will copy the database from stable (production) to our staging sandbox.
# This workflow may fail if changes to the database schema make objects in stable
# incompatible with staging. This should resolve once both schemas match again.

name: Clone Stable Database
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: Clone Stable Database
# This workflow runs at the top of every hour and can be manually run as needed
# The workflow will copy the database from stable (production) to our staging sandbox.
name: Clone Stable Database

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also if the database is missing columns/tables in staging that exist in stable then we do expect this workflow to fail temporarily, correct? If so update this comment to reflect that as that is a very real possibility

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we do, we will need to ensure that the target database has a compatible schema with the source database.


on:
schedule:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment here is incorrect, please update

Expand All @@ -8,16 +13,19 @@ on:
workflow_dispatch:

env:
DESTINATION_ENVIRONMENT: ms
SOURCE_ENVIRONMENT: staging
# sandbox receiving the cloned db
DESTINATION_ENVIRONMENT: staging
Matt-Spence marked this conversation as resolved.
Show resolved Hide resolved
# sandbox we are cloning
SOURCE_ENVIRONMENT: stable
Matt-Spence marked this conversation as resolved.
Show resolved Hide resolved


jobs:
clone-database:
runs-on: ubuntu-24.04
env:
CF_USERNAME: ${{ secrets.CF_MS_USERNAME }}
CF_PASSWORD: ${{ secrets.CF_MS_PASSWORD }}
# must be the github secrets for the receiving sandbox
CF_USERNAME: ${{ secrets.CF_STAGING_USERNAME }}
Matt-Spence marked this conversation as resolved.
Show resolved Hide resolved
CF_PASSWORD: ${{ secrets.CF_STAGING_PASSWORD }}
steps:
- name: Clone Database
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we updating to cf8-cli here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also don't you need to do continue-on-error: true so that if this fails before unsharing the service the last step is triggered? reference

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been using cf8-cli for months (including when researching for this ticket) with no issues so I figured it would be fine to just go ahead and use it here too.

For your second question: I have set the next step to run always(), which does the same thing as continue-on-error in practice. Just two different ways of accomplishing the same thing.

run: |
Expand All @@ -42,6 +50,11 @@ jobs:
# clone from source to destination
cf target -s $SOURCE_ENVIRONMENT
cg-manage-rds clone getgov-$SOURCE_ENVIRONMENT-database getgov-$DESTINATION_ENVIRONMENT-database

# load fixtures in destination
cf target -s $DESTINATION_ENVIRONMENT
cf run-task getgov-staging --command 'python manage.py load' --name fixtures

- name: Cleanup
if: always()
run: cf unshare-service getgov-$DESTINATION_ENVIRONMENT-database -s $SOURCE_ENVIRONMENT -f