Do a release with new deployment model

[angela-tran]

Depends on #284, #286, #287

• Deploy an -rc version to test environment
  • Check out the default branch and make sure it's up-to-date with remote default branch
  • Make an annotated Git tag following release candidate version number format, 2024.01.1-rc1, and push it
  • Check on the Azure DevOps infra pipeline for each transit agency:
    • Did the pipeline run?
    • Was terraform plan successful?
    • Did the pipeline wait for your approval to run terraform apply?
    • Was terraform apply succcessful?
  • Check on the GitHub workflow that does the Docker publish:
    • Did the tests run?
    • Did a new Docker image get published with the expected Docker tags (2024.01.1-rc1, test, and the Git SHA)?
  • QA the app:
    • Does the app respond? (/healthcheck)
      • The app failed to start up because of an error in the eligibility_server.sentry module - fixed in Fix: Bugs with Sentry configuration #398
    • Is the version number correct?
• Deploy a new version to prod environment
  • Check out the release candidate tag
  • Make an annotated Git tag following release version number format, 2024.01.1, and push it
    • git tag -a 2024.01.1
    • git push origin 2024.01.1
  • Check on the Azure DevOps infra pipeline for each transit agency:
    • Did the pipeline run?
    • Was terraform plan successful?
    • Did the pipeline wait for your approval to run terraform apply?
    • Was terraform apply succcessful?
  • Check on the GitHub workflow that does the Docker publish:
    • Did the tests run?
    • Did a new Docker image get published with the expected Docker tags (2024.01.1, prod, and the Git SHA)?
  • Check on the GitHub workflow that creates the GitHub release:
    • Was a release created for the right tag?
    • Were release notes generated?
      • Found an issue with release notes - it diffs against the very beginning to generate release notes which is incorrect
        • Attempting to fix with Refactor: tag fetch #402 and Fix: fetch tags for dev #403
        • Merge to dev went through correctly ✅
        • Tag push for release candidate 2024.01.3-rc1
          • infra pipeline ran even though we thought it wouldn't based on path filter configuration -- reading CI Triggers docs closer, it seems like path filters are combined only with branch filters 👀
        • Tag push for release 2024.01.3 failed due to Fix: avoid tag checkout collision #404
        • Fix: avoid tag checkout collision #404 fix merged to dev ✅ , pushed as 2024.01.4-rc1 ✅ , pushed as 2024.01.4 - release notes generated correctly
  • QA the app:
    • Does the app respond? (/healthcheck)
    • Is the version number correct?
      • The version inside the prod Docker image says 2024.01.4rc1 which is incorrect - is GitHub caching something?

[angela-tran]

Planning on doing this after our winter break!

[angela-tran]

Actually, #393 needs to be done first

[thekaveman]

• Is the version number correct?
  • The version inside the prod Docker image says 2024.01.4rc1 which is incorrect - is GitHub caching something?

OMG I think I finally figured this out. See logs for 2024.01.12-rc1 and 2024.01.12.

It took from 2024.01.5-rc1 until 2024.01.12 (16 total tag pushes!!!) to get this sorted 😓

Debug process

This was extra messy 🚧 👷 🏗️, because I could never reproduce the issue locally (with act or otherwise).

• I temporarily turned off branch protections on dev to avoid PRs / review cycles (back on now)
• I temporarily added a bunch of extra debug output to the docker-publish workflow (removed now)
• I temporarily disabled Docker image building and pushing in the docker-publish workflow (back on now)
• I temporarily disabled the release workflow to avoid publishing meaningless releases (back on now)
• I ignored just about every Azure pipeline run/approval (although there were no Terraform changes, see Fix release-tag trigger in infra pipeline definition files #377)

The logs from the test workflow ended up being really helpful. This workflow is called for RC and release tag pushes by the docker-deploy workflow, so it was always running before these previously broken runs.

I noticed that in the test workflow, it was always installing the correct version of the package (i.e. matching the expected RC or release tag).

Here's an example for of the test workflow for the 2024.01.4 tag, which we saw incorrectly installed as 2024.01.4-rc1 for the docker-publish workflow.

Changes that got it to stick

The changes I made to the docker-publish workflow to align it more closely with the test workflow:

• Remove fetch-depth: 0 and fetch-tags from the actions/checkout step -- since our docker-publish workflow is triggered by a tag push, this tag will be the ref that is checked out. Don't allow extra tags/refs to come in and pollute the git environment
• Remove pip caching in the actions/setup-python step
• Upgrade pip in both the workflow step AND during both phases of the Docker image build
• Remove usage of venv, just pip install directly into the action runner
• Full changelog with all my nonsense: 2024.01.4...2024.01.12

FINALLY going to mark this issue as complete! 🎉 ✨

Thank you for all of your work that led up to this @angela-tran 🙏 🙏

[thekaveman]

Yesssssssssssssssssssssssss

2024-01-25T23:48:09.325111047Z INFO eligibility_server.sentry:42 Enabling Sentry for environment 'prod',
  release '2024.1.12'...
...
2024-01-25T23:48:10.611937797Z INFO eligibility_server.app:31 Starting Eligibility Server 2024.1.12
2024-01-25T23:48:10.795858463Z INFO Starting gunicorn 21.2.0