[ETL-686] Use S3 as GX validations store

[philerooski]

A lot of files touched here, but most of those changes involve shuffling parameters around. The primary changes are:

src/glue/jobs/run_great_expectations_on_parquet.py

• A large part of the GX data context configuration was moved to src/glue/resources/great_expectations.yml. This file is downloaded and formatted by configure_gx_config. The configuration of the datasource (get_batch_request, which now uses fluent datasources rather than the older datasource API) and the expectation suite (add_expectations_from_json) is still done in Python.
• In order to have a cumulative collection of validation reports in our docs, I had to change two things: persisting our validation store between job runs, which meant using S3 as the store backed rather than the local file system, and using checkpoints to update the data docs, rather than the context.build_data_docs method, which builds the entire site from scratch each time and can apparently take quite a while if we are adding to the data docs each day, judging from some GX issue reports. Checkpoints were necessary, since they allow us to use the UpdateDataDocsAction to update our data docs. As far as I can tell, there is no method to do this using a data context method. We also get a StoreValidationResultAction with checkpoints, so that saves us a step.

tests/test_run_great_expectations_on_parquet.py

Because of the big refactor above, the tests changed a lot, as well.

src/scripts/manage_artifacts/artifacts.py

I added the namespaced location of our data docs to the --remove option of this script, so feature branch data docs ought to be cleaned up upon branch deletion.

[rxu17]

Nice work! So much to redo just because of the non-persistent data context before :'). Just had some comments/clarification questions

[rxu17]

LGTM!
Just a failing test: https://github.com/Sage-Bionetworks/recover/actions/runs/11637550526/job/32410945073

[sonarcloud]

Quality Gate passed

Issues
32 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud