[Issue #2190] Fix test setup and expand factories for foreign table load

[chouinar]

Summary

Fixes #2190

Time to review: 10 mins

Changes proposed

Make the load from Oracle logic be tested with actual table data

Add factories for foreign tables by refactoring the staging table factories to inherit from a base class

• Along with this I added relationships to the data models, this just makes the factories be able to create their related data (ie. an opportunity factory can create a synopsis and link them together) - this is purely a SQLAlchemy concept and does not affect the schema in any way.

Explicitly define which tables we want to load (statically, but allow user override for testing)

Adjustments to the way data is generated in the local seed script (to be expanded later)

Context for reviewers

Apologies for the size - this is a lot of work towards adding more testing in our transform logic that will come in future PRs. Think of this as a lot of cleanup and pre-work in order to properly make more thorough testing.

This work started as fixing an issue I noticed where before it would attempt to always load EVERY table we have configured as a foreign/staging table. This would mean in order to add any new table to these schemas, we would need to make sure the foreign data wrapper connection existed in an environment before deploying which would not be ideal. This way lets us create the tables, and test them in a decoupled fashion. For example, I recently added a tgroups table. This needs to be manually tested before we turn it on with the rest of the job.

Additional information

Running locally, you can test some of this by doing:

make volume-recreate
make setup-foreign-tables
make seed-local-legacy-tables
poetry run flask data-migration load-transform --load --no-transform --no-set-current

You can also run something like this after to just have it process a single table

make seed-local-legacy-tables
poetry run flask data-migration load-transform --load --no-transform --no-set-current -t topportunity

[mdragon]

Not sure if it's worth the trouble, but doing TRUNCATE TABLE is often way more efficient (skips writing the DELETEs to the log) for large tables. From my poking around it seems like SQLAlchemy doesn't support it via it's API (which makes sense as not every version of every DB engine supports it) but we could run it as a raw SQL if we wanted to go that route.

[chouinar]

Only a minor change to do - it works fine in this case.

[mdragon]

Looks good, just flagged about running TRUNCATE instead of DELETE but that could just be a note for the future.

[mdragon]

Great!