Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task]: Setup Step Function / Orchestration for Transformations #1809

Open
1 task
acouch opened this issue Apr 23, 2024 · 3 comments · May be fixed by #2004
Open
1 task

[Task]: Setup Step Function / Orchestration for Transformations #1809

acouch opened this issue Apr 23, 2024 · 3 comments · May be fixed by #2004
Assignees
@coilysiren
Labels
project: grants.gov Grants.gov Modernization tickets
Milestone
Search API - ELT ...

Comments

@acouch
Copy link
Collaborator

acouch commented Apr 23, 2024 •
edited
Loading

Summary

Copy existing oracle data task.

Acceptance criteria

  • data loads in AWS from oracle to postgres and transformations work
@acouch acouch added the project: grants.gov Grants.gov Modernization tickets label Apr 23, 2024
@jamesbursa
Copy link
Collaborator

The command it should run is:

poetry run flask data-migration load-transform --load --no-transform --no-set-current

so that only the load part of the process is enabled.

@coilysiren
Copy link
Collaborator

👍

@coilysiren
Copy link
Collaborator

Two notes:

  • we shouldn't be running this in dev right now
  • we want to run it daily, since the performance isn't at the place where we can run it every X minutes

@coilysiren coilysiren linked a pull request May 13, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coilysiren coilysiren

Labels
project: grants.gov Grants.gov Modernization tickets
Projects
Simpler Grants Sprint Planning
Status: Sprint Ready
Milestone
Search API - ELT Implementation
Development

Successfully merging a pull request may close this issue.

[Issue #1809] Add load-transform SFN HHS/simpler-grants-gov
3 participants
@acouch @jamesbursa @coilysiren