✨ export data to postgres through hasura graphql api #15
+2,594
−27
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goal
The goal is to export all the data fetched from GitHub into a database, with history, so that can access the data easily and quickly, and explore the evolution of the data.
Implementation
I'm using Hasura on top of Postgres to define the SQL schema as well as expose a GraphQL API to insert and read the data easily.
Instead of modifying the existing code to insert to postgres instead of writing JSON files, I've written a separate code file
hasura.js
that reads the JSON files generated byindex.js
and loads them into postgres. All of the data is inserted into postgres using one big GraphQL query. Each time, the entire data graph is inserted as new entities in order to keep older entities intact.How to load up the data into Postgres
.env
and fillGITHUB_ID
,GITHUB_OAUTH
andGITHUB_ORGA
npm start generate
docker-compose up -d
HASURA_ADMIN_SECRET=hasura
andHASURA_GRAPHQL_URL=http://localhost:8080/v1/graphql
to.env
node -r dotenv/config src/hasura.js
Examples of queries
Once the data is loaded up, browser to
http://localhost:8080
(password ishasura
). In the GraphiQL tab, we can test some GraphQL queries:History of the number of repositories for a member
History of contributions of a member
Improvements to make
fetched_at
field that is currently on the organization entity and that denote the time that data was fetched, needs to be moved to its own entity.