Using Flat Data With Neo4j

[johnymontana]

Hi Folks 👋

I've been experimenting with using Flat Data to import data into Neo4j.

To make this workflow easier I published an Action called Flat Graph that allows for defining an Action step that includes the database connection credentials and a database query (using Cypher, the query language for Neo4j) to run that handles importing the data fetched by the Flat action in the previous step. For example, here's how you would use it to import data from the Lobsters JSON feed:

name: Flat Graph for Neo4j

on:
  push:
    paths:
      - .github/workflows/flat.yml
  workflow_dispatch:
  schedule:
    - cron: '*/60 * * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Setup deno
        uses: denoland/setup-deno@main
        with:
          deno-version: v1.x
      - name: Fetch newest 
        uses: githubocto/flat@v2
        with:
          http_url: https://lobste.rs/newest.json
          downloaded_filename: newest.json
      - name: Neo4j import
        uses: johnymontana/[email protected]
        with:
          neo4j-user: ${{secrets.NEO4J_USER}}
          neo4j-password: ${{secrets.NEO4J_PASSWORD}}
          neo4j-uri: ${{secrets.NEO4J_URI}}
          filename: 'newest.json'
          cypher-query: >
              UNWIND $value AS article
              MERGE (u:User {username: article.submitter_user.username})
              MERGE (a:Article {id: article.short_id})
              SET a.url           = article.url,
                  a.id_url        = article.short_id_url,
                  a.created_at    = article.created_at,
                  a.title         = article.title,
                  a.score         = article.score,
                  a.flags         = article.flags,
                  a.comment_count = article.comment_count,
                  a.description   = article.description,
                  a.comments_url  = article.comments_url
              MERGE (u)-[:SUBMITTED]->(a)
              WITH article, a
              UNWIND article.tags AS tag
              MERGE (t:Tag {name: tag})
              MERGE (a)-[:HAS_TAG]->(t)        

To see it in action I wrote a tutorial / blog post showing how to use Flat Data + Neo4j.

Would love to know what folks think of this workflow or any suggestions on how to improve it.

Cheers,
Will

[idan]

@johnymontana that's lovely! It's cool to see flat composed into workflows this way.

You mention in the writeup that you could have done the writing to neo4j in a postprocessing step, but you ended up writing the flat-graph action instead. There's nothing at all wrong with that! But I'm curious what pain with the postprocessing led you to write the action instead.

[johnymontana]

Well I started with the postprocessing script, but it was a lot of boilerplate (create a database driver instance, read the downloaded JSON file, pass it as a query parameter, etc) so I thought it would be a nicer abstraction to be able to just write the database query in the same yaml file when defining the action workflow. So the reason for the separate action was the pain of copy/paste boilerplate setup code in the postprocessing script and also some issues with using the Neo4j driver in Deno (but that's a Neo4j problem).