Migrate data from Filesystem to Postgres

[nelsonic]

At present the Node.js MVP saves data to the instance filesystem because that was the simplest way of storing data without having to manage any database.
see: hits-nodejs/lib/db_filesystem.js#L26-L68

This worked well for MVP as it streams the contents of the file each time a request is made and counts the lines in the file as the count. Node.js shines at this because fs.createReadStream is non-blocking.
Anyway, with the migration to phoenix, we need to:

• write an Elixir Script that parses the data in the filesystem and inserts it into PostgreSQL

Tasks

I estimate that this will take me T4h because I have to:

• Log-in to the Linode instance and inspect how many directories have been created
  (this is the number of people - GitHub usernames - using the hits badge)
• zip the data on Linode instance to get it off the instance - because I will be running the Elixir script on my localhost.
• download the data to localhost
• unzip data on localhost
• Download and install Postgres on Mac: https://postgresapp.com/downloads.html
• write the script
  • insert data into hits_dev PostgreSQL on localhost
• run the script
• export the data from hits_dev PostgreSQL on localhost
• load the data on the remote server

[nelsonic]

While attempting to run the tar command:

tar -zcvf logs.tar.gz logs/

as per
https://unix.stackexchange.com/questions/93139/can-i-zip-an-entire-folder-using-gzip
we received the following error:

tar: logs: file changed as we read it

Now deleting the .gz files recursively:

find . -name "*.gz" -type f -delete

https://askubuntu.com/questions/377438/how-can-i-recursively-delete-all-files-of-a-specific-extension-in-the-current-di

Temporarily stopping the node.js server in order to perform the archive.

forever stop server.js

Now running:

tar -zcvf logs.tar.gz logs/

And it's taking forever ... ⏳

[nelsonic]

https://stackoverflow.com/questions/9427553/how-to-download-a-file-from-server-using-ssh
Format:

scp [email protected]:foobar.txt /local/dir

Actual:

scp [email protected]:hits/logs.tar.gz ./logs.tar.gz

Worked:

[nelsonic]

Ecto refresher: https://geoffreylessel.com/2016/from-zero-to-ecto-in-10-minutes

[nelsonic]

Processing large files: https://www.poeticoding.com/processing-large-csv-files-with-elixir-streams

[nelsonic]

Roadblock: dwyl/learn-devops#55

[nelsonic]

https://serverfault.com/questions/264595/can-scp-copy-directories-recursively

scp -rp [email protected]:hits/logs ./

scp -rp [email protected]:hits/logs ./

in-progress:

heading to bed let's see if it works overnight.

[nelsonic]

This is why we can't have nice things: