Neo4j ETL Components

Data import from relational databases to Neo4j.

Download & Install

Download & unzip the latest neo4j-etl-components.zip from https://github.com/neo4j-contrib/neo4j-etl-components/releases

Examples of command usage:

Minimal command line

./bin/neo4j-etl mysql export  --user <user> --password <pwd> --database northwind \
 --destination $NEO4J_HOME/data/databases/graph.db/ --import-tool $NEO4J_HOME/bin  \
 --csv-directory /tmp/northwind

Full set of command line options

./bin/neo4j-etl mysql export  --host 127.0.0.1 --port 3306 --user <user> --password <pwd> \
--database <database> --destination $NEO4J_HOME/data/databases/graph.db/ \
--import-tool $NEO4J_HOME/bin  --csv-directory /tmp  \
--options-file ./import-tool-options.json --force --debug

For detailed usage see also the: tool documentation.

Quick Build Instructions

To run the command line tool, once you cloned the project, build the project locally.

From the root directory, run:

git clone https://github.com/neo4j-contrib/neo4j-etl-components
cd neo4j-etl-components
mvn clean package -DskipTests

Once that has been successfully executed, the scripts are located in the neo-etl-cli/bin directory.

License

This tool is licensed under the GPLv3.

Issues & Feedback & Contributions

• You can raise GitHub issues or send feedback to [email protected]

• Please also join our neo4j-users Slack and ask in the #help-import channel

We’re welcoming all contributions.

• Please first raise an issue so that we can discuss your ideas and avoid duplicate work

• Send your changes as a pull request

• Make sure to provide tests with your changes

Prerequisites

• Maven

• MySql

Integration Tests

You can run the tests with

• a local MySQL instance,

• Vagrant

• or in AWS.

You will need a MySQL user neo4j with password neo4j with admin privileges to run the tests.

To run the tests in AWS, you’ll need an AWS IAM user.

To run the tests using a local MySQL instance:

Grant all privileges to the user neo4j identified by password neo4j

CREATE USER 'neo4j'@'localhost' IDENTIFIED BY 'neo4j';
GRANT ALL PRIVILEGES ON *.* TO 'neo4j'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;

mvn -DPLATFORM=local clean integration-test

To run the tests using Vagrant:

mvn -DPLATFORM=vagrant clean integration-test

To run the tests using AWS:

Note: You need to create AWS Keypair and have the credentials file created to do this

mvn -DPLATFORM=aws -DEC2_SSH_KEY=<name of your EC2 SSH key> clean integration-test

If you don’t supply a PLATFORM parameter, the tests will attempt to use Vagrant.

Performance Tests

Set of tests that are part of the neo4j-etl-it module.

However, they are skipped usually when you run the integration-test target by default. You can run them separately as part of a test suite.

To run performance tests in AWS:

mvn clean dependency:copy-dependencies integration-test -Dtest=PerformanceTestSuite -DfailIfNoTests=false -DPLATFORM=aws -DEC2_SSH_KEY=<name of your EC2 SSH key>

To run performance tests locally,

Ensure that you have the datasets imported in your local mysql instance. You could do this by uncommenting the bits in the code to download the sql file from S3.

mvn clean dependency:copy-dependencies integration-test -Dtest=PerformanceTestSuite -DfailIfNoTests=false -DPLATFORM=local