Nifi-Docker

NiFi docker container customization from the official Apache Nifi Docker Image

1. Build Docker

Pull the latest version of Apache Nifi Docker, build and run

$ docker pull apache/nifi:1.27.0

Option 1: Run without authentication

$ docker run --log-opt max-size=100m --name noharm-nifi -e NIFI_WEB_HTTP_PORT='8080' -p 8080:8080 -d apache/nifi:1.27.0 --restart=always

Open the URL http://localhost:8080/nifi.

Option 2: Run with authentication

docker run --log-opt max-size=100m --name noharm-nifi -e NIFI_WEB_HTTPS_PORT='8443' -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=nifi_noharm -e SINGLE_USER_CREDENTIALS_PASSWORD=nifi_pass   apache/nifi:1.27.0 --restart=always

Watch until the container is ready. You are looking for the "NiFi has started. The UI is available" message.

docker logs -f noharm-nifi

Open the URL https://localhost:8443/nifi

2. Download Connectors and Add Timezone

Dive into container shell, add timezone and download JARs & NARs

$ docker exec --user="root" -it nifi /bin/bash
$ echo "java.arg.8=-Duser.timezone=America/Sao_Paulo" >> conf/bootstrap.conf
$ wget https://raw.githubusercontent.com/noharm-ai/nifi-composer/refs/heads/main/genkeypair.sh
$ chmod +x genkeypair.sh
$ ./genkeypair.sh
$ cd lib

Copy & Paste it

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar
wget https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/23.2.0.0/ojdbc8-23.2.0.0.jar
wget https://jdbc.postgresql.org/download/postgresql-42.7.3.jar
wget https://repo1.maven.org/maven2/org/apache/nifi/nifi-kite-nar/1.15.3/nifi-kite-nar-1.15.3.nar
wget https://truststore.pki.rds.amazonaws.com/sa-east-1/sa-east-1-bundle.pem

• Updated JDBC PostgreSQL Driver at https://jdbc.postgresql.org/download
• Updated JDBC Oracle Driver at https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html
• Updated JDBC MySQL Driver at https://dev.mysql.com/downloads/connector/j/

Configure aws

$ docker exec --user="root" -it noharm-nifi apt update
$ docker exec --user="root" -it noharm-nifi apt install nano vim awscli -y
$ exit
$ docker exec --user="nifi" -it noharm-nifi /bin/bash
$ aws configure
Access key ID and Secret access key: available at https://noharm.odoo.com/odoo/knowledge/34/knowledge/72/knowledge/92
Region: sa-east-1
Output format: None
$ aws s3 ls #para testar o acesso
$ exit
$ docker restart noharm-nifi

4. Access Nifi Web Service

Now you can access your Nifi web service at http://localhost:8080/nifi/ or https://localhost:8443/nifi/

If you are in a VPN environment remember to tunnel 8080 port

$ ssh -f user@ip_address -L 8080:localhost:8080 -N

How to tunnel in Putty (for Windows users)

5. Basic Nifi Usage

This wiki page will cover the main proccess of NoHarm integration: Basic Nifi Usage

Troubleshooting

a. Testing JDBC Connection

• Download tool from http://jdbcsql.sourceforge.net/
• Run: java -jar jdbcsql-1.0.zip -h hostname -p 1521 -U user -P pass -d service -m oracle "SELECT table_name FROM all_tables"

b. Install SQL Plus

• Install:

sudo yum install -y yum install https://download.oracle.com/otn_software/linux/instantclient/216000/oracle-instantclient-basic-21.6.0.0.0-1.el8.x86_64.rpm
sudo yum install -y yum install https://download.oracle.com/otn_software/linux/instantclient/216000/oracle-instantclient-sqlplus-21.6.0.0.0-1.el8.x86_64.rpm

• Run:

sqlplus user/pass@localhost:1521/service

SQLPlus for Debian: https://askubuntu.com/questions/159939/how-to-install-sqlplus

c. Repair "no space left" issue

Prune useless container and images:

$ docker system prune -a

Check fuller folders (if possible):

$ docker exec -t nifi sh -c "du -sh *"

Empty target folder (usually content_repository):

$docker exec -t nifi sh -c "rm content_repository/* -rf"

Restart nifi to repair empty folder:

docker exec -t nifi sh -c "./bin/nifi.sh restart"

You can do the same inside the container:

$ docker exec --user="root" -it nifi /bin/bash

d. Fix "docker.sock permission denied" issue

sudo chown $USER /var/run/docker.sock

e. Restarting Docker service if Docker is down:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

sudo service docker start

e. Changing Properties from HTTPS to HTTP:

• https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L53-L69