Post build configuration clarifications request

[developerdudeone]

Hi Pascal,

I created a new thread in order to focus on the post build required configuration.
I would like to clarify with your help few of the issues I encountered.
Please receive my thanks in advance for your time and help.

Grafana and Monitor Roles

This is more of a question to double check my understanding.
After looking at your diagram the following is my understanding:

• The Monitor and Grafana images run as separate containers from the Relay or Node Producer.
• Prometheus is gathering metrics from both Relay and Node Producer
• Grafana is consolidating the data read by Prometheus into the dashboards.

Regarding the Config folder path for the Grafana and Monitor

As in my last question I assume the required configuration files are outside of the container.
If that is the case should I create the config folder in this path? :

• /home/pi/Monitor/config
• /home/pi/Grafana/config

Regarding the Grafana and Monitor images

The Grafana docker image was built successfully.
During the Docker build for the Monitor image the following error was encountered:

sudo docker build -t cardano_monitor:latest ./Dockerfiles/monitor

Sending build context to Docker daemon 6.144kB
Step 1/35 : FROM ubuntu:focal
---> 306e6575a9e4
Step 2/35 : LABEL name="cardano_monitor"
---> Running in a8ce0c1aa406
Removing intermediate container a8ce0c1aa406
---> 39f5feb92535
Step 3/35 : LABEL description="Monitoring tools for Cardano node and its host"
---> Running in 2a156b07f114
Removing intermediate container 2a156b07f114
---> 0f89c901b641
Step 4/35 : LABEL maintainer="https://github.com/pascallapointe"
---> Running in 9ad753e15c65
Removing intermediate container 9ad753e15c65
---> 1382f2960d41
Step 5/35 : WORKDIR /cardano
---> Running in 6c5e5cb9b6fc
Removing intermediate container 6c5e5cb9b6fc
---> 8321d0cc194f
Step 6/35 : RUN apt-get update
---> Running in 309bd68d5646
Err:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Temporary failure resolving 'ports.ubuntu.com'
Err:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Temporary failure resolving 'ports.ubuntu.com'
Err:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Temporary failure resolving 'ports.ubuntu.com'
Err:4 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Temporary failure resolving 'ports.ubuntu.com'
Reading package lists...
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/focal/InRelease Temporary failure resolving 'ports.ubuntu.com'
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/focal-updates/InRelease Temporary failure resolving 'ports.ubuntu.com'
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/focal-backports/InRelease Temporary failure resolving 'ports.ubuntu.com'
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/focal-security/InRelease Temporary failure resolving 'ports.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Removing intermediate container 309bd68d5646
---> c29dd19a1e0e
Step 7/35 : RUN apt-get install -y wget
---> Running in 46bf5cf9479c
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package wget
The command '/bin/sh -c apt-get install -y wget' returned a non-zero code: 100

on the subsequent run the following was the error:
sudo docker build -t cardano_monitor:latest ./Dockerfiles/monitor
Sending build context to Docker daemon 6.144kB
Step 1/35 : FROM ubuntu:focal
---> 306e6575a9e4
Step 2/35 : LABEL name="cardano_monitor"
---> Using cache
---> 39f5feb92535
Step 3/35 : LABEL description="Monitoring tools for Cardano node and its host"
---> Using cache
---> 0f89c901b641
Step 4/35 : LABEL maintainer="https://github.com/pascallapointe"
---> Using cache
---> 1382f2960d41
Step 5/35 : WORKDIR /cardano
---> Using cache
---> 8321d0cc194f
Step 6/35 : RUN apt-get update
---> Using cache
---> c29dd19a1e0e
Step 7/35 : RUN apt-get install -y wget
---> Running in 0520d480293f
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package wget
The command '/bin/sh -c apt-get install -y wget' returned a non-zero code: 100

[pascallapointe]

From your question i've notice that my diagram can be misleading. I will fix that, I think I wasn't settle yet on what port I would used when a draw that one..!

Real configuration below:
Prometheus is currently listening on port 3200
Grafana is listening on port 3100

Grafana and Monitor Roles

Yes, this is it. When cardano-node is running, it stream metrics on port 12798. Prometheus read from that feed and save it to its database. Then Grafana display a dashboard using data from your prometheus data source.

Actually you can configure prometheus to read from different node, so you wouldn't require to run a prometheus container along with each of your node. But it would require more configuration so I choose to leave it that way.

Regarding the Config folder path for the Grafana and Monitor

This step is optional, it only allow you to override the default configuration.
To answer your question, yes your folder structure will work perfectly, just remember to get it the right folder before creating the container, or to adjust the docker run command with the right path.

Regarding the Grafana and Monitor images

From the log you sent me, I can see that there's been an issue with ubuntu repositories. The apt-get package manager didn't updated properly so wget could not be installed.
Try building again the image from scratch by adding --no-cache to the docker build command.
I've successfully builded the image after seing your issue so it should work now.

Cheers!

[developerdudeone]

Hi Pascal,

Thank you for answering.
What would be the recommended approach on Monitoring?

• Should I run monitoring on my Relay Node or the Node Producer?

All the best,
Alex

[developerdudeone]

Hi Pascal,

Just to let you know that the link for the Installation of db-sync & graphql is broken:

• https://github.com/pasklab/cardano-docker/blob/pool-monitor/Docs/db-sync.md

I assume I need to install these also.

All the best,
Alex

[pascallapointe]

No, you absolutly don't need db-sync nor graphql.
They are part of other development that is not completed yet, the documentation link as leaked from an other private repository.
I will remove the link for now, thanks for letting me know.

[developerdudeone]

Thanks a lot for clarifying.

[developerdudeone]

Hi Pascal,

I have few more questions, I would greatly appreciate if you could find some time and clarify these for me.
These questions are mostly about creating the keys for the Node Producer in the context of a docker container.

Understanding the interaction with a Docker container

I have to admit I still have difficulty understanding how to interact with the container once created, mostly from the point of view of copying files from the container out or to bring in certain files.
I did read the Docker documentation which shows that we can run a command inside of a container with docker exec
In that respect I have this particular question:

• In order to use docker exec is it necessary to have the container running?

Creating the Node Producer Keys

In my case I plan to create these keys on my Docker container of Node Producer Pi while is completely disconnected from internet.
After the creation of these keys I want to copy them from the Node Producer container into a secure location (I have a secure USB).
In that respect my question is:

• Can I copy these keys using docker exec to a local folder on Pi?

Would be this recommended
I wonder if would be more practical that these keys and the cert to be located in the ~cardano/config folder outside of the Docker container... The node configuration files are already outside of the container.
Would be this a security risk?

Note
I noticed that you have a script on updating the keys and the cert.
However I see that the script is using a setting for the Cardano Testnet.
I guess the following option in your script can be replaced with: --mainnet

#!/bin/bash

USERNAME="root"

TESTNET_MAGIC=42

CNODE_BIN="/root/.cabal/bin"
export PATH="${CNODE_BIN}:${PATH}"
export CARDANO_NODE_SOCKET_PATH="/root/node_data/socket"

SLOT_PER_PERIOD=3600
TIP=$(cardano-cli shelley query tip --testnet-magic ${TESTNET_MAGIC} | grep -oP 'SlotNo = \K\d+')
KES_PERIOD=$(expr $TIP / $SLOT_PER_PERIOD)

cardano-cli shelley node issue-op-cert --kes-verification-key-file kes.vkey --cold-signing-key-file node.skey --operational-certificate-issue-counter node.counter --kes-period $KES_PERIOD --out-file node.cert

should look like that:

• TIP=$(cardano-cli shelley query tip --mainnet | grep -oP 'SlotNo = \K\d+')

Thanks a lot for your time and patience,
Alex

[pascallapointe]

Hi Alex,

Understanding the interaction with a Docker container

In order to use docker exec is it necessary to have the container running?

Yes, the container must be running. Think of it as a virtual machine.
You can use theses 2 useful commands to get in the container:

docker exec -it cardano_node bash
docker exec -it --user root cardano bash

NOTE: The later one allow you to login as ROOT user. This is particularly useful when needing to fix file ownership and access right.

Creating the Node Producer Keys
Can I copy these keys using docker exec to a local folder on Pi?

Yes, there is 3 way I can think of:

1. Using the following command will allow you to copy a file from the container to your host:

   docker cp cardano_node:/cardano/YOUR_FILE /home/host_username/some_directory/.
   

   The reverse is also possible:

   docker cp local_file.txt cardano_node:/cardano/.
   

2. You can use the config folder to exchange files between the container and the host since the folder is a "bind" volume. IMPORTANT: If you are using docker-compose to bring up your container, you need to edit the docker-compose.yaml file and set the config volume configuration read-only to false. Otherwise the container can't write to this folder.

3. While cardano-node binary won't work outside the container since there would be missing dependencies that only live inside your container, cardano-cli can be copied to your host system and should work without any issue. This is actually what i'm doing to avoid working inside the container. (I prefer using my colorfull bash terminal..!)

    docker cp cardano_node:/cardano/bin/cardano-cli /home/your_username/.
   

   From there you can generate everything you need using cardano-cli binary.

Would be this recommended

I think that question refer to generating the files and moving them out before connecting to the internet? I don't see any issue as long as you are using only the terminal and properly erase the files later on with rm. The best way of doing it would be having a 3rd Pi that would never be connected to internet, copy cardano-cli binary on it and then generate files from there.

I wonder if would be more practical that these keys and the cert to be located in the ~cardano/config folder outside of the Docker container... The node configuration files are already outside of the container.
Would be this a security risk?

The keys and certificates files that you will need to run the block producer will have to be in that config folder. The scripts to start the producer node included in my dockerfiles will look in that folder to find the following files:

config/kes.skey
config/vrf.skey
config/node.cert

I noticed that you have a script on updating the keys and the cert.

I don't actually use that script...! I might have put it there as a mean to save it before I made that repository public. I'll will remove that file. My repository don't provide "pool creation" instruction, only pool node management. The reason is that I couldn't provide reliable scripts and keep them up to date with cardano docs since I don't have enough free time for it.

But you can definitely update that script to fit your own need!

Cheers,
Pascal

[developerdudeone]

Hi Pascal,

I tried to use docker-compose but I get the following error:

Recreating cardano_node ... error

ERROR: for cardano_node Cannot start service cardano_node: OCI runtime failed: container_linux.go:367: starting container process caused exec: "start-producer.sh": executable file not found in $PATH: unknown

I copied the start-producer.sh in the folders: ~/cardano and also in ~/cardano/config
Do I need to execute docker-compose from your initial folder I downloaded?
Thanks a lot for your time.

All the best,
Alex

[pascallapointe]

Hi Alex!

This error might be happening because there is already an existing container or something.
Make sure to start fresh before using docker-compose.

You can see all running and stopped containers using the following command:

docker ps -a

then remove all of them using:

docker rm d4a5ee703a81
... or with container names ...
docker rm cardano_node

Also, once the cardano_node:1.25.1 image is build, you dont need to copy anything from the repository, all start-up scripts have already been included in the image as you built it.

The only file that need to be copied is docker-compose.yaml because it is using relative path reference. (" ./config, ./db and ./socket)

Get a running relay before trying to start in producer mode. Less chance of something going wrong as it require 3 less config files. You can change the startup mode in docker-compose.yaml by commenting and uncommenting the proper startup script line.

Ok, so in the end, once you have a cardano_node and cardano_monitor built on your system, you need to have the following file and folder structure:

~/Cardano
        ./config
                ./byron-genesis.json
                ./config.json
                ./port.txt
                ./shelley-genesis.json
                ./topology.json
        ./db
        ./socket
        ./docker-compose.yaml

Then you can go in ~/Cardano folder and launch all your container using the following command:

docker-compose up -d

The "-d" is to launch all this in background and get back your cli, you can see what going on with your node using the following command:

docker-compose logs -f --tail=100 cardano_node

Ok, tell me if that all work for you, I would like to know if you made it :)
Also, I just release a new version of this repository, you might be interested in building all images again to get the latest change. The cardano_env will be really faster than the last time you did it, you don't have to compile GHC anymore, just have to install it as ARM is now supported in version 8.10.4

Cheers!

[developerdudeone]

Hi Pascal,

Thanks for taking your time and explaining.
The status of what I am doing is this one:

Status

1. I compiled successfully all images and created their containers for both Relay and Producer
2. Everything started properly in Docker other than two issues I encountered.

Issues

The first issue is related to the Cardano Node.
I started the node with the Topology Updater from your docker compose.
However by looking at the docker logs I could see the that the file named topologyUpdater.sh still makes reference to the old name which is genesis.json
Because of that it fails to get the topology. This particular issue can be seen at the line 23.
23. GENESIS_JSON="$CNODE_CONFIG_DIR}/genesis.json"
Obviously I modified this line with the new name. However maybe you can update this particular line with the new name:

• shelley-genesis.json

Second issue is related to the Grafana
It seems the Grafana wants to create a log folder but it gets into a permission denied from the Ubuntu os.
I did change the permissions on the folder data that other user could create this folder.
However it did not work. The full path it needs is this one:

• /cardano/grafana/data/log

By looking at the Grafana docker file I see you use the user: cardano inside that image.
I am thinking maybe I need to add this cardano user to the docker group:

• sudo user mod -aG docker cardano - I will try it to see what happens

Cardano Node New Version Released

I noticed yesterday that Cardano just released a new version of their node: 1.26.1

Question
You mentioned once that the image once created can be updated easy.
I do not know how to do that, other than recompiling it when is a new version.
Could you give me an example how is done?

Thanks a lot for your time,
Alex

[developerdudeone]

Hi Pascal,

Today trying to run the script updateTopology.sh I was getting an error saying the blockNo is null.
Seems the new version of the node changed some things in the cardano-cli.
The changes can be seen here:

• https://forum.cardano.org/t/topologyupdater-sh-invalid-blockno-null-cardano-cli-query-json-key-changed-with-v1-26-1/56711

Basically the following is changed as of 1.26.1 version:

The ‘tip’ query now additionally returns the epoch at the tip. Some of the names of fields returned have changed and will need to be modified accordingly:
“blockNo” is now “block”
“slotNo” is now “slot”
“headerHash” is now “hash”

In my case since I could not replace the script in the image I recompiled the node again with the script modified.
I tried to copy the cardano-cli from the container as suggested by you, however it did not copied anything on my local folder. Not sure why...

All the best,
Alex

[pascallapointe]

Hi Alex!

Genesis file in topologyUpdater

The genesis.json in topology updater was named this way because, back in September, I tough that byron genesis would be no more required after shelley hard fork, so I've named it genesis.json right away in the script.. So I did run with a copy of shelley-genesis.json copy named genesis.json all this time...! Never change a running system as they say.

So ok, time for change, I did renamed it in the script to shelley-genesis.json. The things is that you'll need to build the image again from the master branch to have the latest change. I think that you need to upgrade to cardano-node 1.26.1, so you'll have to build again anyway.

Grafana issue

The path /cardano/grafana/data/log is in the container, that path doesn't existe on your host system. Also, the cardano user only exist in the container. So running commands without going in the container won't help.

I might have run into this issue and fixed folder ownership and right when I first created that without remembering :/
I'll need to investigate this one further to better assist you. But here what I think is happening:

When you first create the container following the instructions I provided, your create a volume named grafana-data binded to the path /cardano/grafana/data in your grafana container. Here we are leveraging docker volume to avoid loosing everything you do in Grafana everytime you recreate the Grafana container.

The thing going on here is that, when the volume is first created, it belongs to the user root instead of the user cardano ...!

Try changing ownership on that folder launching the chown command using root user:

docker exec -it --user root grafana chown -R /cardano/grafana/data

Tell me if that fix your issue. I'll update my Dockerfile as soon as I can to fix that issue.

How update cardano-node

To update cardano-node on a running setup, do the following:

# Get the latest version of this repository
git pull
git checkout <master or lastest release version>

# Set the 2 env variables
ARCHITECTURE=aarch64
VERSION_NUMBER=1.26.1

#Build the latest version
 docker build \
 --build-arg ARCHITECTURE=${ARCHITECTURE} \
 --build-arg RELEASE=${VERSION_NUMBER} \
 -t cardano_node:${VERSION_NUMBER} Dockerfiles/node

# Tag the newly built image with "latest" tag
docker tag cardano_node:${VERSION_NUMBER} cardano_node:latest

# Docker compose will notice the image change and recreate only the required containers, run:
docker-compose up -d

The script updateTopology.sh

Yes, there is in deed some change blockno and slotno parameters, I'll need to fix that in the dockerfile, than you'll need to build it with the latest changes. This way your container will be able to send its status properly when launching it with start-with-topology.sh.

This will be fix in the next release of this repository. I'll try to do it ASAP

Thanks for providing useful comments on this repository!

[pascallapointe]

Also, to copy cardano-cli to your home directory, run the following:

docker cp cardano_node:/cardano/bin/cardano-cli ~/.