Skip to content

Latest commit

 

History

History
155 lines (96 loc) · 10.4 KB

README.md

File metadata and controls

155 lines (96 loc) · 10.4 KB

Dashboard example for The Things Network

This repository contains a complete example that grabs device data from The Things Network, stores it in a database, and then displays the data using a web-based dashboard.

You can set this up on a "Ubuntu + Docker" VM from the Microsoft Azure store (or on a Ubuntu VM from DreamCompute, or on a Docker droplet from Digital Ocean) with minimal effort. You should set up this service to run all the time so as to capture the data from your devices; you then access the data at your convenience using a web browser.

This dashboard uses docker-compose to set up a group of four primary docker containers, backed by two auxiliary containers:

  1. An instance of Apache, which proxies the other services, handles access control, gets SSL certificates from Let's Encrypt, and faces the outside world.
  2. An instance of Node-RED, which processes the data from the individual nodes, and puts it into the database.
  3. An instance of InfluxDB, which stores the data as time-series measurements with tags.
  4. An instance of Grafana, which gives a web-based dashboard interface to the data.

The auxiliary containers are:

  1. influxdb-backup, which (if configured) runs periodic backups.
  2. postfix, which (if configured) handles outbound mail services for the containers.

To make things more specific, most of the description here assumes use of Microsoft Azure. However, I have tested this on Ubuntu 16 LTS without difficulty (apart from the additional complexity of setting up apt-get to fetch docker, and the need for a manual install of docker-compose), on DreamCompute, and on Digital Ocean. I believe that this will work on any Linux or Linux-like platform that supports docker, docker-compose, and node-red. It's likely to run on a Raspberry Pi... but as of this writing, this has not been tested.

Definitions

  • The host system is the system running Docker and Docker-compose.
  • A container is one of the virtual systems running under Docker on the host system.
  • A file on the host is a file on the host system (typically not visible from within the container(s).
  • A file in container X (or a file in the X container) is a file in a file-system associated with container X (and typically not visible from the host system).

Security

All communication with the Apache server are encrypted using SSL with auto-provisioned certificates from Let's Encrypt. Grafana is the primary point of access for most users, and Grafana's login is used for that purpose.

Access to Node-RED and InfluxDB is via special URLs (base/node-red/ and base/influxdb/, where base is the URL served by the Apache container). These URLs are protected via Apache htpasswd and htgroup file entries. These entries are files in the Apache container, and must be manually edited by an administrator.

The initial administrator's login password for Grafana must be initialized prior to starting the very first; it's stored in grafana/.env. (When you start the Grafana container the first time, it creates grafana.db in the Grafana container, and stores the password at that time. If grafana.db already exists, the password in grafana/.env is ignored.)

Microsoft Azure, by default, will not open any of the ports to the outside world, so you will need to open port 443 for SSL access to Apache.

For concreteness, the following table assumes that base is server.example.com.

To access Open this link Notes
Node-RED https://server.example.com/node-red/ Port number is not needed and shouldn't be used. Note trailing '/' after node-red.
InfluxDB API queries https://server.example.com/influxdb/:8086 Port number is needed. Also note trailing '/' after influxdb.
Grafana https://server.example.com Port number is not needed and shouldn't be used.

This can be visualized as below: Connection Architecture using SSH

Assumptions

  • Your host system must have docker-compose 1.9 or later (for which see https://github.com/docker-compose -- be aware that apt-get normally doesn't grab this; if configured at all, it frequently gets an out-of-date version).

  • The environment variable TTN_DASHBOARD_DATA, if set, points to the common directory for your data. If not set, docker-compose will quit at startup. (This is by design!)

    • ${TTN_DASHBOARD_DATA}node-red will have your local Node-RED data.
    • ${TTN_DASHBOARD_DATA}influxdb will have your local influxdb data (this is what you should back up)
    • ${TTN_DASHBOARD_DATA}grafana will have your dashboards

Composition and External Ports

Within their containers, the individual programs use their usual ports, but these are isolated from the outside world, except as specified by docker-compose.yml.

In docker-compose.yml, the following ports on the docker host are connected to the individual programs.

  • Apache runs on 80 and 443. (All connections to port 80 are redirected to 443 using SSL).

Remember, if your server is running on a cloud platform like Microsoft Azure or AWS, you need to check the firewall and confirm that the ports are open to the outside world.

Installation

Please refer to SETUP.md for detailed set-up instructions.

Data Files

When designing this collection of services, we had to decide where to store the data files. We had two choices: keep them inside the docker containers, or keep them in locations on the host system. The advantage of the the former is that everything is reset when you rebuild the docker images. The disadvantage of the former is that you lose all your data when you rebuild. On the other hand, there's another level of indirection when keeping things on the host, as the files reside in different locations on the host and in the docker containers.

Data files are kept in the following locations by default.

Component Data file location on host Location in container
Node-RED ${TTN_DASHBOARD_DATA}node-red /data
InfluxDB ${TTN_DASHBOARD_DATA}influxdb /data
Grafana ${TTN_DASHBOARD_DATA}grafana /var/lib/grafana

As shown, you can easily change locations on the host (e.g. for testing). You do this by setting the environment variable TTN_DASHBOARD_DATA to the absolute path (with trailing slash) to the containing directory prior to calling docker-compose up. The above paths are appended to the value of TTN_DASHBOARD_DATA. Directories are created as needed.

Normally, this is done by an appropriate setting in the .env file.

Consider the following example:

$ grep TTN_DASHBOARD_DATA .env
TTN_DASHBOARD_DATA=/dashboard-data/
$ docker-compose up -d

In this case, the data files are created in the following locations:

Component Data file location
Node-RED /dashboard-data/node-red
InfluxDB /dashboard-data/influxdb
Grafana /dashboard-data/grafana

Reuse and removal of data files

Since data files on the host are not removed between runs, as long as you don't remove the files between runs, your data will preserved.

Sometimes this is inconvenient, and you'll want to remove some or all of the data. For a variety of reasons, the data files and directories are created owned by root, so you must use the sudo command to remove the data files. Here's an example of how to do it:

source .env
sudo rm -rf ${TTN_DASHBOARD_DATA}node-red
sudo rm -rf ${TTN_DASHBOARD_DATA}influxdb
sudo rm -rf ${TTN_DASHBOARD_DATA}grafana

Node-RED and Grafana Examples

This version requires that you set up Node-RED, the database and the Grafana dashboards manually, but we hope to add a reasonable set of initial files in a future release.

Connecting to InfluxDB from Node-RED and Grafana

There is one point that is somewhat confusing about the connections from Node-RED and Grafana to InfluxDB. Even though InfluxDB is running on the same host, it is logically running on its own virtual machine (created by docker). Because of this, Node-RED and Grafana cannot use localhost when connecting to Grafana. A special name is provided by docker: influxdb. Note that there's no DNS suffix. If you don't use influxdb, Node-RED and Grafana will not be able to connect.

Logging in to Grafana

  • On the login screen, the user name is "admin". The initial password is given by the value of the variable GF_SECURITY_ADMIN_PASSWORD in grafana/.env. Note that if you change the password in grafana/.env after the first time you launch the grafana containder, the admin password does not change. If you somehow lose the previous value of the admin password, and you don't have another admin login, it's very hard to recover; easiest is to remove grafana.db and start over.

Data source settings in Grafana

  • Set the URL (under HTTP Settings) to http://influxdb:8086.

  • Select the database.

  • Leave user and password blank.

  • Click "Save & Test".

Future work

Although the dashboard is already very useful, it's incomplete. Please refer to TODO.md, and also note that we're considering the following. Check in for updates!

  1. Add a script to setup the passwords initially for Grafana and for access to node-red and Influxdb.
  2. Admin script to show roles and maintain the .htpasswd and .htgroup files.
  3. Add the auto-update cron script -- right now you have to restart in order to get the SSL certs updated. Not a big deal, as the patches-requiring-reboot interval is shorter than the life of the certs, but still, this should be fixed.
  4. Switch to phusion for the base image, instead of Ubuntu.
  5. Provide suitable initial files for Grafana and Node-RED, assuming MCCI sensor nodes.
  6. The initial script should prompt for the data base name.

Acknowledgements

This builds on work done by Johan Stokking of The Things Network for the staging environment. Additional adaptation done by Terry Moore of MCCI.

Other contributors: Olivier Girondel, Murugan Chandrasekar.