Merge pull request #58 from KnowWhereGraph/deployment-fixes

Deployment fixes

.gitignore

@@ -1,12 +1,13 @@
+nginx/logs/
 .DS_STORE
-nginx/logs
 graphdb/license/
-nginx/local-certs/
+nginx/letsencrypt/local-certs/*
 nginx/sites/faceted-search/
 nginx/sites/kw-panels/
 nginx/sites/node-browser/
+kwg-api/
 services/
 graphdb/nginx/*
 grafana/persistent_config
 *drawio.bkp
-*.png.bkp
+*.png.bkp

elasticsearch/README.md

@@ -2,13 +2,11 @@
 
 Text search & ranking for KnowWhereGraph
 
-## Deploying
-
-The docker-compose file is used for deploying the service. Before deploying, set the password in the docker-compose file.
-
 ## Credentials
 
-```
+Credentials need to be set for Elasticsearch before the stack is deployed. Set this in the docker-compose.yaml file.
+
+```text
 user: elastic
 pass: <refer to docker-compose.yaml>
 ```

grafana/README.md

@@ -37,3 +37,15 @@ The dashboards are taken from pre-designed packages, which are available on the
 - [Node Exporter](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)
 - [NGINX](https://grafana.com/grafana/dashboards/14900-nginx/)
 - KWG-API: Custom
+
+## Data Sources
+
+Grafana makes use of two data sources
+
+### Prometheus
+
+Prometheus stores the metrics data and requires the prometheus user/pass to connect. This is set in the datasource yaml file, and the credentials should match the prometheus docker-compose credentials.
+
+### Loki
+
+Loki is the log data source. The communication with this service is unauthenticated.

graphdb/README.md

@@ -11,16 +11,10 @@ There are *six* docker-compose files here. The two main flavors are
 1. Preloading: These compose files are used to the first upload of data. There are three (local/stage/prod)
 2. Running: These compose files are used when running GraphDB to serve content. There are three (local/stage/prod)
 
-
-
 ## Data Persistence
 
 Data is persisted on the host machine, _not_ the container. This is achieved by a volume mount between the host and GraphDB's repository data directory which is set in the docker-compose file. Graph DB stores its repository, configuration, and logging data under `/opt/graphdb/home`. This path can be mounted to the local system, persisting the data. When a new container is launched, it will reference the persisted data and load it.
 
-## Deploying
-
-GraphDB deployments should be managed by the repositories root Makefile. Run `make help` for a description of commands and follow the documentation below to learn more about loading data & deploying.
-
 ### Initial Data Load
 
 GraphDB's initial database is constructed using the `importrdf` tool from Ontotext. This runs with GraphDB offline and offers much faster data loading than other options. In this process, GraphDB creates a new repository and inserts data into it. To account for this, separate docker-compose files are needed to manage the offline instances.

loki/docker-compose.yaml

@@ -2,7 +2,7 @@ version: "3"
 
 services:
   loki:
-    image: grafana/loki:3.2.2
+    image: grafana/loki:2.9.11
     container_name: loki
     ports:
       - "3100:3100"

makefile

@@ -34,6 +34,7 @@ repository-setup: # Fetches the GitHub repos needed & configures them
 	git clone https://github.com/KnowWhereGraph/kwg-faceted-search.git nginx/sites/kwg-faceted-search
 	git clone https://github.com/KnowWhereGraph/kwg-api.git kwg-api/
 	git clone https://github.com/KnowWhereGraph/kwg-ontologies.git nginx/sites/onto/
-	cp nginx/robots.txt nginx/sites/robots.txt
+	mkdir nginx/sites/robots/
+	cp nginx/robots.txt nginx/sites/robots/robots.txt
 validate-deployment: # Checks the local deployment files
-	sh validate.sh
+	sh scripts/validate.sh

nginx/README.md

@@ -8,17 +8,19 @@ KnowWhereGraph has a handful of services that require networking capabilities. T
 
 The NGINX configuration files are templated with environment variables, which can be customized in the docker-compose file. For more information on how the configuration files are generated from the templates refer to [this](https://github.com/docker-library/docs/tree/master/nginx#using-environment-variables-in-nginx-configuration-new-in-119) documentation page.
 
-## Deploying
+## Certificates
 
-Deploying nginx without using the make command is *not* recommended and most likely not necessary. If you know what you're doing and need to, you can with
+Certificates are mounted into the nginx container from the local filesystem. Non-local deployments work the same way: the `/etc/letsencrypt` files are mounted into the container.
 
-`docker-compose up`
+### Staging
 
-To bring the service down, run
+New certs can be generated with
 
-`docker-compose down`
+`sudo certbot certonly --webroot --webroot-path nginx/data/certbot/www/  -d staging.knowwheregraph.org`
 
-## Certificates
+ A common problem is that the certs may have different permissions for mounting into the nginx container.
+
+### Local
 
 Certificates for local development need to be manually generated and added to the `local-certs` directory.
 
@@ -39,7 +41,7 @@ https://javorszky.co.uk/2019/11/06/get-firefox-to-trust-your-self-signed-certifi
 
 ## Logging
 
-The NGINX logs are found in the container's /var/logs/nginx, which is mounted locally at `./nginx/logs`. For more verbose logging, refer to the NGINX Docker image documentation and modify the deployment script to include any additional flags.
+The NGINX logs should be checked through grafana or by inspecting the container's logs.
 
 ## Metrics
 

nginx/docker-compose.local.yaml

@@ -9,7 +9,6 @@ services:
       - 'variables.env'
     volumes:
     - ./nginx/templates:/etc/nginx/templates # NGINX config file
-    - ./nginx/logs:/var/log/nginx # NGINX logs
     - ./nginx/sites:/usr/share/nginx/html # Static sites
     - ./nginx/local-certs:/etc/letsencrypt/live/localhost # Local certificates
     depends_on: # These services need to exist for the nginx config to validate

nginx/docker-compose.prod.yaml

@@ -9,9 +9,9 @@ services:
       - 'variables.env'
     volumes:
     - ./nginx/templates:/etc/nginx/templates # NGINX config file
-    - ./nginx/logs:/var/log/nginx # NGINX logs
     - ./nginx/sites:/usr/share/nginx/html # Static sites
-    - ./nginx/local-certs:/etc/letsencrypt/live/stko-kwg.geog.ucsb.edu # Local certificates
+    - /etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/cert.pem:/etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/cert.pem # Local certificates
+    - /etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/privkey.pem:/etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/key.pem # Local certificates
     depends_on: # These services need to exist for the nginx config to validate
       - graphdb
       - kwg-api

nginx/docker-compose.stage.yaml

@@ -9,12 +9,12 @@ services:
       - 'variables.env'
     volumes:
     - ./nginx/templates:/etc/nginx/templates # NGINX config file
-    - ./nginx/logs:/var/log/nginx # NGINX logs
     - ./nginx/sites:/usr/share/nginx/html # Static sites
-    - ./nginx/local-certs:/etc/letsencrypt/live/staging.knowwheregraph.org # Local certificates
+    - ./nginx/data/certbot/www:/var/www/certbot
+    - /etc/letsencrypt/live/staging.knowwheregraph.org/cert.pem:/etc/letsencrypt/live/staging.knowwheregraph.org/cert.pem # Local certificates
+    - /etc/letsencrypt/live/staging.knowwheregraph.org/privkey.pem:/etc/letsencrypt/live/staging.knowwheregraph.org/key.pem # Local certificates
     depends_on: # These services need to exist for the nginx config to validate
       - graphdb
       - kwg-api
     networks:
       - kwg_network
-

nginx/templates/default.conf.template

@@ -1,3 +1,5 @@
+server_names_hash_bucket_size 64;
+
 # Redirect http -> https
 server {
     listen 80 default_server;
@@ -20,9 +22,10 @@ server {
 
     # Certbot certificate challenge
     location /.well-known/acme-challenge/ {
-        root /var/www;
+	alias /var/www/certbot/;
     }
 
+
    location /stub_status {
         stub_status on;
 	allow 172.17.0.0/16;
@@ -685,8 +688,7 @@ server {
 
     # Robots.txt
     location /robots.txt {
-        alias /usr/share/nginx/html/robots.txt;
+        alias /usr/share/nginx/html/robots/robots.txt;
     }
 
-
 }

validate.sh → scripts/validate.sh

@@ -3,27 +3,11 @@ ORANGE='\033[0;31m'
 DEFAULT='\033[0m'
 GREEN='\033[0;32m'
 
-
-BUILD_FILES_PROD := docker-compose.yaml -f nginx/docker-compose.prod.yaml -f nginx/metrics/docker-compose.yaml -f graphdb/docker-compose.prod.yaml -f elasticsearch/docker-compose.yaml -f prometheus/docker-compose.yaml -f kwg-api/docker-compose.prod.yaml -f grafana/docker-compose.prod.yaml -f loki/docker-compose.yaml
-BUILD_FILES_LOCAL := docker-compose.yaml -f nginx/docker-compose.local.yaml -f nginx/metrics/docker-compose.yaml -f graphdb/docker-compose.local.yaml -f elasticsearch/docker-compose.yaml -f prometheus/docker-compose.yaml -f kwg-api/docker-compose.local.yaml -f grafana/docker-compose.local.yaml -f loki/docker-compose.yaml
-BUILD_FILES_STAGE := docker-compose.yaml -f nginx/docker-compose.stage.yaml -f nginx/metrics/docker-compose.yaml -f graphdb/docker-compose.stage.yaml -f elasticsearch/docker-compose.yaml -f prometheus/docker-compose.yaml -f kwg-api/docker-compose.stage.yaml -f grafana/docker-compose.stage.yaml -f loki/docker-compose.yaml
-
-
-
-
-
 echo "\n\n===== KnowWhereGraph Deployment Configuration Test ====="
 echo "=====                                              ====="
 echo "                  Checking SSL Certificates                  "
 echo "                  .........................                  "
 
-# Check for nginx's local certificates
-if [ ! -f ./nginx/local-certs/cert.ped ]; then
-    echo "${ORANGE}Warning${DEFAULT}: Local certificate not found in 'nginx/local-certs' folder!\nCertificates are required for deploying the system. Refer to the README.md for instructions on self signed certs."
-else
-    echo "${GREEN}Good${DEFAULT}: Located local certificates for nginx."
-fi
-
 # Check for graphdb's local certificates
 if [ ! -f ./nginx/local-certs/cert.ped ]; then
     echo "${ORANGE}Warning${DEFAULT}: Local certificate not found in 'graphdb/local-certs' folder!\nGraphDB may not process SSL data properly!"
@@ -64,25 +48,32 @@ fi
 
 # Check for Node Browser dist
 if [ ! -d nginx/sites/node-browser/node-browser/dist/node-browser ]; then
-    echo "${ORANGE}Error${DEFAULT}: Failed to locate the node browser's build artifacts. Try building the source with its docker-compose file.\n"
+    echo "${ORANGE}Error${DEFAULT}: Failed to locate the node browser's build artifacts. Try building the source with its docker-compose file."
 else
     echo "${GREEN}Good${DEFAULT}: Located the node browser build artifacts."
 fi
 
+# Check for ontology repo
+if [ ! -d nginx/sites/onto ]; then
+    echo "${ORANGE}Error${DEFAULT}: Failed to locate the ontology files. Try pulling from GitHub"
+else
+    echo "${GREEN}Good${DEFAULT}: Located the ontology artifacts."
+fi
+
 echo "\n                  Checking Web Artifacts                      "
 echo "                    ......................                      "
 
 
 # Check for void file
-if [ ! -f nginx/sites/void/void.ttl ]; then
+if [ ! -f nginx/sites/onto/void.ttl ]; then
     echo "${ORANGE}Error${DEFAULT}: Void ttl file not detected!\n This file comes from the kwg-ontologies repository. Check the makefile for the clone command."
 else
     echo "${GREEN}Good${DEFAULT}: Located the void.ttl file."
 fi
 
 
 # Check for robots.txt
-if [ ! -f nginx/sites/robots.txt ]; then
+if [ ! -f nginx/sites/robots/robots.txt ]; then
     echo "${ORANGE}Error${DEFAULT}: Failed to locate robots.txt"
 else
     echo "${GREEN}Good${DEFAULT}: Located the robots.txt"
@@ -93,40 +84,25 @@ echo "                    .........................                "
 
 # Check for an exiting Grafana database file
 if [ -f grafana/persistent_storage/grafana.db ]; then
-    echo "${ORANGE}Warning${DEFAULT}: Found an existing Grafana database file. Any Grafana password changes will not persist."
+    echo "${ORANGE}Info${DEFAULT}: Found an existing Grafana database file. Any Grafana password changes will not persist."
 else
-    echo "${GREEN}Good${DEFAULT}: Failed to find an existing Grafana database file. A new one will be created"
+    echo "${GREEN}Info${DEFAULT}: Failed to find an existing Grafana database file. A new one will be created"
 fi
 
 # Check for any existing GraphDB repositories
 
 if [ -d graphdb/ ]; then
-    echo "${ORANGE}Info${DEFAULT}: Found existing GraphDB repositories. These will be loaded when GraphDB starts."
+    echo "${GREEN}Info${DEFAULT}: Found existing GraphDB repositories. These will be loaded when GraphDB starts."
 else
     echo "${GREEN}Info${DEFAULT}: Failed to find any GraphDB repositories. GraphDB will not be started with any repositories."
 fi
 
 echo ""
-echo""
-
-
-echo "\n                  Checking Variables                       "
-echo "                    .........................                "
-
-# Check for an exiting Grafana database file
-if [ -f grafana/persistent_storage/grafana.db ]; then
-    echo "${ORANGE}Warning${DEFAULT}: Found an existing Grafana database file. Any Grafana password changes will not persist."
-else
-    echo "${GREEN}Good${DEFAULT}: Failed to find an existing Grafana database file. A new one will be created"
-fi
-
-# Check for any existing GraphDB repositories
-
-if [ -d graphdb/ ]; then
-    echo "${ORANGE}Info${DEFAULT}: Found existing GraphDB repositories. These will be loaded when GraphDB starts."
-else
-    echo "${GREEN}Info${DEFAULT}: Failed to find any GraphDB repositories. GraphDB will not be started with any repositories."
-fi
+echo ""
 
+echo "\n\n===== KnowWhereGraph Deployment Configuration Test End ====="
+echo "                                                             "
+echo "                  Finished Deployment Check                  "
+echo "                  .........................                  "
 echo ""
-echo""
+echo "=============================================================\n\n\n"