diff --git a/.gitignore b/.gitignore index 68cac4c..e2e3fbd 100644 --- a/.gitignore +++ b/.gitignore @@ -1,12 +1,13 @@ +nginx/logs/ .DS_STORE -nginx/logs graphdb/license/ -nginx/local-certs/ +nginx/letsencrypt/local-certs/* nginx/sites/faceted-search/ nginx/sites/kw-panels/ nginx/sites/node-browser/ +kwg-api/ services/ graphdb/nginx/* grafana/persistent_config *drawio.bkp -*.png.bkp \ No newline at end of file +*.png.bkp diff --git a/architecture/architecture.drawio b/architecture/architecture.drawio index 9da6055..17d8278 100644 --- a/architecture/architecture.drawio +++ b/architecture/architecture.drawio @@ -1,16 +1,16 @@ - + - + - + - + - + @@ -18,7 +18,7 @@ - + @@ -42,11 +42,11 @@ - + - + @@ -55,53 +55,53 @@ - + - + - + - + - + - + - + - + - + - + @@ -109,17 +109,17 @@ - + - + - + @@ -127,99 +127,99 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -228,10 +228,113 @@ - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/elasticsearch/README.md b/elasticsearch/README.md index 09fe6dc..0b23b62 100644 --- a/elasticsearch/README.md +++ b/elasticsearch/README.md @@ -2,13 +2,11 @@ Text search & ranking for KnowWhereGraph -## Deploying - -The docker-compose file is used for deploying the service. Before deploying, set the password in the docker-compose file. - ## Credentials -``` +Credentials need to be set for Elasticsearch before the stack is deployed. Set this in the docker-compose.yaml file. + +```text user: elastic pass: ``` diff --git a/grafana/README.md b/grafana/README.md index 39f6f53..a6559f5 100644 --- a/grafana/README.md +++ b/grafana/README.md @@ -37,3 +37,15 @@ The dashboards are taken from pre-designed packages, which are available on the - [Node Exporter](https://grafana.com/grafana/dashboards/1860-node-exporter-full/) - [NGINX](https://grafana.com/grafana/dashboards/14900-nginx/) - KWG-API: Custom + +## Data Sources + +Grafana makes use of two data sources + +### Prometheus + +Prometheus stores the metrics data and requires the prometheus user/pass to connect. This is set in the datasource yaml file, and the credentials should match the prometheus docker-compose credentials. + +### Loki + +Loki is the log data source. The communication with this service is unauthenticated. diff --git a/graphdb/README.md b/graphdb/README.md index 91aa051..27c0f1e 100644 --- a/graphdb/README.md +++ b/graphdb/README.md @@ -11,16 +11,10 @@ There are *six* docker-compose files here. The two main flavors are 1. Preloading: These compose files are used to the first upload of data. There are three (local/stage/prod) 2. Running: These compose files are used when running GraphDB to serve content. There are three (local/stage/prod) - - ## Data Persistence Data is persisted on the host machine, _not_ the container. This is achieved by a volume mount between the host and GraphDB's repository data directory which is set in the docker-compose file. Graph DB stores its repository, configuration, and logging data under `/opt/graphdb/home`. This path can be mounted to the local system, persisting the data. When a new container is launched, it will reference the persisted data and load it. -## Deploying - -GraphDB deployments should be managed by the repositories root Makefile. Run `make help` for a description of commands and follow the documentation below to learn more about loading data & deploying. - ### Initial Data Load GraphDB's initial database is constructed using the `importrdf` tool from Ontotext. This runs with GraphDB offline and offers much faster data loading than other options. In this process, GraphDB creates a new repository and inserts data into it. To account for this, separate docker-compose files are needed to manage the offline instances. diff --git a/loki/docker-compose.yaml b/loki/docker-compose.yaml index a40fd9f..83f90e4 100644 --- a/loki/docker-compose.yaml +++ b/loki/docker-compose.yaml @@ -2,7 +2,7 @@ version: "3" services: loki: - image: grafana/loki:3.2.2 + image: grafana/loki:2.9.11 container_name: loki ports: - "3100:3100" diff --git a/makefile b/makefile index 0f8f1b6..938bebd 100644 --- a/makefile +++ b/makefile @@ -34,6 +34,7 @@ repository-setup: # Fetches the GitHub repos needed & configures them git clone https://github.com/KnowWhereGraph/kwg-faceted-search.git nginx/sites/kwg-faceted-search git clone https://github.com/KnowWhereGraph/kwg-api.git kwg-api/ git clone https://github.com/KnowWhereGraph/kwg-ontologies.git nginx/sites/onto/ - cp nginx/robots.txt nginx/sites/robots.txt + mkdir nginx/sites/robots/ + cp nginx/robots.txt nginx/sites/robots/robots.txt validate-deployment: # Checks the local deployment files - sh validate.sh + sh scripts/validate.sh diff --git a/nginx/README.md b/nginx/README.md index 62f3948..c31cb38 100644 --- a/nginx/README.md +++ b/nginx/README.md @@ -8,17 +8,19 @@ KnowWhereGraph has a handful of services that require networking capabilities. T The NGINX configuration files are templated with environment variables, which can be customized in the docker-compose file. For more information on how the configuration files are generated from the templates refer to [this](https://github.com/docker-library/docs/tree/master/nginx#using-environment-variables-in-nginx-configuration-new-in-119) documentation page. -## Deploying +## Certificates -Deploying nginx without using the make command is *not* recommended and most likely not necessary. If you know what you're doing and need to, you can with +Certificates are mounted into the nginx container from the local filesystem. Non-local deployments work the same way: the `/etc/letsencrypt` files are mounted into the container. -`docker-compose up` +### Staging -To bring the service down, run +New certs can be generated with -`docker-compose down` +`sudo certbot certonly --webroot --webroot-path nginx/data/certbot/www/ -d staging.knowwheregraph.org` -## Certificates + A common problem is that the certs may have different permissions for mounting into the nginx container. + +### Local Certificates for local development need to be manually generated and added to the `local-certs` directory. @@ -39,7 +41,7 @@ https://javorszky.co.uk/2019/11/06/get-firefox-to-trust-your-self-signed-certifi ## Logging -The NGINX logs are found in the container's /var/logs/nginx, which is mounted locally at `./nginx/logs`. For more verbose logging, refer to the NGINX Docker image documentation and modify the deployment script to include any additional flags. +The NGINX logs should be checked through grafana or by inspecting the container's logs. ## Metrics diff --git a/nginx/data/certbot/www/.gitkeep b/nginx/data/certbot/www/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/nginx/docker-compose.local.yaml b/nginx/docker-compose.local.yaml index 070b7b8..8a24b93 100644 --- a/nginx/docker-compose.local.yaml +++ b/nginx/docker-compose.local.yaml @@ -9,7 +9,6 @@ services: - 'variables.env' volumes: - ./nginx/templates:/etc/nginx/templates # NGINX config file - - ./nginx/logs:/var/log/nginx # NGINX logs - ./nginx/sites:/usr/share/nginx/html # Static sites - ./nginx/local-certs:/etc/letsencrypt/live/localhost # Local certificates depends_on: # These services need to exist for the nginx config to validate diff --git a/nginx/docker-compose.prod.yaml b/nginx/docker-compose.prod.yaml index e0e4897..3b55b1c 100644 --- a/nginx/docker-compose.prod.yaml +++ b/nginx/docker-compose.prod.yaml @@ -9,9 +9,9 @@ services: - 'variables.env' volumes: - ./nginx/templates:/etc/nginx/templates # NGINX config file - - ./nginx/logs:/var/log/nginx # NGINX logs - ./nginx/sites:/usr/share/nginx/html # Static sites - - ./nginx/local-certs:/etc/letsencrypt/live/stko-kwg.geog.ucsb.edu # Local certificates + - /etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/cert.pem:/etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/cert.pem # Local certificates + - /etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/privkey.pem:/etc/letsencrypt/live/stko-kwg.geog.ucsb.edu/key.pem # Local certificates depends_on: # These services need to exist for the nginx config to validate - graphdb - kwg-api diff --git a/nginx/docker-compose.stage.yaml b/nginx/docker-compose.stage.yaml index 8d9895c..cdab3d9 100644 --- a/nginx/docker-compose.stage.yaml +++ b/nginx/docker-compose.stage.yaml @@ -9,12 +9,12 @@ services: - 'variables.env' volumes: - ./nginx/templates:/etc/nginx/templates # NGINX config file - - ./nginx/logs:/var/log/nginx # NGINX logs - ./nginx/sites:/usr/share/nginx/html # Static sites - - ./nginx/local-certs:/etc/letsencrypt/live/staging.knowwheregraph.org # Local certificates + - ./nginx/data/certbot/www:/var/www/certbot + - /etc/letsencrypt/live/staging.knowwheregraph.org/cert.pem:/etc/letsencrypt/live/staging.knowwheregraph.org/cert.pem # Local certificates + - /etc/letsencrypt/live/staging.knowwheregraph.org/privkey.pem:/etc/letsencrypt/live/staging.knowwheregraph.org/key.pem # Local certificates depends_on: # These services need to exist for the nginx config to validate - graphdb - kwg-api networks: - kwg_network - diff --git a/nginx/templates/default.conf.template b/nginx/templates/default.conf.template index c9b227d..7d11d2c 100644 --- a/nginx/templates/default.conf.template +++ b/nginx/templates/default.conf.template @@ -1,3 +1,5 @@ +server_names_hash_bucket_size 64; + # Redirect http -> https server { listen 80 default_server; @@ -20,9 +22,10 @@ server { # Certbot certificate challenge location /.well-known/acme-challenge/ { - root /var/www; + alias /var/www/certbot/; } + location /stub_status { stub_status on; allow 172.17.0.0/16; @@ -685,8 +688,7 @@ server { # Robots.txt location /robots.txt { - alias /usr/share/nginx/html/robots.txt; + alias /usr/share/nginx/html/robots/robots.txt; } - } diff --git a/validate.sh b/scripts/validate.sh similarity index 56% rename from validate.sh rename to scripts/validate.sh index 8391e8d..88d0c84 100644 --- a/validate.sh +++ b/scripts/validate.sh @@ -3,27 +3,11 @@ ORANGE='\033[0;31m' DEFAULT='\033[0m' GREEN='\033[0;32m' - -BUILD_FILES_PROD := docker-compose.yaml -f nginx/docker-compose.prod.yaml -f nginx/metrics/docker-compose.yaml -f graphdb/docker-compose.prod.yaml -f elasticsearch/docker-compose.yaml -f prometheus/docker-compose.yaml -f kwg-api/docker-compose.prod.yaml -f grafana/docker-compose.prod.yaml -f loki/docker-compose.yaml -BUILD_FILES_LOCAL := docker-compose.yaml -f nginx/docker-compose.local.yaml -f nginx/metrics/docker-compose.yaml -f graphdb/docker-compose.local.yaml -f elasticsearch/docker-compose.yaml -f prometheus/docker-compose.yaml -f kwg-api/docker-compose.local.yaml -f grafana/docker-compose.local.yaml -f loki/docker-compose.yaml -BUILD_FILES_STAGE := docker-compose.yaml -f nginx/docker-compose.stage.yaml -f nginx/metrics/docker-compose.yaml -f graphdb/docker-compose.stage.yaml -f elasticsearch/docker-compose.yaml -f prometheus/docker-compose.yaml -f kwg-api/docker-compose.stage.yaml -f grafana/docker-compose.stage.yaml -f loki/docker-compose.yaml - - - - - echo "\n\n===== KnowWhereGraph Deployment Configuration Test =====" echo "===== =====" echo " Checking SSL Certificates " echo " ......................... " -# Check for nginx's local certificates -if [ ! -f ./nginx/local-certs/cert.ped ]; then - echo "${ORANGE}Warning${DEFAULT}: Local certificate not found in 'nginx/local-certs' folder!\nCertificates are required for deploying the system. Refer to the README.md for instructions on self signed certs." -else - echo "${GREEN}Good${DEFAULT}: Located local certificates for nginx." -fi - # Check for graphdb's local certificates if [ ! -f ./nginx/local-certs/cert.ped ]; then echo "${ORANGE}Warning${DEFAULT}: Local certificate not found in 'graphdb/local-certs' folder!\nGraphDB may not process SSL data properly!" @@ -64,17 +48,24 @@ fi # Check for Node Browser dist if [ ! -d nginx/sites/node-browser/node-browser/dist/node-browser ]; then - echo "${ORANGE}Error${DEFAULT}: Failed to locate the node browser's build artifacts. Try building the source with its docker-compose file.\n" + echo "${ORANGE}Error${DEFAULT}: Failed to locate the node browser's build artifacts. Try building the source with its docker-compose file." else echo "${GREEN}Good${DEFAULT}: Located the node browser build artifacts." fi +# Check for ontology repo +if [ ! -d nginx/sites/onto ]; then + echo "${ORANGE}Error${DEFAULT}: Failed to locate the ontology files. Try pulling from GitHub" +else + echo "${GREEN}Good${DEFAULT}: Located the ontology artifacts." +fi + echo "\n Checking Web Artifacts " echo " ...................... " # Check for void file -if [ ! -f nginx/sites/void/void.ttl ]; then +if [ ! -f nginx/sites/onto/void.ttl ]; then echo "${ORANGE}Error${DEFAULT}: Void ttl file not detected!\n This file comes from the kwg-ontologies repository. Check the makefile for the clone command." else echo "${GREEN}Good${DEFAULT}: Located the void.ttl file." @@ -82,7 +73,7 @@ fi # Check for robots.txt -if [ ! -f nginx/sites/robots.txt ]; then +if [ ! -f nginx/sites/robots/robots.txt ]; then echo "${ORANGE}Error${DEFAULT}: Failed to locate robots.txt" else echo "${GREEN}Good${DEFAULT}: Located the robots.txt" @@ -93,40 +84,25 @@ echo " ......................... " # Check for an exiting Grafana database file if [ -f grafana/persistent_storage/grafana.db ]; then - echo "${ORANGE}Warning${DEFAULT}: Found an existing Grafana database file. Any Grafana password changes will not persist." + echo "${ORANGE}Info${DEFAULT}: Found an existing Grafana database file. Any Grafana password changes will not persist." else - echo "${GREEN}Good${DEFAULT}: Failed to find an existing Grafana database file. A new one will be created" + echo "${GREEN}Info${DEFAULT}: Failed to find an existing Grafana database file. A new one will be created" fi # Check for any existing GraphDB repositories if [ -d graphdb/ ]; then - echo "${ORANGE}Info${DEFAULT}: Found existing GraphDB repositories. These will be loaded when GraphDB starts." + echo "${GREEN}Info${DEFAULT}: Found existing GraphDB repositories. These will be loaded when GraphDB starts." else echo "${GREEN}Info${DEFAULT}: Failed to find any GraphDB repositories. GraphDB will not be started with any repositories." fi echo "" -echo"" - - -echo "\n Checking Variables " -echo " ......................... " - -# Check for an exiting Grafana database file -if [ -f grafana/persistent_storage/grafana.db ]; then - echo "${ORANGE}Warning${DEFAULT}: Found an existing Grafana database file. Any Grafana password changes will not persist." -else - echo "${GREEN}Good${DEFAULT}: Failed to find an existing Grafana database file. A new one will be created" -fi - -# Check for any existing GraphDB repositories - -if [ -d graphdb/ ]; then - echo "${ORANGE}Info${DEFAULT}: Found existing GraphDB repositories. These will be loaded when GraphDB starts." -else - echo "${GREEN}Info${DEFAULT}: Failed to find any GraphDB repositories. GraphDB will not be started with any repositories." -fi +echo "" +echo "\n\n===== KnowWhereGraph Deployment Configuration Test End =====" +echo " " +echo " Finished Deployment Check " +echo " ......................... " echo "" -echo"" \ No newline at end of file +echo "=============================================================\n\n\n"