Merge pull request #231 from OpenUpSA/sys/add-cron-jobs

Add cron tasks to app.json to be managed by Dokku in production

.dockerignore

@@ -4,3 +4,7 @@ data
 *.dump
 *.sql
 *.tar
+*.DS_Store
+*.tar
+*.dump
+

app.json

@@ -3,5 +3,31 @@
     "dokku": {
       "predeploy": "python manage.py collectstatic --noinput"
     }
-  }
-}
+  },
+  "cron": [
+    {
+      "command": "0 0 * * 1",
+      "schedule": "bin/output-on-error bin/backup-data-dir.bash"
+    },
+    {
+      "command": "0 1 * * *",
+      "schedule": "bin/output-on-error ./manage.py core_list_malformed_slugs"
+    },
+    {
+      "command": "30 1 * * *",
+      "schedule": "bin/output-on-error ./manage.py core_database_dump /data/media_root/dumps/pg-dump && bin/output-on-error gzip -9 -f /data/media_root/dumps/pg-dump_schema.sql /data/media_root/dumps/pg-dump_data.sql"
+    },
+    {
+      "command": "10 2 * * *",
+      "schedule": "bin/output-on-error bin/update_za_hansard.bash"
+    },
+    {
+      "command": "0 5 * * *",
+      "schedule": "bin/output-on-error python manage.py core_export_to_popolo_json /data/media_root/popolo_json http://www.pa.org.za"
+    },
+    {
+      "command": "30 5 * * *",
+      "schedule": "bin/output-on-error python manage.py core_export_to_popolo_json --pombola /data/media_root/popolo_json http://www.pa.org.za"
+    }
+  ]
+}

docs/INSTALL.md

@@ -20,8 +20,7 @@ dynamic data such as the search indexes, uploaded images, various caches etc.
 These extra directories are by default put inside the `data/` directory, although
 this can be modified using the `DATA_DIR` configuration variable.
 
-Development
------------
+## Development
 
 The code is available via github: https://github.com/OpenUpSA/pombola
 
@@ -41,9 +40,10 @@ Load [demo data](#demo-data) for easy dev setup. (See also [loading a production
 docker compose run --rm app python manage.py loaddata demodata.json
 ```
 
-Load WriteInPublic fixtures - these will let you draft messages but not send them, just to enable trying out the write-to flows locally. Change the slugs and keys to the sandbox ones to send actual messages from your dev environment. 
+Load WriteInPublic fixtures - these will let you draft messages but not send them, just to enable trying out the write-to flows locally. Change the slugs and keys to the sandbox ones to send actual messages from your dev environment.
 
     docker compose run --rm app python manage.py loaddata writeinpublic.json
+
 Start the app, DB and search index:
 
 ```
@@ -58,7 +58,7 @@ docker compose run --rm app python manage.py rebuild_index
 
 Now you can visit the site at [http://localhost:8000](http://localhost:8000)
 
------
+---
 
 Delete the DB, media files and search index to start afresh:
 
@@ -103,9 +103,7 @@ zcat pg-dump_schema.sql.gz | docker compose run --rm db psql postgres://pombola:
 zcat pg-dump_data.sql.gz | docker compose run --rm db psql postgres://pombola:pombola@db/pombola
 ```
 
-
-Production deployment
----------------------
+## Production deployment
 
 ### Elasticsearch
 
@@ -158,7 +156,6 @@ You need to get a PMG API key for a user that is confirmed (`user.confirmed_at`
 
 This key is needed by the [`za_hansard_pmg_api_scraper` command](https://github.com/OpenUpSA/pombola/blob/efcfaf05916ca2cb838a6b570109cae91545905a/pombola/za_hansard/management/commands/za_hansard_pmg_api_scraper.py#L82) that imports committee meetings appearances.
 
-
 ### Create and configure the app
 
 ```
@@ -235,6 +232,8 @@ and `GOOGLE_RECAPTCHA_SECRET_KEY` environment variables.
 
 ### Cron jobs
 
+Note that `app.json` schedules these jobs with Dokku in production.
+
 Cron jobs should only output to stdout or stderr if something went wrong and
 needs an operator's attention. We use `bin/run_management_command_capture_stdout`
 to capture any output and only output it if the command exited with an error

pombola/core/management/commands/core_database_dump.py

@@ -50,7 +50,8 @@ def get_tables_to_dump(self):
             'popolo_name_resolver_entityname',
             'writeinpublic_configuration',
             'za_hansard_sourceparsinglog',
-            'za_hansard_questionparsingerror'
+            'za_hansard_questionparsingerror',
+            'constance_config'
         ])
         tables_to_dump = [
             t for t in tables if t not in tables_to_ignore