Skip to content
This repository has been archived by the owner on Jul 17, 2020. It is now read-only.

Protocol for doing a safe data load and site deploy

Jump to bottom
johnadamsDFID edited this page Nov 3, 2013 · 8 revisions

Step 1: Preparation on your Dev box

Out of the interest of safety of the project and to help prevent problems that could take systems offline you should follow the following protocol when you want to do a new data load.

The dataload should be attempted on your local machine first, steps to do so are as follows

  1. Checkout/clone the master branch from Github.
  2. Ensure mongo is running, if not run the single-node.sh from the shell-scripts
  3. In the terminal cd aid-platform-beta/src/platform/
  4. ./sbt run
  5. when that has loaded go to localhost:9000/admin
  6. login
  7. Select the activity sources that you want if they are not already selected
  8. Click on data load and enter the password
  9. You should see in the terminal that the dataload kicks off, you should follow this and ensure there are no issues with it. If your terminal has lost some of the history, then go to mongo and do db["dataload-audits"].find().

Things to look for in the log:

  • any validation errors, particularly on DFID files (don't worry too much about other organisations' files, unless they are all failing validation)
  • any java/Scala errors that might indicate problems with some of the source data (usually type conversion errors)
  • and You MUST check that he load has finished by rebuilding the ElasticSearch index properly. Should say "Load process completed" or similar.

After the dataload has been completed, you can test that the data is correct by:

  1. Building the Middleman site:
    • Go to the directory src/platform/site in the terminal.
    • Type the command “bundle exec middleman build” and press enter.
    • This will then build the entire middleman site, check out the static files created after its finished.
  2. Testing the search functionality:
    • Go to the directory src/platform in your terminal.
    • Type the command “./sbt run” and press enter.
    • You can now do a simple query by doing something like http://localhost:9000/search?query=1
  3. Checking the data in mongo:
    • db.projects.count()
    • db["funded-projects"].count()
    • db["other-org-projects"].count()

If all of these are successful it should now be completely safe to data load and deploy on live.

Step 2: Load and Deploy on Live

Log into the admin on Live.

  1. Refresh the activity sources to pick up the latest URLs from the IATI Registry. Activity sources -> refresh sources.
  2. Check that the sources you want are all checked (do a quick review), then Save.
  3. Run a Data Load. While that is happening, watch the logs using the aipbeta.pem technique, or by checking the logs in mongo.
  4. If there are no error messages or invalid DFID files, Deploy. This will run Middleman in site generation mode and will use Jenkins to put the pages into the aid-platform-site on GitHub, then deploying to Live.
  5. When deployed to Live, do some basic checks:
    • check that all the top 5s are showing on the front page
    • check the map on http://devtracker.dfid.gov.uk/location/country/ is showing all the countries you expect.
    • run some searches
    • go to Global Poverty Action Fund page and check the partners are listed
    • go into a GPAF partner page
Clone this wiki locally