Skip to content
This repository has been archived by the owner on Jul 17, 2020. It is now read-only.

Solution Overview old

Jump to bottom
johnadamsDFID edited this page Nov 29, 2013 · 1 revision

High Level Description

The Dev Tracker platform aims to provide transparent access to

  • UK Aid Spend from Reporting Organisations down to the recipient regions & countries in a visual manner
  • access to all IATI data via the API

Architecture Overview

The currently deployed solution looks roughly like this,

As Is

There are some problems with the current solution due to a high level of technical debt. Some technologies have become less useful as the architecture evolved and are causing performance bottle necks and needless complexity.

There is an ongoing process of architectural refactoring to reduce this debt and complexity by automating as much of the process as possible and reducing the amount of data transformation required. The new vision looks like this,

To Be

Or from a slightly different point of view,

To Be

Data Flow

The admin site is the main control point for the system. This set of steps covers what needs done on a new/fresh machine.

  • Admin user logs into the admin functionality
  • User selects Organisation sources by first loading them (Refresh Sources) and then selecting them and hitting save
  • User does the same for activity sources
  • User updates the CMS content (country/region names etc.) - this data will already be seeded from the migration step of the install but there may be extra changes if the user desires.
  • Select "Load Data". This downloads, validates and loads selected Org/Activity sources into Neo4J/Mongo, indexes the data for search purposes and exposes the new data via the APi
  • After the data is fully loaded the user can select to "Deploy the Site" - this process takes data aggregated into Mongo and builds the static site.
  • Once the site is generated it is committed to the aid-platform-site repo and eventually Jenkins will trigger an update - and deploy the site into the root of Apache.

Some basic data flow visualisations

As Is Flow

As Is

To Be Flow

To Be

Architecture layers

Three key layers to the architecture.

Synchroniser research/pipeline/synchroniser

Purpose: Pulls data from the IATI Registry into the BaseX data store and into Elastic Search. Runs every 2 hours, and can be forced in an emergency. Files with symbols: - = no change x = deleted, green tick updated. Uses akka to enable multi-threading, site view also provides metrics.

API research/pipeline/api

Purpose: Abstracts the data from the BaseX data store into a standard set of data structures for use by the Site. Launch using play run in api/

routes file defines the API routes.

  • Aggregator. Calls BaseX through xquery files (.xq).
  • Application. Individual activities available through the aPI.
  • ConfigApi. Calls the config files (converted from mongoDB config files).

Site research/pipeline/site

Purpose: Displays the data to the end user in Country, Project and other templated data.

Key changes from previous architecture are in config.rb which is used to generate the site by Middleman. This calls the templates and passes through data from the API.

Clone this wiki locally