This repository contains code specific for InaSAFE Realtime Hazard processing service, or in short (IRH). This service is used in BNPB to gather and integrate Hazard data from various agencies and convert it into InaSAFE Hazard Layers to be further processed by InaSAFE Realtime System.
Statuses:
Architecture:
[
]
This document describes how to get the IRH environment setup. We mainly develops using PyCharm Professional IDE, courtesy of JetBrains. This setup will automatically generate project configuration for PyCharm.
Prerequisites:
- Running MacOS or Ubuntu 16.04/16.10
- Docker Engine installed
- Ansible is installed (pip install ansible or brew install ansible)
- PyCharm professional is installed (tested versions: 2016.3, 2017.1, 2017.2, or 2017.3)
Checkout or clone main repository or your own fork.
git clone [email protected]:inasafe/inasafe-realtime.git
We work in develop branch.
Open the checkout directory in PyCharm as a project. This is important because the ansible scripts in the next step depend on the .idea directory that gets created there. Then close pycharm again while you run the next steps.
In this step, we configure our project generator to fill in some configuration settings.
cd deployment/ansible/development/group_vars/
cp all.sample.yml all.yml
Edit all.yml for your own system.
use_pycharm (set to **yes** if using pycharm, **no** for otherwise)
remote_user (your username)
remote_group (your user’s group)
project_path (the location of the project)
interpreters → inasafe_realtime → ipaddress: (your local machine ipaddress)
Example:
3 use_pycharm: yes
4 remote_user: timlinux
5 remote_group: staff
6 project_path: "/Users/timlinux/dev/docker/inasafe-realtime"
8 interpreters:
9 inasafe_realtime:
10 name: 'InaSAFE Realtime Container'
11 domain_alias: 'docker-realtime-orchestration'
12 ssh_port: &inasafe_realtime_ssh_port 35103
13 pycharm_helpers: /root/.pycharm_helpers
14 ipaddress: &inasafe_realtime_host_ip 192.168.1.113
Generate your project config:
make setup-ansible
Fill in your sudo password (to change /etc/hosts entry) Choose your pycharm version.
Ansible should return something like this:
PLAY RECAP *********************************************************************
localhost : ok=13 changed=3 unreachable=0 failed=0
Failed=0 is the important part here. After this, you are safe to restart PyCharm and open the project.
We mainly uses docker to build our development environment.
For the rest of the instruction, execute make commands from deployment
folder.
This will build the docker images
make build
Upstart your docker containers
make up
To check that your services running, run:
make status
This will run the environment in docker containers and then use a remote interpreter in PyCharm so that you can debug. Open pycharm again. After the process finished, you need to open your PyCharm preference to make the PyCharm aware of the remote python interpreter. You will see these 3 remote interpreters:
InaSAFE Realtime Hazard Container → For the celery worker InaSAFE Realtime Shakemap Monitor Container → Monitor the shakemap folder (for initial shakemap type) InaSAFE Realtime Shakemap Corrected Monitor Container → Monitor shakemap the shakemap folder (for data-informed shakemap type)
If the interpreter is connected properly in PyCharm, you should be able
to see python package lists in Project Interpreter Dialog.
To run the code using PyCharm configuration, choose your appropriate Run configuration from Run/Debug Configuration Dropdown.
Run the “Run Monitoring Service” or "Run Monitoring Service - Shakemap Processed" run configuration in PyCharm to start service for respective shakemap monitoring type.
Run the “Celery Workers” run configuration in PyCharm to start Celery Task service for IRH.
In addition to run configuration in Run mode, you can also run it in
Debug mode by clicking the Debug button. This allows you to use PyCharm
debugging features like breakpoints, etc.
To test your shakemap monitor, you need drop a shake grid.xml file into ‘shakemaps’ directory in this kind of directory structure: shakemaps/<YYMMDDHHmmss>/grid.xml
If it’s working properly, the worker will pick up the grid.xml, and process it.
You can run unittests in PyCharm. To do so, highlight your test files, methods or packages and open context menu, then click Run unittests...
Tests that runs in PyCharm will run in synchronous mode.
Will be implemented later
Will be implemented later
InaSAFE Realtime Hazard services uses Celery to handle loads. Because
of the way celery architecture works, it was optimized for asynchronous
job. However, sometimes debugging only functional usage in synchronous
mode is preferred. That is why the default pycharm unittests run in
synchronous mode. To switch into async mode, you can put additional
configuration in your all.yml file like this:
inasafe_realtime_worker:
environment:
task_always_eager: True
on_travis: False
Explanation for above: put another yaml key called task_always_eager
and on_travis in that inasafe_realtime_worker.environment tree.
After that, run again make setup-ansible to refresh your new configuration
In celery async mode, your test will hang while waiting for task results.
Your task can only be executed by the worker, so make sure to start
Celery Workers Run configuration before running your tests.
InaSAFE Realtime Hazard services will try to communicate with InaSAFE Django via REST API. Your test will fail if you do not put correct credentials for the rest API. Yaml configuration tree for this is in:
realtime_rest_api:
host: *inasafe_realtime_host_ip
port: 61102
user: [email protected]
password: thepaassword
Change the value into your actual InaSAFE Django test instance.
If you don't want to use real django server, you can use preconfigured mock server. Change above value into this:
realtime_rest_api:
host: localhost
port: 8000
user: [email protected]
password: thepaassword
Then run make setup-ansible to refresh your configuration.
Note that this is a mock server that only able to test REST request but don't actually check the content and always returned HTTP 200 code (OK).