Processing Workflow

User Interface is used to Create a Project/Scenario.

The Client API is used by the UI Application to allow the user to create a Project and one or more Scenarios for that project (Note: While the database is setup to handle 'one project, many scenarios', the application currently forces this into a 1:1 relationship). The data is stored in the Application Database specifically in the following tables:

• tsudat_project
• tsudat_scenario
• tsudat_gauge_point
• tsudat_internal_polygon
• tsudat_project_data_set

User Queue's the Scenario to be Run.

When the user has completed defining their scenario, the run_scenario view/endpoint is invoked with the scenario_id https://github.com/AIFDR/tsudat2/blob/master/tsudat/views.py#L835. The run_scenario endpoint calls the run_tsudat_simulation function in tsudat/tasks.py https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L62 asyncrhonously and control returns to the run_scenario view which notifies the user that their scenario has been queued.

Scenario Definition is used to Create an ANUGA Job Work Package for processing and the Job is Run

The code in run_tsudat_simulation does the following things to prepare a directory for ANUGA to process.

• Retrieves the Scenario Information from the DB https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L72
• Creates a directory and subdirectories for the ANUGA job https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L89
• Figures out which UTM Zone the project is in https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L102
• Transforms the Project Bounding Polygon Geometry to this UTM Zone and writes out this Geometry to a text file in the Work Directory https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L116
• Writes out the internal Polygons to text files https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L125
• Clips each DEM specified for use to the project boundary, projects them to UTM, converts to an ASCII Grid File and writes out a projection file https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L146
• Figures out which of the project polygon vertices are on land and writes out a landward boundary file https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L231
• Writes out a file containing the hazard points which are inside the project polygon boundary https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L252
• Writes out any specified gauges to a file https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L252
• Constructs a dictionary of the project parameters and writes this out to a JSON file https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L279

When these steps are complete, run_tsudat is called to run the Job. https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L322

MORE DETAIL ABOUT ANUGA PROCESSING WORKFLOW HERE

• Creating Anuga Run Script
• Running an ANUGA Simulation

Processing Messages from ANUGA

A periodic task is configured to process ANUGA status messages. https://github.com/AIFDR/tsudat2/blob/master/tsudat/tasks.py#L322 These Status Messages are Retrieved from the Message Queue and Processed. START, ABORT, LOG and IDLE messages are used to update the Scenario record in the database with the status, the timestamp and any information passed from ANUGA in a message.

STOP messages indicate that the job has completed processing and is ready for loading back into the GeoNode for presentatio back to the User. When a STOP message is retrieved, the Scenario record in the Database is updated with the timestamp and the received payload (Link to sample payload here) and process_finished_simulation is invoked https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L24

Process Completed Scenarios

The finished Scenario is processed to be included in the GeoNode site by doing the following:

• Output Layers are assigned keywords and permissions and loaded into GeoNode using the create_scenario_output_layer function https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L85 This step is sometimes prone to errors if the server is busy, so is retried 3 times before failing. This step involves the following sub-steps
  • An Abstract is constructed with information about the Scenario https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L94
  • The File is Warped to WGS84 (EPSG:4326) and converted from ASCII Grid to GeoTiff https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L96
  • The GeoTiff is loaded into the GeoNode (and thus into GeoServer and GeoNetwork) using the constructed abstract and permissions https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L111
  • The Layer is associated with the Scenario
  • A style is assigned based on the type of scenario output layer it is (depth, flow, speed)
  • Keywords are assigned (double checking here)
• A Project Map is created in GeoNode with the constituent output Layers and is assigned a descriptive abstract and assigned permissions. https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L58\
• The user is notified that their job is complete with an email https://github.com/AIFDR/tsudat2/blob/master/tsudat/utils.py#L68

When this sequence of events is complete, and the user receives the email indicating their job is complete, they can click on the link in the email to view their project map and download the result layers. The user can also load the scenario back into the UI for tweaking and re-running.