Skip to content

Infrastructure Description

Jump to bottom Edit New page
Lukas edited this page Jan 18, 2016 · 23 revisions

RECAST Frontend

This component is the public-facing service, where people outside of the experiments can request new reprocessings. It also provides a REST API to both create and query these requests.

repositories

CAP

repositories

Docker Registry

We're moving to a docker-based infrastructure to store actual analysis code + data resources. These Docker images should be provided by the analysis team for every major processing step.

  • currently normal docker.io registry
  • CERN internal registry

Worker Node

This is the component that actually receives the request and runs the analysis workflow to produce the new results needed to fulfill the request.

The node runs a Celery worker process which all know about a single Celery Task (recsastbackend.backendtasks:run_analysis) https://github.com/recast-hep/recast-backend/blob/master/recastbackend/backendtasks.py#L191

the celery task does three things:

  1. Setup
  • prepares workdirectory
  • downloads request zipfile from RECAST frontend, unzips
  1. Body
  • run the python function given by the RECAST plugin with the request context
  1. Finish (On Success)
  • collect result files
  • upload back to control center / calling host
  1. Teardown (Always)
  • copy log files to local quanrantine (for debugging)
  • delete workdir

The signature for that celery task is

@shared_task
def run_analysis(setupfunc,onsuccess,teardownfunc,ctx):
    ...

i.e. the task receives the names of these functions in a format module:funcname. In production these are used:

  • setupfunc:
    • recastbackend.backendtasks.setup (for input zip from RECAST frontend retrieved via recast-api, used by recast-prodsub CLI app and Control Center )
    • recastbackend.backendtasks.setupFromURL (for input zip from generic URL, used by recast-localsub CLI app)
  • onsuccess:
    • recastbackend.backendtasks.generic_onsuccess (used by recast-localsub)
    • recastbackend.backendtasks.onsuccess (user by recast-prodsub and Control Center)
  • teadownfunc:
    • recastbackend.backendtasks.cleanup used by both CLI apps

The context prepared by the submission, includes a plugin entry function in the same format module:name which must have the signature:

def pluginfunc_name(ctx):
    pass

currently we have these examples:

Components

This section will include a description of the various services / components expected to run on a worker node

  • supervisord
    • this controls the recast worker process
      • process name recast_worker
      • start script is in /home/recast/start_worker.sh
    • configuration is /etc/supervisord.conf
  • diamond
    • this runs periodic system inspection scripts (like Memory, Cpu etc)
    • configuration is in /etc/diamond/diamond.conf
    • data is archived in /var/log/diamond and sent to statsd server on monitor host (currently testing-headnode
  • statsd
    • python module needed for diamond for sensind system info
  • celery worker
    • see detailed description above
  • RECAST backend
    • see detailed description above
  • RECAST plugin
    • see detailed description above
  • Docker
    • vanilla docker installation, this is called by adage to execute steps in the workflow
  • extra resources
    • CVMFS
      • configuration in /etc/cvmfs/default.local
    • AFS
      • comes preconfigured in CC7
    • recast_auth
      • on provisioning we request a host autoenrolling GRID host certificate via cern-get-certificate and store this in /home/recast/recast_auth together with a script that can retrieve a MyProxy grid proxy. this can be optionally mounted on docker containers to give them access to GRID resources.

Repositories

RECAST plugins (Celery Entry Point)

Workflow Execution (Adage)

  • multiple backends (multiprocessing / Celery? / IPython parallel?)

Repositories

Monitoring

Repositories

Control Center

  • core control center
  • result blueprints

Repositories

Redis Broker / Job Store

  • just vanilla redis store hosted on recast-demo.cern.ch

Misc

Add a custom footer
Add a custom sidebar
Clone this wiki locally