Using Celery

Using Celery

Retrieving Activity Metadata

It is stated in the xAPI spec that, "If an Activity IRI is an IRL, an LRS SHOULD attempt to GET that IRL, and include in HTTP headers: "Accept: application/json, /". This SHOULD be done as soon as practical after the LRS first encounters the Activity id. Upon loading JSON which is a valid Activity Definition from an IRL used as an Activity id, an LRS SHOULD incorporate the loaded definition into its internal definition for that Activity, while preserving names or definitions not included in the loaded definition."

The ADL LRS will now retrieve activity metadata, void statements, and check LRS hooks via celery. This speeds up the processing of incoming statements, especially those that are in large batches, and gives the client a faster response.

Voiding Statements

In case batch statements don't come in the correct order when voiding statements, the LRS will wait until all statements from that batch are stored, then will start worker processes to attempt to void the statements that were specified instead of returning a 404 error when trying to void statements sequentially.

Setup

The LRS will be using Celery as an asynchronous task queue and RabbitMQ as its message broker to retrieve the activity metadata. This will be running completely independent of Django. The libraries required for this should already be installed in your environment (amqp and celery) from the requirements.txt document.

RabbitMQ Setup

For details and more in-depth documentation, visit the celery docs.

1. admin:~$ sudo apt-get install rabbitmq-server
2. admin:~$ sudo rabbitmqctl add_user <username_for_rabbitmq> <password_for_rabbitmq>
3. admin:~$ sudo rabbitmqctl add_vhost <vhost_name>
4. admin:~$ sudo rabbitmqctl set_permissions -p <vhost_name> <username_for_rabbitmq> ".*" ".*" ".*"

Celery Setup

For details and more in-depth documentation, visit the celery docs.

1. Configure /path/to/adl_lrs/settings.ini

   [ampq]
   USERNAME: <username_for_rabbitmq>
   PASSWORD: <password_for_rabbitmq>
   HOST: localhost
   PORT: 5672
   VHOST: <vhost_name>
   

2. Create supervisord.conf file in /path/to/ADL_LRS/

   [unix_http_server]
   file=/tmp/supervisor.sock                       ; path to your socket file
   
   [supervisord]
   logfile=/path/to/logs/supervisord/supervisord.log ; CHANGE THIS PATH
   childlogdir=/path/to/logs/supervisord/            ; CHANGE THIS PATH  
   logfile_maxbytes=50MB                           ; maximum size of logfile before rotation
   logfile_backups=10                              ; number of backed up logfiles
   loglevel=info                                  ; info, debug, warn, trace
   pidfile=/var/run/supervisord.pid                ; pidfile location
   minfds=1024                                     ; number of startup file descriptors
   minprocs=200                                    ; number of process descriptors
   
   [rpcinterface:supervisor]
   supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
   
   [supervisorctl]
   serverurl=unix:///tmp/supervisor.sock         ; use a unix:// URL  for a unix socket
   
   [program:celery]
   ; Set full path to celery program if using virtualenv
   command=/path/to/env/bin/celery worker -A lrs --loglevel=INFO ; CHANGE THIS PATH
   
   
   directory=/path/to/ADL_LRS ; CHANGE THIS PATH
   numprocs=1
   stdout_logfile=/path/to/logs/celery/celery-worker.log        ; CHANGE THIS PATH
   stderr_logfile=/path/to/logs/celery/celery-worker-errors.log ; CHANGE THIS PATH
   autostart=true
   autorestart=true
   startsecs=10
   
   ; Need to wait for currently executing tasks to finish at shutdown.
   ; Increase this if you have very long running tasks.
   stopwaitsecs = 600
   
   ; When resorting to send SIGKILL to the program to terminate it
   ; send SIGKILL to its whole process group instead,
   ; taking care of its children as well.
   killasgroup=true
   
   ; if rabbitmq is supervised, set its priority higher
   ; so it starts first
   priority=998
   
   

3. Create upstart script in /etc/init/ (our file will be called celerylrs.conf). You will need to use sudo for admin privileges.

   description    "supervisor for lrs-celery"
   start on runlevel    [2345]
   stop on runlevel    [!2345]
   
   respawn
   
   setuid <Name of system user LRS is running under>
   chdir /path/to/ADL_LRS
   exec /path/to/env/bin/supervisord --nodaemon
   

   You can stop start/stop the celery tasks by typing sudo {start|stop|restart} celerylrs

   NOTE: If you're using Ubuntu 15+ it uses systemd instead of upstart by default. Follow these instructions for a systemd script

   Create systemd script in /lib/systemd/system/ (our file will be called celerylrs.service). You will need to use sudo for admin privileges.

   [Unit]
   Description=Supervisor to run celery for the LRS
   
   [Service]
   User=<Name of system user LRS is running under>
   WorkingDirectory=/path/to/ADL_LRS
   ExecStart=/path/to/env/bin/supervisord --nodaemon
   Restart=always
   
   [Install]
   WantedBy=multi-user.target
   

   Then run sudo systemctl daemon-reload and sudo systemctl enable celerylrs.service.

   You can stop start/stop the celery tasks by typing sudo systemctl {start|stop|restart} celerylrs