Merge pull request #13 from sara-nl/add-examples

Add examples

This branch adds the examples from the documentation at https://doc.grid.surfsara.nl/en/latest/Pages/Practices/picas/picas_overview.html# to the repository. There are 3 examples. Local, slurm and Grid. They are based on running on the SURF instance of CouchDB as PiCaS backend. 

There are some small but crucial changes, especially in the decoder of the tokens. This is done with jsons, and in this version of PiCaS the syntax changed, which is visible in the change in documents.py and examples having a open().read(). Before it was just open() and the decoder could handle it, but not any more for unclear reasons.

In any case: the examples work in all cases and the CICD runs properly.

.github/workflows/python-app.yml

@@ -1,7 +1,7 @@
 # This workflow will install Python dependencies, run tests and lint with a single version of Python
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
 
-name: Python application
+name: Unit Tests
 
 on:
   push:
@@ -34,7 +34,7 @@ jobs:
         python -m pip install --upgrade pip
         pip install -U .
         pip install -U ".[test]"
-        pip install flake8 pytest
+        pip install flake8 pytest nose
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
     - name: Lint with flake8
       run: |

.gitignore

@@ -20,8 +20,11 @@ pip-log.txt
 .coverage
 .tox
 
-#Translations
+# Translations
 *.mo
 
-#Mr Developer
+# Mr Developer
 .mr.developer.cfg
+
+# PiCaS DB information
+scripts/picasconfig.py

README.md

@@ -1,16 +1,20 @@
 picasclient
-===========
+-----------
+
+![CICD](https://github.com/sara-nl/picasclient/actions/workflows/python-app.yml/badge.svg)
 
 Python client using CouchDB as a token pool server.
 
-## Installation
+Installation
+============
 
-Run
+To install run
 ```
 pip install -U .
 ```
 
-## Testing
+Testing
+=======
 
 First, install the test dependencies with 
 ```
@@ -22,10 +26,128 @@ flake8 picas tests
 nosetests tests
 ```
 
+Examples
+========
+
+## Setting up the examples
+
+The scripts directory contains examples to use the picasclient. There are examples for running locally (laptop, cluster login), slurm and the Grid (https://www.egi.eu/). 
+To run the examples, first you need to have a CouchDB instance running that functions as the token broker that keeps the tokens and then worker machines can approach the broker to get work. To set up this db, see the [SURF documentation](https://doc.grid.surfsara.nl/en/latest/Pages/Practices/picas/picas_overview.html#picas-server-1).
+
+Once this server is running, you can run the PiCaS examples:
+ - Local
+ - Slurm
+ - Grid
+
+To approach the DB, you have to fill in the `script/picasconfig.py` with the information to log in to your CouchDB instance and the database you want use for storing the work tokens.
+
+Next you have to send some tokens with work to the CouchDB instance. You can send two types of work in this example. For very fast running jobs, send the `quickExample.txt` file with:
+
+```
+python pushTokens.py quickExample.txt
+```
+
+
+Now we are ready to run the examples!
+
+## Running locally
+
+To run the local example do:
+
+```
+python local-example.py
+```
+
+If all goes well you should see output like:
+
+```
+-----------------------
+Working on token: token_0
+_id token_0
+_rev 4-8b04da64c0a536bb88a3cdebe12e0a87
+type token
+lock 1692692693
+done 0
+hostname ui-01.spider.surfsara.nl
+scrub_count 0
+input echo "bash-echo"
+exit_code 0
+-----------------------
+```
+
+The token in de database will have attachments with the regular and error output of the terminal. There will find the output file `logs_token_0.out`, containing the output of the input command:
+
+```
+echo "bash-echo"
+>>> bash-echo
+```
+
+Once the script is running, it will start polling the CouchDB instance for work. Once the work is complete, the script will finish.
+
+## Running on Slurm
+
+To run on slurm, first open the `slurm-example.sh` file and make sure your python virtual env or conda/mamba environment is loaded.
+Then you have to add tokens to CouchDB using the same setup procedure as mentioned above, with the pushTokens methods.
+
+To start the slurm job that runs the PiCaS client do:
+
+```
+sbatch slurm-example.sh
+```
+
+Now in a slurm job array (you can set the number of array jobs in the script at `--array`) and each job will start polling the CouchDB instance for work. Once the work is complete, the jobs will finish.
+
+## Running on Grid
+
+First we need to create a tar of the picas code, so that it can be sent to the Grid:
+
+```
+tar cfv grid-sandbox/picas.tar ../picas/
+```
+
+Secondly, the CouchDB python API needs to be available too, so download and extract it:
+
+```
+wget https://files.pythonhosted.org/packages/7c/c8/f94a107eca0c178e5d74c705dad1a5205c0f580840bd1b155cd8a258cb7c/CouchDB-1.2.tar.gz
+```
+
+Now you can start the example from a grid login node with (in this case DIRAC is used for job submission):
+
+```
+dirac-wms-job-submit fractals.jdl
+```
+
+And the status and output can be retrieved with the usual DIRAC commands, while in the token you see the status of the token and the attachments with the log files.
+
+## Running the long jobs
+
+To send longer running code (it takes up to 30 minutes per token), do:
+
+```
+./createTokens
+>>> /tmp/tmp.JoLqcdYZRD
+```
+
+And pass the output file to the push tokens code:
+
+```
+python pushTokens.py /tmp/tmp.abc123
+```
+
+Now the tokens are available in the database. Now, the binary for the calculation needs to be built:
+
+```
+cc src/fractals.c -o bin/fractals -lm
+```
+
+And finally, the `*-example.py` code needs to call a different command:
+
+```
+command = "/usr/bin/time -v ./process_task.sh " + "\"" +token['input'] + "\" " + token['_id'] + " 2> logs_" + str(token['_id']) + ".err 1> logs_" + str(token['_id']) + ".out"
+```
 
-## Travis build status
+So adjust the `*-example.py` python code for whichever way you want to run it (locally, slurm, grid) and start running the way described above!
 
-[![Build Status](https://travis-ci.org/sara-nl/picasclient.svg?branch=master)](https://travis-ci.org/sara-nl/picasclient)
 
 ## QuantifiedCode Automated code review  
 

examples/createTokens

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+PARAM_Q_MIN=100
+PARAM_Q_MAX=260
+PARAM_Q_STEP=84
+
+PARAM_D_MIN=256
+PARAM_D_MAX=8192
+PARAM_D_STEP=2024
+
+PARAM_M_MIN=400
+PARAM_M_MAX=10000
+PARAM_M_STEP=4000
+
+function testNumerical {
+  if [ $1 -eq $1 2> /dev/null ]; then
+    return
+  fi
+  echo "$1 is not numerical!"
+  exit 1
+}
+
+while getopts "q:d:m:" opt; do
+  testNumerical $OPTARG
+  case $opt in
+    q)
+      PARAM_Q_STEP=$OPTARG
+      ;;
+    d)
+      PARAM_D_STEP=$OPTARG
+      ;;
+    m)
+      PARAM_M_STEP=$OPTARG
+      ;;
+  esac
+done
+
+PARAMFILE=$( mktemp )
+
+PARAM_Q=$PARAM_Q_MIN
+PARAM_D=$PARAM_D_MIN
+PARAM_M=$PARAM_M_MIN
+
+STR_PARAM_Q=""
+
+while [ $PARAM_Q_MAX -ge $PARAM_Q ]; do
+  # Ugly hack in order not to have to use bc
+  if [ $[ $PARAM_Q / 10 ] -eq 0  ]; then
+    STR_PARAM_Q="0.00${PARAM_Q}"
+  elif [ $[ $PARAM_Q / 100 ] -eq 0 ]; then
+    STR_PARAM_Q="0.0${PARAM_Q}"
+  else
+    STR_PARAM_Q="0.${PARAM_Q}"
+  fi
+  while [ $PARAM_D_MAX -ge $PARAM_D ]; do
+    while [ $PARAM_M_MAX -ge $PARAM_M ]; do
+      echo "-q ${STR_PARAM_Q} -d ${PARAM_D} -m ${PARAM_M}" >> $PARAMFILE
+      PARAM_M=$[ $PARAM_M+$PARAM_M_STEP ]
+    done
+    PARAM_M=$PARAM_M_MIN
+    PARAM_D=$[ $PARAM_D+$PARAM_D_STEP ]
+  done
+  PARAM_D=$PARAM_D_MIN
+  PARAM_Q=$[ $PARAM_Q+$PARAM_Q_STEP ]
+done
+
+echo ${PARAMFILE}

examples/fractals.jdl

@@ -0,0 +1,12 @@
+[
+  ParameterStart=0;
+  ParameterStep=1;
+  Parameters=5;
+
+  Executable = "/bin/sh";
+  Arguments = "startpilot.sh";
+  Stdoutput = "parametricjob.out";
+  StdError = "parametricjob.err";
+  InputSandbox = {"grid-sandbox/CouchDB-1.2.tar.gz", "grid-sandbox/picas.tar", "grid-sandbox/startpilot.sh", "grid-sandbox/grid-example.py", "grid-sandbox/process_task.sh", "bin/fractals", "picasconfig.py"};  
+  OutputSandbox = {"parametricjob.out", "parametricjob.err"};
+]

examples/grid-sandbox/grid-example.py

@@ -0,0 +1,80 @@
+'''
+@helpdesk: SURF helpdesk <[email protected]>
+
+usage: python grid-example.py
+description:
+    Connect to PiCaS server
+    Get the next token in todo View
+    Fetch the token parameters, e.g. input value
+    Run main job (process_task.sh) with the input argument
+    When done, return the exit code to the token
+    Attach the logs to the token
+
+
+'''
+
+#python imports
+import os
+import time
+import couchdb
+import picasconfig
+
+#picas imports
+from picas.actors import RunActor
+from picas.clients import CouchDB
+from picas.iterators import TaskViewIterator
+from picas.modifiers import BasicTokenModifier
+from picas.executers import execute
+
+class ExampleActor(RunActor):
+    def __init__(self, db, modifier, view="todo", **viewargs):
+        super(ExampleActor, self).__init__(db, view=view, **viewargs)
+        self.modifier = modifier
+        self.client = db
+
+    def process_task(self, token):
+        # Print token information
+        print("-----------------------")
+        print("Working on token: " +token['_id'])
+        for key, value in token.doc.items():
+            print(key, value)
+        print("-----------------------")
+
+        # Start running the main job
+        # /usr/bin/time -v ./process_task.sh [input] [tokenid] 2> logs_[token_id].err 1> logs_[token_id].out
+        command = "/usr/bin/time -v ./process_task.sh " + "\"" +token['input'] + "\" " + token['_id'] + " 2> logs_" + str(token['_id']) + ".err 1> logs_" + str(token['_id']) + ".out"
+
+        out = execute(command,shell=True)
+
+        ## Get the job exit code in the token
+        token['exit_code'] = out[0]
+        token = self.modifier.close(token)
+        #self.client.db[token['_id']] = token # necessary?
+
+        # Attach logs in token
+        curdate = time.strftime("%d/%m/%Y_%H:%M:%S_")
+        try:
+            logsout = "logs_" + str(token['_id']) + ".out"
+            log_handle = open(logsout, 'rb')
+            token.put_attachment(logsout, log_handle.read())
+
+            logserr = "logs_" + str(token['_id']) + ".err"
+            log_handle = open(logserr, 'rb')
+            token.put_attachment(logserr, log_handle.read())
+        except:
+            print("excepted attachemnt")
+            pass
+
+def main():
+    # setup connection to db
+    client = CouchDB(url=picasconfig.PICAS_HOST_URL, db=picasconfig.PICAS_DATABASE, username=picasconfig.PICAS_USERNAME, password=picasconfig.PICAS_PASSWORD)
+    print("Connected to the database %s sucessfully. Now starting work..." %(picasconfig.PICAS_DATABASE))
+    # Create token modifier
+    modifier = BasicTokenModifier()
+    # Create actor
+    actor = ExampleActor(client, modifier)
+    # Start work!
+    actor.run()
+
+if __name__ == '__main__':
+    main()