Running on COMPS

Here's what I did. (Using past tense, since this is a report of one way to do it, not a generalized recipe yet)

Published idmlaser (my version) to our pypi.

python3 -m twine upload --verbose --repository-url https://packages.idmod.org/api/pypi/idm-pypi-production/ dist/idmlaser-0.0.5-py3-none-any.whl

Created a Singularity Definition File

Bootstrap: docker
From: rockylinux:9

%post
    dnf -y install python3-pip
    dnf -y install gcc-c++
    dnf -y install sudo
    dnf -y install epel-release
    dnf clean all

    python3 -m pip install pip --upgrade
    python3 -m pip install idmlaser -i https://packages.idmod.org/api/pypi/pypi-production/simple

%runscript

%environment
    export INPUT_ROOT=Assets
    export HEADLESS=1

%labels
    Author [email protected]

%help
    Container for running LASER prototype on COMPS

Saved this to laser.def.

Created SIF locally

singularity build sifs/laser.sif laser.def

You'll need to have singularity installed to do that. Installing singularity can be a challenge. This can be done on COMPS too.

Uploaded to COMPS and Recorded Asset Collection Id

python3.9 -m COMPS create_asset_collection sifs/

I copy-pasted the AC ID from the console to a file called laser.id. You can use my AC ID now if you want.

Alternative: Make SIF on COMPS

h/t Josh Suresh for this code

#!/usr/bin/env python

from idmtools.core.platform_factory import Platform
from idmtools_platform_comps.utils.singularity_build import SingularityBuildWorkItem

if __name__ == '__main__':
    platform = Platform("CALCULON")
    sbi = SingularityBuildWorkItem(name="Create sif from Artifactory dockerfile", definition_file="my_local_laser.def", image_name="my_new_laser.sif")
    sbi.tags = dict(my_key="my_value")
    sbi.run(wait_until_done=True, platform=platform)
    if sbi.succeeded:
        # Write ID file
        sbi.asset_collection.to_id_file("laser.id")

Created local collection of input files

python3.9 -m idmlaser.utils.build_template_workspace

And follow instructions to select a directory and a scenario.

Copy relevant input files to local inputs/ dir

After running both of my example scenarios I had:

├── inputs
│   ├── demographics_settings.py
│   ├── fits.npy
│   ├── modeled_pop.csv.gz
│   └── settings.py
├── inputs_ew
│   ├── attraction_probabilities.csv
│   ├── cbrs_ew.csv
│   ├── cities.csv
│   ├── demographics_settings.py
│   ├── engwal_modeled.csv.gz
│   ├── fits.npy
│   └── settings.py
├── laser.def
├── laser.id
├── run_laser_with_sif.py
├── sifs
│   └── laser.sif

"inputs" was the single-node synthetic demo. inputs_ew was for England and Wales. Note: I had to copy fits.npy by hand because my script has a bug which doesn't generate this for you yet.

Created run script.

See here: https://github.com/InstituteforDiseaseModeling/laser/blob/jb_modulify/jb/run_laser_on_comps.py (currently broken)

Make sure run script points to a valid inputs dir. E.g.,

    task.common_assets.add_directory('inputs_ew')

Run

python3.9 run_laser_on_comps.py

Wait 10 seconds for the single node example or 3-4 minutes for the England & Wales one, assuming they are running for 20 years.

Observe Results

You can see a couple of successful COMPS runs here and here.

Run At Scale (Sweeps)

Code here.

There is a run script run_laser_with_sif.py which uses a new Task: PyConfiguredSingularityTask, which is in the file laser_task.py. The run script demonstrates how to run a 3x3x3x3 multidimensional sweep.

A Note About Assets, Paths, and Common Errors

There are "Common Assets" and "Transient Assets".

• Common Assets are the same across an Experiment and live in the Assets sub-directory.
• Transient Assets vary across an Experiment, per Simulation, and live in the top level (main) sub-directory. So any files or scripts that are part of a sweep will likely be per-simulation transient assets in the top-level directory. This of course matters when loading or importing files. The top level directory should always be in the python path. But the Assets directory will need to be added to the sys.path list for an import to work on a file in there.

A Note About Specifying COMPS Resources

We use the add_schedule_config function to specify node_group and other resources. See full documentation here.