Skip to content

Latest commit

 

History

History
486 lines (367 loc) · 16.2 KB

wcdo.org

File metadata and controls

486 lines (367 loc) · 16.2 KB

wcdo

Do stuff with Wire-Cell in a Singularity container.

Getting started

Installation

Download/copy the following script somewhere in your $PATH and make sure it is executable.

To assist in getting updates you may wish to clone this repository and symlink wcdo.sh into a directory in your $PATH.

Quick start usage

Pick a directory with several GB of space to work in and run a few wcdo.sh commands run in your native OS (the container host OS):

$ mkdir -p ~/wcdo/example
$ cd ~/wcdo/example
$ wcdo.sh init
$ wcdo.sh wct
$ wcdo.sh get-image sl7 
$ wcdo.sh make-project myproj sl7
$ ./wcdo-myproj.sh

With this last command, you are now inside the container environment. Many bash functions are available and may be discovered with tab-completion:

[sl7]bviren@hierocles:example> wcdo-TAB

Most have a ”wcdo-*-help” function that will print some help:

[sl7]bviren@hierocles:example> wcdo-wirecell-path-help

Set or display the WIRECELL_PATH

usage: wcdo-wirecell-path [ default <path> <path> ... ]

With no argument, print the current path.

A special "<path>" given as the literal string "default" will cause
the default directories for the container to be used.

Otherwise a <path> argument is prepended.

Note: the UPS product "wirecell" destroys the WIRECELL_PATH so this
function will be useful after any "setup" of "wirecell" UPS product.

Getting started commands

Each of the wcdo.sh native OS commands above are now described. Most importantly, users should read the last section describing the make-project command and customizing the files it produces.

init

$ wcdo.sh init

This prepares your current working directory to follow some conventions.

  • makes some sub-directory stubs
  • initializes an empty UPS products directory
  • downloads a recent copy of wcdo.rc for later use

wct

This command downloads Wire-Cell Toolkit source and data/cfg packages.

$ wcdo.sh wct [access branch]

Running this is the same as individually running:

$ wcdo.sh wct-data [access branch]
$ wcdo.sh wct-cfg [access branch]
$ wcdo.sh wct-source [access branch]

The optional [access branch] arguments control which GitHub URL is used and which branch of the packages is checked out.

The access option is one of:

anonymous
(default) anonymous (non-authenticated) HTTP Git access to GitHub is used.
developer
use authenticated access requiring user to supply credentials.

The branch specifies which point in the code history to retrieve:

~ master :: (default) tip of the master branches are retrieved

X.Y.x
(the .x is literal) a release branch is retrieved.

The results of these commands are just normal local git repositories. You are free to evolve them as you wish either on in your native OS or in the container.

get-image

This downloads a Singularity image based on its name. By default it simply downloads the image to the current working directory. You may instead supply a local cache directory in order to avoid redownloading the same image. A cache directory also results in the image being symlinked so you can save disk space. Finally, you can provide some other URL from which to find images. Get a little help by running with no arguments:

$ ./wcdo.sh get-image 
usage: wcdo get-image <name> [dir [url]]
        <name>	an Singularity image name
        <dir>	a local directory to find images
        <url>	base URL to find images

make-project

This command generates files in the current working directory. Some of these files should be modified by the user and some should not. Scripts (ending in .sh) contain code that executes inside your native OS. RC files (.rc) contain code that executes inside the container.

These generated files should not be modified manually:

wcdo.rc
a downloaded file holding many helpful bash functions.
wcdo-<name>.rc
a generated RC file which defines shell environment for the project.
wcdo-<name>.sh
a generated script which is used to enter the project of the given name.

These files are either generated or may be created by the user. The user should modify them. They will not be overwritten by wcdo.

wcdo-local.sh
Any shell code written here will be run by any wcdo-<name>.sh script.
wcdo-local-<name>.sh
Any shell code written here will be only by the matching wcdo-<name>.sh script.
wcdo-local.rc
Any shell code written here will be run in the container for any project.
wcdo-local-<name>.rc
Any shell code written here will be run in the container for only the matching project.

Projects

As already introduced, using the container is organized around the concept of a “project”. The idea is to bundle all the information for a particular task so that you may easily return to it days or weeks later after all the annoying details have long left your mind.

A wcdo project is defined by the working directory and the set of wcdo-*.{rc,sh} scripts as described above.

In principle, one may produce multiple sets of the wcdo-*.{rc,sh} files in the same working directory. However, it is possible (even easy) to produce conflicting configurations which can not share the same /wcdo working directory. For example, there is a single MRB-controlled development area mounted under /wcdo/src/mrb which can not support multiple sets of source packages that different wcdo projects may want.

To assure no conflicts, it is recommended to maintain only one wcdo working directory per project. The amount of extra storage required to maintain this separation is minimal assuming /cvmfs or a local UPS “products” area is shared between the projects.

Customizing project creation

When making many projects it can help to customize some defaults, for example the use of an image cache. This can be done in $HOME/.wcdo/main-local.sh. For example:

$ cat ~/.wcdo/main-local.sh
# Stuff here can override wcdo.sh operation.
wcdo_image_cache="$HOME/public_html/simg"

Customizing native side

When can customize how a project’s generated start script wcdo-<name>.sh behaves in three places:

wcdo-local-<name>.sh
this should hold any customization for the particular project. It will only be included from wcdo-<name>.sh.
wcdo-local.sh
in a wcdo project workspace this file will be included in every wcdo-<name>.sh. It is ideal for any commonalities between otherwise variations which may otherwise be expressed in wcdo-local-<name>.sh.
$HOME/.wcdo/project-local.sh
if existing, this is included in every wcdo-<name>.sh and is the place to provide global settings. For example, if directory holding the singularity command is not normally in your $PATH that variable could be updated here.

Customizing container side

You may wish to modify or augment the environment provided already by wcdo.rc and wcdo-<name>.rc. Making a project creates a skeleton of wcdo-local-<name>.rc which names several variable which help to reduce how many things you have to remember. You may also want to call a few commands each time you enter the project’s container.

For example, here is a modified version of a generated wcdo-local-<name>.rc stub which I use for testing Clang-based builds:

$ cat wcdo-local-cldev.rc
#!/bin/bash

# This is a local wcdo rc file for project cldev.
# It was initally generated but is recomended for customizing by you, dear user.
# It is included at the end of the main RC files.
  
# These are optional but required if wcdo-mrb-* commands are to be used.
wcdo_mrb_project_name="larsoft"
wcdo_mrb_project_version="v07_07_00"
wcdo_mrb_project_quals="c2:prof"

# Additional variables may be usefully set since this file was
# first generated.  

# It is perhaps useful to end this with some command to be called 
# on each entry to the contaner.
# The wcdo-* functions try to be idempotent.
source /cvmfs/larsoft.opensciencegrid.org/products/setup
path-prepend $wcdo_ups_products PRODUCTS
wcdo-mrb-init
wcdo-mrb-add-source larwirecell bviren_v070200 v07_02_00

I have another almost identical “project” which is for testing GCC-based builds. The only difference besides the “project” name is the wcdo_mrb_project_quals setting.

More examples of configuration are given below.

In-container environment

The directory structure is opinionated in order to reduce options/complexity. All user-accessible files are mounted at:

/wcdo

This is the same directory as used wcdo.sh init etc in the section Getting started. The user’s home directory is also typically mounted by Singularity and additional directories may be mounted by modifying the native-side .sh files.

As mentioned above, the environment is populated with various wcdo-* functions to try to make life with UPS/mrb easier. TAB-completion will show a list. Running a function ending in -help will provide some guidance for using its associated function. You can also dump their function bodies to see exactly what they do with the usual Bash type command:

$ type wcdo-mrb-goto 
wcdo-mrb-goto is a function
wcdo-mrb-goto () 
{ 
    if [ -z "$MRB_TOP" ]; then
        wcdo-mrb-init;
    fi;
    goto $MRB_TOP
}

Any global environment variables are named wcdo_* and are kept to a minimum and are meant to hold some static “wcdo project” information as typically set inside the native-side .sh files.

Extended example of use

Building WCT source

Here we modify the generated “stub” file wcdo-local-<name>.rc. Replace ”<name>” here and below with the project name you chose. Configure some variables that will be used to operate on the MRB-controlled source area and to set up UPS. With the comments deleted we have something like:

wcdo_mrb_project_name="larsoft"
wcdo_mrb_project_version="v07_07_00"
wcdo_mrb_project_quals="e17:prof"
wcdo-ups-init
path-append $wcdo_ups_products PRODUCTS

One can then run the container:

$ ./wcdo-<name>.sh

To start clean I remove any previous UPS products named “wirecell” that I might have made prior:

[sl7]bviren@hierocles:example> rm -rf /wcdo/lib/ups/wirecell
[sl7]bviren@hierocles:example> setup wirecell v0_9_1a -q e17:prof
[sl7]bviren@hierocles:example> wcdo-ups-declare wirecell DEVEL
[sl7]bviren@hierocles:example> setup wirecell DEVEL -q e17:prof

At this point, take note that UPS does not setup up the environment properly. In particular, LD_LIBRARY_PATH has no entry. But, one can go on to configure, build and install:

[sl7]bviren@hierocles:example> wcdo-ups-wct-configure-source
[sl7]bviren@hierocles:wct> ./wcb -p --notests install

At this point the lib directory exists and one must re-~setup~ the wirecell UPS “product” in order to unbreak the UPS environment. You also must reset the WIRECELL_PATH because the UPS “product” breaks it. UPS actively hates its users.

[sl7]bviren@hierocles:wct> echo $LD_LIBRARY_PATH |tr ':' '\n'|grep wirecell
[sl7]bviren@hierocles:wct> setup wirecell DEVEL -q e17:prof
[sl7]bviren@hierocles:wct> echo $LD_LIBRARY_PATH |tr ':' '\n'|grep wirecell
/wcdo/lib/ups/wirecell/DEVEL/Linux64bit+4.4-2.17-sl7-5-e17-prof/lib64

[sl7]bviren@hierocles:wct> wcdo-wirecell-path default

[sl7]bviren@hierocles:wct> ./wcb -p --alltests
...
  tests that fail 0/109

Adding idempotent instructions to wcdo-local-<name>.rc

In some cases you want to enter a container and assure some complex state. However, you don’t want laboriously retype command. Nor do you want to simply replay them in a simple script as rerunning them may not be needed and it wastes time.

Here is an example of a wcdo-local-<name>.rc file which has commands written in an “idempotent” manner. When placed in the workspace, its commands will be replayed every time the project container is entered and on subsequent entries the commands will complete quickly. In this example, the commands assure that the Wire-Cell Toolkit is built against a version of the externals from UPS. It also handles many of the anoyingly stateful UPS setup and resetup.

wcdo_mrb_project_name="larsoft"
wcdo_mrb_project_version="v07_07_00"
wcdo_mrb_project_quals="c2:prof"
wcdo_mrb_project_qualsm=$(echo $wcdo_mrb_project_quals | tr : -)
wcdo_mrb_project_quals_=$(echo $wcdo_mrb_project_quals | tr : _)

wcdo-ups-init

bv_wct_ups_version=v0_9_1a
bv_wct_dev_version=v091a_devel
bv_wct_ups_dir=/wcdo/lib/ups/wirecell/$bv_wct_dev_version
bv_wct_ups_bin_dir=$bv_wct_ups_dir/$(ups flavor)-${wcdo_mrb_project_qualsm}/bin

path-append $wcdo_ups_products PRODUCTS
if [ ! -d $bv_wct_ups_dir ] ; then
    setup wirecell $bv_wct_ups_version -q $wcdo_mrb_project_quals
    wcdo-ups-declare wirecell $bv_wct_dev_version
fi
setup wirecell $bv_wct_dev_version -q $wcdo_mrb_project_quals
cd /wcdo/src/wct/
if [ ! -d /wcdo/src/wct/build ] ; then
    wcdo-ups-wct-configure-source
fi
if [ ! -f /wcdo/src/wct/build/apps/wire-cell ] ; then
    ./wcb -p --notests
fi
if [ ! -f $bv_wct_ups_bin_dir/wire-cell ] ; then
    ./wcb -p --notests install
fi
setup wirecell $bv_wct_dev_version -q $wcdo_mrb_project_quals
wcdo-wirecell-path default
if [ ! -f build/apps/test_dlopen ] ; then
    ./wcb -p --alltests
fi
echo "wire-cell --help"
wire-cell --help

Command reuse

Going further, the above commands can be factored into a single Bash function, added to $HOME/wcdo/project-local.rc so that it may be easily reused in many different wcdo-local-<name>.rc files. Here’s how that might look:

Now the wcdo-local-<name>.rc file in its entirety is just a couple lines:

#!/bin/bash
wcdo-ups-init
bv-ups-wct-build v0_9_1a v091a_devel c2:prof

The $HOME/.wcdo/project-local.rc then receives idempotent script from above cleaned up a bit.

bv-ups-wct-build () {
    if [ -z "$SETUP_UPS" ] ; then
        echo "bv-ups-wct-build: need ups set up"
        return
    fi

    local oldver="$1" ; shift
    local newver="$1" ; shift
    local quals="$1"; shift
    local qualsm="$(echo $quals | tr : -)"

    local instdir=/wcdo/lib/ups/wirecell/$newver
    local bindir=$instdir/$(ups flavor)-${qualsm}/bin

    path-append $wcdo_ups_products PRODUCTS
    if [ ! -d $instdir ] ; then
        setup wirecell $oldver -q $quals
        wcdo-ups-declare wirecell $newver
    fi
    setup wirecell $newver -q $quals
    cd /wcdo/src/wct/
    if [ ! -d /wcdo/src/wct/build ] ; then
        wcdo-ups-wct-configure-source
    fi
    if [ ! -f /wcdo/src/wct/build/apps/wire-cell ] ; then
        ./wcb -p --notests
    fi
    if [ ! -f $bindir/wire-cell ] ; then
        ./wcb -p --notests install
    fi
    setup wirecell $newver -q $quals
    wcdo-wirecell-path default
    if [ ! -f build/apps/test_dlopen ] ; then
        ./wcb -p --alltests
    fi
    echo "wire-cell --help"
    wire-cell --help
}

Sharing with others

The wcdo system greatly helps to capture the state of complex software development in a small set of text files. In particular, when a problem (or a solution!) is discovered, it may be reproduced, bundled up and communicated to others.

Some steps to capture your work and share it may go like:

  1. Reproduce the problem starting again in a clean work space.
  2. Reproduce the problem entirely by adding commands to your native or container side “local” files wcdo-local-<name>.{rc,sh} and without relying on any customization in files under $HOME/.wcdo/.
  3. Bundle the state of your wcdo files by running ./wcdo-<name>.sh bundle.
  4. Provide the small wcdo-bundle-<name>.* files to others.