Skip to content

SIRF and Python

Jump to bottom
Kris Thielemans edited this page Jun 18, 2024 · 21 revisions

SIRF can be called from Python. This page documents some issues that you might encounter.

Python versions

Up to SIRF version 2.2.0 we supported both Python 2 and 3. SIRF 3.0 dropped Python 2 support (it might still work, but don't try!).

Version conflicts

If you have more than 1 Python version installed, you could encounter conflicts when installing or using SIRF. It is important that you use the same version of python throughout the installation, and of course also when running SIRF.

For instance, many people have both python2.7 and python3, or have different virtualenv's. You will have to create a separate SIRF build for every version of Python that you want to use. Obviously, you will also have to install Python packages for each. Using an environment such as Anaconda might help.

If you have conflicts during installation, or CMake picked up the wrong version, we recommend to first try to set the CMake variable PYTHON_EXECUTABLE (use an absolute path!). For instance,

    # find out where python3 sits
    which python3
    cmake -DPYTHON_EXECUTABLE=/wherever/it/was/python3

If that fails, set both PYTHON_LIBRARY and PYTHON_INCLUDE_DIR. On many systems python-config or python3-config can help (but not when using Anaconda).

Please make sure that you always use the Python version that you used to build SIRF.

Linux

As an example, you could have

$ python3-config --ldflags
-L/usr/lib/python3.4/config-3.4m-x86_64-linux-gnu -L/usr/lib -lpython3.4m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
$ python3-config --cflags
-I/usr/include/python3.4m -I/usr/include/python3.4m  -Wno-unused-result -g -fstack-protector-strong -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

This tells you that appropriate settings are

PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.4m.so
PYTHON_INCLUDE_DIR=/usr/include/python3.4m

MacOS and brew

(This text is likely out-of-date).

You could have a look at the Python Guide for OSX. For instance, use

    brew install python

and then find the relevant location of your Python executable (CMake will then attempt to find your Python library and include directories). If you installed python3 with brew, the following might work:

    PY_INST=$(ls -d1 /usr/local/Cellar/python/3.*/Frameworks/Python.framework/Versions/3.*|tail -n 1)
    PY_VER=$(basename ${PY_INST})
    PY_EXE=$PY_INST/bin/python$PY_VER
    cmake -DPYTHON_EXECUTABLE=$PY_EXE <normal_options>

(This tries to find the (alphabetically) last Python 3 version, and then assigns its executable to PY_EXE).

Note that specifying PYTHON_EXECUTABLE=/usr/local/bin/python for CMake will sadly not work.

SuperBuild installation

After building and installing SIRF with the SuperBuild, several environment files for different shells (such as bash and csh) are created. These files contain all the necessary information about the binary installation to avoid conflicts of versions of software.

Before using SIRF, these files should be sourced in a shell session. In particular, they set the variables

  • SIRF_PYTHON_EXECUTABLE which points to the python executable that was used during the build.
  • PYTHONPATH which points to the location of the python modules for SIRF

Python packages

Use the SIRF requirements.txt to install packages. Note that if you used the SIRF-SuperBuild, you can find it in /wherever/you/built/sources/SIRF/requirements.txt.

We also recommend a Python IDE, e.g. spyder or VS Code.

Using pip

If you are using pip install (either with the SuperBuild or manually installing the dependencies of SIRF), you need to ensure that the same version of python is used throughout the installation. Assuming that this executable is stored in SIRF_PYTHON_EXECUTABLE, the following might work:

$SIRF_PYTHON_EXECUTABLE -m pip install --upgrade pip setuptools wheel
$SIRF_PYTHON_EXECUTABLE -m pip install -U -r requirements.txt

Using Anaconda/miniconda

Check our page for using conda to install dependencies.

Using your OS package manager

(This option is not recommended and the text is out-of-date).

On a debian based system you would issue

sudo apt-get install python-scipy  python-docopt python-numpy python-h5py python-matplotlib python-libxml2 python-psutil python-tk python-nose python-coverage

Run-time conflicts with shared libraries

Despite best efforts, it is possible that SIRF is compiled using a version of a library that is different from what is used by Python itself, for example if you compile SIRF with a more recent compiler than your normal system compiler. CMake should normally warn you about this with something like

  Cannot generate a safe runtime search path for target _pysirfreg because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
      /opt/Python/Python-3.6/lib

This might lead to run-time errors. Below is one example that we encountered on CentOS 7.

OMP library conflicts

We did encounter a corner case on CentOS 7 when using Python 2.7 with gcc 4.9 (possibly because the Python version was out-of-date). There was a conflict that the libgomp.so.1 version used by Python was too old. As more recent versions of libgomp should contain older versions as well, this can be resolved by using LD_PRELOAD.

LD_PRELOAD=/lib64/libgomp.so.1 python
Clone this wiki locally