sif_parser

A small package to read Andor Technology Multi-Channel files.

This package was renamed from sif_reader.

Install

This package can be installed via pip

pip install sif_parser

or if you have git installed in your system, you can also do

pip install git+https://www.github.com/fujiisoup/sif_parser

Basic usage

It provides the following methods,

sif_parser.np_open

Read '.sif' file and return as a np.ndarray for image and an OrderedDict for metadata.

>>> import sif_parser
>>> data, info = sif_parser.np_open('/path/to/file.sif')
>>> data
array([[[887.  , 881.25, 875.65, ..., 866.05, 870.  ],
        [905.6 , 872.7 , 900.7 , ..., 871.4 , 866.45],
        ...,
        [885.6 , 879.4 , 873.5 , ..., 883.6 , 877.  ],
        [879.4 , 873.  , 880.5 , ..., 881.  , 867.  ]]],
      dtype=float32)
>>> info
OrderedDict([('SifVersion', 65559),
             ('ExperimentTime', 1254330082),
             ('DetectorTemperature', -100.0),
             ...
            ])

If your calibration data is included in the file, this will be included as info['Calibration_data'] or info['Calibration_data_for_frame_1'].

Lazy load

If your data is very big but you are only interested in a certain part of the file, you can use lazy load feature.

>>> data, info = sif_parser.np_open('path/to/file', lazy='memmap')  # <-- it only reads the header.
>>> data.shape  # <-- we know the shape
 (1900, 74, 84)
>>> da = data[10]  # <-- we can even index the data, BEFORE actually reading the file  
>>> np.array(da)  # <-- Read only the 10th frame and store it into the memory

We can use either lazy='memmap' and lazy='dask'.
With lazy='memmap', we use [np.memmap](https://numpy.org/doc/stable/reference/generated/numpy. memmap.html), where we create an off-memory data that points the sif file. Withlazy='dask', dask.Array will be returned. See dask for the details. For this option, dask must be installed in your system.

sif_parser.xr_open('/path/to/file.sif'):

xarray must be installed to use this method.

Read 'sif' file and return as a xr.DataArray. The metadata is stored in xr.DataArray.attrs. The calibration data and timestamps are stored as coordinates.

xarray is a very useful package to handle multi-dimensional arrays with metadata. See xarray project for the details.

>>> sif_parser.xr_open('testings/examples/image.sif')
<xarray.DataArray (Time: 1, height: 512, width: 512)>
array([[[887.  , 881.25, 875.65, ..., 866.05, 870.  ],
        [905.6 , 872.7 , 900.7 , ..., 871.4 , 866.45],
        [922.6 , 883.95, 899.  , ..., 864.6 , 864.8 ],
        ...,
        [880.65, 857.95, 883.55, ..., 866.  , 875.55],
        [885.6 , 879.4 , 873.5 , ..., 883.6 , 877.  ],
        [879.4 , 873.  , 880.5 , ..., 881.  , 867.  ]]],
      dtype=float32)
Coordinates:
  * Time     (Time) float64 0.0
Dimensions without coordinates: height, width
Attributes:
    SifVersion:            65559
    ExperimentTime:        1254330082
    DetectorTemperature:   -100.0
    ...

Lazy load

Lazy load is also possible for xr_open. To do so, just pass either lazy='memmap' or lazy='dask'.

sif_parser.np_spool_open('/path/to/spool_files'):

Read from a directory the binary files and metadata generated via spooling and return a np.array. Spooling acquisition save your data directly on disk when reading from your camera. When spooling acquisition is enabled, a directory is created in your PC and the data is written directly on the hard disk as it is being acquired (see the Andor SDK Manual for more details).

Spooling acquisition normally generates the following files by default and must be present in the directory:

• 1 file with the extension *.sifx. This is the header of the file containing the metadata. You could also read it by using the method sif_parser.np_open('/path/to/my_file.sifx', ignore_corrupt=True).
• 1 file with the extension *.ini. This file contains information on the image format such as number of pixels by row (AOIWidth) number of rows and (AOIHeight),and padding bytes (AOIStride), pixel encoding, etc. (See the Andor SDK manual for more details).
• 1 or set of files with the extension *spool.dat containing the actual image data as binary files.

>>> data, info = sif_parser.np_spool_open('/path/to/spool_files')

>>> data
array([[[2873, 2861, 2876, ..., 4016, 4185, 4086],
         [2846, 2730, 2915, ..., 4101, 4136, 4290],
         ...,
         [8269, 8247, 8554, ..., 4177, 3988, 4072],
         [8332, 8224, 9474, ..., 4112, 4056, 4124]]], 
      dtype=uint32),
>>> info
OrderedDict([('SifVersion', 65567),
              ('ExperimentTime', 1688045153),
              ('DetectorTemperature', 0.0),
              ...
            ])

Utils

sif_parser.utils.extract_calibration

The Calibration_data entry of info contains coefficients of a cubic polynomial used to calculate the wavelengths of an image. To facilitate this sif_parser.utils contains the extract_calibration function, which returns the wavelength of each pixel.

data, info = sif_parser.np_open('path/to/file.sif')
wavelengths = sif_parser.utils.extract_calibration(info)

sif_parser.utils.parse

Used to parse a .sif file into a 2 column numpy array as wavelengths and counts.

import pandas as pd
import sif_parser


# parse the 'my_pl.sif' file
(data, info) = sif_parser.utils.parse('my_pl.sif')

# place data into a pandas Series
df = pd.Series(data[:, 1], index = data[:, 0])

CLI

Installs a command line interface (CLI) named sif_parser that can be used to convert .sif files to .csv.

Convert all .sif files in the current directory to .csv.

sif_parser

Convert all .sif files ending in pl in the current directly into a single .csv.

sif_parser --join *pl.sif

Use as a plugin for PIL

NOTE!! This feature was removed. See the issue #7

Previously, we have provided a plugin for PIL,

from PIL import image
import sif_parser.plugin

I = Image.open('/path/to/file.sif')

History

This plugin is originally developed by soemraws based on Marcel Leutenegger's MATLAB script.

Current status

Andor has changed sif format for many times. Although I have tested this package with as many kinds of sif files as I have (the test suit is always checking the compatibility, as the badge above shows), it might be still incompatible with your particular sif file.

If your file cannot be read by this script, please raise an issue in github. If you send me your file, I can add your file into the test suit (I have a private repo in order to keep your sif file private).

Contribution is also very welcome.

License of original MATLAB script

Copyright (c) 2006, Marcel Leutenegger All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution
• Neither the name of the Ecole Polytechnique Fédérale de Lausanne, Laboratoire d'Optique Biomédicale nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.