fitsio can't write large arrays and doesn't raise an error

[londumas]

Running on fitsio.__version__ = 0.9.11 and on sp.__version__ = 1.2.0, with the minimalist example bellow. This code shows that fitsio can't write large arrays but does not raise any error.
In the case of main(coef=2), where we save a large array, the output fits file has 6 HDUs instead of 11, and has a corrupted last HDU: OSError: FITSIO status = 107: tried to move past end of file.

Do you have an idea of why fitsio can't write the file? Is it too large?
Do you have an idea why fitsio is not raising any writing error?

import numpy as np
import fitsio

def main(coef):

    ### Create
    n1 = 50
    n2 = 50
    nHDU = 10

    out = fitsio.FITS('test.fits.gz','rw',clobber=True)

    for i in range(nHDU):

        print(i)

        out_list = []
        out_names = []

        out_names += ['A']
        out_list += [np.zeros(n1*n2*coef*coef)]

        out_names += ['B']
        out_list += [np.zeros(n1*n2*coef*coef)]

        out_names += ['C']
        out_list += [np.zeros(n1*n2*coef*coef)]

        out_names += ['D']
        out_list += [np.zeros(n1*n2*coef*coef)]

        out_names += ['E']
        out_list += [np.zeros((n1*n2*coef*coef,n1*n2*coef*coef))]

        out.write(out_list,names=out_names)

    out.close()

    ### Read
    h = fitsio.FITS('test.fits.gz')
    print('There are {} HDUs'.format(len(h)))
    if len(h) != nHDU+1:
        print(h[-1]['A'][:])

    return

main(coef=1)
main(coef=2)

returns

0
1
2
3
4
5
6
7
8
9
There are 11 HDUs
0
1
2
3
4
5
6
7
8
9
There are 6 HDUs
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-4-02941757b203> in <module>()
     46 
     47 main(coef=1)
---> 48 main(coef=2)

<ipython-input-4-02941757b203> in main(coef)
     41     print('There are {} HDUs'.format(len(h)))
     42     if len(h) != nHDU+1:
---> 43         print(h[-1]['A'][:])
     44 
     45     return

/uufs/chpc.utah.edu/sys/installdir/python/3.6.3/lib/python3.6/site-packages/fitsio/fitslib.py in __getitem__(self, arg)
   3324         if isrows:
   3325             # rows was entered: read all current column subset
-> 3326             return self.read(rows=res)
   3327 
   3328         # columns was entered.  Return a subset objects

/uufs/chpc.utah.edu/sys/installdir/python/3.6.3/lib/python3.6/site-packages/fitsio/fitslib.py in read(self, **keys)
   3298 
   3299         if self.is_scalar:
-> 3300             data = self.fitshdu.read_column(self.columns, **keys)
   3301         else:
   3302             c=keys.get('columns',None)

/uufs/chpc.utah.edu/sys/installdir/python/3.6.3/lib/python3.6/site-packages/fitsio/fitslib.py in read_column(self, col, **keys)
   1887         """
   1888 
-> 1889         res = self.read_columns([col], **keys)
   1890         colname = res.dtype.names[0]
   1891         data = res[colname]

/uufs/chpc.utah.edu/sys/installdir/python/3.6.3/lib/python3.6/site-packages/fitsio/fitslib.py in read_columns(self, columns, **keys)
   2011             colnumsp = colnums[:].copy()
   2012             colnumsp[:] += 1
-> 2013             self._FITS.read_columns_as_rec(self._ext+1, colnumsp, array, rows)
   2014 
   2015             for i in xrange(colnums.size):

OSError: FITSIO status = 107: tried to move past end of file

[ngbusca]

@londumas can you try without the .gz ? I've encountered problems with large files when gzipping in the past.

[londumas]

@ngbusca, indeed changing this line out = fitsio.FITS('test.fits.gz','rw',clobber=True) by this line out = fitsio.FITS('test.fits','rw',clobber=True) fixes everything. Strange!

[esheldon]

I think this problem is occurring in cfitsio, the library that is wrapped by this python package. It might be that this error should be reported to the cfitsio maintainers. However, in that case you would need to write the example in C.

[londumas]

@esheldon, thank you for the answer. I have no idea on how to write an example in C for this bug. Is there anyway you or someone else could do it?

[rainwoodman]

Was the cfitsio compiled with large file support?

https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/f_user/node13.html

[esheldon]

I think this is now the default

https://github.com/esheldon/fitsio/blob/master/cfitsio3430/configure#L4523

but maybe this is broken for gzip

[esheldon]

certainly I routinely read and write files larger than 2.1G

[esheldon]

Where do we stand on this issue? Does it work for the latest version (1.1.3), for which we have updated cfitsio version to 3.49?

[londumas]

@esheldon, I still have no ways to try on the latest version, since I can not pip install it (issue #298 (comment)). But I can say, that as 1.0.5, it is still there

[esheldon]

Have you tried using conda to install?

[londumas]

@esheldon, no I have not. If you give me the set of commands, I be happy to try.

[esheldon]

If you have an anaconda install it is easy

conda install fitsio

If conda-forge isn't in your channels, do this first

conda config --add channels conda-forge 

That will add conda forge to your ~/.condarc.

Note, I usually make sure that conda-forge is first in the list so it gets packages from there first, e.g. make sure that file looks like this

channels:
  - conda-forge
  - defaults

but that might be a matter of taste

[londumas]

Just tested on fitsio=1.1.3, with conda, and still an issue.

[akira-okumura]

I have the same issue with fitsio 1.1.8 that I installed from conda-forge on Cent OS 7 and macOS (ARM).

[esheldon]

I think this is an issue with cfitsio

[beckermr]

Is there a flag for building that we didn't set that allows it to run on larger arrays?

[esheldon]

fitsio works fine on large arrays, the issue seems to be specifically for .gz files