Too many file access using the Python interfacte

[qezlou]

@rainwoodman

Hi Yu,
Admins on TACC have raised an issue with our code using thebigfile python interface. Apparently, it invokes too many queries to the filesystem, similar to your comment on README file:

The Python binding under MPI invoked more meta-data queries to the file system than we would like to be, though for small scale applications (thousands of cores) it is usually adequate

Not quite sure if this #43 fixes this though, since my issue is with reading not writing files.

Any starter clues, like which source file I need to dive into to fix this?

[qezlou]

After a bit of digging in the code:

1. Remove the MPI_Barrier after big_file_mpi_close #43 doesn't fix the issue we have here which is over-accessing file during file read
2. The python wrapper for open() and Dataset() don't use the MPI routines implemented for C (crc/bigfile-mpi.c) instead invoke the serial version, i.e. src/bigfile.c. It is already mentioned in the poster [https://github.com/rainwoodman/bigfile/tree/documents] that

Python binding is based on the Non-MPI API, with its own MPI interface;

However, in contrary to the C-MPI, python wrapper opens the file on all processes for read. After a dataset is indexed, the flow is :

Dataset() in bigfile.__init__.py ---> Dataset.read() in pyxbigfile.pyx ---> big_file_read_records() in src/bigfile.h and not the MPI version , big_file_mpi_read_records in src/bigile-mpi.h.

This might be why the I/O is overwhelmed with large number of processes. I am planning to try mpi4py.File.Open and Read() and check the I/O.

[sbird]

Yup, that would do it!

[rainwoodman]

One reason the python api did not use the c MPI api was that back then it was hard to create a python surrogate for the c MPI communicator object. Recent mpi4py offers a way ( I recall still not a public api but it is intended to be supported) of creating a python surrogate from a c pointer, so a rewrite based on that could be nice.

…

On Fri, Aug 16, 2024 at 4:02 PM Simeon Bird ***@***.***> wrote: Yup, that would do it! — Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBWTC6ISZWQBFVEWBCPMTZR2AJNAVCNFSM6AAAAABLRELZE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUGQZDGOJVGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>