v0.9.1

Release Notes

• Metadata: Replace MDHIM-based support with tree-based group rpcs for file operations
  • UnifyFS itself no longer depends on MPI (unless configured with --enable-mdhim)
  • MPI is currently still needed for the example programs that ship with UnifyFS
• Add write flattening through use of segment trees
• Improved HDF5 and MPI-IO support
• Implement unifyfs-stage infrastructure for --stage-in and --stage-out support
• Add support for truncate and ftruncate
• Add support for chdir and getcwd
• Optimization to satisfy read requests from local extent info if possible
• Implement and use common log-based I/O methods
• Support per-file sync
• Remove runstate file
• Update unifyfs utility to wait until all servers become ready
• Improvements to error reporting
• Update GOTCHA dependency to v1.0.3
• Support for relative paths by using spath dependency
• Drop NUMA, FlatCC, and LevelDB dependencies
• Performance improvements and code cleanup
• Improvements to example programs
• Addition of more unit and integration tests
  • Update GitLab CI to run on multiple architectures
  • Add UNIFYFS prefix to CI envars to avoid conflicts
  • Improve errno handling in unit tests
• Updated documentation

Config option changes

• Removal or rename of all UNIFYFS_SHMEM and UNIFYFS_SPILLOVER options
• UNIFYFS_CLIENT_CWD to set effective starting current working directory
• UNIFYFS_CLIENT_LOCAL_EXTENTS to service reads from local data if possible
• UNIFYFS_CLIENT_RECV_DATA_SIZE to set max size of memory buffer for receiving data from server.
• UNIFYFS_CLIENT_WRITE_INDEX_SIZE to set max size of memory buffer for storing write log metadata.
• UNIFYFS_CLIENT_WRITE_SYNC to sync data to server after every write
• UNIFYFS_LOGIO_CHUNK_SIZE to set data chunk size. Formerly UNIFYFS_SHMEM_CHUNK_BITS
• UNIFYFS_LOGIO_SHMEM_SIZE to set max size of data in shared memory. Formerly UNIFYFS_SHMEM_CHUNK_MEM
• UNIFYFS_LOGIO_SPILL_SIZE to set max size of data in spillover file. Formerly UNIFYFS_SPILLOVER_SIZE
• UNIFYFS_LOGIO_SPILL_DIR to set path to spillover data directory. Formerly UNIFYFS_SPILLOVER_META/DATA_DIR
• UNIFYFS_SERVER_INIT_TIMEOUT to set timeout (s) to wait for servers to be ready for clients

Numerous bug fixes

• handle read extent with no registered data chunks
• invoke correct rpc for truncate bcast tree
• fix stat_cmd() when errno != 0 from prior call
• flush spillover file before invoking sync rpc with server
• single server startup hangs when using the unifyfs CLI tool
• race condition on server pidfile creation
• truncate filesize issues due to client caching of extents
• lio_listio detection during configure
• use argobots mutex for svcmgr state synchronization
• use separate argobots mutex for reqmgr request locking
• handle EINTR during posix_fallocate
• stat file time fixes
• segfault in remote read creation
• remove logio spill files on server exit
• avoid double free on client->reqmgr
• race condition between server and client during mread
• logio fixes for empty shmem or spill
• fix server application cleanup deadlock
• duplicate client id assignment
• fix overrun in parsing LSF host list
• fix chdir unit test errno expected value
• fix for potential trailing chars in intercept_path check
• return number of bytes written in fid_write and fid_logio_write
• fix to use original file descriptor when delegating to a wrapper
• do not unlink superblock file on unmount
• handle read rpcs that have no write index values
• fix offset when reading from spillover file

Potential Hang

Note that there is currently a potential hang that may be encountered when using UnifyFS at larger scales.
This hang starts to show up around 1024 processes (256 compute nodes w/ ppn=4) and is currently being investigated.