v2.0

Release Notes

• Add resource manager support for Flux (#798)
• Scalability improvements (#785)
• Add C and Fortran wrappers for MPI_Init_thread (#774)
• Improved ls tool - can now list files from all the server processes (#764)
• Rework server bootstrap completion (#800)
• Change the OpenSSL MD5 API function from the deprecated MD5 to the new EVP_Digest (#791)
• New I/O wrappers:
  • fopen64 (#771)
  • stat64, fstat64, __xstat64, __fxstat64 and __lxstat64 (#775)
• Documentation:
  • Embed UnifyFS Quickstart and ECP Tutorial videos/slides on Overview page
  • Update to use new .readthedocs.yaml file
    • Enable reproducible builds
  • Add UnifyFS paper citation
  • Add details to PnetCDF docs
• Testing:
  • Add tests for stat(), lstat() and fstat() (#773)

Bug Fixes

• compile and link agains the OpenSSL detected by the configure script (#799)
• fortran MPI wrappers have wrong parameter types (#795)
• handle the case where h5cc -show returns multiple lines (#794)
• do not assume file must exist after an open call (#789)
• fix: remove potential double free (#787)
• fix warning about too small buffer in write() call (#786)
• define stat wrappers depending on whether symbol is found (#779)
• call PMPI_Finalize() before unmount the client (#778)
• fstat() after close should fail with EBADF (#773)

v1.1

Release Notes

• Update Mochi-Margo usage for compatibility with newer dependency versions (#765)
  • Primarily tested with Mochi-Margo v0.13.1, Mercury v2.2.0, and libfabric v1.13.2
    • Enables compatibility with Mochi-Margo v0.10-v0.13.1*
• Updates to handle O_EXCL and EEXIST (#746)
• Add config for optional sleep after unlink() (#745)
• Made basic changes to the coll_request struct to allow returning data from broadcast RPCs (#758)
• New unit tests
• Update GitHub Actions testing
• Documentation Updates
  • Added distinct ROMIO, HDF5, and PnetCDF limitation sections

*Not all Mochi-Margo versions within this range have been fully tested. Create an issue if any versions cause a problem.

Config Option Changes

• New UNIFYFS_CLIENT_UNLINK_USECS for optional sleep after client-to-server unlink rpc
• New UNIFYFS_CLIENT_EXCL_PRIVATE config option to toggle whether O_EXCL implies a private file (default: on)
  • Set UNIFYFS_CLIENT_EXCL_PRIVATE=0 to create shared files using O_EXCL

Bug Fixes

• allow create on existing file
• avoid EEXIST error and truncate if needed when opening an existing file
• fix unlock bug in shmem logio when no spill available (#766)

v1.0.1

Release Notes

• Changes to support Node Local extents (#725)
  • Allows reads to be executed locally
  • Optimization for:
    • read-only workload where data is cached in unifyfs node-locally.
    • producer/consumer use cases like MuMMI are colocated in the same node.
  • New client configuration for node_local_extents (UNIFYFS_CLIENT_NODE_LOCAL_EXTENTS)
• Implement new unifyfs-ls utility (#736)
• Add optional LD_PRELOAD library (#737)
  • New --enable-preload config option
• Improvements to pkg-config support
• Add a new unifyfs-static.pc for builds that want to use linker wrapping
• Add new standalone Makefile.examples in order to build the example programs from an installed UnifyFS
• Add dup(), dup2(), futimens(), and utimensat() wrappers
• Additional unit tests
• GitHub Actions updates/fixes
• Documentation updates

Config Option Changes

• New UNIFYFS_CLIENT_NODE_LOCAL_EXTENTS option for node-local reads optimization
• New --enable-preload configure time option

Bug Fixes

• only use MPI flags on MPI client libs
• fix for unifyfsd launch using srun
• remove unnecessary includes in examples
• fixes for a working PMI2-based startup and clean termination of request manager threads
• fix next_page_align unused function compiler erro

v1.0

Release Notes

• Add client API for I/O middleware libraries (#622, #651, #662, #665, #667, #719)
• Update Mochi-Margo usage (new required version - v0.9.6) (#703)
• Build separate libraries for MPI auto-mount (#640)
• Improve file staging (#614, #631, #717)
• Install unifyfs-stage in bin (#710)
• Ensure logio data is persisted to storage on user fsync() (#655)
  • Configure with new UNIFYFS_CLIENT_FSYNC_PERSIST option
• Add server.local_extents mode (#695)
• Add client unlink callbacks (#661)
• Add heartbeat RPC to detect failed clients (#653)
• Enable servers to use PMI to get rank and count (#626)
• Release logio storage for extents on file unlink (#656)
• Offload server-server request processing to service manager threads (#627)
• Cray compiler wrapper support for lx_find_mpi.m4
• Update example programs
• Migrate from Travis CI to GitHub Actions
• Unit tests
  • Client API tests
  • Add lipsum generator/checks
  • Update unifyfs-stage tests
  • Print failed tests at end of run
• Continuous Integration tests
  • Run sub-sets of tests in independent jobs
  • Laminated and non-laminated file tests
  • New stage/transfer tests
  • Test on additional compilers
• Documentation updates
  • API for I/O Middleware
  • Known limitations of UnifyFS
  • Determining UnifyFS compatibility with VerifyIO
  • General reorganization and updates

Config Option Changes

• New UNIFYFS_CLIENT_FSYNC_PERSIST to persist data to storage on calls to fsync()
• New UNIFYFS_MARGO_CLIENT_TIMEOUT to adjust rpc timeouts between client and server
• New UNIFYFS_MARGO_SERVER_TIMEOUT to adjust rpc timeouts between servers
• New UNIFYFS_MARGO_LAZY_CONNECT to speed up bootstrap
• New UNIFYFS_SERVER_LOCAL_EXTENTS to use server extents to service local reads without consulting file owner
• Set UNIFYFS_DAEMONIZE default to off
• Remove unused UNIFYFS_CONSISTENCY config

Bug Fixes

• avoid request signal if RM thread exiting (#721)
• disallow unifyfs_create() of existing file (#720)
• PMIx initialization bug (#712)
• fix for serial transfer from non-zero rank client
• fix to use flexible arrays (#696)
• define _STAT_VER if it's not already defined
• fix various memory leaks
• initialize unifyfs_io_request result and state in dispatch (#683)
• fix for failed reads when client.local_extents=1 (#680)
• fix attempt to sync unlinked file (#679)
• allow remote clients to read non-shared files once they are laminated
• fix undefined refs for inline logio functions
• avoid concurrent logio updates by the client and server
• defer ABT_init to Margo (#659)
• fix bug in unifyfs_fskv_init() (#654)
• remove unused app_id parameter to unifyfs_mount()
• unlink global file not found locally (#638)
• avoid laminate broadcast for private files
• allocate extent_tree when inode is allocated
• disable unnecessary dispatcher signaling
• use max bulk size in client-server transfers
• only broadcast truncate when new size is smaller
• avoid metaget storm on file open by using server cached attributes
• build fix for --enable-pmix
• remove racy client read pthread condition signal/wait
• fix unifyfs start/terminate on SLURM
• allow variable logio header size (#635)
• fix build failure from undefined reference to major, minor (#629)
• fix for missing req completion notice for hole extent
• fix for write after laminate
• eliminate shadow copy of file attributes in client metadata
• fix to clean up logio shmem regions on server exit

Potentially Impactful Issues

• Server may hang at shutdown on multi-node jobs with large number of clients

v0.9.2

Release Notes

• Minimum Automake version of 1.15 or newer now required
• Update for newer versions of dependencies (Margo v0.9.1, Mercury v2.0.0)
  • Select OFI over BMI when both NA are available for mercury
  • margo_init() - fallback to ofi+sockets if ofi+tcp fails
• Use RPC to return read data to client
• Avoid interleaved log messages
• Add support for setting cores-per-server via environ
• Define UNIFYFS_SUPER_MAGIC value as default returned by statfs
  • Setting UNIFYFS_CLIENT_SUPER_MAGIC=0 will revert to old behavior of returning TMPFS_MAGIC
• Add UNIFYFS_LOG_ON_ERROR support
• Update unifyfs-config and pkgconfig input files
• Redirect mercury log messages to our server log
• Update build
  • make GOTCHA a fully optional (but preferred) dependency
    • Note: MPI-IO and HDF5 support are lost when building without GOTCHA
  • Don't build anything needing Fortran support if GOTCHA not found
    • Note: GOTCHA dependency now required when enabling Fortran support
  • Add a --with-spath config option
  • Skip unit tests needing spath when dependency not available
• Update Travis CI to test on multiple version of GCC
• Update unit tests
  • Add read of increasing size unit tests
  • Add statfs tests
  • Addition of statfs test suite for testing UnifyFS with different config options
• Update CI tests
  • Refactor structure of suites
  • Add writeread with shuffle tests
  • Add MPI-IO tests
  • Add producer/consumer workflow tests
• Update bootstrap.sh
• Documentation updates

Config option changes

• New UNIFYFS_LOG_ON_ERROR option
• New UNIFYFS_CLIENT_SUPER_MAGIC option
• Set UNIFYFS_DAEMONIZE default to on
• Removal of UNIFYFS_CLIENT_RECV_DATA_SIZE option

Bug fixes

• avoid truncate on creating a new file
• fix size_t/unsigned param mismatch
• remove forced call to stat in write_laminate
• fix pkgconfig support
• testutil fix for data checking on unaligned write
• fix test suites to call unifyfs_unmount
• unit test fixes for correct errno usage
• mercury NA SM - enable kernel.yama.ptrace_scope
• fix minor bugs in write and read examples
• fix to execute filesize rpc only for (unlaminated) regular files
• fix to open example (sysio-open) to separate the create and open rank
• fix inode extent add race
• fix for truncate broadcast ENOENT
• fix for laminate bcast when inode does not exist
• fix race condition on file create

Potential Hang

Note that there is currently a potential hang that may be encountered when using UnifyFS at larger scales.
This hang starts to show up around 1024 processes (256 compute nodes w/ ppn=4) and is currently being investigated.

v0.9.1

Release Notes

• Metadata: Replace MDHIM-based support with tree-based group rpcs for file operations
  • UnifyFS itself no longer depends on MPI (unless configured with --enable-mdhim)
  • MPI is currently still needed for the example programs that ship with UnifyFS
• Add write flattening through use of segment trees
• Improved HDF5 and MPI-IO support
• Implement unifyfs-stage infrastructure for --stage-in and --stage-out support
• Add support for truncate and ftruncate
• Add support for chdir and getcwd
• Optimization to satisfy read requests from local extent info if possible
• Implement and use common log-based I/O methods
• Support per-file sync
• Remove runstate file
• Update unifyfs utility to wait until all servers become ready
• Improvements to error reporting
• Update GOTCHA dependency to v1.0.3
• Support for relative paths by using spath dependency
• Drop NUMA, FlatCC, and LevelDB dependencies
• Performance improvements and code cleanup
• Improvements to example programs
• Addition of more unit and integration tests
  • Update GitLab CI to run on multiple architectures
  • Add UNIFYFS prefix to CI envars to avoid conflicts
  • Improve errno handling in unit tests
• Updated documentation

Config option changes

• Removal or rename of all UNIFYFS_SHMEM and UNIFYFS_SPILLOVER options
• UNIFYFS_CLIENT_CWD to set effective starting current working directory
• UNIFYFS_CLIENT_LOCAL_EXTENTS to service reads from local data if possible
• UNIFYFS_CLIENT_RECV_DATA_SIZE to set max size of memory buffer for receiving data from server.
• UNIFYFS_CLIENT_WRITE_INDEX_SIZE to set max size of memory buffer for storing write log metadata.
• UNIFYFS_CLIENT_WRITE_SYNC to sync data to server after every write
• UNIFYFS_LOGIO_CHUNK_SIZE to set data chunk size. Formerly UNIFYFS_SHMEM_CHUNK_BITS
• UNIFYFS_LOGIO_SHMEM_SIZE to set max size of data in shared memory. Formerly UNIFYFS_SHMEM_CHUNK_MEM
• UNIFYFS_LOGIO_SPILL_SIZE to set max size of data in spillover file. Formerly UNIFYFS_SPILLOVER_SIZE
• UNIFYFS_LOGIO_SPILL_DIR to set path to spillover data directory. Formerly UNIFYFS_SPILLOVER_META/DATA_DIR
• UNIFYFS_SERVER_INIT_TIMEOUT to set timeout (s) to wait for servers to be ready for clients

Numerous bug fixes

• handle read extent with no registered data chunks
• invoke correct rpc for truncate bcast tree
• fix stat_cmd() when errno != 0 from prior call
• flush spillover file before invoking sync rpc with server
• single server startup hangs when using the unifyfs CLI tool
• race condition on server pidfile creation
• truncate filesize issues due to client caching of extents
• lio_listio detection during configure
• use argobots mutex for svcmgr state synchronization
• use separate argobots mutex for reqmgr request locking
• handle EINTR during posix_fallocate
• stat file time fixes
• segfault in remote read creation
• remove logio spill files on server exit
• avoid double free on client->reqmgr
• race condition between server and client during mread
• logio fixes for empty shmem or spill
• fix server application cleanup deadlock
• duplicate client id assignment
• fix overrun in parsing LSF host list
• fix chdir unit test errno expected value
• fix for potential trailing chars in intercept_path check
• return number of bytes written in fid_write and fid_logio_write
• fix to use original file descriptor when delegating to a wrapper
• do not unlink superblock file on unmount
• handle read rpcs that have no write index values
• fix offset when reading from spillover file

Potential Hang

Note that there is currently a potential hang that may be encountered when using UnifyFS at larger scales.
This hang starts to show up around 1024 processes (256 compute nodes w/ ppn=4) and is currently being investigated.

v0.9.0

Release Notes

• Rename from UnifyCR to UnifyFS
• Completed integration of Mercury RPC into UnifyFS
• Updated stat() behavior to return consistent size values
• Designed, implemented new metadata layer for swappable metadata service
• UnifyFS Transfer API to support VeloC
• Improved resource manager support, LSF
• Implemented support for lamination operation with chmod
• Initial Fortran support
  • Implemented support for mount/unmount
• Performance improvements and code cleanup
• Numerous bug fixes
  • Correctness on read
  • Cleanup on server shutdown
  • etc
• Updated documentation
• Added new example programs
• Integrated continuous integration testing suite, Gitlab
• Spack package updates
• Added build/configure capability alternative to Spack

v0.2.0

Tag for version 0.2.0