Version 2.1.4

Change Log

[2.1.4] - 2023-07-17

This release is a bugfix release on the 2.1.x branch.

(7e919aa) fs: fixed gcc-13 compilation warnings

Tested With

rocksdb-7.0.2
libzbd-2.0.2

Contributors (Alphabetical)

Przemyslaw Skibinski

Thanks everyone!

Version 2.1.3

Change Log

[2.1.3] - 2023-06-08

This release is a bugfix release on the 2.1.x branch.

(424c988) ci: Add github action for running percona-server smoke test
(7e0fac2) ci: Update percona smoke test schedule
(cea45ab) fs: fixed clang-16 compilation warnings

Tested With

rocksdb-7.0.2
libzbd-2.0.2

Contributors (Alphabetical)

Andreas Hindborg
Dennis Maisenbacher
Yura Sorokin

Thanks everyone!

Version 2.1.2

Change Log

[2.1.2] - 2022-10-07

This release is a bugfix release on the 2.1.x branch.

(5177e71) util: Fix occurances where back() might be called on an empty string

Tested With

rocksdb-7.0.2
libzbd-2.0.2

Contributors (Alphabetical)

Dennis Maisenbacher

Thanks everyone!

Version 2.1.1

Change Log

[2.1.1] - 2022-10-04

This release is a bugfix release on the 2.1.x branch.

(755c062) Fix AddDirSeparatorAtEnd where back() can not be called on an empty string
(9e26704) Enable basic checks for pull requests to any branch

Tested With

rocksdb-7.0.2
libzbd-2.0.2

Contributors (Alphabetical)

Dennis Maisenbacher
Yura Sorokin

Thanks everyone!

Version 2.1.0

Change Log

[2.1.0] - 2022-05-31

This release is a small feature release on the 2.x branch.

• (c7385f2) Allow restoration of single files from backup
• (252628c) Fix a bug where memcopy was used incorrectly. This bug caused an
  illegal instruction exception when ZenFS was running in Alpine Linux.

Tested With

• percona-server-8.0.28-19
• rocksdb-7.0.2
• libzbd-2.0.2

Contributors (Alphabetical)

Aravind Ramesh
Hans Holmberg

Thanks everyone!

Version 2.0.0

Change Log

[2.0.0] - 2022-04-19

This release contains performance related improvements, adds features for better Percona MySQL integration, tighter integration with TerarkDB along with the usual bug fixes. ZenFS is now also supporting RocksDB 7.0 which introduced a plugin-api breaking change.

NOTE: A new on-disk format requires manual data migration. See the section "ZenFS on-disk file formats" in README.md for details.
NOTE: ZenFS now requires C++17 to compile.

Highlights:

• Drastically improved max write latencies.
  A rework of the allocator and the introduction of a new on-disk file format now ensures that WAL writes will not suffer from zone allocation latency. Max ZenFS WAL write latency is now dropped from 800+ ms to < 20ms.

• Synced writes performance improvements
  With the introduction of a file format for WAL files that allows in-lined metadata, synced writes throughput has doubled for writes of less than 4k.

• Reduced memory footprint
  With improvments done to the allocator code, write buffer size requirements can now be relaxed from 2GB to the default 64MB without a performance penalty.

• Hardlinks support
  ZenFS now supports hard links, enabling Myrocks/RocksDB snapshots to be done without any space overhead. This also improves bulk loading memory usage and peformance.

• Reduced zenfs utilty binary size
  The zenfs utility is now per default stripped of its debug symbols, reducing the size from 200MB+ to 6MB

• Added support for TerarkDB
  Inclusion of metrics for ZenFS performance introspection. A new API enables collaborative garbage collection, solving space amplification issues in TerarkDB when separating keys and values.

• Various bug fixes, including more robust internal path handling

Tested With

• percona-server-8.0.28-19
• rocksdb-7.0.2
• libzbd-2.0.2

Features/Additions

• General storage driver features
  • e8b0f9a fs: Add GetZenFSChildrenNoLock
  • a66adfb fs: Add IsDirectoryNoLock
  • 6deb5d8 fs: Add AreFilesSame() api support
  • b63b373 fs: Add NumFileLinks() api support
  • ec1519c fs: Add LinkFile api support to ZenFS.
  • b4bd4f5 fs: make zenfs compatible with latest plugin registration API
  • 36ae6d9 fs: add extent migration API for better collaborative GC
  • d48b3fe fs: increase zone extent size to 64 bits
  • 5eb3d4a fs: implement file recovery based on active extent information
  • f3eb10f fs: store active extent start
  • 7c25c8b fs: reset unused io zones on file deletes
  • 7383605 fs: Introduce token API for active/open zone limits
  • 3c0ec5c fs: add supporting for ZoneSnapshot::UsedCapacity().
  • 9be61ea fs:Add support for getting ZBD status.
• Utility
  • 9b3fef6 fs, util: Introduce rmdir utility command
  • 60628ba util: add rename command to zenfs utility
  • b9dbcde util: add delete command to zenfs utility
  • 8467b45 util: add link command to zenfs utility
  • 0e4b829 util: create directories if missing for backup/restore
  • 32c9ddf util: add a simple command description
• Build
  • 1de221b build: Use pkg-config to specify dependencies
  • c699570 util: Check that pkg-config did not fail
• CI
  • f66248d ci: Bump RocksDB and percona-server versions for c++17 support
  • 0c9d24f ci: Checkout zbdbench from github
  • f824ddb ci: Add nightly build against 6.29, 7.0 branches
  • fcb252c ci: Run utils test on nightly master test execution
  • 2c6ca96 ci: Print compile output to stdout during smoke test build
  • b8c5166 ci: Run smoke test with debug enabled on PR
  • 012b67a ci: Add github action to run MTR tests on a weekly schedule
  • 7a7de38 ci: Add short performance test on XFS
  • 1517dc9 ci: Add long performance test on XFS
  • f3e8a9b ci: Add matrix build for zenfs long_performance
  • 302ad34 ci: Add disk-select utility to test execution
  • 154475b ci: Capture make output to logs
  • d638f68 ci: Run tests/long_performance on weekly schedule
  • 70dc5cf ci: Run tests/quick_performance on daily schedule
  • 66d15b7 ci: Add zenfs utils test
  • 13f9a47 ci: Add nightly build against rocksdb master
• Misc
  • aa8d6dd metrics: add zone append trace
  • 1b8e0c8 README: Add link to RocksDB repository
  • 211939f README: add section about on-disk-file-formats
  • 79ad8f9 metrics: add more metrics explanition comments & refine metrics
  • ab0f426 log: enable logger by default and add garbage stats log
  • a6a619e zenfs: add WITH_TERARKDB macro and refine zenfs.cc plugin include path

Fixes/Changes

• General storage driver fixes
  • 4a199b0 fs: Fix GetZenFSChildrenNoLock to return relative children paths.
  • 9761656 zbd: Fix partial I/O handling in Append and DirectRead
  • 241b9ba fs: Fix extent blocks compute error for sparse recovery
  • 542c6cd fs: format all input paths into ZenFS lexically
  • 63ec787 fs: Fix RenameFile for directory renames
  • 8d31adb fs: Refactor, rename ZoneFile::fileSize to file_size_
  • 2507514 fs: make RenameFile() thread safe
  • c798d69 fs: Make OpenWritableFile() thread safe
  • 13f9f04 fs: add DeleteFileNoLock()
  • 485fbf1 fs: add SyncFileMetaDataNoLock()
  • aaa3297 fs: Make GetChildren() thread safe
  • 882b73b fs: Rename GetFileInternal() to GetFileNoLock()
  • 0966ca4 fs: list root files correctly in GetChildren
  • 47e6522 io: fix wrong assertion condition
  • 383474a fs: persist metadata for buffered, non-sparse files on DataSync
  • 3bd845b fs: limit sparse format to WAL files
  • f2a2887 fs: disable direct reads for sparse files
  • 9fe3d9a fs: check if a zone file IsDeleted before syncing metadata
  • 66efab6 fs: mark up ZoneFiles as deleted in DeleteFile
  • 167b6f4 fs: Fix a bug where ZonedWritableFile bufferes are not flushed
  • 33ce781 fs: drop empty-file assertion for writable files
  • 4968742 fs: ensure that metadata is persisted before releasing active zone
  • f4bcdc2 fs: improve on-disk-format incompatibility error message
  • e841b5d fs: Align direct and unaligned read requests to blocksize boundary
  • 7c3f4fc fs: Fix ZoneFile:: GetIOType wrong return value
  • 058e125 fs: fix api ReuseWritableFile() as per rocksdb definition
  • a2dc783 fs: ensure ReopenWritableFile() matches rocksdb definition
  • c262f4f fs: Fix issue cause by used_capacity_ being signed
  • a68e2f6 fs: Fix use of uninitialized value in ZonedBlockDevice destructor
  • ba372b1 fs,util: Fix a bug where zone locks are not handled correctly
  • 4e44dfc zbd: FinishCheapestZone shouldn't fail if all non-busy zones are empty or full.
  • 8641057 fs: build failure when compiling rocksdb in debug mode
  • dd926ac fs: Split buffered and non-buffered appends
  • 0874bf1 fs: refactor ZoneFile::Append
  • 99212a1 fs: priortize open resources for WAL to avoid starvation
  • 3a45ace fs: avoid finishes in allocator by relaxing the life time diff requirement
  • cca35a9 fs: avoid zone threshold finishes when doing WAL allocations
  • a268225 fs: In AllocateIOZone, do not zero output argument
  • 36130b8 fs: Remove io_zones_mtx
  • 402f01c fs: Extract AllocateEmptyZone from AllocateIOZone
  • 5fdeb15 fs: Extract GetBestOpenZoneMatch from AllocateIOZone
  • 614f255 fs: Extract FinishCheapestIOZone from AllocateIOZone
  • 52a156e fs: Extract ApplyFinishTreshold from AllocateIOZone
  • 45ada8a fs: Propagate error on reset failure in ResetUnusedIOZones
  • 09c846c fs: Extract ResetUnusedIOZones from AllocateIOZone
  • bf0a69c fs: Refactor name of AllocateZone to AllocateIOZone
  • e1d388b fs: In ZoneFile::CloseActiveZone(), release zone before returning
  • 830e435 fs: Remove Zone::CloseWR
• Utility
  • 08466d1 util: refactor zenfs_tool_restore()
  • 224eb4a util: Ensure restore command errors out if path is not specified
  • 17eed1f util: Fix --version subcommand to display ZENFS_VERSION of version.h
  • 7fed50d util: improve zenfs path handling for list, backup and restore
  • 5a3b6db util: remove spurious prints during backup
  • 253eff4 utils: strip debug symbols from zenfs utility binary
• CI
  • 2ae5f89 ci: Fix percona-server version for MTR test case
  • 1334699 test: Workaround bug in dbstress in rocksdb 6.29.3
  • eb53908 ci: Bump percona-server version to 8.0.26-17
  • dbafdd4 ci: Continue with the remaining MTR tests on failure.
  • dacec24 ci: Split individual MTR test scenarios in seperate run commands.
  • b6eda97 ci: Bump rocksdb to v6.29.3
  • d81936d ci: Move to podman container manager
  • b85c670 ci: fix: Use disk-select to allocate disks for mtr tests
  • 799b8c6 ci: Fix the percona-server setup for the MTR Test action.
  • d22067d ci: Change RocksDB dev branch from master to main
  • cb61970 ci: Bump linter action to v4.4.1
  • a59c498 ci: Fix pulling ZenFS for zbdbench CI action.
• Misc
  • f0a172d metrics: fix typo of the metrics in AllocateIOZone
  • 9ccd356 metrics: refactor metrics, reduce code complexity

Contributors (Alphabetical)

Andreas Hindborg
Aravind Ramesh
Dennis Maisenbacher
Hans Holmberg
Jorgen S. Hansen
Kuankuan Guo
Liu Ruicheng
Roy Guo
hml1006

Thanks everyone!

Version 1.0.2

Change Log

[1.0.2] - 2022-03-01

This release is a bugfix release on the 1.0.x branch.

Tested with:

• percona-server-8.0.26-16
• rocksdb-6.25.3
• libzbd-2.0.2

Changed

• 398f7f4 util: create directories if missing for backup/restore
• df8c264 util: improve zenfs path handling for list, backup and restore
• 3d1844e util: remove spurious prints during backup
• 50605c3 fs: list root files correctly in GetChildren

Contributors for This Release

Hans Holmberg

Thanks everyone!

Version 1.0.1

Change Log

[1.0.1] - 2022-01-07

This release is a bugfix release on the 1.0.x branch.

Tested with:

• percona-server-8.0.26-16
• rocksdb-6.25.3
• libzbd-2.0.2

Changed

• d7e468d fs: Fix use of uninitialized value in ZonedBlockDevice destructor
• 52b9c51 fs,util: Fix a bug where zone locks are not handled correctly

Contributors for This Release

Andreas Hindborg
Mohit Joshi
Yura Sorokin

Thanks everyone!

Version 1.0.0

Change Log

[1.0.0] - 2021-12-15

This release comes with a number of fixes to the locking scheme that increase
stability of ZenFS under heavy load. We also include a number of commits that
allow Zenfs to build under TerakDB in addition to MySQL/Percona Server.

Tested with:

• percona-server-8.0.26-16
• rocksdb-6.25.3
• libzbd-2.0.1

Added

• Storage driver features and enhancements
  • ed07d77 fs: add modification time for files.
  • 1bc2a11 fs: enable zenfs to handle zonesize more than 4GB
  • e55f277 util: enable backup single file or directory
  • 93506a6 fs: check the device scheduler for mq-deadline
  • 7cad1d7 fs: Add ZenFSMetrics to ZenFS
  • fee6ebe fs: Add snapshot API
  • 99cb288 fs, util: Introducing semi automatic versioning for ZenFS.
• Utility features and enhancements
  • 2bcc876 util: enhance dump tool
  • cb7ff4a util: add metadata analysis tool
  • 7e4d482 util: add dump metadata command
  • d983ad4 util: allow backups of already mounted filesystems for testing purposes
• Test
  • 7b2d60f test: add zenfs utility related test cases to test suite.
  • 95975aa tests: add convenience script for zenfs performance testing
  • 30444c9 test: introduce crashtesting
  • 7ed549d Allow running tests from any directory
  • e91137b Allow configuration of paths in test script
  • ab42302 tests: print and store time spent for tests and test sets
• CI
  • 5c75717 ci: add clang-format check
  • 1dbc7aa ci: add zenfs directory as part of lint check.
  • d318f03 ci: clang-formatted zenfs utility code
  • 2a5482b Add github action to run smoke tests
  • 794dcb7 ci: Remove docker images after run
  • f93bd5c ci: Always remove docker images
  • 79244a7 ci: Always collect results
  • 234531e ci: zbdbench workflow
• Documentation
  • e555e81 docs: add corporate code of conduct Add official Western Digital code of conduct governing project.
  • f0171d2 docs: add manual page for zenfs utility
  • d6b38af docs: Update README with minimum rocksdb version required for ZenFS.
  • f65ffb9 docs: Add community section to README.md

Changed

• General storage driver fixes
  • fed67e8 fs: check the correct fd for direct reads
  • 225215f fs: use read fd when reading zone report from ZNS device
  • 685e206 fs: change variable addr_space_sz type to uint64_t from size_t
  • 98ea024 fs: fixed problem with ZenFS::GetChildren() returning error for empty directories
  • df612d0 fs: minor fix, remove circular dependency
  • fba91dc fs: ensure ZoneFile object is reference counted
  • 0680965 fs: Do not silently ignore zone close errors
  • 24077dd fs: Add error flag to report zone close errors
  • f95fa5b fs: Make error messages unique for open device errors
  • 86ab5cb fs: Refactor ZoneFile for code reuse
  • 71a5ec1 fs: Correct impl of ZenFS::DeleteFile when target does not exist
  • 999b22f fs: Fix a bug in ZenFS::RenameFile()
• Locking improvements for storage driver
  • ba81b69 fs: only acquire lock once on deletion
  • 7d0be5e fs: consistent view when zone full
  • 5dd349c fs: Prevent data race when closing writable zone
  • 146a542 fs: Handle faulty Acquire and Release calles on zones.
  • fafe584 fs: Add missing lock as required by std::condition_variable
  • d65cd8a fs: Remove unused lock operation
  • 759c555 fs: Move locking operation closer to protected area
  • cb8db57 fs: Make ZonedBlockDevice::zone_resources_mtx_ private
  • a45dd1a fs: Replace explicit lock/unlock ops with scoped locks
• Utility fixes
  • 5d037e1 util: fix zenfs list sub command to handle improper path argument
  • 210698d util: fixed memory leaks in zenfs utility
  • 725e0c4 util: zenfs mkfs does not create the aux-dir immediately
• General usability
  • 62a248d util: remove user message
  • f1315e6 util: fixed command line help string
  • c0c4b4e fs: Improve error messages in ZoneFile::Append
• Documentation
  • 270a467 Fix instruction errors in README
  • 5ca1df7 fs: Update docstring for ZonedBlockDevice::zone_resources_mtx_
• Build system
  • d0d661a Fixed problem with hardcoded CXX compiler in the zenfs utility Makefile
  • 9c5e830 Fix clang-12 compilation issues
  • d12914d Fix gcc-11 compilation issues
  • a517230 Add make dependency files to gitignore.
  • e473e50 util: Allow extra compiler/linker flags when building zenfs utility

Contributors

Alan Tse
Alex Chi
Andreas Hindborg
Aravind Ramesh
BaronStack
Bo Jiang
chenchanglong
Dennis Maisenbacher
Hans Holmberg
Kuankuan Guo
Liu Ruicheng
Matias Bjørling
Przemyslaw Skibinski
Yura Sorokin

Thanks everyone!

Version 0.2.0

This release adds bug fixes, some improvements and a testing framework for module-level tests. It is an intermediate release as we move towards production-readiness.