Skip to content

Commit

Permalink
Squashed 'src/c-blosc2/' changes from f344bb7c3..f4d595baa
Browse files Browse the repository at this point in the history
8fbdb4310 Getting ready for release 2.10.2
1c5441f46 Fix redefinition of PATH_MAX macro with mingw
d1908567b Fix CMake Install: Find*.cmake Modules
d0594f5ed CMake: Update FindZLIB_NG For Target
5a7adf4be CMake: Fix installing cmake files in mingw
42ec0a71e Be more explicit on errors when loading dynamic plugin symbols
2bd8eddd1 Post 2.10.1 release actions done
3b4827b85 Fix some compiler warnings
029b6404a Getting ready for release 2.10.1
d24f7e923 Fix new typos found by codespell
d510951c4 Inline Comments & Cleaning
2eec1cdde CMake: Install Blosc2Config.cmake
002509336 Do not return error when file does not exist
e11cb2d21 Post 2.10.0 release actions done
7c0b97dd2 Getting ready for release 2.10.0
90975f887 Merge branch 'froody-bytedelta'
4966a176a Fix test and buggy filter renamed to bytedelta_buggy
e5ff89ecc Use another filter ID for fixed bytedelta
f14aa157e Better check for zstd presence
1a26298a1 Use srand and rand for Win compat
88c46fd90 Fix some issues on win and scenarios where zstd is not there
49b5dbd94 Fix some warnings
d62e35c2d Correctly cycle buffers
9dc3b01e4 Test for filters
b34c1d276 Fix: CMake External Zlib
4b4097723 blosc2.h: Avoid C++20 Designated Init
bcc861040 CMake: Quote $ENV{CMAKE_OSX_ARCHITECTURES}
b62c8d09d Better check for zstd presence
d4d99a261 Use srand and rand for Win compat
da9d9de46 Fix some issues on win and scenarios where zstd is not there
f3ad130ba Fix some warnings
9da341204 Correctly cycle buffers
0e3751790 Test for filters
7e59a4559 Fix: CMake External Zlib
26cd3cbdd blosc2.h: Avoid C++20 Designated Init
09a3bccbf CMake: Quote $ENV{CMAKE_OSX_ARCHITECTURES}
7614d284e Fix discrepancy between SIMD/Non-SIMD versions of bytedelta
d1aa36719 Better test for bytedelta
bcc3825d2 Rename Btune Balance to Tradeoff
0787e1b26 Assorted minor documentation fixes
37ea3288e Post 2.9.3 release actions done
48e7cdf4a Getting ready for release 2.9.3
d55bfcd68 Fix #522
425e8a9a5 Fix for #519 #520 #521
144ed29ae Move images
c2d499595 Add new images for blosc2 pipeline
86fc0f66a Check chunksize <= MAX_BUFFERSIZE
5486f5ca4 Replace the superseded cmake option COMPILE_FLAGS with COMPILE_OPTIONS
08e4e1a4e Fix windows
092e30d33 move load_lib to blosc-private
fd6f6ef76 Improvements on README for plugins
804640269 Add a section on citing Blosc work
2f52ee807 Fix. Malloc array with variable size
fd6df7577 Declare strindes length acording to ndim
876316bcf Make test_frame_get_offsets not fail when different comp params are used
190292cbc Do not reduce maxout if intr_codec
2fffef723 Do not use dict when btune active
7ee73133a unnecessary whitespace before a quoted newline
b2fe10bad Macros with multiple statements should be enclosed in a do - while loop
257cc203a Single statement macros should not use a do {} while (0) loop
c12cd6ea1 `inline` keyword should sit between storage class and type
7b3da0373 `calloc` uses number as first arg, sizeof is generally wrong
2eab185da Revert previous changes to vendored code
1d4a50f6d Add CMAKE_DL_LIBS to CMakeLists.txt to make dlopen available on non-win platforms
57f7d6937 Complete tunes -> tuners transition
298e08589 Fix compiler warning
7e31de3e3 Clean up header file inclusion, sequel of c4f28da
ae71d5e72 Do not use BLOSC_EXPORT in source files
22f69e16a Macros starting with if should be enclosed by a do - while loop to avoid possible if/else logic defects
02a39a60b Macros with multiple statements should be enclosed in a do - while loop
c4c2d910e Macros should not use a trailing semicolon
3362954d9 `inline` keyword should sit between storage class and type
c8ebc2472 Fix incorrect nested ifdef's from a3e56ee
a3e56eeca Clean up existing header guards, add missing ones
4450f1549 Forgot to pclose() in all cases in ca60217
0420bd4f6 Clean up use of 'extern "C"' in header files
868b53ec5 Add a BLOSC_TRACE_INFO
53e83654a Post 2.9.2 release actions done

git-subtree-dir: src/c-blosc2
git-subtree-split: f4d595baa9b573818412143862a67f8ecfdffe84
  • Loading branch information
t20100 committed Sep 4, 2023
1 parent a9befe7 commit 616c34e
Show file tree
Hide file tree
Showing 123 changed files with 2,311 additions and 1,041 deletions.
9 changes: 4 additions & 5 deletions ANNOUNCE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
# Announcing C-Blosc2 2.9.2
# Announcing C-Blosc2 2.10.2
A fast, compressed and persistent binary data store library for C.

## What is new?

This a maintenance release with improved support for dynamic plugins and
fixes for some corner cases when handling incompressible data. Also,
many other small fixes and improvements have been included. An upgrade to
this release is recommended.
This is a maintenance release with also several improvements for helping
integration of C-Blosc2 in other projects (thanks to Alex Huebl). Also,
some fixes for MinGW platform are in (thanks to Biswapriyo Nath).

For more info, please see the release notes in:

Expand Down
84 changes: 84 additions & 0 deletions Blosc2Config.cmake.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# only add PUBLIC dependencies as well
# https://cmake.org/cmake/help/latest/manual/cmake-packages.7.html#creating-a-package-configuration-file
include(CMakeFindDependencyMacro)

# Search in <PackageName>_ROOT:
# https://cmake.org/cmake/help/v3.12/policy/CMP0074.html
if(POLICY CMP0074)
cmake_policy(SET CMP0074 NEW)
endif()

# locate the installed FindABC.cmake modules
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}/Modules")

# this section stores which configuration options were set
set(HAVE_THREADS @Threads_FOUND@)
set(HAVE_IPP @HAVE_IPP@)
set(HAVE_ZLIB_NG @HAVE_ZLIB_NG@)
set(DEACTIVATE_IPP @DEACTIVATE_IPP@)
set(DEACTIVATE_ZLIB @DEACTIVATE_ZLIB@)
set(DEACTIVATE_ZSTD @DEACTIVATE_ZSTD@)
set(PREFER_EXTERNAL_LZ4 @PREFER_EXTERNAL_LZ4@)
set(PREFER_EXTERNAL_ZLIB @PREFER_EXTERNAL_ZLIB@)
set(PREFER_EXTERNAL_ZSTD @PREFER_EXTERNAL_ZSTD@)

# find dependencies and their targets, which are used in our Blosc2Targets.cmake
# additionally, the Blosc2_..._FOUND variables are used to support
# find_package(Blosc2 ... COMPONENTS ... ...)
# this enables downstream projects to express the need for specific features.
if(WIN32)
if(HAVE_THREADS)
find_dependency(Threads)
set(Blosc2_THREADS_FOUND TRUE)
else()
set(Blosc2_THREADS_FOUND FALSE)
endif()
else()
find_dependency(Threads)
set(Blosc2_THREADS_FOUND TRUE)
endif()

if(NOT DEACTIVATE_IPP AND HAVE_IPP)
find_dependency(IPP)
set(Blosc2_IPP_FOUND FALSE)
else()
set(Blosc2_IPP_FOUND TRUE)
endif()

if(PREFER_EXTERNAL_LZ4)
find_dependency(LZ4)
endif()
set(Blosc2_LZ4_FOUND TRUE)

if(DEACTIVATE_ZLIB)
set(Blosc2_ZLIB_FOUND FALSE)
elseif(NOT DEACTIVATE_ZLIB AND PREFER_EXTERNAL_ZLIB)
if(HAVE_ZLIB_NG)
find_dependency(ZLIB_NG)
else()
find_dependency(ZLIB)
endif()
set(Blosc2_ZLIB_FOUND TRUE)
endif()

if(DEACTIVATE_ZSTD)
set(Blosc2_ZSTD_FOUND FALSE)
elseif(NOT PREFER_EXTERNAL_ZSTD AND PREFER_EXTERNAL_ZSTD)
find_dependency(ZSTD)
set(Blosc2_ZSTD_FOUND TRUE)
endif()

# define central Blosc2::blosc2_shared/static targets
include("${CMAKE_CURRENT_LIST_DIR}/Blosc2Targets.cmake")

# check if components are fulfilled and set Blosc2_<COMPONENT>_FOUND vars
# Blosc2_FIND_COMPONENTS is a list set by find_package(... COMPONENTS ... ...)
# likewise Blosc2_FIND_REQUIRED_... per component specified
foreach(comp ${Blosc2_FIND_COMPONENTS})
if(NOT Blosc2_${comp}_FOUND)
if(Blosc2_FIND_REQUIRED_${comp})
set(Blosc2_FOUND FALSE)
endif()
endif()
endforeach()

139 changes: 121 additions & 18 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ endif()

# Propagate CMAKE_OSX_ARCHITECTURES env variable into CMAKE_SYSTEM_PROCESSOR
if(DEFINED ENV{CMAKE_OSX_ARCHITECTURES})
if($ENV{CMAKE_OSX_ARCHITECTURES} STREQUAL "arm64")
if("$ENV{CMAKE_OSX_ARCHITECTURES}" STREQUAL "arm64")
set(CMAKE_SYSTEM_PROCESSOR arm64)
endif()
endif()
Expand Down Expand Up @@ -385,23 +385,6 @@ if(NOT DEFINED BLOSC_INSTALL)
endif()
endif()

# uninstall target
if(BLOSC_INSTALL)
include(GNUInstallDirs)
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/blosc2.pc.in"
"${CMAKE_CURRENT_BINARY_DIR}/blosc2.pc"
@ONLY)
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/blosc2.pc"
DESTINATION "${CMAKE_INSTALL_LIBDIR}/pkgconfig" COMPONENT DEV)
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/cmake_uninstall.cmake.in"
"${CMAKE_CURRENT_BINARY_DIR}/cmake_uninstall.cmake"
@ONLY)
add_custom_target(uninstall
COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_BINARY_DIR}/cmake_uninstall.cmake)
endif()

# include directories
include_directories(include)
if(BUILD_PLUGINS)
Expand Down Expand Up @@ -440,6 +423,126 @@ if(BUILD_EXAMPLES)
add_subdirectory(examples)
endif()

# collecting SOURCES is now complete
if(BUILD_SHARED)
target_sources(blosc2_shared PRIVATE ${SOURCES})
endif()
if(BUILD_STATIC)
target_sources(blosc2_static PRIVATE ${SOURCES})
endif()
if(BUILD_TESTS)
target_sources(blosc_testing PRIVATE ${SOURCES})
endif()

# install targets
if(BLOSC_INSTALL)
include(GNUInstallDirs)

# C++ files
install(FILES ${PROJECT_SOURCE_DIR}/include/blosc2.h DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} COMPONENT DEV)
install(FILES ${PROJECT_SOURCE_DIR}/include/b2nd.h DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} COMPONENT DEV)
install(FILES
${PROJECT_SOURCE_DIR}/include/blosc2/blosc2-export.h
${PROJECT_SOURCE_DIR}/include/blosc2/blosc2-common.h
${PROJECT_SOURCE_DIR}/include/blosc2/blosc2-stdio.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/blosc2 COMPONENT DEV)
if(BUILD_PLUGINS)
install(FILES
${PROJECT_SOURCE_DIR}/include/blosc2/filters-registry.h
${PROJECT_SOURCE_DIR}/include/blosc2/codecs-registry.h
${PROJECT_SOURCE_DIR}/include/blosc2/tuners-registry.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/blosc2 COMPONENT DEV)
endif()

if(BUILD_SHARED)
install(TARGETS blosc2_shared
LIBRARY COMPONENT LIB
ARCHIVE COMPONENT DEV
RUNTIME COMPONENT LIB)
endif()
if(BUILD_STATIC)
install(TARGETS blosc2_static COMPONENT DEV)
endif()

# config files
include(CMakePackageConfigHelpers)

# we need a general location for Unix and Windows to install our
# Blosc2Config.cmake files to. This is defined in CMake:
# https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-search-procedure
if(NOT Blosc2_INSTALL_CMAKEDIR)
if(CMAKE_INSTALL_CMAKEDIR)
set(Blosc2_INSTALL_CMAKEDIR "${CMAKE_INSTALL_CMAKEDIR}/Blosc2")
else()
if(WIN32 AND NOT MINGW)
set(Blosc2_INSTALL_CMAKEDIR "cmake")
else()
set(Blosc2_INSTALL_CMAKEDIR "${CMAKE_INSTALL_LIBDIR}/cmake/Blosc2")
endif()
endif()
endif()

# CMake config file
# This stores our targets and find and populates the targets we depend on,
# including third party interface libraries that we added.
set(Blosc2_INSTALL_TARGET_NAMES)
if(BUILD_SHARED)
list(APPEND Blosc2_INSTALL_TARGET_NAMES blosc2_shared)
endif()
if(BUILD_STATIC)
list(APPEND Blosc2_INSTALL_TARGET_NAMES blosc2_static)
endif()
configure_file(
${PROJECT_SOURCE_DIR}/Blosc2Config.cmake.in
${CMAKE_CURRENT_BINARY_DIR}/Blosc2Config.cmake
@ONLY
)
install(TARGETS ${Blosc2_INSTALL_TARGET_NAMES}
EXPORT Blosc2Targets
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
)
install(EXPORT Blosc2Targets
FILE Blosc2Targets.cmake
NAMESPACE Blosc2::
DESTINATION ${Blosc2_INSTALL_CMAKEDIR}
)
write_basic_package_version_file("Blosc2ConfigVersion.cmake"
VERSION ${BLOSC2_VERSION_STRING}
COMPATIBILITY SameMajorVersion
)
install(
FILES
${CMAKE_CURRENT_BINARY_DIR}/Blosc2Config.cmake
${CMAKE_CURRENT_BINARY_DIR}/Blosc2ConfigVersion.cmake
DESTINATION ${Blosc2_INSTALL_CMAKEDIR}
)

# CMake Find*.cmake files used in Blosc2Config.cmake
install(
DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/cmake/
DESTINATION ${Blosc2_INSTALL_CMAKEDIR}/Modules
)

# pkg-config .pc file
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/blosc2.pc.in"
"${CMAKE_CURRENT_BINARY_DIR}/blosc2.pc"
@ONLY)
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/blosc2.pc"
DESTINATION "${CMAKE_INSTALL_LIBDIR}/pkgconfig" COMPONENT DEV)

# uninstaller
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/cmake_uninstall.cmake.in"
"${CMAKE_CURRENT_BINARY_DIR}/cmake_uninstall.cmake"
@ONLY)
add_custom_target(uninstall
COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_BINARY_DIR}/cmake_uninstall.cmake)
endif()


# packaging
if(NOT BLOSC_IS_SUBPROJECT)
Expand Down
17 changes: 16 additions & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ New features in C-Blosc2

* **More filters:** besides `shuffle` and `bitshuffle` already present in C-Blosc1, C-Blosc2 already implements:

- `bytedelta`: calculates the difference between bytes in a block that has been shuffle already. We have `blogged about bytedelta <https://www.blosc.org/posts/bytedelta-enhance-compression-toolset/>`_.
- `bytedelta`: calculates the difference between bytes in a block that has been shuffled already. We have `blogged about bytedelta <https://www.blosc.org/posts/bytedelta-enhance-compression-toolset/>`_.

- `delta`: the stored blocks inside a chunk are diff'ed with respect to first block in the chunk. The idea is that, in some situations, the diff will have more zeros than the original data, leading to better compression.

Expand Down Expand Up @@ -227,6 +227,21 @@ Tweeter feed
Follow `@Blosc2 <https://twitter.com/Blosc2>`_ so as to get informed about the latest developments.


Citing Blosc
============

You can cite our work on the different libraries under the Blosc umbrella as:

.. code-block:: console
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Acknowledgments
===============

Expand Down
6 changes: 3 additions & 3 deletions README_CHUNK_FORMAT.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ A regular chunk is composed of a header and a blocks section::

Also, there are the so-called lazy chunks that do not have the actual compressed data,
but only metainformation about how to read it. Lazy chunks typically appear when reading
data from persistent media. A lazy chunk has the header and bstarts sections in place
and in addition, they have an additional trailer for allowing to read the data blocks::
data from persistent media. A lazy chunk has header and bstarts sections in place and
in addition, an additional trailer for allowing to read the data blocks::

+---------+---------+---------+
| header | bstarts | trailer |
Expand Down Expand Up @@ -184,7 +184,7 @@ compression, and finally a list of compressed data streams::
| bstarts | dict | streams |
+=========+======+=========+

Each block is equal-sized as specified by the `blocksize` header field. The size of the last block that can be shorter
Each block is equal-sized as specified by the `blocksize` header field. The size of the last block can be shorter
or equal to the rest.

**Block starts**
Expand Down
57 changes: 57 additions & 0 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,63 @@
Release notes for C-Blosc2
==========================

Changes from 2.10.1 to 2.10.2
=============================

* Several fixes for the CMake system. Thanks to Axel Huebl. See PR #541 and #542.

* Several fixes for mingw plaform. Thanks to Biswapriyo Nath. See PR #540 and #543.


Changes from 2.10.0 to 2.10.1
=============================

* `blosc2_remove_urlpath(const char *urlpath)` does not return an error
when path does not exist.

* Changes in CMake installer to conserve targets and properties
on install, so CMake users do not need to write `FindBlosc2.cmake`
files anymore. This also helps to preserve transitive dependencies on
CMake targets, especially useful for fully static builds, e.g., for
Python wheels. Thanks to @ax3l (Axel Huebl). See PR #537.

* Fix new typos. Thanks to @DimitriPapadopoulos. See PR #538.


Changes from 2.9.3 to 2.10.0
============================

* bytedelta filter has been fixed. For backward compatibility, the old
bytedelta filter is still available as `BLOSC_FILTER_BYTEDELTA_BUGGY`
symbol, with the same ID (34) than before. The new, fixed bytedelta
filter has received a new ID (35) and it can be used via the usual
`BLOSC_FILTER_BYTEDELTA` symbol. That means that old data written with
the buggy bytedelta filter should be decompressed without issues.
Thanks to @foody (Tom Birch) for the fix. See #531, #532 for more info.

* Filter buffers are correctly cycled now. Now it is possible to use e.g.
shuffle and bitshuffle filters in the pipeline. Thanks to @foody (Tom Birch)
for the fix. See #528 and PR #530.

* Assorted fixes for allowing better inclusion in external projects.
Thanks to @ax3l (Axel Huebel). See #525, #527 and #529.

* Minor fixes in the documentation. Thanks to @ivilata (Ivan Vilata).
See #523.


Changes from 2.9.2 to 2.9.3
===========================

* Thanks to Dimitri Papadopoulos for an extensive set of improvements in
documentation and code.

* `load_lib` is now a private function. Before was public, but
never meant to be.

* Several fixes for bugs discovered by the fuzzer.


Changes from 2.9.1 to 2.9.2
===========================

Expand Down
3 changes: 1 addition & 2 deletions RELEASING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,7 @@ Post-release actions
version to the next minor one (i.e. X.Y.Z --> X.Y.(Z+1).dev).

- Create new headers for adding new features in ``RELEASE_NOTES.md``
and empty the release-specific information in ``ANNOUNCE.md`` and
add this place-holder instead:
and add this place-holder instead:

#XXX version-specific blurb XXX#

Expand Down
4 changes: 3 additions & 1 deletion THANKS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,6 @@ Thanks

* Nathan Moinvaziri for his outstanding work on the security side of the things via `fuzzer testing <https://google.github.io/oss-fuzz/>`_.

* Marta Iborra for her implementation of sparse storage for persistent super-chunks.
* Marta Iborra for her implementation of sparse storage for persistent super-chunks and her attention to detail in many other aspects of the library.

* Dimitri Papadopoulos for an extensive set of improvements in documentation and code.
Loading

0 comments on commit 616c34e

Please sign in to comment.