IDIOMS Update & BULKI v0.1 (#203)

* fix cmake mercury_util not found issue * update for Julia support * fix hdf5.h not found for src/tools * update container config * add libhdf5-dev for Github Actions * update CMake for HDF5 in tools * update logic for finding HDF5 * update * remove use system hdf5 * delete useless find library * update findHDF5 * Feature/dart (#11) Update to avoid fixing compilation issue on src/tools (due to : HDF5 cannot be found) * Use cc on Perlmutter (#161) Dr. Tang fixed a compilation issue in NERSC CI where HDF5 cannot be detected even if the cray-parallel-hdf5 module is loaded on Perlmultter. * update with fixes on tools and llsm example * add gitignore for llsm * update gitignore * Feature/dart (#12) * fix formatting * update clangformat10 * update base dockerfile * Add clang-format10 to docker container. Also fixed clang-format. Add clang-format10 to docker container. Also fixed clang-format. * Fix pdc ls (#154) * pdc import, export, ls compiled successfully * removed requested files * formatting issues * changed install tools * gets checkpoint files * grabbing checkpoint files from within sub-directories, minor comments * Committing clang-format changes * Committing clang-format changes * Fix a few issues with pdc_ls * Committing clang-format changes --------- Co-authored-by: nickaruwang <[email protected]> Co-authored-by: Nick Wang <[email protected]> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <[email protected]> * update documentation * update document update document * sync branch sync branch * no UUID module is required * update document and make UUID an optional package * update docker repo name * updating docker repo name and make UUID optional * Complete support for Docker and Github Codespace (#157) Include support for Docker and Github Codespace so we can run our dev environment with the support of Docker. * SQLite and RocksDB support for KVtags (#165) SQLite and RocksDB support for KVtags * fix round for tag delete * update test * bulki update * BULKI base type worked * BULKI all tests done * new index code * update * update new test * update csv bench * update * update script * adding python scripts for generating large metadata set for LLSM application * update json schema * better json validator * update importer * update code for non-MPI compatibility * update llsm converter * update LLSM data converter * split files * update .gitignore * update * add timing info * update * update tag size * detect object creation failure * update * update object name with date * update for robustness * update * update JMD_DEBUG option * update output for overall output * update inttypes.h * update * update extractor * update inttypes.h * update converter * update importer information * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <[email protected]> Co-authored-by: Jean Luca Bez <[email protected]> * fix issue * fixed search issues * update for infix * update * index persistence still needs improvement * update * enable cache by default (#187) * Removing PDC macro (#189) * Removing gres option for ctest * Removing gres option from scripts * Update check for core * Remove PDC macro * Committing clang-format changes --------- Co-authored-by: Hyunju Oh <[email protected]> Co-authored-by: Jean Luca Bez <[email protected]> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * update * range query done' * range query local test passed * multi-condition in progress * clean up code * add comments * new benchmark * update * update range query test * update cmake: * update * update * update * update * someta range query * someta range query * someta range query * fix value serialization * update * update double free * update * update * update * fixed pointer issue * rb_tree delete fixed, now need to check index persistence * update * update * update * update * update * update * update * update * update * update * update * update * update * BDCATS fix (#193) * Fix issues with bdcats_batch * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * update * clean up code * update test sh * IDIOMS persistence DONE * update * remove old kvtag benchmarks * update * update * update changes * dart info * update * multi data type for the same key, supported now * Monitoring changes from feature/dart to develop (#18) Major changes: * IDIOMS -> affix-based query benchmark * IDIOMS -> Simulation Test * IDIOMS -> Multi data type supported for the same key * IDIOMS -> Range Query and Exact Query for Numeric Values * IDIOMS -> benchmark for numeric values (exact search and range query) * IDIOMS -> Index Persistence * BULKI -> A data serialization and deserialization mechanism. * fix CMakeLists.txt * update * update format * update BULKI interface order * BULKI API sorted * add idioms ci test * Feature/dart (#20) 1. add documentation about BULKI and IDIOMS query conditions 2. add ci test for IDIOMS 3. optimized BULKI to save space on its metadata fields. * Feature/dart (#22) update version * update * update * update * remove unnecessary .bin file * update * update --------- Co-authored-by: Houjun Tang <[email protected]> Co-authored-by: nickaruwang <[email protected]> Co-authored-by: Nick Wang <[email protected]> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <[email protected]> Co-authored-by: Hyunju Oh <[email protected]> Co-authored-by: Hyunju Oh <[email protected]>
hpc-io · Sep 10, 2024 · 2e5de75 · 2e5de75
1 parent 7d9fc1c
commit 2e5de75
Show file tree

Hide file tree

Showing 117 changed files with 18,090 additions and 3,094 deletions.
diff --git a/.docker/dev_base.Dockerfile b/.docker/dev_base.Dockerfile
@@ -4,13 +4,55 @@
 ARG ARCH
 FROM ${ARCH}ubuntu:jammy
 
-RUN echo "ARCH=${ARCH}" && sleep 3
-
 ARG ARCH_CODE
 
-RUN echo "ARCH_CODE=${ARCH_CODE}" && sleep 3
+ENV JULIA_HOME=/opt/julia
+ENV WORK_SPACE=/home/project
+ENV LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric
+ENV LIBFABRIC_DIR=$WORK_SPACE/install/libfabric
+ENV MERCURY_SRC_DIR=$WORK_SPACE/source/mercury
+ENV MERCURY_DIR=$WORK_SPACE/install/mercury
+
+ENV PDC_SRC_DIR=$WORK_SPACE/source/pdc
+ENV PDC_DIR=$WORK_SPACE/install/pdc
+
+ENV LD_LIBRARY_PATH="$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH"
+ENV PATH="$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH"
+ENV LD_LIBRARY_PATH="$MERCURY_DIR/lib:$LD_LIBRARY_PATH"
+ENV PATH="$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH"
+
+
+ENV MERCURY_CMAKE_FLAGS="-DCMAKE_INSTALL_PREFIX=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DNA_USE_OFI=ON -DNA_USE_SM=OFF -DNA_OFI_TESTING_PROTOCOL=tcp "
+ENV CLANG_FORMAT_PATH=$WORK_SPACE/software/clang-format-lint-action/clang-format/clang-format10
+
 # Install necessary tools, MPICH, UUID library and developer files
-RUN apt-get update && apt-get install -y \
+RUN echo "ARCH=${ARCH}" && echo "ARCH_CODE=${ARCH_CODE}" && sleep 3 && mkdir -p $WORK_SPACE && \
+    mkdir -p $WORK_SPACE/software && \
+    mkdir -p $WORK_SPACE/source && \
+    mkdir -p $WORK_SPACE/install && \
+    mkdir -p $LIBFABRIC_SRC_DIR && \
+    mkdir -p $MERCURY_SRC_DIR && \
+    mkdir -p $PDC_SRC_DIR && \
+    mkdir -p $LIBFABRIC_DIR && \
+    mkdir -p $MERCURY_DIR && \
+    mkdir -p $PDC_DIR && \
+    rm -rf $LIBFABRIC_SRC_DIR/* && \
+    rm -rf $MERCURY_SRC_DIR/* && \
+    rm -rf $PDC_SRC_DIR/* && \
+    rm -rf $LIBFABRIC_DIR/* && \
+    rm -rf $MERCURY_DIR/* && \
+    rm -rf $PDC_DIR/* && \
+    echo "export LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric" > $WORK_SPACE/pdc_env.sh && \
+    echo "export LIBFABRIC_DIR=$WORK_SPACE/install/libfabric" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export MERCURY_SRC_DIR=$WORK_SPACE/source/mercury" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export MERCURY_DIR=$WORK_SPACE/install/mercury" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export PDC_SRC_DIR=$WORK_SPACE/source/pdc" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export PDC_DIR=$WORK_SPACE/install/pdc" >> $WORK_SPACE/pdc_env.sh && \
+    echo 'export LD_LIBRARY_PATH=$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh  && \
+    echo 'export PATH=$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh && \
+    echo 'export LD_LIBRARY_PATH=$MERCURY_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh \
+    echo 'export PATH=$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh && \
+    apt-get update && apt-get install -y \
     build-essential \
     git \
     mpich \
@@ -28,107 +70,47 @@ RUN apt-get update && apt-get install -y \
     wget \
     axel \
     curl \
+    bc \
     vim \
     nano \
     gdb \
     cgdb \
     curl \
     valgrind \
-    python3 
-
-# Install Oh My Bash
-RUN bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohmybash/oh-my-bash/master/tools/install.sh)" && \
-    sed -i 's/OSH_THEME="font"/OSH_THEME="powerline-multiline"/g' ~/.bashrc
-
-# Install Julia
-
-RUN echo "https://julialang-s3.julialang.org/bin/linux/aarch64/1.6/julia-1.6.7-linux-aarch64.tar.gz" > /julia_url_arm64v8.txt && \
-    echo "https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.7-linux-x86_64.tar.gz" > /julia_url_amd64.txt
-
-RUN echo $(cat /julia_url_${ARCH_CODE}.txt) && sleep 3
-
-RUN mkdir -p /opt/julia && wget -O - $(cat /julia_url_${ARCH_CODE}.txt) | tar -xz -C /opt/julia --strip-components=1 && \
-    ln -s /opt/julia/bin/julia /usr/local/bin/julia
-
-RUN rm -rf /tmp/julia_url_*.txt
-
-ENV JULIA_HOME=/opt/julia
-
-# Install Rust
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
-RUN echo 'source $HOME/.cargo/env' >> ~/.bashrc
-
-
-# Set WORK_SPACE environment variable and create necessary directories
-ENV WORK_SPACE=/home/project
-RUN mkdir -p $WORK_SPACE
-
-# Install clang-format repo
-RUN mkdir -p $WORK_SPACE/software
-RUN cd $WORK_SPACE/software && git clone https://github.com/DoozyX/clang-format-lint-action.git
-ENV CLANG_FORMAT_PATH=$WORK_SPACE/software/clang-format-lint-action/clang-format/clang-format10
-
-# Clone the repositories
-WORKDIR $WORK_SPACE/source
-RUN git clone https://github.com/ofiwg/libfabric.git && \
-    git clone https://github.com/mercury-hpc/mercury.git --recursive
-
-ENV LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric
-ENV LIBFABRIC_DIR=$WORK_SPACE/install/libfabric
-ENV MERCURY_SRC_DIR=$WORK_SPACE/source/mercury
-ENV MERCURY_DIR=$WORK_SPACE/install/mercury
-
-ENV PDC_SRC_DIR=$WORK_SPACE/source/pdc
-ENV PDC_DIR=$WORK_SPACE/install/pdc
-
-RUN mkdir -p $LIBFABRIC_SRC_DIR && \
-    mkdir -p $MERCURY_SRC_DIR && \
-    mkdir -p $PDC_SRC_DIR && \
-    mkdir -p $LIBFABRIC_DIR && \
-    mkdir -p $MERCURY_DIR && \
-    mkdir -p $PDC_DIR
-
-
-# Save the environment variables to a file
-RUN echo "export LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric" > $WORK_SPACE/pdc_env.sh && \
-    echo "export LIBFABRIC_DIR=$WORK_SPACE/install/libfabric" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export MERCURY_SRC_DIR=$WORK_SPACE/source/mercury" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export MERCURY_DIR=$WORK_SPACE/install/mercury" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export PDC_SRC_DIR=$WORK_SPACE/source/pdc" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export PDC_DIR=$WORK_SPACE/install/pdc" >> $WORK_SPACE/pdc_env.sh
-
-
-# Build and install libfabric
-WORKDIR $LIBFABRIC_SRC_DIR
-RUN git checkout v1.18.0 && \
+    python3 && \
+    cd $WORK_SPACE/software && \
+    bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohmybash/oh-my-bash/master/tools/install.sh)" && \
+    sed -i 's/OSH_THEME="font"/OSH_THEME="powerline-multiline"/g' ~/.bashrc && \
+    echo "https://julialang-s3.julialang.org/bin/linux/aarch64/1.6/julia-1.6.7-linux-aarch64.tar.gz" > $WORK_SPACE/software/julia_url_arm64v8.txt && \
+    echo "https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.7-linux-x86_64.tar.gz" > $WORK_SPACE/software/julia_url_amd64.txt && \
+    echo $(cat $WORK_SPACE/software/julia_url_${ARCH_CODE}.txt) && sleep 3 && \
+    mkdir -p /opt/julia && wget -O - $(cat $WORK_SPACE/software/julia_url_${ARCH_CODE}.txt) | tar -xz -C /opt/julia --strip-components=1 && \
+    ln -s /opt/julia/bin/julia /usr/local/bin/julia && \
+    rm -rf $WORK_SPACE/software/julia_url_*.txt && \
+    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \
+    echo 'source $HOME/.cargo/env' >> ~/.bashrc && \
+    git clone https://github.com/DoozyX/clang-format-lint-action.git && \
+    git clone https://github.com/ofiwg/libfabric.git ${LIBFABRIC_SRC_DIR} && \
+    git clone https://github.com/mercury-hpc/mercury.git --recursive ${MERCURY_SRC_DIR} && \
+    cd $LIBFABRIC_SRC_DIR && \
+    git checkout v1.18.0 && \
     ./autogen.sh && \
     ./configure --prefix=$LIBFABRIC_DIR CC=mpicc CFLAG="-O2" && \
     make clean && \
     make -j 8 && make install && \
-    make check
-
-ENV LD_LIBRARY_PATH="$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH"
-ENV PATH="$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH"
-RUN echo 'export LD_LIBRARY_PATH=$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh  && \
-    echo 'export PATH=$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh
-
-
-# Build and install Mercury
-WORKDIR $MERCURY_SRC_DIR
-ENV MERCURY_CMAKE_FLAGS="-DCMAKE_INSTALL_PREFIX=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DNA_USE_OFI=ON -DNA_USE_SM=OFF -DNA_OFI_TESTING_PROTOCOL=tcp "
-RUN git checkout v2.2.0 \
-    mkdir -p build 
-WORKDIR ${MERCURY_SRC_DIR}/build
-RUN cmake $MERCURY_CMAKE_FLAGS ../ && \
+    make check && \
+    cd $MERCURY_SRC_DIR && \
+    git checkout v2.2.0 && \
+    mkdir -p build && \
+    cd ${MERCURY_SRC_DIR}/build && \
+    cmake $MERCURY_CMAKE_FLAGS ../ && \
     make -j 16 && make install && \
-    ctest
-
-# Set the environment variables
-ENV LD_LIBRARY_PATH="$MERCURY_DIR/lib:$LD_LIBRARY_PATH"
-ENV PATH="$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH"
-RUN echo 'export LD_LIBRARY_PATH=$MERCURY_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh \
-    echo 'export PATH=$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh
-
+    ctest && \
+    apt-get autoremove -y && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/* && \
+    rm -rf /tmp/* && \
+    rm -rf /var/tmp/* 
 
 ENV PDC_CMAKE_FLAGS="-DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DCMAKE_INSTALL_PREFIX=$PDC_DIR -DPDC_ENABLE_MPI=ON -DMERCURY_DIR=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc -DMPI_RUN_CMD=mpiexec "
 

diff --git a/.gitignore b/.gitignore
@@ -19,3 +19,9 @@ build
 
 # ignore docs build dir
 docs/build
+
+# ignore venv
+**/*venv
+
+# ignore .zed
+.zed
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -198,6 +198,9 @@ if(SUPPRESS_IGNORABLE_WARNINGS)
     set(SUPPRESSED_LIST "-Wno-deprecated-non-prototype" ${SUPPRESSED_LIST})
   else()
     set(SUPPRESSED_LIST "-Wno-maybe-uninitialized" ${SUPPRESSED_LIST})
+    set(SUPPRESSED_LIST "-Wno-alloc-size-larger-than" ${SUPPRESSED_LIST})
+    # set(SUPPRESSED_LIST "-Wno-use-after-free" ${SUPPRESSED_LIST})
+    set(SUPPRESSED_LIST "-Wno-stringop-truncation" ${SUPPRESSED_LIST})
   endif()
   set(SUPPRESSED_LIST "-Wno-sign-compare" ${SUPPRESSED_LIST})
   set(SUPPRESSED_LIST "-Wno-format" ${SUPPRESSED_LIST})
@@ -207,6 +210,8 @@ if(SUPPRESS_IGNORABLE_WARNINGS)
   set(SUPPRESSED_LIST "-Wno-unused-function" ${SUPPRESSED_LIST})
   set(SUPPRESSED_LIST "-Wno-unused-result" ${SUPPRESSED_LIST})
   set(SUPPRESSED_LIST "-Wno-unused-but-set-variable" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-inline" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-array-bounds" ${SUPPRESSED_LIST})
 
   if(CMAKE_C_COMPILER_ID MATCHES "GNU|Clang")
     add_compile_options(${SUPPRESSED_LIST})

diff --git a/benchmark/idioms/CMakeLists.txt b/benchmark/idioms/CMakeLists.txt
@@ -0,0 +1,68 @@
+cmake_minimum_required (VERSION 2.8.12)
+
+# Setup cmake policies.
+foreach(p
+    CMP0012
+    CMP0013
+    CMP0014
+    CMP0022 # CMake 2.8.12
+    CMP0025 # CMake 3.0
+    CMP0053 # CMake 3.1
+    CMP0054 # CMake 3.1
+    CMP0074 # CMake 3.12
+    CMP0075 # CMake 3.12
+    CMP0083 # CMake 3.14
+    CMP0093 # CMake 3.15
+  )
+  if(POLICY ${p})
+    cmake_policy(SET ${p} NEW)
+  endif()
+endforeach()
+
+project(METADATA_JSON_LOADER C)
+
+set(JMD_LDR_EXT_INCLUDE_DIRS "")
+set(JMD_LDR_EXT_LIBRARIES "")
+
+set(CMAKE_BUILD_TYPE RelWithDebInfo CACHE STRING "Choose the type of build." FORCE)
+# set(JMD_DEBUG ON CACHE BOOL "Enable debug mode.")
+option(JMD_DEBUG_MODE "Enable debug mode." ON)
+if(JMD_DEBUG_MODE)
+  add_definitions(-DJMD_DEBUG=1)
+endif()
+
+option(USE_SYSTEM_MPI "Use system-installed MPI." ON)
+if(USE_SYSTEM_MPI)
+  find_package(MPI)
+  if(MPI_FOUND)
+    add_definitions(-DJMD_LDR_ENABLE_MPI=1)
+    SET(CMAKE_C_COMPILER ${MPI_C_COMPILER})
+    SET(CMAKE_CXX_COMPILER ${MPI_CXX_COMPILER})
+    set(JMD_LDR_EXT_INCLUDE_DIRS ${MPI_C_INCLUDE_PATH}
+      ${JMD_LDR_EXT_INCLUDE_DIRS}
+      )
+    set(JMD_LDR_EXT_LIBRARIES ${MPI_C_LIBRARIES} ${JMD_LDR_EXT_LIBRARIES})
+  endif()
+endif()
+
+find_package(PDC REQUIRED)
+if(PDC_FOUND)
+    #message(STATUS "PDC include directory: ${PDC_INCLUDE_DIR}")
+  set(JMD_LDR_EXT_INCLUDE_DIRS ${PDC_INCLUDE_DIR}
+    ${JMD_LDR_EXT_INCLUDE_DIRS}
+    )
+  set(JMD_LDR_EXT_LIBRARIES pdc ${JMD_LDR_EXT_LIBRARIES})
+endif()
+
+
+add_library(cjson cjson/cJSON.c)
+
+add_executable(metadata_json_loader 
+            metadata_json_loader.c 
+            metadata_json_processor.h 
+            metadata_json_printer.c 
+            metadata_json_printer.h 
+            metadata_json_importer.c 
+            metadata_json_importer.h)
+target_link_libraries(metadata_json_loader ${PDC_EXT_LIB_DEPENDENCIES} pdc cjson ${JMD_LDR_EXT_LIBRARIES})
+target_include_directories(metadata_json_loader PUBLIC ${PDC_EXT_INCLUDE_DEPENDENCIES} ${JMD_LDR_EXT_INCLUDE_DIRS})