granne (graph-based retrieval of approximate nearest neighbors) is a Rust library for ANN-search based on Hierarchical Navigable Small World (HNSW) graphs and is used in Cliqz Search. For some background and motivation behind granne, please read Indexing Billions of Text Vectors.
Note: granne is still under active development. A more stable release (with documentation) is coming soon.
- Memory-mapped
- Multithreaded index creation
- Extensible indexes (add elements to an already built index)
- Python bindings
- Dense
floatorint8elements (cosine distance)
granne is dependent on BLAS (https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) for some computations. This applies both to the rust and python versions. On Debian/Ubuntu both libblas-dev and libopenblas-dev should work, with the latter being significantly faster.
On Mac OS there seems to be some issue (maybe
this one)
with the default BLAS library. A workaround is to install e.g. openblas and link to that instead.
# build
cargo build --release
# test
cargo test
# bench
cargo +nightly bench
To quickly install:
pip install setuptools_rust
pip install .
To build python wheels for python 2.7, 3.4, 3.5 and 3.6 (requires docker).
docker build -t granne_manylinux docker/manylinux/
docker run -v $(pwd):/granne/ granne_manylinux /opt/build_wheels.sh
The output is written to wheels/ and can be installed by
pip install granne --no-index -f wheels/
...
...
*granne is Swedish and means neighbor