Bump pynndescent to 0.5.13 to fix import error #15

antoninkriz · 2025-01-21T00:19:28Z

Hello, this might be the final solution to issues #9 and #12 as the Import currently still fails.

Neofuzz's dependency pynndescent does not correctly support NumPy 2.x.x prior to version 0.5.13. The fix lmcinnes/pynndescent#242 required for this support was shipped after the release 0.5.12 currently used by Neofuzz.

MRE

Python 3.12 or 3.13 (did not test others)

$ python -m venv venv
$ source venv/bin/activate
$ pip install neofuzz==0.3.3
$ python -c 'import neofuzz'

Fix (workaround)

The simplest fix is to forcefully bump pynndescent from 0.5.12 to 0.5.13 and ignore pip screaming about package requirements of Neofuzz.

$ # continuation from MRE
$ pip install pynndescent==0.5.13
$ python -c 'import neofuzz'

Logs - MRE

$ mkdir neofuzztest
$ cd neofuzztest
$ python -m venv venv
$ source venv/bin/activate
$ pip install neofuzz
Collecting neofuzz
  Downloading neofuzz-0.3.3-py3-none-any.whl.metadata (6.9 kB)
Collecting joblib>=1.4.0 (from neofuzz)
  Downloading joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting numpy>=0.22.0 (from neofuzz)
  Downloading numpy-2.2.2-cp312-cp312-macosx_14_0_arm64.whl.metadata (62 kB)
Collecting pynndescent==0.5.12 (from neofuzz)
  Downloading pynndescent-0.5.12-py3-none-any.whl.metadata (6.8 kB)
Collecting scikit-learn>=1.1.0 (from neofuzz)
  Downloading scikit_learn-1.6.1-cp312-cp312-macosx_12_0_arm64.whl.metadata (31 kB)
Collecting thefuzz>=0.22.0 (from neofuzz)
  Downloading thefuzz-0.22.1-py3-none-any.whl.metadata (3.9 kB)
Collecting tokenizers>=0.19.0 (from neofuzz)
  Downloading tokenizers-0.21.0-cp39-abi3-macosx_11_0_arm64.whl.metadata (6.7 kB)
Collecting scipy>=1.0 (from pynndescent==0.5.12->neofuzz)
  Downloading scipy-1.15.1-cp312-cp312-macosx_14_0_arm64.whl.metadata (61 kB)
Collecting numba>=0.51.2 (from pynndescent==0.5.12->neofuzz)
  Downloading numba-0.61.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (2.7 kB)
Collecting llvmlite>=0.30 (from pynndescent==0.5.12->neofuzz)
  Downloading llvmlite-0.44.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (4.8 kB)
Collecting threadpoolctl>=3.1.0 (from scikit-learn>=1.1.0->neofuzz)
  Downloading threadpoolctl-3.5.0-py3-none-any.whl.metadata (13 kB)
Collecting rapidfuzz<4.0.0,>=3.0.0 (from thefuzz>=0.22.0->neofuzz)
  Using cached rapidfuzz-3.11.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (11 kB)
Collecting huggingface-hub<1.0,>=0.16.4 (from tokenizers>=0.19.0->neofuzz)
  Downloading huggingface_hub-0.27.1-py3-none-any.whl.metadata (13 kB)
Collecting filelock (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading fsspec-2024.12.0-py3-none-any.whl.metadata (11 kB)
Collecting packaging>=20.9 (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pyyaml>=5.1 (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading PyYAML-6.0.2-cp312-cp312-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting requests (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting tqdm>=4.42.1 (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting numpy>=0.22.0 (from neofuzz)
  Downloading numpy-2.1.3-cp312-cp312-macosx_14_0_arm64.whl.metadata (62 kB)
Collecting charset-normalizer<4,>=2 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading charset_normalizer-3.4.1-cp312-cp312-macosx_10_13_universal2.whl.metadata (35 kB)
Collecting idna<4,>=2.5 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting urllib3<3,>=1.21.1 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.0->neofuzz)
  Downloading certifi-2024.12.14-py3-none-any.whl.metadata (2.3 kB)
Downloading neofuzz-0.3.3-py3-none-any.whl (10 kB)
Downloading pynndescent-0.5.12-py3-none-any.whl (56 kB)
Downloading joblib-1.4.2-py3-none-any.whl (301 kB)
Downloading scikit_learn-1.6.1-cp312-cp312-macosx_12_0_arm64.whl (11.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 10.6 MB/s eta 0:00:00
Downloading thefuzz-0.22.1-py3-none-any.whl (8.2 kB)
Downloading tokenizers-0.21.0-cp39-abi3-macosx_11_0_arm64.whl (2.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 9.9 MB/s eta 0:00:00
Downloading huggingface_hub-0.27.1-py3-none-any.whl (450 kB)
Downloading llvmlite-0.44.0-cp312-cp312-macosx_11_0_arm64.whl (26.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.2/26.2 MB 10.3 MB/s eta 0:00:00
Downloading numba-0.61.0-cp312-cp312-macosx_11_0_arm64.whl (2.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 9.4 MB/s eta 0:00:00
Downloading numpy-2.1.3-cp312-cp312-macosx_14_0_arm64.whl (5.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 10.5 MB/s eta 0:00:00
Using cached rapidfuzz-3.11.0-cp312-cp312-macosx_11_0_arm64.whl (1.4 MB)
Downloading scipy-1.15.1-cp312-cp312-macosx_14_0_arm64.whl (24.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.9/24.9 MB 10.5 MB/s eta 0:00:00
Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)
Downloading fsspec-2024.12.0-py3-none-any.whl (183 kB)
Downloading packaging-24.2-py3-none-any.whl (65 kB)
Downloading PyYAML-6.0.2-cp312-cp312-macosx_11_0_arm64.whl (173 kB)
Downloading tqdm-4.67.1-py3-none-any.whl (78 kB)
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Using cached filelock-3.16.1-py3-none-any.whl (16 kB)
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Downloading certifi-2024.12.14-py3-none-any.whl (164 kB)
Downloading charset_normalizer-3.4.1-cp312-cp312-macosx_10_13_universal2.whl (196 kB)
Using cached idna-3.10-py3-none-any.whl (70 kB)
Downloading urllib3-2.3.0-py3-none-any.whl (128 kB)
Installing collected packages: urllib3, typing-extensions, tqdm, threadpoolctl, rapidfuzz, pyyaml, packaging, numpy, llvmlite, joblib, idna, fsspec, filelock, charset-normalizer, certifi, thefuzz, scipy, requests, numba, scikit-learn, huggingface-hub, tokenizers, pynndescent, neofuzz
Successfully installed certifi-2024.12.14 charset-normalizer-3.4.1 filelock-3.16.1 fsspec-2024.12.0 huggingface-hub-0.27.1 idna-3.10 joblib-1.4.2 llvmlite-0.44.0 neofuzz-0.3.3 numba-0.61.0 numpy-2.1.3 packaging-24.2 pynndescent-0.5.12 pyyaml-6.0.2 rapidfuzz-3.11.0 requests-2.32.3 scikit-learn-1.6.1 scipy-1.15.1 thefuzz-0.22.1 threadpoolctl-3.5.0 tokenizers-0.21.0 tqdm-4.67.1 typing-extensions-4.12.2 urllib3-2.3.0

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
$ python -V
Python 3.12.7
$ python -c 'import neofuzz'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/neofuzz/__init__.py", line 1, in <module>
    from neofuzz.process import Process, char_ngram_process
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/neofuzz/process.py", line 7, in <module>
    import pynndescent
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/pynndescent/__init__.py", line 5, in <module>
    from .pynndescent_ import NNDescent, PyNNDescentTransformer
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/pynndescent/pynndescent_.py", line 23, in <module>
    import pynndescent.sparse_nndescent as sparse_nnd
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/pynndescent/sparse_nndescent.py", line 24, in <module>
    EMPTY_GRAPH = make_heap(1, 1)
                  ^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 443, in _compile_for_args
    raise e
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 376, in _compile_for_args
    return_val = self.compile(tuple(argtypes))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 904, in compile
    cres = self._compiler.compile(args, return_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 80, in compile
    status, retval = self._compile_cached(args, return_type)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 94, in _compile_cached
    retval = self._compile_core(args, return_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 107, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler.py", line 739, in compile_extra
    return pipeline.compile_extra(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler.py", line 439, in compile_extra
    return self._compile_bytecode()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler.py", line 505, in _compile_bytecode
    return self._compile_core()
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler.py", line 481, in _compile_core
    raise e
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler.py", line 473, in _compile_core
    pm.run(self.state)
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 363, in run
    raise e
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 356, in run
    self._runPass(idx, pass_inst, state)
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass
    mutated |= check(pss.run_pass, internal_state)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 272, in check
    mangled = func(compiler_state)
              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typed_passes.py", line 112, in run_pass
    typemap, return_type, calltypes, errs = type_inference_stage(
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typed_passes.py", line 93, in type_inference_stage
    errs = infer.propagate(raise_errors=raise_errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typeinfer.py", line 1066, in propagate
    errors = self.constraints.propagate(self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typeinfer.py", line 160, in propagate
    constraint(typeinfer)
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typeinfer.py", line 708, in __call__
    attrty = typeinfer.context.resolve_getattr(ty, self.attr)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typing/context.py", line 282, in resolve_getattr
    attrty = self.resolve_module_constants(typ, attr)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numba/core/typing/context.py", line 347, in resolve_module_constants
    attrval = getattr(typ.pymod, attr)
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/antoninkriz/neofuzztest/venv/lib/python3.12/site-packages/numpy/__init__.py", line 400, in __getattr__
    raise AttributeError(
AttributeError: `np.infty` was removed in the NumPy 2.0 release. Use `np.inf` instead.
$ # FAIL :(

Logs - fix (workaround

$ pip install pynndescent==0.5.13
Collecting pynndescent==0.5.13
  Using cached pynndescent-0.5.13-py3-none-any.whl.metadata (6.8 kB)
Requirement already satisfied: scikit-learn>=0.18 in ./venv/lib/python3.12/site-packages (from pynndescent==0.5.13) (1.6.1)
Requirement already satisfied: scipy>=1.0 in ./venv/lib/python3.12/site-packages (from pynndescent==0.5.13) (1.15.1)
Requirement already satisfied: numba>=0.51.2 in ./venv/lib/python3.12/site-packages (from pynndescent==0.5.13) (0.61.0)
Requirement already satisfied: llvmlite>=0.30 in ./venv/lib/python3.12/site-packages (from pynndescent==0.5.13) (0.44.0)
Requirement already satisfied: joblib>=0.11 in ./venv/lib/python3.12/site-packages (from pynndescent==0.5.13) (1.4.2)
Requirement already satisfied: numpy<2.2,>=1.24 in ./venv/lib/python3.12/site-packages (from numba>=0.51.2->pynndescent==0.5.13) (2.1.3)
Requirement already satisfied: threadpoolctl>=3.1.0 in ./venv/lib/python3.12/site-packages (from scikit-learn>=0.18->pynndescent==0.5.13) (3.5.0)
Downloading pynndescent-0.5.13-py3-none-any.whl (56 kB)
Installing collected packages: pynndescent
  Attempting uninstall: pynndescent
    Found existing installation: pynndescent 0.5.12
    Uninstalling pynndescent-0.5.12:
      Successfully uninstalled pynndescent-0.5.12
ERROR: pip\'s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
neofuzz 0.3.3 requires pynndescent==0.5.12, but you have pynndescent 0.5.13 which is incompatible.
Successfully installed pynndescent-0.5.13

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
$ python -c 'import neofuzz' 
$ # OK :)

x-tabdeveloping · 2025-01-21T07:35:03Z

pyproject.toml

@@ -12,7 +12,7 @@ packages = [{include = "neofuzz"}]
 [tool.poetry.dependencies]
 python = ">=3.8"
 scikit-learn = ">=1.1.0"
-pynndescent = "0.5.12"
+pynndescent = "0.5.13"


Maybe we could relax this somewhat instead of locking it to a single version. "~0.5.13" for instance would be a bit more flexible.

x-tabdeveloping · 2025-01-21T08:18:34Z

Can I also ask you to bump the Neofuzz version, so that I can put out a new release on PyPI?

antoninkriz · 2025-01-21T08:26:19Z

Thank you for quick reply!
Both tilde version specification of pynndescent and package version bump should be done.

x-tabdeveloping · 2025-01-21T09:00:52Z

Awesome @antoninkriz ! Thanks for the contribution, it is much appreciated.

Update pynndescent to 0.5.13

8984790

x-tabdeveloping reviewed Jan 21, 2025

View reviewed changes

antoninkriz added 2 commits January 21, 2025 09:24

Use tilde to allow newer patch versions of pynndescent

87b70e4

Bump package version

73fb696

x-tabdeveloping merged commit 7faf044 into x-tabdeveloping:main Jan 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump pynndescent to 0.5.13 to fix import error #15

Bump pynndescent to 0.5.13 to fix import error #15

antoninkriz commented Jan 21, 2025

x-tabdeveloping Jan 21, 2025

x-tabdeveloping commented Jan 21, 2025

antoninkriz commented Jan 21, 2025

x-tabdeveloping commented Jan 21, 2025

Bump pynndescent to 0.5.13 to fix import error #15

Bump pynndescent to 0.5.13 to fix import error #15

Conversation

antoninkriz commented Jan 21, 2025

MRE

Fix (workaround)

Logs - MRE

Logs - fix (workaround

x-tabdeveloping Jan 21, 2025

Choose a reason for hiding this comment

x-tabdeveloping commented Jan 21, 2025

antoninkriz commented Jan 21, 2025

x-tabdeveloping commented Jan 21, 2025