TypeError: metaclass conflict with cell2cell in reticulate

[rbutleriii]

Moving over from saezlab/liana#152 as this seems to be a cell2cell issue. I am trying to get cell2cell to load on a cluster, in R using reticulate, but I keep getting a metaclass conflict:

Error in py_module_import(module, convert = convert) :
  TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Run `reticulate::py_last_error()` for details.

Initially I figured it was a conflict with the ccc_protocols env_python_gpu.yml, but I have also tried setting up a test environment in isolation and it has the same problem. Built using the following:

conda create -n test -y python=3.10
conda activate test
pip install cell2cell

When trying to import it in R, I get the following:

library(reticulate, quietly = TRUE)
Sys.setenv(RETICULATE_PYTHON = "~/.local/share/r-miniconda/envs/test/bin/python")

c2c <- reticulate::import(module = "cell2cell")

> reticulate::py_last_error()

── Python Exception Message ────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/share/r-miniconda/envs/test/lib/python3.10/site-packages/cell2cell/__init__.py", line 3, in <module>
    from cell2cell import analysis
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/share/r-miniconda/envs/test/lib/python3.10/site-packages/cell2cell/analysis/__init__.py", line 1, in <module>
    from cell2cell.analysis.cell2cell_pipelines import (initialize_interaction_space, BulkInteractions, SingleCellInteractions)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/share/r-miniconda/envs/test/lib/python3.10/site-packages/cell2cell/analysis/cell2cell_pipelines.py", line 6, in <module>
    import scanpy
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/__init__.py", line 16, in <module>
    from . import plotting as pl
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/plotting/__init__.py", line 1, in <module>
    from ._anndata import (
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/plotting/_anndata.py", line 28, in <module>
    from . import _utils
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 93, in _run_hook
    module = hook()
  File "/home/rrbutler/R/x86_64-pc-linux-gnu-library/4.2/reticulate/python/rpytools/loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/plotting/_utils.py", line 35, in <module>
    class _AxesSubplot(Axes, axes.SubplotBase, ABC):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

── R Traceback ─────────────────────────────────────────────────────────────────────────────────────────────────────────
    ▆
 1. └─reticulate::import(module = "cell2cell")
 2.   └─reticulate:::py_module_import(module, convert = convert)

Trying to call it directly in python:

Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cell2cell as c2c
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rrbutler/.local/share/r-miniconda/envs/test/lib/python3.10/site-packages/cell2cell/__init__.py", line 3, in <module>
    from cell2cell import analysis
  File "/home/rrbutler/.local/share/r-miniconda/envs/test/lib/python3.10/site-packages/cell2cell/analysis/__init__.py", line 1, in <module>
    from cell2cell.analysis.cell2cell_pipelines import (initialize_interaction_space, BulkInteractions, SingleCellInteractions)
  File "/home/rrbutler/.local/share/r-miniconda/envs/test/lib/python3.10/site-packages/cell2cell/analysis/cell2cell_pipelines.py", line 6, in <module>
    import scanpy
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/__init__.py", line 16, in <module>
    from . import plotting as pl
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/plotting/__init__.py", line 1, in <module>
    from ._anndata import (
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/plotting/_anndata.py", line 28, in <module>
    from . import _utils
  File "/home/rrbutler/.local/lib/python3.10/site-packages/scanpy/plotting/_utils.py", line 35, in <module>
    class _AxesSubplot(Axes, axes.SubplotBase, ABC):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Updating to the latest devel version of cell2cell doesn't help, issue is the same with 0.7.1->0.7.4.

[earmingol]

Hi @rbutleriii

From the errors, it seems to be an issue with the tool scanpy, which is imported by cell2cell.

I suggest trying older or newer versions of scanpy and see what happens.

Here there is a detailed table of versions tht we have tested: https://www.cell.com/cell-reports-methods/fulltext/S2667-2375(24)00089-4?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS2667237524000894%3Fshowall%3Dtrue#secsectitle0160

Scanpy 1.9 and matplotlib 3.7.3 should make your environment work.

[rbutleriii]

Hmm, perhaps the link didn't resolve correctly, but it brought up the zenodos for ccc_protocols (the yml I am using) and tc2c_benchmark. I went ahead and tried scanpy 1.9.1 (1.10.4 is the latest) and still nothing. Looks like in the recent past scanpy developed an issue with matplotlib 3.7 scverse/scanpy#3029. Trying a number of scenarios for install, but another issue appears to be pandas 2+ if you roll the matplot back to <3.7 scverse/scanpy#2564. I will keep at it and follow up if that resolves everything.

[earmingol]

If you scroll down a bit in the link you should find the table with the versions (look for Software Requirements Table). Also pandas 2+ generates a lot of issues since they deprecated many functions, I suggest using pandas 1.5.1 or so.
Also, my bad I meant scanpy 1.9.0