Python 3.12 Support

[xavierholt]

Is there a plan to support Python 3.12?

Ubuntu 24.04 is out now, and it only comes with Python 3.12; none of the older Python versions are available in the official APT repos. It would be great if I could install ProDy on a new Ubuntu box, but I've seen several comments to the effect that it isn't compatible with Python 3.12 (and my attempts to install it via pip have all failed with some sort of build system error).

I might be able to help out with this - is there a list of changes that would be required for 3.12 support?

[jamesmkrieger]

First of all, compatibility with the latest numpy and scipy, I think. I think there were some changes to installation too. @atbogetti, do you know?

help would be great. Thanks. Just remember to keep back compatibility with python 3.9 and 2.x

Are you sure you can’t get earlier python with anaconda? Lots of things don’t support python 3.11 or 3.12 yet.

[lenny-mavs]

I am having compatibility issues when trying to install ProDy and MDAnalysis in the same Anaconda environment, as downgrading all of the mutual dependencies causes one or the other to not work correctly. I would greatly appreciate a version that can use the more recent releases of numpy, scipy, biopython, etc.

[jamesmkrieger]

Could you maybe use an older version of mdanalysis?

[lenny-mavs]

Yes, that solved the issue quite quickly!

[jamesmkrieger]

Great. Glad to hear

[diogomart]

@xavierholt @jamesmkrieger me and @sforli took a look and the C extension dynamics/rtbtools seems to be a culprit. One guess are the numpy headers needing updating to 1.26 or later. Not sure this helps much but may be a start. If we continue looking into it and make progress we will post here.

[jamesmkrieger]

It looks like this is a problem is several of the C extensions, not just dynamics/rtbtools. The following header is found in dynamics/rtbtools, dynamics/smtools, sequence/msaio, sequence/msatools and sequence/seqtools:
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION

[jamesmkrieger]

It looks like this is a problem is several of the C extensions, not just dynamics/rtbtools. The following header is found in dynamics/rtbtools, dynamics/smtools, sequence/msaio, sequence/msatools and sequence/seqtools: #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION

this isn't the problem per se. What it does it masks a warning and the warning says to add it to do so

[jamesmkrieger]

It looks like the problem is that there's a new numpy C API that we need to use instead. The old one has been deprecated for a while but I guess it's now completely removed in the new versions

@SHZ66, do you think you could have a look at it please?

[jamesmkrieger]

@xavierholt @jamesmkrieger me and @sforli took a look and the C extension dynamics/rtbtools seems to be a culprit. One guess are the numpy headers needing updating to 1.26 or later. Not sure this helps much but may be a start. If we continue looking into it and make progress we will post here.

You're absolutely right that the only problem is with rtbtools and there's a particular function dblock_projections2. If I comment this function out then everything compiles files. Whether it runs fine (excluding RTB) is another question of course

[jamesmkrieger]

The error is with the the first line of the function that declares the variables and then we get errors that they are undeclared and after that errors with their use

[jamesmkrieger]

ok! I fixed it!!!

The problem was that the name I was coming from one of the imports (see https://stackoverflow.com/questions/76668359/error-when-compiling-c-flie-with-cblas-h-getting-error-error-expected-identifi). Calling it Ii fixed the problem

[jamesmkrieger]

ok, now I have managed to fix some other issues with missing modules like imp too

[jamesmkrieger]

now all the checks are passing in python 3.12 too!

[xavierholt]

This is great! Is there anything I can do to help with testing?

[jamesmkrieger]

more testing is always helpful, I guess. If you could try installing it and running a few things, that would be great. Thanks

[xavierholt]

I've been able to build your branch with Numpy 1.26.4 in both Ubuntu 22.04 (Python 3.10) and Ubuntu 24.04 (Python 3.12). That works flawlessly. I'm running into issues trying to run the test suite afterward, though. When I run pytest, I get two variations of this error:

___ ERROR collecting prody/tests/dynamics/test_enms.py ___
prody/tests/dynamics/test_enms.py:39: in <module>
    gnm.buildKirchhoff(ATOMS)
prody/dynamics/gnm.py:156: in buildKirchhoff
    dist2 = kdtree.getDistances() ** 2
prody/kdtree/kdtree.py:273: in getDistances
    if self.getCount():
prody/kdtree/kdtree.py:291: in getCount
    return self._kdtree.neighbor_get_count()
E   AttributeError: 'C KDTree' object has no attribute 'neighbor_get_count'

And three variations of this error:

___ ERROR collecting prody/tests/sequence/test_msafile.py ___
ImportError while importing test module '/opt/ProDy/prody/tests/sequence/test_msafile.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
prody/tests/sequence/test_msafile.py:18: in <module>
    FASTA = parseMSA(pathDatafile('msa_Cys_knot.fasta'))
prody/sequence/msafile.py:588: in parseMSA
    from .msaio import parseFasta as parser
E   ModuleNotFoundError: No module named 'prody.sequence.msaio'

Do you have any suggestions for working around this? I'm running in a virtual environment with the dependencies installed via pip. I'm hoping to be able to run the tests without running pip install . (running that before running pytest didn't fix the errors above).

[jamesmkrieger]

Have you done the “python setup.py --build_ext”step? It looks like you’re missing the C/C++ extensions. We had a similar problem of the test suite not finding them, but I think it was with nose not pytest

[xavierholt]

I've been running python3 setup.py build. According to the output from that command, it runs build_py and build_ext as subtasks.

[xavierholt]

Ah! I got past it - the GitHub workflow scripts had the hint I needed. The build_ext step needs the --inplace flag. That makes it copy a bunch of .so files into the source tree where the tests can find them.

[jamesmkrieger]

Ah ok. Good to know