Devops: Address deprecation warnings from internal code and dependencies

[sphuber]

This addresses the majority of deprecation warnings that are emitted under Python 3.9. The Python 3.12 run has quite a number more but they are mostly caused by dependencies and so they will need updates upstream. These will be taken care of at a later point in time.

The remaining warnings are the following:

 tests/test_dataclasses.py::TestCifData::test_ase_primitive_and_conventional_cells_ase
  /opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/ase/atoms.py:1205: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
    atoms *= rep

This should be fixed by ase.

tests/cmdline/commands/test_code.py::test_code_duplicate_non_interactive[vim -cwq]
  /home/runner/work/aiida-core/aiida-core/src/aiida/orm/utils/builders/code.py:17: AiidaDeprecationWarning: This module is deprecated. To create a new code instance, simply use the constructor. (this will be removed in v3)
    warn_deprecation('This module is deprecated. To create a new code instance, simply use the constructor.', version=3)

This is because verdi code duplicate still requires the deprecated CodeBuilder. Addressing this will require quite some work and so is reserved for a separate PR.

tests/cmdline/commands/test_code.py::test_code_export
tests/cmdline/commands/test_code.py::test_code_create[sleep 1; vim -cwq-core.code.installed]
tests/cmdline/commands/test_code.py::test_code_create[sleep 1; vim -cwq-core.code.portable]
tests/cmdline/commands/test_code.py::test_code_create[sleep 1; vim -cwq-core.code.containerized]
  /home/runner/work/aiida-core/aiida-core/src/aiida/cmdline/groups/dynamic.py:145: AiidaDeprecationWarning: Relying on `_get_cli_options` is deprecated. The options should be defined through a `pydantic.BaseModel` that should be assigned to the `Config` class attribute. (this will be removed in v3)
    warn_deprecation(

This would be taken care of by #6190

tests/engine/test_work_chain.py::TestWorkchain::test_tocontext_schedule_workchain
tests/engine/processes/calcjobs/test_calc_job.py::test_multi_codes_with_mpi[code_info_with_mpi_none-None-None-False-0]
  /opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/asyncio/base_events.py:686: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
    _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/engine/processes/calcjobs/test_calc_job.py::test_multi_codes_with_mpi[code_info_with_mpi_none-None-None-False-0]
  /opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/asyncio/base_events.py:672: RuntimeWarning: coroutine 'Process.step_until_terminated' was never awaited
    self._ready.clear()
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

These are due to the test test_call_on_process_finish in tests/engine/test_runners.py. The fix would be to close the new event loop that is created in the runner fixture, or simply use the Runner class to take care of it. This works for Python 3.9, however, it fails for Python 3.12. The behavior of asyncio.get_event_loop changed in Python 3.12 where now it raises if no event loop has been set. This first needs to be addressed in plumpy before we can fix it in aiida-core.

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (35d7ca6) 77.70% compared to head (911b1f2) 77.64%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
src/aiida/transports/util.py	57.15%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6272      +/-   ##
==========================================
- Coverage   77.70%   77.64%   -0.05%     
==========================================
  Files         548      548              
  Lines       40230    40237       +7     
==========================================
- Hits        31255    31239      -16     
- Misses       8975     8998      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sphuber]

@unkcpz and @mbercx this is ready for review.

[unkcpz]

I'll take a look later. FYI, the docker test failed because reach the rate limit.

[unkcpz]

Thanks @sphuber. Just one small question, the rest looks all good.

[unkcpz]

I don't understand why ignore the ValueError specifically?

[sphuber]

This will be raised iof the file handle was already closed, which can happen

[unkcpz]

I see, thanks.

[unkcpz]

Also since this PR is on deprecation, would you mind to take a look at the deprecation in doc buildhttps://readthedocs.org/projects/aiida-core/builds/23431980/ ? I assume it will follow the same rules of change to get rid of those.

[sphuber]

Also since this PR is on deprecation, would you mind to take a look at the deprecation in doc buildhttps://readthedocs.org/projects/aiida-core/builds/23431980/ ? I assume it will follow the same rules of change to get rid of those.

You mean the warn_deprecation( messages in the build log? Those are just there because it imports the code to do the auto API docs and so it hits our own deprecation warnings. There is not much we can do there, can we?

[unkcpz]

Those are just there because it imports the code to do the auto API docs and so it hits our own deprecation warnings. There is not much we can do there, can we?

I see your point, if the warnings are from import InstalledCode which inherit the Code and the warning will anyway raised, correct? Then, I don't think there is much we can do.

But I think this one show in the doc should be suppressed.

[sphuber]

But I think this one show in the doc should be suppressed.

Ah yeah, that one should be fixed. This is really an incorrect deprecation, so I removed the deprecation altogether.

How did you find this deprecation warning? Just by going through the built docs and you happened to stumble on it when reading the tutorial? Or is there an easier way to find all deprecations? The docs build log contains a bunch of hits of our warn_deprecation but I don't understand where they are coming from. The logs seem all mangled and are impossible to read.

[unkcpz]

Just by going through the built docs and you happened to stumble on it when reading the tutorial?

yep 😁. You know I translate doc to Chinese, I spot in when review the auto-translation PR.

The logs seem all mangled and are impossible to read.

True, it is hard to read, I neither have clue on how to fix. But if the sphinx command run locally, the output is well indented with line breaks. Might be a bit easy to read. I think all those warnings are raised during import phase in apidoc build. Thus not too much to check and fix.

[sphuber]

Thanks for the review @unkcpz