Enable Virtual Environments in testing

[causten]

The test cases of "make check" can fail over time due to upstream changes when explicit versions are not captured. This is important for official releases (release/rocm-rel-5.x). The Develop and Master branch should keep up with latest possible to catch issues but need to fix upstream dependencies once in to release branches. An example of this is Protobuf dropping support for Python3.6. Our rocm-rel-5.2 stream now doesn't build all of it's CI because Protobuf was allowed to float. So unit tests now fail on OS' that have python3.6

I'd like to see our build support Virtual Environments to easily pin python packages at to specific configurations

[pfultz2]

The virtual environments need to be setup as a test command, but needs to run before the python onnx tests. Its probably best to use ctest's fixtures to do this(although we dont need a cleanup) to ensure it will always set up the virtual env first. We can probably add a function to setup a virtualenv fixture like this:

function(add_py_venv_fixture FIXTURE_NAME VIRTUAL_ENV_DIR REQUIREMENTS_FILE)
...
endfunction()

Then add_py_test we would use the fixture and run the python in the virtual env.

Here is a blog about ctest's fixtures:

https://crascit.com/2016/10/18/test-fixtures-with-cmake-ctest/

[ahsan-ca]

Draft PR is up for questions/feedback: #2191

[ahsan-ca]

During discussions with Paul, he suggested to disable Python 2.7 in test/PythonModules.cmake. The motivation for this change is that it would help us to use venv instead of virtualenv. venv is part of the Python standard library whereas virtualenv is a third-party tool. venv is availabe in Python 3.3 and later versions.
PR for disabling Python 2.7: #2204

[ahsan-ca]

Reopened as changes in the other PR for virtual env, once it is ready, need to be merged in as well.

[ahsan-ca]

Addressed review comments on the draft PR. It is up for review again. In the meanwhile, I am testing things out on sles docker container.

[ahsan-ca]

Discussed the duplication of test names issue and why I was setting up virtual env for different Python versions with Paul on Friday. He suggested to check for existence of tests before setting up virtual env to avoid the unique name issue. I have made the changes and put up a PR for review.
I have also added a separate requirements.txt file for sles.
PR is ready for review: #2191

[ahsan-ca]

Regarding the requirements file for the virtual env for ubuntu and sles, we discussed in office hours that we would ideally like to keep one requirements file.

In the testing for the required packages, I saw two test cases failing:

test_py_3.6_gpu_async
test_py_3.6_backend

1. test_py_3.6_gpu_async complains about a missing shared library libamdhip64.so.
This library can be found at /opt/rocm/lib on sles. In /etc/ld.so.conf.d/rocm.conf, I see /opt/rocm/lib being specified.
Also, in /etc/ld.so.conf we have:

/usr/local/lib64
/usr/local/lib
include /etc/ld.so.conf.d/*.conf

It should pick up the library but not sure why it is not picking that up. I was able to make the test case pass by specifying the full path to the library in the test case.

2. test_py_3.6_backend complained about missing onnx and pytest packages initially. After installing those packages (onnx==1.10.2, pytest==6.0.1), it errors out because it cannot download the models:

Start downloading model bvlc_alexnet from https://s3.amazonaws.com/download.onnx/models/opset_9/bvlc_alexnet.tar.gz
Failed to prepare data for model bvlc_alexnet: HTTP Error 403: Forbidden

To fix this I installed a more recent version of onnx==1.14.1. This gives error because future feature annotations is not defined in python3.6. They are available from python3.7:

  File "/usr/lib64/python3.6/site-packages/onnx/backend/test/runner/__init__.py", line 5
    from __future__ import annotations
    ^
SyntaxError: future feature annotations is not defined

If I try to install an older version of onnx==1.13.1, I hit the error ERROR: No matching distribution found for protobuf<4,>=3.20.2 since I had installed protobuf==4.0.0rc1.

If I install protobuf==3.19.6 and try to install onnx==1.13.1, I get ERROR: Could not find a version that satisfies the requirement protobuf<4,>=3.20.2 (from onnx).

If I try to install protobuf==3.20.2, I get the following error:

ERROR: Could not find a version that satisfies the requirement protobuf==3.20.2 (from versions: 2.0.0b0, 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0, 3.1.0, 3.1.0.post1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0, 3.3.0, 3.4.0, 3.5.0.post1, 3.5.1, 3.5.2, 3.5.2.post1, 3.6.0, 3.6.1, 3.7.0rc2, 3.7.0rc3, 3.7.0, 3.7.1, 3.8.0rc1, 3.8.0, 3.9.0rc1, 3.9.0, 3.9.1, 3.9.2, 3.10.0rc1, 3.10.0, 3.11.0rc1, 3.11.0rc2, 3.11.0, 3.11.1, 3.11.2, 3.11.3, 3.12.0rc1, 3.12.0rc2, 3.12.0, 3.12.1, 3.12.2, 3.12.4, 3.13.0rc3, 3.13.0, 3.14.0rc1, 3.14.0rc2, 3.14.0rc3, 3.14.0, 3.15.0rc1, 3.15.0rc2, 3.15.0, 3.15.1, 3.15.2, 3.15.3, 3.15.4, 3.15.5, 3.15.6, 3.15.7, 3.15.8, 3.16.0rc1, 3.16.0rc2, 3.16.0, 3.17.0rc1, 3.17.0rc2, 3.17.0, 3.17.1, 3.17.2, 3.17.3, 3.18.0rc1, 3.18.0rc2, 3.18.0, 3.18.1, 3.18.3, 3.19.0rc1, 3.19.0rc2, 3.19.0, 3.19.1, 3.19.2, 3.19.3, 3.19.4, 3.19.5, 3.19.6, 4.0.0rc1, 4.0.0rc2, 4.21.0rc1, 4.21.0rc2, 4.21.0)
ERROR: No matching distribution found for protobuf==3.20.2

TLDR: I cannot get the required version of protobuf to install that will support onnx version clsoe to onnx 1.14.1.

[ahsan-ca]

Updates for the above 2 issues:
1. test_py_3.6_gpu_async Missing shared library issue is resolved by specifying the full path to the library. This is tracked under #2263

2. test_py_3.6_backend I got feedback from the team to try an older version of onnx that could satisfy other dependencies for python3.6. The oldest version I can use for onnx is 1.13.1 because the URL for the real models was fixed in that release via Update URL for real models from ONNX Runtime onnx/onnx#4865.
When trying to install onnx 1.13.1, it complains about an incompatible version of protobuf:

ERROR: Could not find a version that satisfies the requirement protobuf<4,>=3.20.2 (from onnx==1.13.1) (from versions: 2.0.0b0, 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0, 3.1.0, 3.1.0.post1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0, 3.3.0, 3.4.0, 3.5.0.post1, 3.5.1, 3.5.2, 3.5.2.post1, 3.6.0, 3.6.1, 3.7.0rc2, 3.7.0rc3, 3.7.0, 3.7.1, 3.8.0rc1, 3.8.0, 3.9.0rc1, 3.9.0, 3.9.1, 3.9.2, 3.10.0rc1, 3.10.0, 3.11.0rc1, 3.11.0rc2, 3.11.0, 3.11.1, 3.11.2, 3.11.3, 3.12.0rc1, 3.12.0rc2, 3.12.0, 3.12.1, 3.12.2, 3.12.4, 3.13.0rc3, 3.13.0, 3.14.0rc1, 3.14.0rc2, 3.14.0rc3, 3.14.0, 3.15.0rc1, 3.15.0rc2, 3.15.0, 3.15.1, 3.15.2, 3.15.3, 3.15.4, 3.15.5, 3.15.6, 3.15.7, 3.15.8, 3.16.0rc1, 3.16.0rc2, 3.16.0, 3.17.0rc1, 3.17.0rc2, 3.17.0, 3.17.1, 3.17.2, 3.17.3, 3.18.0rc1, 3.18.0rc2, 3.18.0, 3.18.1, 3.18.3, 3.19.0rc1, 3.19.0rc2, 3.19.0, 3.19.1, 3.19.2, 3.19.3, 3.19.4, 3.19.5, 3.19.6, 4.0.0rc1, 4.0.0rc2, 4.21.0rc1, 4.21.0rc2, 4.21.0)
ERROR: No matching distribution found for protobuf<4,>=3.20.2 (from onnx==1.13.1)

There is no version of protobuf for python3.6 and onnx1.13.1 that satisfies the requirement protobuf<4,>=3.20.2.

I also tried the other option to use --no-use-pep563 that Umang sugested to disable future feature annotations, that also did not help.

I tried backporting future-annotations library that provides future annotations support (pip3 install future-annotations), but that also does not seem to help.

Looks like we would have to disable this test for python3.6.

[ahsan-ca]

Updated PR to add support for python version specific requirement files. We now have a base requirements file, python 3.6 requirement file and a common requirement file for all other python versions. This can be extended to add more python version specific requirement files if they are needed.

[ahsan-ca]

I got feedback on my PR yesterday and I have addressed all the comments.
The new PR sets up different virtual envs for onnx backend and other tests. It also disables running onnx backend tests for python 3.6. The requirement file for backend onnx test has been separated. The PR with these changes is up for review again.

[ahsan-ca]

Re-opening as the main PR needs to be merged in.

[ahsan-ca]

This PR removed the try-except clause for importing numpy tests because the tests would silently exit if numpy was not found. It is preferable to have the tests fail if the numpy requirement is not satisfied, rather than silently exiting.