Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add benchmarks to test runs #2410

Open
wants to merge 17 commits into
base: gh/vmoens/22/base
Choose a base branch
from

Update

deef000
Select commit
Loading
Failed to load commit list.
Open

[CI] Add benchmarks to test runs #2410

Update
deef000
Select commit
Loading
Failed to load commit list.
PyTorch Bot / Dr.CI completed Sep 18, 2024 in 0s

Dr.CI classification results

{"FAILED":[{"workflowId":10911636220,"workflowUniqueId":110013764,"id":30284718257,"runnerName":"GitHub Actions 404","authorEmail":"[email protected]","name":"Wheels / build-wheel-windows (3.10, 3.10.3)","jobName":"build-wheel-windows (3.10, 3.10.3)","conclusion":"failure","completed_at":"2024-09-17T21:48:43.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636220/job/30284718257","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-17T21:48:32.000000000Z"},{"workflowId":10911636254,"workflowUniqueId":60501564,"id":30284706930,"runnerName":"i-0c8a370f4423e781b","authorEmail":"[email protected]","name":"Habitat Tests on Linux / tests (3.9, 12.1) / linux-job","jobName":"tests (3.9, 12.1) / linux-job","conclusion":"failure","completed_at":"2024-09-17T21:58:08.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636254/job/30284706930","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":["RuntimeError: Command docker exec -t e86b7c7e29fe1a7a46f7fc6c383a6b3b8e70b1130f8cca8732362d8d0b34197d /exec failed with exit code 134"],"failure_lines":["RuntimeError: Command docker exec -t e86b7c7e29fe1a7a46f7fc6c383a6b3b8e70b1130f8cca8732362d8d0b34197d /exec failed with exit code 134"],"failure_context":["+ bash /pytorch/rl/.github/unittest/linux_libs/scripts_habitat/run_test.sh","+ bash /pytorch/rl/.github/unittest/linux_libs/scripts_habitat/install.sh","+ bash /pytorch/rl/.github/unittest/linux_libs/scripts_habitat/setup_env.sh","+ cp /pytorch/rl/.github/unittest/linux_libs/scripts_habitat/10_nvidia.json /usr/share/glvnd/egl_vendor.d/10_nvidia.json","+ this_dir=/pytorch/rl/.github/unittest/linux_libs/scripts_habitat","++ pwd","++ cd .github/unittest/linux_libs/scripts_habitat","+++ dirname .github/unittest/linux_libs/scripts_habitat/run_all.sh","+ apt-get dist-upgrade -y","+ apt-get install -y g++ gcc","+ apt-get install -y libglvnd0 libgl1 libglx0 libegl1 libgles2","+ apt-get install -y libglfw3 libgl1-mesa-glx libosmesa6 libglew-dev"],"time":"2024-09-17T21:48:15.000000000Z"},{"workflowId":10911636267,"workflowUniqueId":82165822,"id":30284736898,"runnerName":"i-09a10d468660d3387","authorEmail":"[email protected]","name":"Unit-tests on Linux / tests-cpu (3.12) / linux-job","jobName":"tests-cpu (3.12) / linux-job","conclusion":"failure","completed_at":"2024-09-17T21:53:50.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636267/job/30284736898","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":["RuntimeError: Command docker exec -t fca59c26b9d8b0ab15bb60ea49929f0dfb1ca8af9d811d5ec95dd73c7a36bba9 /exec failed with exit code 1"],"failure_lines":["RuntimeError: Command docker exec -t fca59c26b9d8b0ab15bb60ea49929f0dfb1ca8af9d811d5ec95dd73c7a36bba9 /exec failed with exit code 1"],"failure_context":["+ /pytorch/rl/conda/bin/conda env update --file /pytorch/rl/.github/unittest/linux/scripts/environment.yml --prune","+ __conda_exe env update --file /pytorch/rl/.github/unittest/linux/scripts/environment.yml --prune","+ case "$cmd" in","+ local cmd=env","+ conda env update --file /pytorch/rl/.github/unittest/linux/scripts/environment.yml --prune","+ pip install virtualenv","+ pip3 install pip --upgrade","+ /pytorch/rl/conda/bin/conda env config vars set MUJOCO_GL=glfw PYOPENGL_PLATFORM=glfw DISPLAY=:0 SDL_VIDEODRIVER=dummy LAZY_LEGACY_OP=False","+ __conda_exe env config vars set MUJOCO_GL=glfw PYOPENGL_PLATFORM=glfw DISPLAY=:0 SDL_VIDEODRIVER=dummy LAZY_LEGACY_OP=False","+ case "$cmd" in","+ local cmd=env","+ conda env config vars set MUJOCO_GL=glfw PYOPENGL_PLATFORM=glfw DISPLAY=:0 SDL_VIDEODRIVER=dummy LAZY_LEGACY_OP=False"],"time":"2024-09-17T21:49:04.000000000Z"},{"workflowId":10911636267,"workflowUniqueId":82165822,"id":30284738333,"runnerName":"i-0333d1f811cdc4c99","authorEmail":"[email protected]","name":"Unit-tests on Linux / tests-cpu-oldget (3.12) / linux-job","jobName":"tests-cpu-oldget (3.12) / linux-job","conclusion":"failure","completed_at":"2024-09-17T21:53:54.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636267/job/30284738333","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":["RuntimeError: Command docker exec -t 94e16e5347ef67a702af142dcf42521e9fc5adfe510817f622adf86bc5656ee3 /exec failed with exit code 1"],"failure_lines":["RuntimeError: Command docker exec -t 94e16e5347ef67a702af142dcf42521e9fc5adfe510817f622adf86bc5656ee3 /exec failed with exit code 1"],"failure_context":["+ /pytorch/rl/conda/bin/conda env update --file /pytorch/rl/.github/unittest/linux/scripts/environment.yml --prune","+ __conda_exe env update --file /pytorch/rl/.github/unittest/linux/scripts/environment.yml --prune","+ case "$cmd" in","+ local cmd=env","+ conda env update --file /pytorch/rl/.github/unittest/linux/scripts/environment.yml --prune","+ pip install virtualenv","+ pip3 install pip --upgrade","+ /pytorch/rl/conda/bin/conda env config vars set MUJOCO_GL=glfw PYOPENGL_PLATFORM=glfw DISPLAY=:0 SDL_VIDEODRIVER=dummy LAZY_LEGACY_OP=False","+ __conda_exe env config vars set MUJOCO_GL=glfw PYOPENGL_PLATFORM=glfw DISPLAY=:0 SDL_VIDEODRIVER=dummy LAZY_LEGACY_OP=False","+ case "$cmd" in","+ local cmd=env","+ conda env config vars set MUJOCO_GL=glfw PYOPENGL_PLATFORM=glfw DISPLAY=:0 SDL_VIDEODRIVER=dummy LAZY_LEGACY_OP=False"],"time":"2024-09-17T21:49:06.000000000Z"},{"workflowId":10911636267,"workflowUniqueId":82165822,"id":30284738963,"runnerName":"i-03f1a2d6318100460","authorEmail":"[email protected]","name":"Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job","jobName":"tests-olddeps (3.8, 11.6) / linux-job","conclusion":"failure","completed_at":"2024-09-17T22:24:56.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636267/job/30284738963","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":["test/test_exploration.py::TestConsistentDropout::test_consistent_dropout[device0-True-0.5]"],"failure_lines":["FAILED test/test_exploration.py::TestConsistentDropout::test_consistent_dropout[device0-True-0.5] - TypeError: init() takes 1 positional argument but 2 were given"],"failure_context":["+ bash .github/unittest/linux_olddeps/scripts_gym_0_13/batch_scripts.sh","+ bash .github/unittest/linux_olddeps/scripts_gym_0_13/setup_env.sh","+ TD_GET_DEFAULTS_TO_NONE=1","+ export TD_GET_DEFAULTS_TO_NONE=1","+ TF_CPP_MIN_LOG_LEVEL=0","+ export TF_CPP_MIN_LOG_LEVEL=0","+ TORCH_VERSION=nightly","+ export TORCH_VERSION=nightly","+ RELEASE=0","+ export RELEASE=0","+ [[ refs/pull/2410/merge =~ release/* ]]","+ TAR_OPTIONS=--no-same-owner"],"time":"2024-09-17T21:49:07.000000000Z"}],"FLAKY":[],"BROKEN_TRUNK":[{"workflowId":10911636266,"workflowUniqueId":106193614,"id":30294295561,"runnerName":"GitHub Actions 437","authorEmail":"[email protected]","name":"Build Windows Wheels / pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cpu","jobName":"pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cpu","conclusion":"failure","completed_at":"2024-09-18T03:34:53.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636266/job/30294295561","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-18T03:34:17.000000000Z"},{"workflowId":10911636266,"workflowUniqueId":106193614,"id":30294295793,"runnerName":"GitHub Actions 177","authorEmail":"[email protected]","name":"Build Windows Wheels / pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cuda11_8","jobName":"pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cuda11_8","conclusion":"failure","completed_at":"2024-09-18T03:34:56.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636266/job/30294295793","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-18T03:34:17.000000000Z"},{"workflowId":10911636266,"workflowUniqueId":106193614,"id":30294295989,"runnerName":"GitHub Actions 304","authorEmail":"[email protected]","name":"Build Windows Wheels / pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cuda12_1","jobName":"pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cuda12_1","conclusion":"failure","completed_at":"2024-09-18T03:35:02.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636266/job/30294295989","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-18T03:34:18.000000000Z"},{"workflowId":10911636266,"workflowUniqueId":106193614,"id":30294296161,"runnerName":"GitHub Actions 220","authorEmail":"[email protected]","name":"Build Windows Wheels / pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cuda12_4","jobName":"pytorch/rl (pytorch/rl, .github/scripts/td_script.sh, .github/scripts/version_script.bat, python ... / upload / wheel-py3_9-cuda12_4","conclusion":"failure","completed_at":"2024-09-18T03:34:53.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636266/job/30294296161","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-18T03:34:18.000000000Z"},{"workflowId":10911636244,"workflowUniqueId":79519303,"id":30284713564,"runnerName":"i-0220d2b200be8fa8d","authorEmail":"[email protected]","name":"Unit-tests on Windows / unittests-cpu / windows-job","jobName":"unittests-cpu / windows-job","conclusion":"failure","completed_at":"2024-09-18T03:47:02.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636244/job/30284713564","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":["RuntimeError: Compiler: cl is not found."],"failure_lines":["RuntimeError: Compiler: cl is not found."],"failure_context":["+ pytest --junitxml=test-results/junit.xml -v --durations 200 --ignore test/test_distributed.py --ignore test/test_rlhf.py","+ python -m torch.utils.collect_env","+ LAZY_LEGACY_OP=False","+ export LAZY_LEGACY_OP=False","+ BATCHED_PIPE_TIMEOUT=60","+ export BATCHED_PIPE_TIMEOUT=60","+ MAX_IDLE_COUNT=60","+ export MAX_IDLE_COUNT=60","+ CKPT_BACKEND=torch","+ export CKPT_BACKEND=torch","++ [[ cpu != \c\p\u ]]","++ [[ 3 -eq 5 ]]"],"time":"2024-09-17T21:48:25.000000000Z"},{"workflowId":10911636220,"workflowUniqueId":110013764,"id":30284718660,"runnerName":"GitHub Actions 186","authorEmail":"[email protected]","name":"Wheels / build-wheel-windows (3.11, 3.11)","jobName":"build-wheel-windows (3.11, 3.11)","conclusion":"failure","completed_at":"2024-09-17T21:48:44.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636220/job/30284718660","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-17T21:48:33.000000000Z"},{"workflowId":10911636220,"workflowUniqueId":110013764,"id":30284719002,"runnerName":"GitHub Actions 416","authorEmail":"[email protected]","name":"Wheels / build-wheel-windows (3.12, 3.12)","jobName":"build-wheel-windows (3.12, 3.12)","conclusion":"failure","completed_at":"2024-09-17T21:48:43.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636220/job/30284719002","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-17T21:48:34.000000000Z"},{"workflowId":10911636220,"workflowUniqueId":110013764,"id":30284717837,"runnerName":"GitHub Actions 310","authorEmail":"[email protected]","name":"Wheels / build-wheel-windows (3.9, 3.9)","jobName":"build-wheel-windows (3.9, 3.9)","conclusion":"failure","completed_at":"2024-09-17T21:48:43.000000000Z","html_url":"https://github.com/pytorch/rl/actions/runs/10911636220/job/30284717837","head_branch":"gh/vmoens/22/head","pr_number":2410,"head_sha":"deef0007127a5cec1341b94c69b5433b1bc4512f","head_sha_timestamp":"2024-09-17T21:48:01.000000000Z","failure_captures":[],"failure_lines":[],"failure_context":[],"time":"2024-09-17T21:48:32.000000000Z"}],"UNSTABLE":[]}