test_publish_and_manipulate and test_get_dandiset_published fail locally

[yarikoptic]

This is with

• Add an option to dump content of Audit table from Docker env upon completion of testing #1484
  state (0.63.0-6-g782e959b) but I don't think it should relate anyhow since that PR is only for the finalization of the session fixture, so shouldn't effect individual tests

============================================================================= short test summary info ==============================================================================
FAILED dandi/tests/test_dandiapi.py::test_publish_and_manipulate - ValueError: Dandiset 000008 is Pending: {
FAILED dandi/tests/test_dandiapi.py::test_get_dandiset_published[True] - ValueError: Dandiset 000021 is Pending: {
FAILED dandi/tests/test_dandiapi.py::test_get_dandiset_published[False] - ValueError: Dandiset 000022 is Pending: {
FAILED dandi/tests/test_dandiapi.py::test_get_dandiset_published_no_version_id[True] - ValueError: Dandiset 000023 is Pending: {
FAILED dandi/tests/test_dandiapi.py::test_get_dandiset_published_no_version_id[False] - ValueError: Dandiset 000024 is Pending: {
=================================================== 5 failed, 707 passed, 1 skipped, 2 xfailed, 8 xpassed in 1119.58s (0:18:39) ====================================================
DANDI_TESTS_AUDIT_CSV=/tmp/audit-full.csv python -m pytest -v dandi/  153.27s user 13.15s system 14% cpu 18:41.56 total

@jwodder - any immediate ideas?

I have

❯ docker images | grep dandi
dandiarchive/dandiarchive-api                        latest                         5bc326d92640   19 hours ago    749MB
dandiarchive/dandiarchive-api                        <none>                         cff5f21b9a7c   2 months ago    734MB

[jwodder]

@yarikoptic Those tests involve publishing a Dandiset, and it seems that the publication did not complete in time. What happens if you rerun the tests?

[yarikoptic]

they were and still are (on master) failing consistently on my attempts. Full log http://www.oneukrainian.com/tmp/pytest-fail-20240815.txt

I also note that there is 100% CPU celery (not sure what could keep it that busy) - do you (or may be @jjnesbitt or @mvandenburgh ) have an idea what it could be doing or how to figure it out?

celery container logs show that it completes its jobs in milliseconds

[2024-08-15 15:51:30,720: INFO/MainProcess] Task dandiapi.api.tasks.calculate_sha256[47f0c15a-7d49-4208-aeda-9c8e01c18980] received: ((UUID('a33573ab-acc0-4d14-b823-91afa51b82b2'),), {})
[2024-08-15 15:51:30,736: INFO/ForkPoolWorker-2] dandiapi.api.tasks.calculate_sha256[47f0c15a-7d49-4208-aeda-9c8e01c18980]: Calculating sha256 checksum for asset blob a33573ab-acc0-4d14-b823-91afa51b82b2
[2024-08-15 15:51:30,742: INFO/ForkPoolWorker-2] Task dandiapi.api.tasks.calculate_sha256[47f0c15a-7d49-4208-aeda-9c8e01c18980] succeeded in 0.02144717297051102s: None
[2024-08-15 15:51:30,909: INFO/MainProcess] Task dandiapi.api.tasks.calculate_sha256[b4f586df-dadf-4841-ae26-a7e733c4dd60] received: ((UUID('5bdd712a-04cc-40ac-b73f-98f614743b27'),), {})
[2024-08-15 15:51:30,910: INFO/ForkPoolWorker-2] dandiapi.api.tasks.calculate_sha256[b4f586df-dadf-4841-ae26-a7e733c4dd60]: Calculating sha256 checksum for asset blob 5bdd712a-04cc-40ac-b73f-98f614743b27
[2024-08-15 15:51:30,914: INFO/ForkPoolWorker-2] Task dandiapi.api.tasks.calculate_sha256[b4f586df-dadf-4841-ae26-a7e733c4dd60] succeeded in 0.005075494991615415s: None
[2024-08-15 15:53:31,583: INFO/MainProcess] Task dandiapi.api.tasks.calculate_sha256[11a11275-94c1-40ff-adec-e87f012e4baa] received: ((UUID('091711e2-af2e-44f5-8fec-85eb49ce1ff9'),), {})
[2024-08-15 15:53:31,585: INFO/ForkPoolWorker-2] dandiapi.api.tasks.calculate_sha256[11a11275-94c1-40ff-adec-e87f012e4baa]: Calculating sha256 checksum for asset blob 091711e2-af2e-44f5-8fec-85eb49ce1ff9
[2024-08-15 15:53:31,588: INFO/ForkPoolWorker-2] Task dandiapi.api.tasks.calculate_sha256[11a11275-94c1-40ff-adec-e87f012e4baa] succeeded in 0.004132542992010713s: None
[2024-08-15 15:53:31,758: INFO/MainProcess] Task dandiapi.api.tasks.calculate_sha256[9170e368-2588-4999-a1c4-4f2c6ce910ef] received: ((UUID('3b7be16d-38e1-4f8f-8046-36936890eac8'),), {})
[2024-08-15 15:53:31,759: INFO/ForkPoolWorker-2] dandiapi.api.tasks.calculate_sha256[9170e368-2588-4999-a1c4-4f2c6ce910ef]: Calculating sha256 checksum for asset blob 3b7be16d-38e1-4f8f-8046-36936890eac8
[2024-08-15 15:53:31,763: INFO/ForkPoolWorker-2] Task dandiapi.api.tasks.calculate_sha256[9170e368-2588-4999-a1c4-4f2c6ce910ef] succeeded in 0.004479025024920702s: None
[2024-08-15 15:53:31,938: INFO/MainProcess] Task dandiapi.api.tasks.calculate_sha256[2da50b13-a501-4881-ad8d-0ccd1c6e1f8c] received: ((UUID('1e95e5b4-8fe2-4f6a-8d99-9fbf86646a72'),), {})
[2024-08-15 15:53:31,939: INFO/ForkPoolWorker-2] dandiapi.api.tasks.calculate_sha256[2da50b13-a501-4881-ad8d-0ccd1c6e1f8c]: Calculating sha256 checksum for asset blob 1e95e5b4-8fe2-4f6a-8d99-9fbf86646a72
[2024-08-15 15:53:31,943: INFO/ForkPoolWorker-2] Task dandiapi.api.tasks.calculate_sha256[2da50b13-a501-4881-ad8d-0ccd1c6e1f8c] succeeded in 0.004295543010812253s: None