Smoketest: bundle notification fails (flaky test)

[chmreid]

Am seeing an error happening in the smoketest during the "integration test" stage on the master branch on Allspark. The error happens while checking that a bundle UUID that was added matches the bundle UUID in a notification. This error was triggered by an unrelated change (a pull request that was just updating a docstring in the swagger API (#2682).

Link to Allspark job: https://allspark.dev.data.humancellatlas.org/HumanCellAtlas/data-store/-/jobs/58611

/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-bundles-all --replica gcp --per-page 500 --prefix f62ced5c --no-paginate
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-event --replica aws --uuid f62ced5c-de55-438b-812b-cbd74ccc4ea8 --version 2019-12-05T203643.903513Z
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-events --replica aws --per-page 10 --from-date 2019-12-05T203643.903513Z --no-paginate
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-event --replica gcp --uuid f62ced5c-de55-438b-812b-cbd74ccc4ea8 --version 2019-12-05T203643.903513Z
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-events --replica aws --per-page 10 --from-date 2019-12-05T203643.903513Z --no-paginate
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-bundle --uuid f62ced5c-de55-438b-812b-cbd74ccc4ea8 --version 2019-12-05T203643.903513Z --reason 'smoke test' --replica aws
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-bundle --uuid f62ced5c-de55-438b-812b-cbd74ccc4ea8 --version 2019-12-05T203643.903513Z --reason 'smoke test' --replica gcp
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-bundles-all --replica aws --per-page 500 --no-paginate
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss get-bundles-all --replica gcp --per-page 500 --no-paginate
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss post-search --es-query='{}' --replica gcp --no-paginate > /dev/null
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid 0a903c6f-5790-46b8-955b-1f534ef9f1f7 --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid 4d34bb5d-22ec-4a58-bb3c-88bdf9266cb6 --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid 0121131e-1c9a-4dc5-882f-b384ba6194a0 --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid da5a4f71-db49-44a1-a6cc-b34d78de55ed --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid c7c3d1c8-5e84-4cd5-bae0-b3bdde71524c --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid 69ee46b9-a77f-4f14-a1ed-a580d0f3af7f --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid 891c98f3-94ed-4009-bf56-867d342aecbf --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid f9772810-37d2-4843-9019-5e106968c092 --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid a92ad0f7-6719-4cda-9eb3-2d0597ad3bcd --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid eef06ac2-40ef-4ba3-9211-4e322b76c0cd --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid 5cced67d-2f0a-4cc6-96ca-75f6336b5367 --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica gcp --uuid 4d9f5ed8-c29c-4391-9029-8b556c67d196 --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid ba565336-731f-4c1e-b96a-827a1f203b04 --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid 8fef0fda-afde-4788-a7f5-1dd1eeb0ba63 --subscription-type jmespath
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid 92fa0f71-aae0-429f-9660-aa1baca32bf8 --subscription-type elasticsearch
/builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp/venv/bin/hca dss delete-subscription --replica aws --uuid 6603fe84-0272-412a-84a6-5fce3ac571a1 --subscription-type elasticsearch
cleaning up: /builds/HumanCellAtlas/data-store/smoketest-30wygj4v.tmp

======================================================================
FAIL: test_smoketest (__main__.Smoketest) [aws: Check the notifications. gcp, a92ad0f7-6719-4cda-9eb3-2d0597ad3bcd, notifications/71e5bb16-0511-4201-93a1-de2afd8aa90c]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_smoketest.py", line 139, in smoketest
    self.assertEqual(bundle_uuid, notification['match']['bundle_uuid'])
AssertionError: '49a6c4d9-a04d-4b4c-ad23-a9ad79595d8c' != '519021d8-d47c-4c4b-ae15-66b5d9f7eb3a'
- 49a6c4d9-a04d-4b4c-ad23-a9ad79595d8c
+ 519021d8-d47c-4c4b-ae15-66b5d9f7eb3a


----------------------------------------------------------------------
Ran 1 test in 198.982s

FAILED (failures=1)
Makefile:45: recipe for target 'tests/test_smoketest.py' failed
make[1]: Leaving directory '/builds/HumanCellAtlas/data-store'
Makefile:55: recipe for target 'integration_test' failed
make[1]: *** [tests/test_smoketest.py] Error 1
make: *** [integration_test] Error 2

[chmreid]

Job succeeded when re-run, so this one can be filed under flaky tests

[amarjandu]

I wonder if this is caused by a bundle creation event in the DSS at the same time as a smoketest, perhaps we can do some jmespath filtering on the query to specify which UUID subscribe to....