CI | Investigation of Tests That Are Failing in the CI (Encryption tests, namespace caching)

[shirady]

Environment info

• NooBaa Version: master (5.18.0)
• Platform: CI

Actual behavior

1. We have 3 tests that used to be inconsistent in failing, and in that period, that were failing in the CI even after rerunning them.

Expected behavior

1. To have tests that always pass.

Steps to reproduce

1. You can revert the commit in the PR CI | Update Ceph S3 Tests + Ignore Failing Tests #8621, or you can remove the skip in the files test_encryption.js and test_namespace_cache.js and rerun it in the CI and make sure that it passed (a couple of times).

More information - Screenshots / Logs / Other output

2024-12-22T09:06:14.1390374Z   4 failing
2024-12-22T09:06:14.1395851Z
2024-12-22T09:06:14.1415872Z   1) namespace caching: read scenarios that object is cached
2024-12-22T09:06:14.1416497Z        read from hub if read from cache fails:
2024-12-22T09:06:14.1416872Z
2024-12-22T09:06:14.1417262Z       AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2024-12-22T09:06:14.1417634Z
2024-12-22T09:06:14.1417856Z   assert(read_obj_stream_time > params.object_md.cache_last_valid_time)
2024-12-22T09:06:14.1418181Z
2024-12-22T09:06:14.1418279Z       + expected - actual
2024-12-22T09:06:14.1418444Z
2024-12-22T09:06:14.1418542Z       -false
2024-12-22T09:06:14.1418729Z       +true
2024-12-22T09:06:14.1418908Z
2024-12-22T09:06:14.1419235Z       at Context.<anonymous> (src/test/unit_tests/test_namespace_cache.js:730:9)
2024-12-22T09:06:14.1419813Z       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-12-22T09:06:14.1420672Z
2024-12-22T09:06:14.1421902Z   2) Encryption tests
2024-12-22T09:06:14.1422346Z        Check master keys in system
2024-12-22T09:06:14.1422698Z          create cloud pools succefully:
2024-12-22T09:06:14.1423008Z      Target already in use
2024-12-22T09:06:14.1423249Z
2024-12-22T09:06:14.1423344Z
2024-12-22T09:06:14.1427915Z   3) Encryption tests
2024-12-22T09:06:14.1428349Z        Check master keys in system
2024-12-22T09:06:14.1428806Z          update connections succefully - accounts + pools:
2024-12-22T09:06:14.1429242Z      TypeError: Cannot read properties of null (reading 'cloud_pool_info')
2024-12-22T09:06:14.1429702Z       at /root/node_modules/noobaa-core/src/test/unit_tests/test_encryption.js:430:40
2024-12-22T09:06:14.1430377Z       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-12-22T09:06:14.1430752Z       at async Promise.all (index 4)
2024-12-22T09:06:14.1431078Z       at async Context.<anonymous> (src/test/unit_tests/test_encryption.js:417:13)
2024-12-22T09:06:14.1431739Z
2024-12-22T09:06:14.1463857Z   4) Encryption tests
2024-12-22T09:06:14.1464236Z        Check master keys in system
2024-12-22T09:06:14.1464637Z          delete pools succefully:
2024-12-22T09:06:14.1465051Z      No such pool: email4-cloud-pool

Attached the log file from the CI:
0_run-unit-tests.txt