[QA] Flaky API tests with Posix FS filesystem driver

[phil-davis]

We are running API test pipelines in reva that use the new PosixFS driver for storage.
That was introduced in PR cs3org/reva#4562

Today there were a few unexpected failures. When drone CI was restarted, they passed. So there are some "flaky" things. It might be timing between what the tests do, and how the PosixFS driver behaves when updating storage, etags, etc. in the back end. Or it might be???

This was noticed in reva PR cs3org/reva#4715

I have opened this issue as a place to mention unexpected test failures, and that will help us see if there is some pattern, and understand if we need to adjust test timing, or if there is some more consistent back-end problem or...

[phil-davis]

Some POSIX FS test scenarios failed unexpectedly. Example:
https://drone.owncloud.com/cs3org/reva/4694/34/7

      | spaces           | "C++ file.cpp" |
        Failed step: When user "Alice" uploads file with content "uploaded content" to "C++ file.cpp" using the WebDAV API
        TestHelpers\WebDavHelper::getPersonalSpaceIdForUser webdav propfind for user Alice failed with status 404 - so the personal space id cannot be discovered (Exception)

runsh: Total unexpected failed scenarios throughout the test run:
coreApiWebdavUpload/uploadFile.feature:53

[phil-davis]

Also there was:
https://drone.owncloud.com/cs3org/reva/4694/31/7

    Examples:
      | ocs-api-version | ocs-status-code |
      | 1               | 100             |
        Failed step: And uploading content to a public link shared file with password "%public%" should work using the new public WebDAV API
        Checking the content of the last public shared file after downloading with the new public WebDAV API
        The content was expected to be 'test new', but actually is 'test newata'. HTTP status was 200
        Failed asserting that two strings are equal.
        --- Expected
        +++ Actual
        @@ @@
        -'test new'
        +'test newata'

runsh: Total unexpected failed scenarios throughout the test run:
coreApiSharePublicLink1/createPublicLinkShare.feature:88

https://drone.owncloud.com/cs3org/reva/4694/33/7

      | new              |
        Failed step: And these etags should not have changed:
        WebDavPropertiesContext::theseEtagsShouldNotHaveChanged
        The etag '"169e21c9cdd598d3743b8a83802fe00e"' of element '/upload/file.txt' of user 'Alice' changed to '"a6a2af996d4739063f6ac0a1c4d51d20"'.
        Failed asserting that 1 matches expected 0.

runsh: Total unexpected failed scenarios throughout the test run:
coreApiWebdavEtagPropagation2/copyFileFolder.feature:116

[aduffeck]

I started to look into this problem end of last week. I could sometimes reproduce it locally at first, but unfortunately it no longer fails after I added some debug outbut, even after removing the debug output for whatever strange reason. Also my attempts to get helpful logs by running a shorter version of the pipeline (cs3org/reva#4716) did not yield that issue yet even after 30 runs. So... I'm still on it.

[phil-davis]

Thanks @aduffeck - hopefully we get a few more "random" test fails, and can see a pattern to give us a clue.

[aduffeck]

This should be fixed with the merge of cs3org/reva#4719