Can't build or test locally

[odersky]

Compiler version

3.6 nightly

Minimized example

clean
scala3-bootstrapped/clean
scaladoc/clean
scaladoc-js-main/clean
scaladoc-js-main/test

Output

[error] error while loading scaladoc,
[error] TASTy file /Users/odersky/workspace/dotty/scaladoc-js/out/bootstrap/scaladoc-js-common/scala-3.6.0-RC1-bin-SNAPSHOT-nonbootstrapped/classes/dotty/tools/scaladoc/CodeSnippets.tasty could not be read, failing with:
[error]   Backward incompatible TASTy file has version 28.5-experimental-1, produced by Scala 3.6.0-RC1-bin-SNAPSHOT-nonbootstrapped-git-0cf25db,
[error]   expected stable TASTy from 28.0 to 28.5, or exactly 28.6-experimental-1.
[error]   The source of this file should be recompiled by a later version.
[error]   Usually this means that the classpath entry of this file should be updated.
[error]   Please refer to the documentation for information on TASTy versioning:
[error]   https://docs.scala-lang.org/scala3/reference/language-versions/binary-compatibility.html

Expectation

No Tasty conflict. Note also that it claims that the version it has 28.6-experimental-1 is among the versions that it expected, yet there's still an error.

This is a blocker for me to debug the test failures in my latest commit. Without this being fixed we cannot expect any forward progress in fixing issues.

[hamzaremmal]

So far, I'm not able to reproduce this locally 😥

[odersky]

I cleaned my out directory and that fixed it. It seems there were artefacts from previous runs that were not cleaned by scala3-boostrapped/clean. Probably worth checking why that happens.

The original error is now gone on my side as well. But I still get the following test failure:

[error] All tests passed after retry, indicating that some tests are flaky.
[error] This build is configured to reject flaky tests.
[error] Failed: Total 661, Failed 0, Errors 0, Passed 661, Ignored 7
[error] (scala3-compiler-bootstrapped / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1488 s (24:48), completed 18 Sep 2024, 14:32:30

Do we know which tests are flaky and can we do something about it?

Even so, it's workable now, so removing the blocker status.

[hamzaremmal]

Do we know which tests are flakey and can we do something about it?

I believe that this error message was introduced by the new develocity plugin. We do publish reports in https://develocity.scala-lang.org/ so maybe some extra information can be found there (/cc @adpi2 )

[adpi2]

Do we know which tests are flakey and can we do something about it?

I also think this message comes from the Develocity plugin. Probably you should see the failing tests in the logs, if you scroll back enough. Or maybe they are flagged as flaky in the test report.

@dotta do you have an example of sbt test report with flaky tests?

[adpi2]

I cleaned my out directory and that fixed it. It seems there were artefacts from previous runs that were not cleaned by scala3-boostrapped/clean. Probably worth checking why that happens.

The corrupted TASTy file was in the scaladoc-js-common project. To clean it you should do scaladoc-js-common/clean.

[adpi2]

Also it's worth noting there were no flaky test in the entire history of https://develocity.scala-lang.org/. So it could be some changes locally or some corrupted state that cause the flakiness in your build @odersky.

[hamzaremmal]

Also it's worth noting there were no flaky test in the entire history of https://develocity.scala-lang.org/. So it could some changes locally or some corrupted state that cause the flakiness in your build @odersky.

Both @odersky and I had this message locally. For me, it was on a "fresh" clone of the repository (after running git clean -xfd).

[hamzaremmal]

I also think this message comes from the Develocity plugin. Probably you should see the failing tests in the logs, if you scroll back enough. Or maybe they are flagged as flaky in the test report.

No test failed for me locally, only this message appears at the end.

[adpi2]

No test failed for me locally, only this message appears at the end.

Can you share the logs?

[hamzaremmal]

scala3.log

I've just rerun the scala3-bootstrapped/test command now and it shows two errors + the fact that some tests are flaky. I'm wondering if it's something that we have missed in the previous run.

[hamzaremmal]

The previous run actually had the same errors, we just missed them 😅. I'm now wondering now why that the flakiness of that test wasn't detected before introducing develocity; last change was 3 months ago.

[dotta]

Do we know which tests are flakey and can we do something about it?

I also think this message comes from the Develocity plugin. Probably you should see the failing tests in the logs, if you scroll back enough. Or maybe they are flagged as flaky in the test report.

@dotta do you have an example of sbt test report with flaky tests?

@adpi2 Would you have a link to the specific build scan that was published when the output in this message was produced?

Asking because I had a quick look at build scans with test failures, and I noticed the second execution seems to be skipped in some cases (see this build scan for instance), and I wonder if that might explain why the test isn't marked flaky. But I'd prefer to have a look at the specific build scan as there may be a good reason why the build scan I was looking at is the way it is.

Btw, see https://docs.gradle.com/enterprise/sbt-plugin/#test_case_details_page for an example of a flaky test reported in the test case details page. And here you can see an actual build scan from the apache pekko project with a flaky test execution reported (beware though that the linked page will eventually disappear due to the data retention window).