Results from symlinked tests are also reused even though output validator flags are different. That is definitely incorrect behaviour.

Agreed, this is not what we want.

@austrin @ghamerly Do you see any reason why it would be dangerous to fix this?

I cannot think of a reason why this should be prohibited. This is a question for @simonlindholm who wrote the code originally, I think.

Looking at the code, it seems to me that these symlink checks were added for some quite particular use case, where one also wanted the optimization of re-using results. So, it makes some sense that someone who wanted that particular use cases added a guard (the error) to catch when results were re-used dangerously.

We should definitely fix this. I'm somewhat inclined to simply remove the optimization of re-using results when test cases are identical and I/O-files are symlink:ed - it feels like a very corner-case scenario for the code complexity. If we do want to keep it, we should change what are now error messages to instead simply disable the re-use of results.

It is very much not a corner case scenario; it would at least double judging times for scoring problems with test groups, where we more often than not include earlier test groups in later ones.