Fix ambiguous warning around languages that are "not recommended" for output validators

[xiaowuc1]

Though verifyproblem may seem to properly support languages like Java, the production instance of Kattis does not. This was observed in https://open.kattis.com/problems/sequenceguessing. The warning that is emitted is unclear about why the language is not recommended, so either Kattis should maintain parity with verifyproblem and the warning is dropped, or the warning is promoted to an error.

[niemela]

The problem format specification does not say anything about what languages may be used for validators. For this reason we made this a warning, and not an error. I would argue that this is technically correct simply because using Java for validators is allowed by the spec, but I agree that it is very unhelpful :(.

Maybe we should add system specific "strict" flags. E.g. "--kattis" which would among other things treat validator in Java as error. What do you think?

There are other limits that are not specified in the spec, e.g. the size of input, maximum run time and similar things. Reasonable people might disagree on what these limits should be, so it's hard to pick global limits, but the flags suggested above could be allowed to be more opinionated.

[xiaowuc1]

Based on your last paragraph, I sense that the thing to do is make warning messages generally more helpful, and possibly to set "reasonable" limits in the places you mention and issue warnings when those are exceeded.

Therefore, the validator warning might instead say "Java is not recommended because it may not be supported by your CCS" which invites the user to verify whether it will work properly or not - this permits people who are using a CCS where a Java validator would work to implement their validator in Java even if some other CCS would not support it.

I do not like the idea of adding flags, if only because the source of truth (what the CCS does in its production instance) might not actually be whatever verifyproblem claims to be, and keeping those in sync sounds like it could be problematic, especially if folks are running different versions of some CCS.

[niemela]

possibly to set "reasonable" limits in the places you mention and issue warnings when those are exceeded.

Well, the problem is that "reasonable limits" wary by quite a lot. E.g. having input data as large as, say, 10GB is really no issue if you're just running a single contest, but it quickly add up if you have thousands of problems on an online judge (such as Kattis). So what is a reasonable maximum input size? 100MB, 10GB? I can see arguments for both of those (and maybe even beyond, in both directions. That's several orders of magnitude.

I do not like the idea of adding flags, if only because the source of truth (what the CCS does in its production instance) might not actually be whatever verifyproblem claims to be

But that would at least allow us to communicate expectations. My assumptions is that every system that want a flag to decide what that flag should do. I.e. we would only add "--kattis" is Kattis asks for it (and I do think that would be useful for us at Kattis, since users could then know earlier about specific requirements such as this), and we would only add "--domjudge" if DOMjudge ask for it.

Honestly, I think it's way more likely that there would be such additional requirements for an online judge/SaaS system, since many (all?) of the limits we would set depends more on the underlying system than the CCS, it's just that in the online judge/SaaS case those are connected. Also, see below.

keeping those in sync sounds like it could be problematic, especially if folks are running different versions of some CCS.

But it's actually even worse. Many (all?) of the actual incompatibilities we've run into in the past has been language versions (not really CCS version, they tend to be very stable). If you're sample solutions are written in C++20, but the CCS is set up with C++17 it might fail. Problemtools provides a few Docker images with "reasonable" language defaults, but that's obviously not guaranteed to be what you actually want.