Minimal changes for multi-pass problem support. #2725
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vmcj
reviewed
Sep 28, 2024
|
I don't think you significantly reduce logic branching with inheritance as you still need to check the type somehow and do different things in the different cases. Additionally, the resulting SQL tables would basically have the same columns except for one or two attributes which is ugly IMHO. |
vmcj
reviewed
Sep 29, 2024
| )] | ||
| #[Assert\GreaterThan(0)] | ||
| #[Serializer\Exclude] | ||
| private ?int $multipassLimit = null; |
There was a problem hiding this comment.
I think the code becomes much easier if we already set this to 2 (like the comment implies). Now this is only a constant which we use in other parts of the code.
It would lose the information if a problem uses the default value but I don't think we'll need that often in practice?
There was a problem hiding this comment.
Fair point, this was leaking info into other classes. I changed just the comment here and the getter below to do the right thing.
Note that this does not include support for interactive multi-pass problems. Currently, we only upload the result of the last pass of each test case run, which we should obviously fix in the long run but it's easier to do in separate changes. General overview of multi-pass problems: A multi-pass problem requires a validator and that validator is in charge of deciding whether another pass is being run. In that case, the file `feedbackdir/nextpass.in` will contain the input of the next pass. The problem specification includes an upper limit of passes (most often 2), and if the validator generates more passes than the limit, it is treated as internal error. Progress towards DOMjudge#2307. Tested with RagnarGrootKoerkamp/BAPCtools#393: ``` [Sep 28 09:49:05.007] judgedaemon[301583]: Judge started on goo-0 [DOMjudge/8.4.0DEV/3c8c16477] [Sep 28 09:49:05.007] judgedaemon[301583]: 🔏 Executing chroot script: 'chroot-startstop.sh check' [Sep 28 09:49:05.015] judgedaemon[301583]: Registering judgehost on endpoint default: http://localhost/domjudge/api [Sep 28 09:49:05.194] judgedaemon[301583]: No submissions in queue (for endpoint default), waiting... [Sep 28 09:49:22.049] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:22.050] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/100/118 [Sep 28 09:49:22.051] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:22.194] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:22.578] judgedaemon[301583]: 💻 Compilation: (absolute.py) 'correct' [Sep 28 09:49:22.646] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:22.647] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:22.882] judgedaemon[301583]: ✔ ...done in 0.075s (CPU: 0.030s), result: correct [Sep 28 09:49:22.882] judgedaemon[301583]: 🔄 Running pass 2... [Sep 28 09:49:23.117] judgedaemon[301583]: ✔ ...done in 0.044s (CPU: 0.026s), result: correct [Sep 28 09:49:23.196] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:23.196] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/101/119 [Sep 28 09:49:23.196] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:23.217] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:23.370] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:23.675] judgedaemon[301583]: 💻 Compilation: (deterministic.py) 'correct' [Sep 28 09:49:23.745] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:23.745] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:23.923] judgedaemon[301583]: ✔ ...done in 0.042s (CPU: 0.026s), result: correct [Sep 28 09:49:23.923] judgedaemon[301583]: 🔄 Running pass 2... [Sep 28 09:49:24.102] judgedaemon[301583]: ✔ ...done in 0.044s (CPU: 0.025s), result: correct [Sep 28 09:49:24.174] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:24.175] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/102/120 [Sep 28 09:49:24.175] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:24.193] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:24.336] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:24.708] judgedaemon[301583]: 💻 Compilation: (rand.py) 'correct' [Sep 28 09:49:24.780] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:24.780] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:25.039] judgedaemon[301583]: ✔ ...done in 0.129s (CPU: 0.094s), result: correct [Sep 28 09:49:25.039] judgedaemon[301583]: 🔄 Running pass 2... [Sep 28 09:49:25.304] judgedaemon[301583]: ✗ ...done in 0.120s (CPU: 0.091s), result: wrong-answer [Sep 28 09:49:25.734] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:25.734] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/103/121 [Sep 28 09:49:25.734] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:25.951] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:26.271] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:26.778] judgedaemon[301583]: 💻 Compilation: (no_output.py) 'correct' [Sep 28 09:49:26.846] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:26.847] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:27.092] judgedaemon[301583]: ✗ ...done in 0.055s (CPU: 0.025s), result: no-output [Sep 28 09:49:27.298] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:27.298] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/104/122 [Sep 28 09:49:27.298] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:27.319] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:27.454] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:27.763] judgedaemon[301583]: 💻 Compilation: (string.py) 'correct' [Sep 28 09:49:27.836] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:27.836] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:28.012] judgedaemon[301583]: ✗ ...done in 0.068s (CPU: 0.027s), result: wrong-answer [Sep 28 09:49:28.225] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:28.225] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/105/123 [Sep 28 09:49:28.225] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:28.246] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:28.381] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:28.727] judgedaemon[301583]: 💻 Compilation: (negative.py) 'correct' [Sep 28 09:49:28.803] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:28.803] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:29.045] judgedaemon[301583]: ✗ ...done in 0.089s (CPU: 0.056s), result: wrong-answer [Sep 28 09:49:29.250] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:29.250] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/106/124 [Sep 28 09:49:29.250] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:29.270] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:29.425] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:29.761] judgedaemon[301583]: 💻 Compilation: (negative_second_pass.py) 'correct' [Sep 28 09:49:29.833] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:29.833] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:30.034] judgedaemon[301583]: ✔ ...done in 0.080s (CPU: 0.057s), result: correct [Sep 28 09:49:30.034] judgedaemon[301583]: 🔄 Running pass 2... [Sep 28 09:49:30.252] judgedaemon[301583]: ✗ ...done in 0.082s (CPU: 0.060s), result: wrong-answer [Sep 28 09:49:30.446] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:30.446] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/107/125 [Sep 28 09:49:30.446] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:30.467] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:30.612] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:30.923] judgedaemon[301583]: 💻 Compilation: (extra_output.py) 'correct' [Sep 28 09:49:30.992] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:30.992] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:31.170] judgedaemon[301583]: ✔ ...done in 0.055s (CPU: 0.025s), result: correct [Sep 28 09:49:31.170] judgedaemon[301583]: 🔄 Running pass 2... [Sep 28 09:49:31.366] judgedaemon[301583]: ✗ ...done in 0.060s (CPU: 0.028s), result: wrong-answer [Sep 28 09:49:31.569] judgedaemon[301583]: ⇝ Received 1 'judging_run' judge tasks (endpoint default) [Sep 28 09:49:31.570] judgedaemon[301583]: Working directory: /home/sitowert/domjudge/output/judgings/goo-0/endpoint-default/108/126 [Sep 28 09:49:31.570] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:31.590] judgedaemon[301583]: 🔒 Executing chroot script: 'chroot-startstop.sh start' [Sep 28 09:49:31.725] judgedaemon[301583]: 📋 Verifying versions. [Sep 28 09:49:32.062] judgedaemon[301583]: 💻 Compilation: (float.py) 'correct' [Sep 28 09:49:32.132] judgedaemon[301583]: 🏃 Running testcase 7... [Sep 28 09:49:32.132] judgedaemon[301583]: 🔄 Running pass 1... [Sep 28 09:49:32.327] judgedaemon[301583]: ✗ ...done in 0.043s (CPU: 0.022s), result: wrong-answer [Sep 28 09:49:32.522] judgedaemon[301583]: 🔓 Executing chroot script: 'chroot-startstop.sh stop' [Sep 28 09:49:32.541] judgedaemon[301583]: No submissions in queue (for endpoint default), waiting... ```
vmcj
approved these changes
Sep 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note that this does not include support for interactive multi-pass problems.
Currently, we only upload the result of the last pass of each test case run, which we should obviously fix in the long run but it's easier to do in separate changes.
General overview of multi-pass problems:
A multi-pass problem requires a validator and that validator is in charge of deciding whether another pass is being run. In that case, the file
feedbackdir/nextpass.inwill contain the input of the next pass. The problem specification includes an upper limit of passes (most often 2), and if the validator generates more passes than the limit, it is treated as internal error.Progress towards #2307.
Tested with RagnarGrootKoerkamp/BAPCtools#393: