Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not grade empty groups #215

Merged
merged 3 commits into from
May 1, 2023

Conversation

ghamerly
Copy link
Contributor

@ghamerly ghamerly commented Jan 6, 2023

Fixes #189

This causes verifyproblem to give a verdict of AC with score of 0 to any empty group, rather than calling the grader on an empty group. It also changes the way the default_grader reports a judge error (reporting a score as well).

These changes are mostly only useful for use in verifyproblem with the -d flag, which can filter out test files so that groups appear empty.

@niemela
Copy link
Member

niemela commented Jan 7, 2023

Could this ever happen without using -d? I.e. is there something explicitly disallowing empty groups? If not, it might be "more correct" to have the verdict be AC with accept_score instead?

@ghamerly
Copy link
Contributor Author

ghamerly commented Jan 7, 2023

@niemela There was previously nothing in verifyproblem that reported a warning or an error if a test case group was empty (actually empty, not just filtered out with -d). With the commit I just added (4e6179e), I have made that situation an error. If you think it's better treated as a warning, I'm open to that.

As far as using accept_score instead of 0 as the default score (when a test group is empty, due to filtering), we can do that easily (I have written but not committed the code that would do that). However, I'm not sure that the semantics are quite correct to apply it this way, because accept_score is used on each test case, not on each test group. Thoughts?

Instead, return the default verdict (AC) and score (0).

Empty groups should only occur when using -d to filter out some data; we
may want to warn about real empty groups somewhere else.
The specification says that the verdict must come with a score. Even
though a score does not make sense for a judge error verdict, this
sticks with the specification, and prevents the caller (verifyproblem)
from crashing.
@pehrsoderman pehrsoderman merged commit 9ecb800 into Kattis:develop May 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Default grader now crashes on empty test groups with mode min
3 participants