Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print formatted results to file #55

Closed
emichael opened this issue Jul 11, 2023 · 0 comments · Fixed by #56
Closed

Print formatted results to file #55

emichael opened this issue Jul 11, 2023 · 0 comments · Fixed by #56
Labels
enhancement New feature or request

Comments

@emichael
Copy link
Owner

emichael commented Jul 11, 2023

All of the current grading infrastructure is based on the output from stderr/stdout. This is suboptimal for a couple reasons. First, it means that the output must be parsed by grading scripts, which is error prone and subject to potential mischief. Second, stderr/stdout necessarily contain all of students' logging statements. Asking students to disable all logging before submitting has historically not had a high success rate, and sufficiently verbose logging can result in very large outputs of test runs.

There should be a JUnit test listener which, when a global flag is set, logs results as they happen and at the end of a test run outputs them to a file in a structured format (probably JSON). This output should optionally contain a copy of stdout/stderr, which can be obtained with the TeeStdOutErr utility. I'm not sure what the default should be.

Both of these configuration options should probably be accessible through run_tests.py. Students might want to use this test output themselves.

Ultimately, some sort of schema for the test output would be really useful. It would likely evolve over time, but at least we would have something that grading scripts (in this repo and developed by other instructors for their use cases) could reference.

@emichael emichael added the enhancement New feature or request label Jul 11, 2023
@emichael emichael linked a pull request Jul 11, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant