Use GitHub hooks to post a comment with error logs from CI jobs, for posterity

[jchavarri]

I wanted to check a CI report from a few months back but the logs are not available anymore (the commit exists).

I understand it's impossible to keep logs forever, but maybe the CI job could post a summary of the errors encountered using GitHub hooks so the opam-repository PR will keep it as part of the history?

[mseri]

Ping @shonfeder, should we move the issue to the CI infrastructure repo?

[shonfeder]

Sure :) This is also a question for the opam repository itself tho, if the suggestion is to start posting the errors as comments. That could create quite a lot of noise potentially, I'd think.

[mseri]

I would see it more like a manual github bot that answers to a request to save all or part of the error logs, or a button on the CI to say "publish error summary on github" and sends a comment (but I can see risk of abuse of this)

[shonfeder]

I'm not opposed to using github issues as a persistent storage mechanism, But perhaps we can take a step back from implementation ideas and start by describing the problem to be solved.

@jchavarri could you elaborate on

I wanted to check a CI report from a few months back but the logs are not available anymore (the commit exists).

to help explain a bit what motivated you look for a quite old CI log and how you'd hoped to use it?

[jchavarri]

Sure. I was triaging some opam installation error reported by a user of an opam plugin (here) in a particular platform (macOS) and I wanted to see if somehow these errors had appeared in CI at the time one of the packages involved (opam-solver.2.1.6) had been published.

In hindsight, now that I'm explaining it, it doesn't make a lot of sense (if opam CI was failing, then the package wouldn't have been published) 😓 so I'm not sure if this issue makes sense, let me know if you think it is worth pursuing.

[shonfeder]

Thank for the context! It seems like there is definitely a clear need motivating the request, and I'm happy to keep at the discovery until we get a clear articulation of the need and how we might best meet it.

Given the context, I'm unsure whether the kind of utility @mseri is describing would have been useful, because you would have known at that the time that you would need these logs months later.

if opam CI was failing, then the package wouldn't have been published

Ideally, this is how it would work, but the CI is not yet reliable enough for us to always practice the discipline we should (and will soon): things sometimes get merged in without a fully passing status for one reason or another. That said, if all you wanted to check was whether the CI passed on the platform before merging, I think you're right that the root fix here is just for us to get the CI reliable and then improve our merging discipline so that packages can ONLY be merged when the CI is green.

That said, I suspect there may also be a need for

1. Persistent storage of logs beyond our current bounds.
2. An easier way of surveying and discussing error results.

My main point of uncertainty is whether using comments on github is the right interface for meeting those needs.