Troubleshooting knowledge base and tool

[squaremo]

From @simongottschlag in #flux-contributors:

When someone says “X doesn’t work” and it’s caused by something outside of flux, do you gather all these issues under a specific tag or something? It would be nice to build a small extension/tool/test or something that is possible to use to easily troubleshoot issues.

I’m looking at an issue right now where an end use seems to have both issues accessing the GitHub API as well as a token that has weird characters in it.
Would be really sweet to ask someone who reports an issue to “Please run: fluxctl test XYZ” or something like that and it’s easy to add tests for external access, token verification etc. Would be a small investment each time a new test or verification is added but would make life so much easier when helping end users figure out that an issue isn’t Flux related.

I think it would be nice to have a separate knowledge base and perhaps a small CLI that have a separate release cadence than the full suite that can be updated when needed.
Knowledge base that explains issues that we've seen and how to identify them as well as a tool that we either directly do the tests with.

Could be flux-kb and fluxkbcli and when troubleshooting something you could run:

fluxkbcli validate notification-controller

or

fluxkbcli validate github

And each kb should have a specific command that can be used to test it. After a few months, perhaps a year, we should have a large enough knowledge base and test suite to be able to have most people be able to solve non-flux-issue without actually creating an issue.

But would be an investment (time) to begin with but would most likely save lots of time in the future.

It should be really easy to find out stuff like:

• Do I have access to GitHub from a POD?
• Does my token work with statuses for the repos I have?
• Push a test status to GitHub.

etc.

[squaremo]

I summarised this elsewhere as: "build a knowledge base of common flux setup gotchas, and a tool that automates the troubleshooting where it can be done".

One thing I like about this suggestion is that it can be pursued almost entirely in parallel with development of the flux controllers, CLI, and so on. A bit of co-ordination might be in order -- for example, printing error codes in flux that the troubleshooting tool knows how to process (either to fetch more information, or to do further checks).

There's an argument that the CLI for this should just be flux; I think it changes little if the knowledge base and perhaps a library are kept separate, and integrated into flux as a subcommand. The main advantage is gained from having the knowledge base grow independently, and flux is released often enough that automated checks can be incorporated and distributed fairly quickly.

I love the idea, implicit in the original suggestion, that the tooling has access to the "live" knowledge base -- e.g., to the git repo where all the advice is kept.

[simongottschlag]

One reason to have a separate cli would be that requirements for it wouldn't be as high as the normal cli since it's only purpose would be for troubleshooting issues.

Since it may require lots of dependencies they wouldn't be needed for the normal cli and it would be possible for it to be much larger.