[EPIC] Compliance Monitor: improve architecture and quality of life, add features

[mbuechse]

The following sections are to keep notes about ideas for future developments; they will each be turned into issues before work starts.

Improve architecture

Quoting @martinmo:

The software engineer in me suggests to introduce a better layering of the application (or separation, however wou want to call it). At the moment, the business logic is tangled with all the technical details of request parsing, auth stuff and database operations

Of course, I totally agree -- now that we have a running deployment (which goes a long way to show that the approach is viable), some real structure is in order!

• Compliance Monitor: improve architecture #663
• Reduce inter-template redundancy #738

Quality of life improvements

• adding new test subjects is rather tedious and error prone
  • we have to create ssh keys, encrypt them (as well as application credentials) via zuul-cli, change multiple config files, restart the service etc.
  • among other things, I propose to use a single ssh key for all test subjects that run on our Zuul instance
• currently, the service has to be restarted to reload static config
  • I propose to add an endpoint (or a signal handler for SIGHUP) that can be used to tell the service to reload static config (bootstrap.yaml as well as the certificate-scope yaml files)

Tasks:

• Compliance Monitor: use the same ssh key for all test subjects that run on our Zuul instance #664
• Compliance Monitor: test all subjects with but one Zuul job #665
• Compliance Monitor: allow reconfiguration without restart #666

Introduce badges

The table currently lists all versions of the scope that the subject complies with, instead of showing a badge; we should convert this list into a badge as follows:

• if an effective version is in the list: badge green
• otherwise: if an obsolete (not yet deprecated) version is in the list: badge yellow
• otherwise: badge red

Maybe the list is not sufficient to compute the badge, because it doesn't tell us anything about versions that could have been complied with, but for some reason, some test results are missing. Then the question is: why are they missing? If the test couldn't complete due to unreliable infrastructure, then it's a bug on the end of the test subject, and the badge can as well be red. If, however, the test couldn't complete due to a programming error on our end, then we better use the grace period of 7 days to fix it, and then a grey badge would also be wrong. So, I think we can actually proceed with the list-to-badge-conversion as outlined above.

Add Prometheus exporter

(if there is demand?)

• see https://prometheus.io/docs/instrumenting/writing_exporters/
• for some reason, I wanted to have a look at this Python file (just putting it here for myself in case I need it later)