Implement Kebechet Managers SLO/SLI metrics

[xtuchyna]

Problem statement

Related to #560 (section Managers SLO/SLI)

As a maintainer,
I would like to have a look at dashboard with all Kebechet metrics

so that I can see which managers are used and performing well, which needs attention (slow responding time, critical number of rejected requests, and etc.) and which managers are not used (so even if they need attention, they can be of a low-priority).

Acceptance criteria

Create following metrics

• Manager name
• Manager % usage
  (comparisson to the overall number of repositories that have kebechet installed - if a manager has at least one reference in repository history, count it as a usage)

For

• thoth advise manager
• version manager
• Store SLI/SLO related metrics on ceph #603
• test metrics for 2 or 3 kebechet repositories
• include example metrics file in PR

Extension

• Extend SLI/SLO metrics for other managers #604

and also

• Have a discussion on what other SLI/SLO metrics should be added, add all of the approved metrics here after

UPDATE after discussion (https://docs.google.com/document/d/1jz4j07Q8gkv9H-23BgOp_4qN3uI29JsuP-_kTKEj-e8/edit#bookmark=id.66cutwjvs6o2):

Additional SLI metrics:

• work with usage count as with timestamps ( ? either each usage timestamp individually or a binned aggregation - e.g. monthly )
• changes to the source code (e.g. update and version managers), furthermore for specific managers specific files
  (e.g. update manager and contributor comparison - how much did bot commit to the Pipfile in opposed to how much did contributor)

Additional SLO metrics:

• response time (objective is to have some desired mean response time over all repositories)
• response rate (e.g. objective is to have at least X % repositories satisfying response time under defined threshold Y seconds)
• completion rate (ratio between created and closed by bot)
• overall usage (e.g. update manager has objective to be used at least in X repositories)
• repository specific (e.g. repo that has some changes in last month in source code should use version / update manager at least X times in month)
• compare release frequency with version manager usage

[goern]

@xtuchyna ping

[xtuchyna]

@goern have a look at #574 (comment) 🙂

[xtuchyna]

@goern @Gkrumbach07 so currently the update advise and version managers are implemented (look at PR to see data files example). I suggest splitting the Issue so we can track the features better. WDYT?

[goern]

@xtuchyna ja, pls go ahead and split this issue

[goern]

/sig user-experience

[Gkrumbach07]

Condensing issue due to inactivity
The following metrics for version, update, and advise managers are added:

• advise
  • repository_usage_count
  • total_source_code_lines_changed_by_bot
  • percent_source_code_changes_by_bot
• version
  • repository_usage_count
• update
  • repository_usage_count
  • total_source_code_lines_changed_by_bot
  • percent_source_code_changes_by_bot
• repositories_missing_issue_metric
• repositories_missing_pull_request_metric

closing this issue and propagating uncompleted issues to parent EPIC
/close

[sesheta]

@Gkrumbach07: Closing this issue.

In response to this:

Condensing issue due to inactivity
The following metrics for version, update, and advise managers are added:

• advise
  • repository_usage_count
  • total_source_code_lines_changed_by_bot
  • percent_source_code_changes_by_bot
• version
  • repository_usage_count
• update
  • repository_usage_count
  • total_source_code_lines_changed_by_bot
  • percent_source_code_changes_by_bot
• repositories_missing_issue_metric
• repositories_missing_pull_request_metric

closing this issue and propagating uncompleted issues to parent EPIC
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.