multi-factor priority plugin: associate jobs with projects

[cmoussa1]

It was mentioned briefly in #247 to add "project" support to both the flux-accounting database and the multi-factor priority plugin. The first part is completed (by PR #255); now it is time to work on the latter. To recap, here is a quick summary of the functionality introduced by #255:

There now exists projects and default_project column for every association in the association_table where projects can be listed for associations to submit jobs under. Below is a quick example:

$ flux account add-user --username=user1234 --bank=bank_A --projects="project_A,project_B,project_C"

projectA will become the association's default project (since it was the first project listed). A user's default project can also be edited with the edit-user command:

$ flux account edit-user user1234 --default_project=project_B

If an association is added to the association_table without specifying any projects, a default project (*) is added for the association automatically. Every association who is added to the association_table belongs to the * project, but will only have their default project set to * if they do not already have another default project.

The latter half of adding basic project support is to be able to communicate this information to the multi-factor priority plugin, where it can be validated and set during job submission.

---

The Requirement

It was mentioned some in discussion in #255 about the need to associate jobs with projects that have arbitrary start and end dates. It will be calculated from archived jobs or something along those lines. At the very least, there is a requirement to associate jobs in the job-archive DB with a project name.

The Implementation

It should be fairly straightforward to send project data (both the projects that each association belongs to as well as the projects listed in the project_table) to the multi-factor priority plugin, where it can be unpacked and stored in an internal data structure. From there, jobs submitted with a project name can have it validated in job.validate, and jobs that specify an invalid project name or a project they do not belong to will be rejected with an appropriate rejection message.

Jobs that do successfully associate with a project, on the other hand, will need to keep that data somewhere in the job record. What comes to mind at first is keeping this data in jobspec. If a project is specified on the command line, for example, then it can be found in the attributes.system section of the jobspec:

"atttributes": {
  "project": "projectA"
}

But if a project is not specified on the command line (e.g an association is using their default project), then I think the plugin should be able to annotate the jobspec with the project name. Similar to the idea brought up in #272, perhaps I can use jobspec-update to add the project name to the jobspec. I played with this some yesterday afternoon, and I believe I was able to successfully post an update, as shown from a job's eventlog:

{"timestamp": 1666106606.9670131,"name":"jobspec-update","context":{"attributes.system.project":"projectA"}}

However, like it is stated in RFC 21, this only affects the Flux instance's view of the job, so when you fetch the jobspec via flux job info $my_job jobspec, it does not show up under "attributes.system". But, I think as long as the key-value pair shows up in the job record somehow in the job-archive, that should be satisfactory. Do we know if that is the case?

[cmoussa1]

This should now be closed by #443 as projects should now be annotated with an association's jobs. I can re-open more specific issues if need be.