plugin: add node count tracking for an association's running jobs #442
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: The Association class has no member to track an association's current node count across all of their running jobs. Add a new member called "cur_nodes" which represents an association's current node count across all of their running jobs. Add a default value in the case where we are creating a special temporary association while the plugin waits for flux-accounting data to be loaded in. Add cur_node values to the accounting unit tests as well as to the expected output of the plugin.query tests.
Problem: The priority plugin needs a way to count the number of nodes an association is using for their job. Add two new helper functions to the priority plugin. extract_nodelist () will be responsible for extracting the "nodelist" key-value pair out of a job's R. process_nodelist () is responsible for counting the number of nodes from this nodelist by using flux-core's hostlist library. Include flux-core's hostlist library in the plugin and in the plugin's Makefile.
Problem: The priority plugin does not increment the cur_nodes count when an association's job is about to run. Add an increment of an association's cur_nodes count by the number of nodes their job is going to use in the callback for job.state.run.
Problem: The priority plugin does not decrement the cur_nodes count when an association's job is finished running. Add a decrement of an association's cur_nodes count by the number of nodes their job used in the callback for job.state.inactive.
Problem: flux-accounting has no tests for keeping track of the number of nodes an association is using across all of their running jobs. Add some tests that track the number of nodes an association is using while they submit jobs.
cmoussa1
added
new feature
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
The priority plugin has no way to track the number of nodes an association is using across all of their running jobs.
This PR looks to add basic node count tracking to the priority plugin, specifically in the callbacks for
job.state.runandjob.state.inactive.Two new helper functions are added to the plugin to help do this -
extract_nodelist ()andprocess_nodelist ().extract_nodelist ()will look for the"nodelist"key-value pair andprocess_nodelist ()will take that string and use flux-core's hostlist library to count the number of nodes. Then, thecur_nodescount for the association is incremented by the number of nodes returning byprocess_nodelist (). A similar process is followed in thejob.state.inactivecallback where the number of nodes returned byprocess_nodelist ()is decremented from the association's total current nodes count.A basic set of tests are also added to simulate submitting a couple of different-sized jobs and ensuring that the priority plugin can accurately keep track of them as they are submitted and finish running.
Note that this PR does not look to add actual enforcement of an association's
max_nodeslimit, but rather just looks to track thecur_nodesnumber for an association. Once this looks good, I can create a follow-up PR that tries to add the actual enforcement of a node count limit.