all pending jobs killed after Flux update

[grondo]

After a flux-core update to v0.58.0, all the pending jobs on fluke were killed with an exception from the mf_priority plugin:
e.g.

[root@fluke108:~]# flux job eventlog -H f2m1t75LN7vo
[Jan05 18:00] submit userid=61494 urgency=16 flags=0 version=1
[  +0.018063] jobspec-update attributes.system.bank="guests"
[  +0.018097] jobspec-update attributes.system.bank="guests"
[  +0.018144] validate
[  +0.034171] depend
[  +0.034206] priority priority=625
[Jan08 08:20] flux-restart
[  +0.000034] exception type="mf_priority" severity=0 note="not a member of guests"
[  +0.000062] priority priority=16
[  +0.000083] clean

The user, testqe, is in the guests bank though:

$ flux account view-bank --users guests | grep testqe
testqe            61494             guests            1                 175899119.293700070.00625           100                                 

[cmoussa1]

Without trying to construct a reproducer (yet), I believe this type of exception message is raised when a job's user/bank information is being updated in job.state.priority and the plugin cannot find a valid user/bank entry for the user that this job is submitted under:

flux-accounting/src/plugins/mf_priority.cpp

Lines 676 to 679 in 1a84215

	if (bank_it == it->second.end ()) {
	flux_jobtap_raise_exception (p, FLUX_JOBTAP_CURRENT_JOB,
	"mf_priority", 0,
	"not a member of %s", bank);

Looking at the timestamps of the eventlog, it looks like the job was:

1. submitted successfully (and received a priority)
2. Flux was restarted
3. the priority plugin was loaded
4. jobs were reprioritized before the plugin received any flux-accounting data, so it rejected this job and presumably all pending jobs.

I'll see if I can reproduce this behavior.

[cmoussa1]

OK, I actually think I was able to reproduce this without having to restart Flux and instead just unloading/reloading the plugin. If I have a number of jobs in SCHED state (i.e they've received a priority and are waiting to run) and I reload the plugin without updating it with flux-accounting information and call reprioritize on all jobs:

(flux.Flux().rpc("job-manager.mf_priority.reprioritize"))

they will have an exception raised on them saying that the plugin cannot find a valid user/bank entry for those previously held jobs.

Is there a process in restarting Flux where jobs could be reprioritized? My thinking is that might have been what caused this.

In any case, the plugin should probably handle this case more gracefully so a bunch of users' jobs don't get canceled if Flux gets restarted.

I'll need to test this, but off the top of my head, I think I can include a check in the callback for job.state.priority that checks the plugin's internal map for data before deciding what to do with the job going through reprioritization.

If the plugin's internal map is empty (i.e it is waiting for flux-accounting information), it can continue to hold the job in PRIORITY until it loads some information. This would be similar to the behavior in the callback for job.validate.

[grondo]

Nice job debugging @cmoussa1!

All jobs are prioritized any time a jobtap plugin is loaded, so I had assumed this would happen after mf_priority.so is loaded.

[grondo]

Closed by #407?

[cmoussa1]

Ah, yes, should be closed by #407 - sorry that I didn't close this yesterday. Closing now