job.state.priority: remove raising exception when no aux item found

[cmoussa1]

Scenario

A job held in PRIORITY and waiting to be allocated resources gets reprioritized and sent back to job.state.priority or job.priority.get after a restart (clearing its aux items) when a plugin is already loaded and seen the job.

Problem

The multi-factor priority plugin raises an exception on a job in job.state.priority when it cannot find the aux item containing the association information for the job. However, when a system instance is reloaded, the aux items set on a job are cleared. When jobs are reprioritized and are sent back to job.state.priority or job.priority.get, the plugin will raise an exception on the job because it can't unpack the aux item containing the flux-accounting information associated with that job. See the below eventlog for an example:

...
{"timestamp":1737671575.5923743,"name":"priority","context":{"priority":19668}}
{"timestamp":1737736372.7067723,"name":"priority","context":{"priority":19620}}
{"timestamp":1737757970.5174642,"name":"priority","context":{"priority":19743}}
{"timestamp":1737801156.5343764,"name":"priority","context":{"priority":19865}}
{"timestamp":1737822769.2824628,"name":"priority","context":{"priority":19160}}
{"timestamp":1737998525.3946743,"name":"exception","context":{"type":"mf_priority","severity":0,"note":"internal error: bank info is missing","userid":767}}
{"timestamp":1737998525.3947248,"name":"clean"}

---

This PR removes raising a job exception in job.state.priority in the case where the job does not have an aux item for the accounting information associated with the job. Instead, it attempts to perform another lookup for the flux-accounting information for the association that submitted the job, which is already what it does when a job is submitted before the plugin is loaded with any flux-accounting information.

Fixes #575

[cmoussa1]

@grondo I believe I've found the source of those job failures I was telling you about a few days ago with the jobs that failed to load the Association aux item after a system instance restart (I wrote up a description with my thoughts in #575). I'm in the process of trying to add a sharness test for this case but I can't seem to figure out how to simulate this scenario. I've added a [WIP] commit to this PR to get me started, but I could use some advice on how to actually execute the restart and send the pending job back to PRIORITY with the aux items cleared. Do you have any advice or are there any examples that I can look at in flux-core for this?

[grondo]

You can take a look at t3200-instance-restart.t which uses a series of flux start commands to simulate restarts.
O/w, you could try just reloading the job-manager module and restart the flux-accounting python service to see if that reproduces the original issue.

[cmoussa1]

OK, I believe I have a working test file now to reproduce the behavior I was describing in #575, so I've dropped [WIP] from that commit.

[grondo]

This LGTM and seems to be a nice solution! Just one comment suggestion inline.

[grondo]

Suggestion: Amend the comment to indicate why the aux item may not be set or the bank info is missing. This may help a future developer. E.g. "This could be due to either ..."

[cmoussa1]

Good suggestion, thanks! I've just force-pushed up an amendment to this comment as well as referenced RFC 21 for the job state diagram; the image in that RFC is super helpful.

[cmoussa1]

Thanks for reviewing this @grondo! I'll set MWP here