Signaling jobs before their allocation expires

[SteVwonder]

How do we want to support the functional equivalent of Slurm's --signal=sig_num>[@<sig_time>] argument, which is defined as:

When a job is within sig_time seconds of its end time, send it the signal sig_num.

Related work: libyogrt

[SteVwonder]

My initial thought here is to create a job shell plugin that is in charge of setting up a timer callback and then signalling the job processes at the appropriate time. This information could be specified in jobspec via a user attribute. Maybe attributes.user.signal?

Strawman example:

- attributes:
    user:
      signal:
        time: 60s
        number: USR1

Where time is an FSD, and number is either an int or one of the POSIX signals without the SIG prefix.

[SteVwonder]

A slight alternative could be:

- attributes:
    user:
      signal-time-remaining:
        when: 60s
        how: USR1

[grondo]

Yeah, a job shell plugin makes sense here.

However, I would say all arguments to job shell plugins should go under attributes.system.shell.options not under attributes.user (I had thought the attributes.user dict was supposed to be ignored by all Flux components)

[SteVwonder]

Doh!!! You are totally right. Sorry about that 🤦 . sed s/user/system.shell/g

[grondo]

If we use attributes.system.shell.options.signal, and steal Slurms sig@time format, then users could trivially set this option without adding a new option to the flux-mini utilities, e.g.

 $ flux mini submit -o signal=SIGUSR1@60s ...

However, generic shell options set in this way have the drawback of no error checking until a solution for calling the job shell in "validate" mode is implemented.