Push job output into a file and then read that file

[rowleya]

The JobManager currently uses a JobOutputPipe to forward the output log from a job to a local file, simultaneously storing it in a file for later debugging. Instead, the output should be pushed to a file directly (i.e. using the equivalent of > outputfile.txt) and then this file should be read dynamically to keep track of the output. This would then allow the job to continue to run even if the JobManager should disappear.

[dkfellows]

I'm not sure that that will work by itself. The issue is that the job is also part of the process group of the parent, so that if the parent is killed (the usual reason for death) the child is also killed. Subprocess management in Java is pretty primitive…

[rowleya]

I think the subprocess does continue to run from my experience (maybe this is Windows feature - worth a quick test to verify). This is actually the preferred option since I would like to shut down the JobManager to do updates to it whilst leaving the child process running if at all possible (i.e. assuming no interface changes are made that make it impossible for the child to talk to the parent after resuming).

As it now stands, the parent and child processes can run on different machines; there are two executors here, the XenVMExecutor where the actual process runs on a VM separate from the parent JobManager; and a LocalExecutor that runs the process locally. The needs of the XenVmExecutor mean that the log file is live-uploaded to the JobManager during the running of the process in general. If this option is disabled, the log is instead pushed to the JobManager at the end. In either case though the JobManager is contacted through a socket. The JobOutputPipe might therefore be obsolete in any case.