separate job inputs and outputs in different dirs
having all files in the same directory (inputs, outputs and allgo.log) is an issue:
- it is cumbersome when implementing an app that processes all submitted files (without having to give the file names in the job parameters), because the developer has to filter the allgo.log from the list of files
- the UI cannot distinguish between input and output files and present separately
- if a job crashes or if we want to put a worker node down, the controller need to reset the job dir to the initial state to be able to reschedule the job
A possible idea is to:
- put input files in
<DATASTORE>/<JOB_ID>/in
and make it read-only (for the job) - put output files in
<DATASTORE>/<JOB_ID>/out
This would break the allgo API. So if we want to do it, we should rather do it in Q3.