short jobs vs. long jobs
We have an issue with long jobs. If all workers are busy with long jobs, then no new job are launched.
- short jobs should have the priority over long jobs
- at least two worker slots should be reserved exclusively for short jobs
how to tell apart short and long jobs ?
- (smart solution) using the job inputs (files + parameters) do some machine learning to make an estimation of duration of the job and feed the data to a scheduler that understands priorities
- (quick solution) ask the user which queue to use (in the job submission form), and implement a job timeout according to the priority. Eg:
- interactive -> highest priority, max duration: 1mn
- standard -> normal priority, max duration: 10mn
- batch -> low priority, max duration: ?? hours
To upload designs, you'll need to enable LFS and have admin enable hashed storage. More information