increase isolation of job containers
to improve the security (jobs inputs cannot be trusted and the webapp developers are unreliable), we should use all the sandboxing capabilities available in docker
jobs should be run
- with no network (--net=none)
- with no capabilities (--cap-drop=all)
- as an ordinary user (currently they are run as the default user, which is very likely 'root')
Additionally, on the worker nodes the docker filesystem (/var/lib/docker) should be mounted with 'nosuid' (to prevent becoming root event inside the container).
This will very likely break several apps, so it is better to do it in Q3.
Note: if we remove the network support, some users will complain because they used the network to implement real-time progress report. While it is already possible to stream app progress using the 'allgo.log' (which is now streamed), it would be better to provide a standard way to report job progress (the allgo UI will benefit from it too).