Stream job logs and job state updates to the user
Note: this PR depends on !58 (merged) and !59 (merged), you should review them first (the actual changes are in 1bb4acf4 if you want to review it immediately)
This PR makes several changes.
In the controller:
-
duplicates the logs produced by the jobs. Initially they were only stored into allgo.log, now they are also forwarded to the container output (using the 'tee' command) so that the controller can read them
-
add a log_task that reads the logs from docker and feeds them into the redis db key "log:job:ID" (this is implemented with aiohttp in order to be fully asynchronous)
-
store the job state in a new redis key "state:job:ID"
-
send notification to the redis pubsub 'notify:aio' channel when the job state has changed or when new logs are available
In the allgo.aio frontend:
- implement the /aio/jobs/ID/events endpoints which streams all job events & logs to the client (using json formatted messages)
In django:
- refactor the JobDetail view and template to update the page
dynamically for job updates (state/logs)
- allgo.log is read only when the job is already terminated. Otherwise the page uses the /aio/jobs/ID/events channel to stream the logs
- the state icon is patched on the page when the state changes, except for the DONE state which triggers a full page reload (because there are other parts to be updated)