Add an auxiliary HTTP server (allgo.aio) for serving asynchronous requests
There are two purposes:
- implement server push (using long-lived HTTP requests) for:
- sending status updates for the jobs and sandboxes
- live-streaming of the job logs
- have a really async implementation for pushing image manifests into the registry (the preliminary implementation in 5451a6df was blocking)
It is implemented with aiohttp.web (a lighweight HTTP framework, similar to what we can do with flask but asynchronously).
The alternative would have been to use the django channels plugin, but:
- it went through a major refactoring (v2) recently
- it requires replacing unicorn with an ASGI server (daphne)
- django-channels and daphne are not yet debian, and for the HTTP server i would prefer an implementation for which we have stable security updates
(anyway this can be ported to django-channels later)
The nginx config redirects the /aio/ path to this server (and the image manifests pushes too).
The allgo.aio server interacts only with the nginx, reddis and django containers. It relies totally on the django server for authenticating the user and for accessing the mysql db (so there is no ORM).
NOTE: in this design the django server has to trust blindly the requests coming from the allgo.aio server (for some endpoints). To prevent security issues, the nginx configuration is modified to set the header 'X-Origin: nginx'. Thus django knowns who he can trust.
This commits implements only the image pushs. Job updated and logs streaming will come in a later pull request.
The PR provides also the javascript (json_seq_event_listener()) and python (JsonSeqStreamResponse) primitives for streaming json objects in the application/json-seq format (RFC 7464).