Updating process of environment deployment
Update environments deployments
For the continuous integration process, we need to do some slight improvement to our deployment process :
- Create a process of dynamic review environments deployment
- Add a deployment of MUTE that uses Netflux, and a deployment of MUTE that uses Pulsar
Dynamic review environments
Whenever we do a Merge Request, there is no other way for the developer to simply add its version of the app to the server.
The idea in this Merge Request is to remove the chore that would be manually modifying the app, the reverse proxy configuration on the server, all in all to make the app accessible online.
Actually, the precise idea is that, in the case of any merge request, the review environment would be automatically setup, without the need of the developer's input (Unless the review process needs to be changed, IE : a strong change on MUTE)
What are the changes that were needed to make the process viable.
App Side :
- Created an environment file named :
environment.review.ts
. This file is like the other environment file except the host address is dynamic.- When we build the docker image, we create a json file named
reviewEnv.json
that stores the review branch name as a string following this structure :{ "reviewBranch": "review-name-of-the-branch" }
- In the MUTE code, the file is read and the review branch name is retrieved. It is then assigned to the
signalingHost
variable which concatenate the current review branch to-signaling.mute.loria.fr
.
- When we build the docker image, we create a json file named
- Every review environment should then be listed like this :
review-name-of-the-branch.mute.loria.fr
as MUTE app andreview-name-of-the-branch-sigver.mute.loria.fr
- The process is now fully automatized. Developer has no need to modify the file except if he made significant changes to the signaling system.
Gitlab-CI :
- Created a script (
conf/scripts/find-available-review-ports.sh
) that searches for available port on the host machine, starting from port 8040 - Added a
build_review_environments_docker_image
job that :- calls the
.prepare_review_variables_publish
job who creates thereviewEnv.json
file that contains the current review branch's name - creates an image for MUTE and Sigver tagged with the current git branch and pushes them to the gitlab container registry.
- Publishing the sigver image with the current review branch as it facilitates the server administration
- calls the
- Added a
deploy_to_review
job that :- gets available review environment port (by calling the
find-available-review-ports.sh
script) and puts these value in a dynamically createdreview.env
file - deploy the review environment container on the host machine
- gets available review environment port (by calling the
What's left to do :
-
Stop and delete the container/ the image on the host once the merge request has been successfully merged (see scripts/remove-review-env.sh
- tested manually) -
Delete remaining review environment image in the container registry (We will use a cleanup policy as the process using gitlab API is too complex)
Edited the cleanup policy to keep tags that match this regex(^v{1}.*$)|(libp2p)|(netflux)
and remove tags that match this regex(^review-.*$)
every week.
Reverse Proxy configuration :
We need to be able to dynamically add/remove domain configuration to our reverse proxy. This process should be, like the process defined before, automatically triggered. The idea is that the reverse proxy should be updated when a new review environment docker container starts running on the server. The reverse proxy should also have the review environment removed from its configuration once the merge request has been accepted.
What can we do with the API
We can achieve this by updating the Caddy configuration using its API.
(We can get the current Caddy structure by typing curl "http://localhost:2019/config/" | jq
, giving us a json representation of the reverse proxy configuration)
Using one of the already defined handle
in the file, we can use the same configuration by slightly modifying some value, principally the address where the reverse proxy should listen to, and also the name of the host. We will be using an id for each configuration to make the process of deleting easier
We could update the configuration via its API, by calling a script during the deployment job in gitlab-ci.
Adding a new review environment to our reverse proxy
Adding a new review environment to the reverse proxy is fairly simple. The script is called at the very end of the pipeline.
It verify if a configuration has already been set on the proxy. We do this verification because the script will be called each time a commit is pushed on a merge request. If no configuration has been set, then we add it to caddy via its API. The code is visible in scripts/add-review-env-proxy.sh
.
Removing a review environment
Removing the review environment is a seemingly much harder task. We have to update the reverse proxy configuration by removing the review environment, but we also have to stop and remove the containers and image from the machine, and also remove the environment from the gitlab UI.
This deletion script should be called either every time or manually when we push a commit on main.
There is two steps in the deletion job via Gitlab-CI, called stop_review_environment
, in the conf/gitlab-ci/deploy.gitlab-ci.yml
file :
- The job is called when the review environment is stoppped (as defined in the
on_stop
attribute of thedeploy_to_review_environment
job). The review environment can be stopped in two ways : either the merge request is merged, either the user manually stops the environment in the Gitlab UI. Either way, the environment will be effectively closed. - the
scripts/remove-review-env.sh
file is called, stopping the docker containers and updating the reverse proxy configuration by removing the review environment
What's left to do :
-
Add a configuration on the reverse proxy dynamically on the host ( scripts/add-review-env-proxy.sh
) -
Delete the configuration once the merge request has been successfully merged (see scripts/remove-review-env.sh
- tested manually)
Deployment of MUTE with netflux or Pulsar
-
Deployment of MUTE using netflux
Deploying a legacy environment of MUTE that uses netflux when a push on main is made
Deploying an environment of MUTE that uses Pulsar is postponed as other issues are prioritized. This task (#38 (closed)) will be tackled in the future.
This merge request is linked to issue #36 (closed) and #33 (closed) (and subsequent tasks)