A GitLab upgrade is scheduled for Monday, May 12, 2025. The service will be unavailable for a few minutes in the morning. We'll keep you posted on the progress of the upgrade on the Mattermost channel: https://mattermost.inria.fr/devel/channels/gitlab. We recommend that you do not work on the platform until an announcement indicates that maintenance is complete.
" 💡 The `run_command` function is polymorphic in its `roles` parameter. You can pass a `Roles` object as return by `provider.init`, a list of `Host` or a single `Host`. Check its documentation using SHIFT+TAB for instance.\n",
"There are some specific shortcuts when the remote actions is a remote (shell) command: `.stdout`, `.stderr`, `.rc`"
"<div class=\"alert alert-info\">\n",
"💡 There are some specific shortcuts when the remote actions is a remote (shell) command: `.stdout`, `.stderr`, `.rc`\n",
"</div>"
]
},
{
...
...
@@ -272,8 +312,10 @@
"id": "operating-testimony",
"metadata": {},
"source": [
"By default the user is `root` (this is common to all EnOSlib's provider).\n",
"If you want to run command as your regular Grid'5000 user you can tell the command to `sudo` back to your regular user using `run_as` (the SSH login is still `root` though)"
"<div class=\"alert alert-info\">\n",
"💡 By default the user is `root` (this is common to all EnOSlib's provider).\n",
"If you want to run command as your regular Grid'5000 user you can tell the command to `sudo` back to your regular user using `run_as` (the SSH login is still `root` though)\n",
"</div>"
]
},
{
...
...
@@ -283,6 +325,7 @@
"metadata": {},
"outputs": [],
"source": [
"# get my username on g5k (this line is generic and useful if you share your code with someone else)\n",
Make sure you've run the one time setup for your environment
<ul>
<li>⚠️ Make sure you've run the one time setup for your environment</li>
<li>⚠️ Make sure you're running this notebook under the right kernel</li>
</ul>
</div>
%% Cell type:code id:funded-azerbaijan tags:
``` python
importenoslibasen
# Display some general information about the library
en.check()
# Enable rich logging
_=en.init_logging()
```
%% Cell type:markdown id:familiar-intention tags:
## Setup on Grid'5000
EnOSlib uses `Providers` to ... provide resources. They transform an abstract resource configuration into a concrete one.
To do so, they interact with an infrastructure where they get the resources from. There are different providers in EnOSlib:
- Vbox/KVM to work with locally hosted virtual machines
- Openstack/Chameleon to work with bare-metal resources hosted in the Chameleon platform
- FiT/IOT lab to work with sensors or low profile machines
- VmonG5k to work with virtual machines on Grid'5000
- Distem to work with lxc containers on Grid'5000
-**Grid'5000**, with options to configure several networks easily
A providers eases the use of the platform by internalizing some of the configuration tasks (e.g automatically managing the reservation on G5k, network configuration ...)
### Describing the resources
For the purpose of the tutorial we'll reserve 2 nodes in the production environment.
First we build a configuration object describing the wanted resources: `machines` and `networks`.
We can pass the `Configuration` object to the `G5k` provider.
%% Cell type:code id:alike-immigration tags:
``` python
provider=en.G5k(conf)
roles,networks=provider.init()
```
%% Cell type:markdown id:asian-stretch tags:
Inspecting the ressources we own for the experiment's lifetime:
- roles: this is somehow a dictionnary whose keys are the role names and the associated values are the corresponding list of hosts
- networks: similar to roles but for networks
%% Cell type:code id:sunset-voluntary tags:
``` python
roles
```
%% Cell type:code id:regional-procedure tags:
``` python
# list of host on a given role
roles["control"]
```
%% Cell type:code id:nonprofit-excess tags:
``` python
# a single host
roles["control"][0]
```
%% Cell type:code id:latest-affiliate tags:
``` python
networks
```
%% Cell type:markdown id:danish-circle tags:
`provider.init` is idempotent. In the Grid'5000 case, you can call it several time in a row. The same reservation will reloaded and the roles and networks will be the same.
%% Cell type:code id:furnished-angle tags:
``` python
roles,networks=provider.init()
roles
```
%% Cell type:code id:blocked-marker tags:
``` python
# sync some more information in the host data structure (for illustration purpose here)
💡 The `run_command` function is polymorphic in its `roles` parameter. You can pass a `Roles` object as return by `provider.init`, a list of `Host` or a single `Host`. Check its documentation using SHIFT+TAB for instance.
By default the user is `root` (this is common to all EnOSlib's provider).
<divclass="alert alert-info">
💡 By default the user is `root` (this is common to all EnOSlib's provider).
If you want to run command as your regular Grid'5000 user you can tell the command to `sudo` back to your regular user using `run_as` (the SSH login is still `root` though)
</div>
%% Cell type:code id:surprising-junior tags:
``` python
# get my username on g5k (this line is generic and useful if you share your code with someone else)
print("This is printed, so the execution can continue")
```
%% Cell type:markdown id:extreme-carol tags:
### Remote actions
Tools like Ansible, Puppet, Chef, Terraform ... are shipped with a set of predefined remote actions to ease the administrator life.
Actions like copying file, adding some users, managing packages, making sure a line is absent from a configuration file, managing docker containers ... are first-class citizens actions and brings some nice garantees of correctness and idempotency.
EnOSlib wraps Ansible module and let you use them from Python (without writting any YAML file). You can call any module by using the `actions` context manager:
In the following we install docker (using g5k provided script) and a docker container. We also need to install the python docker binding on the remote machine so that Ansible can interact with the docker daemons on the remote machines. This block of actions is idempotent.
%% Cell type:code id:organic-vanilla tags:
``` python
withen.actions(roles=roles)asa:
# installing the docker daemon
# prepending with a guard to make the command idempotent
Sometime you need to fire a process on some remote machines that needs to survive the remote connection that started it. EnOSlib provides a `keyword` argument for this purpose and can be used when calling modules (when supported).
%% Cell type:code id:thorough-heritage tags:
``` python
# synchronous execution, will wait until the end of the shell command
results=en.run_command("for i in $(seq 1 10); do sleep 1; echo toto; done",roles=roles)
results
```
%% Cell type:code id:protected-complexity tags:
``` python
# The remote command will be daemonize on the remote machines
results=en.run_command("for i in $(seq 1 10); do sleep 1; echo toto; done",roles=roles,background=True)
results
```
%% Cell type:code id:returning-trout tags:
``` python
# you can get back the status of the daemonized process by reading the remote results_file
# but we need to wait the completion, so forcing a sleep here (one could poll the status)
There's an alternative way to pass a variable to a task: using `extra_vars`.
The difference with the previous case (python interpreted variables) is the fact that the variable is interpolated right before execution happens on the remote node.
One could imagine the the value is broadcasted to all nodes and replaced right before the execution.
To indicate that we want to use this kind of variables, we need to pass its value using the `extra_vars` dictionnary and use a template (`{{ ... }}`) in the task description.
