Mentions légales du service

Skip to content
Snippets Groups Projects
Commit aa287c66 authored by SIMONIN Matthieu's avatar SIMONIN Matthieu
Browse files

doc/up jupyter

parent 146f2573
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id:racial-puppy tags:
# Setup and basic objects
This series of notebooks illustrates the use of EnOSlib on Grid'5000.
Get started with EnOSlib on Grid'5000.
---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib
---
This is the first notebooks of a series that will let you discover the main features of EnOSlib/Grid'5000.
If you want to actually execute them you'll need to setup your environment properly.
We sum up here the different step to achieve this process.
1. Get a Grid'5000 account
- Register using this [page](https://www.grid5000.fr/w/Grid5000:Get_an_account).
Pay attention to the fact that uploading a SSH key (public part) is mandatory to perform any EnOSlib action
- Make sure the SSH connection is ready. You can follow this [tutorial](https://www.grid5000.fr/w/Getting_Started)
Pay attention to the fact that uploading a SSH key (public part) is mandatory to perform any EnOSlib action.
- Make sure the SSH connection is ready. You can follow this [tutorial](https://www.grid5000.fr/w/Getting_Started).
2. Install EnOSlib
- Follow the steps [here](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#installation)
---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib
---
- Follow the steps [here](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#installation).
%% Cell type:markdown id:8b961d12-5752-4181-a672-a00013c20163 tags:
## Testing the import
%% Cell type:code id:indirect-delicious tags:
``` python
import enoslib as en
```
%% Cell type:markdown id:818254cb-c26f-4457-bf41-a2229eb8243d tags:
## Resources abstractions
In this notebook we won't execute anything remotely, instead we'll just cover some basic abstractions provided by the library.
We start with the abstractions of the resources (machines and networks that are usually givn by an infrastructure)
### Host
An host is anything we can connect to and act on it. Most of the time it corresponds to a machine reachable through SSH.
The datastructure reflects this.
Usually you don't instantiate hosts manually but they are brought to you by EnOSlib (because they depend on a scheduler decision).
%% Cell type:code id:665c334b-6c6e-4ca1-95ed-010f683d81b0 tags:
``` python
bare_host = en.Host("192.168.0.1")
host_with_alias = en.Host("192.168.0.2", alias="one_alias")
host_with_alias_and_username = en.Host("192.168.0.3", alias="one_alias", user="foo")
```
%% Cell type:code id:ae526e0b-ca1a-429b-a3a1-a7d3623c9e28 tags:
``` python
bare_host
```
%% Output
Host(address='192.168.0.1', alias='192.168.0.1', user=None, keyfile=None, port=None, extra={}, net_devices=set())
%% Cell type:code id:2fb3ec36-0b5d-4aa3-99bc-702f308e3e89 tags:
``` python
host_with_alias
```
%% Output
Host(address='192.168.0.2', alias='one_alias', user=None, keyfile=None, port=None, extra={}, net_devices=set())
%% Cell type:code id:10e17d3b-9888-486a-bb58-642151b9d42e tags:
``` python
host_with_alias_and_username
```
%% Output
Host(address='192.168.0.3', alias='one_alias', user='foo', keyfile=None, port=None, extra={}, net_devices=set())
%% Cell type:markdown id:2436ad88-9b65-4c17-9c43-6ca33db0b5b7 tags:
The local machine can be represented by an instance of the `LocalHost` object. This is a specialization of an `Host`, the connection to this host will be made using sub-processes (instead of SSH). We can see it in the `extra` attribute of the `LocalHost` object. This `extra` attribute is actually interpreted when we won't to do some "remote" actions on our hosts.
%% Cell type:code id:1d7daa8d-605b-4940-bf9b-37e7d814b92f tags:
``` python
localhost = en.LocalHost()
localhost
```
%% Output
LocalHost(address='localhost', alias='localhost', user=None, keyfile=None, port=None, extra={'ansible_connection': 'local'}, net_devices=set())
%% Cell type:markdown id:a5a50353-9275-4a0e-bf0e-49fd1cd01ec4 tags:
Other types of Hosts are possible. The library has a `DockerHost` which represents a docker container we want to reach using the docker line protocol. One needs to specify where this container is running by passing an host instance.
%% Cell type:code id:06e734e9-4276-46a3-851e-04257ea22b68 tags:
``` python
docker_host = en.DockerHost("alias", "container_name", host_with_alias_and_username)
docker_host
```
%% Output
DockerHost(address='container_name', alias='alias', user='foo', keyfile=None, port=None, extra={'ansible_connection': 'docker', 'ansible_docker_extra_args': '-H ssh://foo@192.168.0.3', 'mitogen_via': 'foo@192.168.0.3'}, net_devices=set())
%% Cell type:markdown id:d62c0172-bea2-4d17-b414-6aa24dd7f070 tags:
### Roles
%% Cell type:markdown id:7cad7f3c-49ee-4a08-964f-4367b16d88a8 tags:
During an experiment your hosts will serve different purposes: some will host the system you are studying while other will install third party tools to inject some load, observe ...
A natural way of configuring differently several sets of hosts is to tag them and group them according to their tags.
The `Roles` datastructure serves this purpose: it lets you group your hosts based on tags. It follow a `dict-like` interface.
%% Cell type:code id:f0e99de3-5565-4203-b1e1-c767f850192b tags:
``` python
h1 = en.Host("10.0.0.1")
h2 = en.Host("10.0.0.2")
h3 = en.Host("10.0.0.3")
roles = en.Roles()
roles["tag1"] = [h1, h2]
roles["tag2"] = [h3]
roles["tag3"] = [h2, h3]
roles
```
%% Output
{'tag1': [Host(address='10.0.0.1', alias='10.0.0.1', user=None, keyfile=None, port=None, extra={}, net_devices=set()), Host(address='10.0.0.2', alias='10.0.0.2', user=None, keyfile=None, port=None, extra={}, net_devices=set())], 'tag2': [Host(address='10.0.0.3', alias='10.0.0.3', user=None, keyfile=None, port=None, extra={}, net_devices=set())], 'tag3': [Host(address='10.0.0.2', alias='10.0.0.2', user=None, keyfile=None, port=None, extra={}, net_devices=set()), Host(address='10.0.0.3', alias='10.0.0.3', user=None, keyfile=None, port=None, extra={}, net_devices=set())]}
%% Cell type:markdown id:8549ef96-1e0c-4d9e-ab58-abb119355177 tags:
### Network and Networks
`Network` and `Networks` are the same as `Host` and `Roles` but for networks:
- `Network` represent a single Network
- `Networks` represent a "Roles" of Network: networks indexed by their tags
.
Networks are usually given by an infrastructure and thus you won't really instantiate `Network` nor `Networks` by yourself.
More precisely there exists a specific subclass of `Network` per infrastructure which will be returned automatically by EnOSlib when needed.
Moreover `Network` datastructure isn't exposed in EnOSlib at the top level, let's see however how a `DefaultNetwork` can look like. A `DefaultNetwork` is a very common abstraction of a network that allows to represent a basic network with optionnally a pool of free ips/macs address. For instance a subnet or a vlan on Grid5000 are represented by a specific `DefaultNetwork`.
%% Cell type:code id:dbbd79c8-4440-44d4-a264-1aa71115cd1a tags:
``` python
from enoslib.objects import DefaultNetwork
```
%% Cell type:code id:d7788d7c-245c-4069-ab06-bb66d022c5dd tags:
``` python
one_network = DefaultNetwork("192.168.1.0/24")
one_network_with_a_pool_of_ips = DefaultNetwork("192.168.1.0/24", ip_start="192.168.1.10", ip_end="192.168.1.100")
```
%% Cell type:code id:e7952082-7aab-4a3d-8ff7-1a6f6a102dc6 tags:
``` python
one_network
```
%% Output
<enoslib.objects.DefaultNetwork at 0x7f38a8d91790>
%% Cell type:code id:a4aa289d-357f-4003-9a77-4cfa6284df72 tags:
``` python
one_network_with_a_pool_of_ips
```
%% Output
<enoslib.objects.DefaultNetwork at 0x7f38a8d91a50>
%% Cell type:code id:337d3e84-c08a-4846-8b64-93f70d491e51 tags:
``` python
# get one free ip
ip_gen = one_network_with_a_pool_of_ips.free_ips
next(ip_gen)
```
%% Output
IPv4Address('192.168.1.10')
%% Cell type:markdown id:5c58ab35-a319-4f27-923c-08153a53ed70 tags:
......
%% Cell type:markdown id:preceding-chorus tags:
# Remote actions and variables
Changing the state of remote resources
---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib
---
**Prerequisites**:
- A Grid'5000 account
- A working EnOSlib environment and Jupyter (not included in EnOSlib dependencies, but `pip install jupyterlab` will install it)
---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib
---
%% Cell type:code id:handed-forestry tags:
``` python
import enoslib as en
# get some logging info
import logging
logging.basicConfig(level=logging.INFO)
```
%% Cell type:markdown id:provincial-madonna tags:
## Setup on Grid'5000
### Describing the resources
%% Cell type:code id:southeast-feature tags:
``` python
# claim the resources
network = en.G5kNetworkConf(id="n1", type="prod", roles=["my_network"], site="rennes")
conf = (
en.G5kConf.from_settings(job_type="allow_classic_ssh", job_name="rsd-01")
.add_network_conf(network)
.add_machine(
roles=["control"], cluster="parasilo", nodes=1, primary_network=network
)
.add_machine(
roles=["compute"],
cluster="parasilo",
nodes=1,
primary_network=network,
)
.finalize()
)
conf
```
%% Output
Conf@0x7f6e70218610
{
"dhcp": true,
"force_deploy": false,
"env_name": "debian10-x64-nfs",
"job_name": "rsd-01",
"job_type": "allow_classic_ssh",
"key": "/home/msimonin/.ssh/id_rsa.pub",
"queue": "default",
"walltime": "02:00:00",
"resources": {
"machines": [
{
"roles": [
"control"
],
"primary_network": "n1",
"secondary_networks": [],
"cluster": "parasilo",
"nodes": 1
},
{
"roles": [
"compute"
],
"primary_network": "n1",
"secondary_networks": [],
"cluster": "parasilo",
"nodes": 1
}
],
"networks": [
{
"id": "n1",
"type": "prod",
"roles": [
"my_network"
],
"site": "rennes"
}
]
}
}
%% Cell type:markdown id:proprietary-cricket tags:
### Reserving the resources
%% Cell type:code id:approximate-sharp tags:
``` python
provider = en.G5k(conf)
roles, networks = provider.init()
```
%% Output
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from lille
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from luxembourg
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from lyon
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from nancy
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from nantes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from sophia
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Submitting {'name': 'rsd-01', 'types': ['allow_classic_ssh'], 'resources': "{cluster='parasilo'}/nodes=1+{cluster='parasilo'}/nodes=1,walltime=02:00:00", 'command': 'sleep 31536000', 'queue': 'default'} on rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Waiting for 1815745 on rennes [2021-08-24 17:13:18]
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Waiting for 1815745 on rennes [2021-08-24 17:13:18]
INFO:enoslib.infra.enos_g5k.g5k_api_utils:All jobs are Running !
%% Cell type:markdown id:unnecessary-romantic tags:
Inspecting the ressources we own for the experiment's lifetime:
- roles: this is somehow a dictionnary whose keys are the role names and the associated values are the corresponding list of hosts
- networks: similar to roles but for networks
%% Cell type:code id:formal-valley tags:
``` python
roles
```
%% Output
{'control': [Host(address='parasilo-17.rennes.grid5000.fr', alias='parasilo-17.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())], 'compute': [Host(address='parasilo-24.rennes.grid5000.fr', alias='parasilo-24.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())]}
%% Cell type:code id:split-colombia tags:
``` python
# list of host on a given role
roles["control"]
```
%% Output
[Host(address='parasilo-17.rennes.grid5000.fr', alias='parasilo-17.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())]
%% Cell type:code id:owned-duration tags:
``` python
# a single host
roles["control"][0]
```
%% Output
Host(address='parasilo-17.rennes.grid5000.fr', alias='parasilo-17.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())
%% Cell type:code id:sacred-valuable tags:
``` python
networks
```
%% Output
WARNING:enoslib.infra.enos_g5k.objects:gateway is not yet implemented for <class 'enoslib.infra.enos_g5k.objects.G5kEnosProd6Network'> on the G5k side
{'my_network': [<enoslib.infra.enos_g5k.objects.G5kEnosProd4Network object at 0x7f6e70234650>, <enoslib.infra.enos_g5k.objects.G5kEnosProd6Network object at 0x7f6e6ff27a90>]}
%% Cell type:code id:rapid-skating tags:
``` python
# sync some more information in the host data structure (for illustration purpose here)
en.sync_info(roles, networks)
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
[WARNING]: No inventory was parsed, only implicit localhost is available
PLAY [all] ************************************************************************************************************************************************
TASK [hostname] *******************************************************************************************************************************************
[started TASK: hostname on parasilo-17.rennes.grid5000.fr]
[started TASK: hostname on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
INFO:enoslib.api:extra_vars = {'enos_action': 'check_network', 'facts_file': '/home/msimonin/workspace/repos/enoslib/_tmp_enos_/facts.json', 'ansible_python_interpreter': 'python3'}
PLAY [Gather facts for all hosts] *************************************************************************************************************************
TASK [Gathering Facts] ************************************************************************************************************************************
ok: [parasilo-17.rennes.grid5000.fr]
ok: [parasilo-24.rennes.grid5000.fr]
TASK [setup] **********************************************************************************************************************************************
ok: [parasilo-17.rennes.grid5000.fr]
ok: [parasilo-24.rennes.grid5000.fr]
PLAY [Utils functions] ************************************************************************************************************************************
TASK [Gathering Facts] ************************************************************************************************************************************
ok: [parasilo-17.rennes.grid5000.fr]
ok: [parasilo-24.rennes.grid5000.fr]
TASK [utils : include] ************************************************************************************************************************************
included: /home/msimonin/workspace/repos/enoslib/enoslib/ansible/roles/utils/tasks/check_network.yml for parasilo-17.rennes.grid5000.fr, parasilo-24.rennes.grid5000.fr
TASK [utils : Dump network information in a file] *********************************************************************************************************
changed: [parasilo-17.rennes.grid5000.fr]
TASK [utils : Create the fake interfaces] *****************************************************************************************************************
skipping: [parasilo-17.rennes.grid5000.fr]
skipping: [parasilo-24.rennes.grid5000.fr]
PLAY RECAP ************************************************************************************************************************************************
parasilo-17.rennes.grid5000.fr : ok=5 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
parasilo-24.rennes.grid5000.fr : ok=4 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
{'control': [Host(address='parasilo-17.rennes.grid5000.fr', alias='parasilo-17.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices={NetDevice(name='eno1', addresses=set()), NetDevice(name='eno2', addresses=set()), NetDevice(name='lo', addresses={IPAddress(network=None, ip=IPv6Interface('::1/128')), IPAddress(network=None, ip=IPv4Interface('127.0.0.1/8'))}), NetDevice(name='eno3', addresses=set()), BridgeDevice(name='br0', addresses={IPAddress(network=None, ip=IPv6Interface('fe80::eef4:bbff:fed1:1b8/64')), IPAddress(network=<enoslib.infra.enos_g5k.objects.G5kEnosProd4Network object at 0x7f6e70234650>, ip=IPv4Interface('172.16.97.17/20'))}, bridged=['eno1']), NetDevice(name='eno4', addresses=set())})], 'compute': [Host(address='parasilo-24.rennes.grid5000.fr', alias='parasilo-24.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices={NetDevice(name='eno1', addresses=set()), NetDevice(name='eno2', addresses=set()), NetDevice(name='lo', addresses={IPAddress(network=None, ip=IPv6Interface('::1/128')), IPAddress(network=None, ip=IPv4Interface('127.0.0.1/8'))}), NetDevice(name='eno3', addresses=set()), BridgeDevice(name='br0', addresses={IPAddress(network=None, ip=IPv6Interface('fe80::eef4:bbff:fed1:1018/64')), IPAddress(network=<enoslib.infra.enos_g5k.objects.G5kEnosProd4Network object at 0x7f6e70234650>, ip=IPv4Interface('172.16.97.24/20'))}, bridged=['eno1']), NetDevice(name='eno4', addresses=set())})]}
%% Cell type:markdown id:caring-glance tags:
## Acting on remote nodes
### run a command, filter results
%% Cell type:code id:political-praise tags:
``` python
results = en.run_command("cat /etc/apt/sources.list", roles=roles)
results
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [cat /etc/apt/sources.list] **************************************************************************************************************************
[started TASK: cat /etc/apt/sources.list on parasilo-17.rennes.grid5000.fr]
[started TASK: cat /etc/apt/sources.list on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='cat /etc/apt/sources.list', status='OK', payload={'cmd': 'cat /etc/apt/sources.list', 'stdout': 'deb http://deb.debian.org/debian buster main contrib non-free\ndeb-src http://deb.debian.org/debian buster main contrib non-free\ndeb http://deb.debian.org/debian buster-updates main contrib non-free\ndeb-src http://deb.debian.org/debian buster-updates main contrib non-free\ndeb http://security.debian.org/ buster/updates main contrib non-free\ndeb-src http://security.debian.org/ buster/updates main contrib non-free\ndeb http://deb.debian.org/debian buster-backports main contrib non-free\ndeb-src http://deb.debian.org/debian buster-backports main contrib non-free', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:13:58.997119', 'end': '2021-08-24 17:13:58.999810', 'delta': '0:00:00.002691', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'cat /etc/apt/sources.list', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['deb http://deb.debian.org/debian buster main contrib non-free', 'deb-src http://deb.debian.org/debian buster main contrib non-free', 'deb http://deb.debian.org/debian buster-updates main contrib non-free', 'deb-src http://deb.debian.org/debian buster-updates main contrib non-free', 'deb http://security.debian.org/ buster/updates main contrib non-free', 'deb-src http://security.debian.org/ buster/updates main contrib non-free', 'deb http://deb.debian.org/debian buster-backports main contrib non-free', 'deb-src http://deb.debian.org/debian buster-backports main contrib non-free'], 'stderr_lines': [], '_ansible_no_log': False})
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='cat /etc/apt/sources.list', status='OK', payload={'cmd': 'cat /etc/apt/sources.list', 'stdout': 'deb http://deb.debian.org/debian buster main contrib non-free\ndeb-src http://deb.debian.org/debian buster main contrib non-free\ndeb http://deb.debian.org/debian buster-updates main contrib non-free\ndeb-src http://deb.debian.org/debian buster-updates main contrib non-free\ndeb http://security.debian.org/ buster/updates main contrib non-free\ndeb-src http://security.debian.org/ buster/updates main contrib non-free\ndeb http://deb.debian.org/debian buster-backports main contrib non-free\ndeb-src http://deb.debian.org/debian buster-backports main contrib non-free', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:13:59.006244', 'end': '2021-08-24 17:13:59.008895', 'delta': '0:00:00.002651', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'cat /etc/apt/sources.list', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['deb http://deb.debian.org/debian buster main contrib non-free', 'deb-src http://deb.debian.org/debian buster main contrib non-free', 'deb http://deb.debian.org/debian buster-updates main contrib non-free', 'deb-src http://deb.debian.org/debian buster-updates main contrib non-free', 'deb http://security.debian.org/ buster/updates main contrib non-free', 'deb-src http://security.debian.org/ buster/updates main contrib non-free', 'deb http://deb.debian.org/debian buster-backports main contrib non-free', 'deb-src http://deb.debian.org/debian buster-backports main contrib non-free'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:willing-senate tags:
``` python
one_result = results.filter(host=roles["control"][0].alias)[0]
one_result
```
%% Output
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='cat /etc/apt/sources.list', status='OK', payload={'cmd': 'cat /etc/apt/sources.list', 'stdout': 'deb http://deb.debian.org/debian buster main contrib non-free\ndeb-src http://deb.debian.org/debian buster main contrib non-free\ndeb http://deb.debian.org/debian buster-updates main contrib non-free\ndeb-src http://deb.debian.org/debian buster-updates main contrib non-free\ndeb http://security.debian.org/ buster/updates main contrib non-free\ndeb-src http://security.debian.org/ buster/updates main contrib non-free\ndeb http://deb.debian.org/debian buster-backports main contrib non-free\ndeb-src http://deb.debian.org/debian buster-backports main contrib non-free', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:13:58.997119', 'end': '2021-08-24 17:13:58.999810', 'delta': '0:00:00.002691', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'cat /etc/apt/sources.list', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['deb http://deb.debian.org/debian buster main contrib non-free', 'deb-src http://deb.debian.org/debian buster main contrib non-free', 'deb http://deb.debian.org/debian buster-updates main contrib non-free', 'deb-src http://deb.debian.org/debian buster-updates main contrib non-free', 'deb http://security.debian.org/ buster/updates main contrib non-free', 'deb-src http://security.debian.org/ buster/updates main contrib non-free', 'deb http://deb.debian.org/debian buster-backports main contrib non-free', 'deb-src http://deb.debian.org/debian buster-backports main contrib non-free'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:specified-restriction tags:
``` python
one_result.payload["stdout"]
```
%% Output
'deb http://deb.debian.org/debian buster main contrib non-free\ndeb-src http://deb.debian.org/debian buster main contrib non-free\ndeb http://deb.debian.org/debian buster-updates main contrib non-free\ndeb-src http://deb.debian.org/debian buster-updates main contrib non-free\ndeb http://security.debian.org/ buster/updates main contrib non-free\ndeb-src http://security.debian.org/ buster/updates main contrib non-free\ndeb http://deb.debian.org/debian buster-backports main contrib non-free\ndeb-src http://deb.debian.org/debian buster-backports main contrib non-free'
%% Cell type:markdown id:coastal-poverty tags:
There are some specific shortcuts when the remote actions is a remote (shell) command: `.stdout`, `.stderr`, `.rc`
%% Cell type:code id:gentle-implementation tags:
``` python
print(f"stdout = {one_result.stdout}\n", f"stderr={one_result.stderr}\n", f"return code = {one_result.rc}")
```
%% Output
stdout = deb http://deb.debian.org/debian buster main contrib non-free
deb-src http://deb.debian.org/debian buster main contrib non-free
deb http://deb.debian.org/debian buster-updates main contrib non-free
deb-src http://deb.debian.org/debian buster-updates main contrib non-free
deb http://security.debian.org/ buster/updates main contrib non-free
deb-src http://security.debian.org/ buster/updates main contrib non-free
deb http://deb.debian.org/debian buster-backports main contrib non-free
deb-src http://deb.debian.org/debian buster-backports main contrib non-free
stderr=
return code = 0
%% Cell type:markdown id:searching-favorite tags:
### Filtering hosts on which the command is run
`run_command` acts on remote hosts. Those hosts can be given as a `Roles` type (output of `provider.init`) or as a list of `Host` or a single `Host`.
%% Cell type:code id:israeli-delivery tags:
``` python
# some roles
en.run_command("date", roles = roles)
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [date] ***********************************************************************************************************************************************
[started TASK: date on parasilo-17.rennes.grid5000.fr]
[started TASK: date on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='date', status='OK', payload={'cmd': 'date', 'stdout': 'Tue 24 Aug 2021 05:14:11 PM CEST', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:14:11.041558', 'end': '2021-08-24 17:14:11.044053', 'delta': '0:00:00.002495', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'date', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Tue 24 Aug 2021 05:14:11 PM CEST'], 'stderr_lines': [], '_ansible_no_log': False})
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='date', status='OK', payload={'cmd': 'date', 'stdout': 'Tue 24 Aug 2021 05:14:11 PM CEST', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:14:11.050523', 'end': '2021-08-24 17:14:11.052991', 'delta': '0:00:00.002468', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'date', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Tue 24 Aug 2021 05:14:11 PM CEST'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:excessive-novelty tags:
``` python
# a list of hosts
en.run_command("date", roles = roles["control"])
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [date] ***********************************************************************************************************************************************
[started TASK: date on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='date', status='OK', payload={'cmd': 'date', 'stdout': 'Tue 24 Aug 2021 05:14:24 PM CEST', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:14:24.781335', 'end': '2021-08-24 17:14:24.783784', 'delta': '0:00:00.002449', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'date', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Tue 24 Aug 2021 05:14:24 PM CEST'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:genetic-aurora tags:
``` python
# a single host
en.run_command("date", roles=roles["control"][0])
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [date] ***********************************************************************************************************************************************
[started TASK: date on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='date', status='OK', payload={'cmd': 'date', 'stdout': 'Tue 24 Aug 2021 05:14:29 PM CEST', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:14:29.312216', 'end': '2021-08-24 17:14:29.314718', 'delta': '0:00:00.002502', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'date', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Tue 24 Aug 2021 05:14:29 PM CEST'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:markdown id:roman-rental tags:
### Remote actions
Tools like Ansible, Puppet, Chef, Terraform ... are shipped with a set of predefined remote actions to ease the administrator life.
Actions like copying file, adding some users, managing packages, making sure a line is absent from a configuration file, managing docker containers ... are first-class citizens actions and brings some nice garantees like correctness and idempotency.
There are 1000+ modules available:
https://docs.ansible.com/ansible/2.9/modules/list_of_all_modules.html
---
EnOSlib wraps Ansible module and let you use them from Python (without writting any YAML file). You can call any module by using the `actions` context manager:
In the following we install docker (using g5k provided script) and a docker container. We also need to install the python docker binding on the remote machine so that Ansible can interact with the docker daemons on the remote machines. This block of actions is idempotent.
%% Cell type:code id:transparent-tourism tags:
``` python
with en.actions(roles=roles) as a:
# prepending with `which docker ||` makes this task idempotent
a.shell("which docker || /grid5000/code/bin/g5k-setup-docker")
a.pip(name="docker", state="present")
a.docker_container(image="nginx", state="started", name="my_webserver")
# we can capture the outputs if we really want to
results = a.results
results
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [which docker || /grid5000/code/bin/g5k-setup-docker] ************************************************************************************************
[started TASK: which docker || /grid5000/code/bin/g5k-setup-docker on parasilo-17.rennes.grid5000.fr]
[started TASK: which docker || /grid5000/code/bin/g5k-setup-docker on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [pip] ************************************************************************************************************************************************
[started TASK: pip on parasilo-17.rennes.grid5000.fr]
[started TASK: pip on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [docker_container] ***********************************************************************************************************************************
[started TASK: docker_container on parasilo-17.rennes.grid5000.fr]
[started TASK: docker_container on parasilo-24.rennes.grid5000.fr]
[DEPRECATION WARNING]: The container_default_behavior option will change its default value from "compatibility" to "no_defaults" in community.docker
2.0.0. To remove this warning, please specify an explicit value for it now. This feature will be removed from community.docker in version 2.0.0.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='which docker || /grid5000/code/bin/g5k-setup-docker', status='OK', payload={'cmd': 'which docker || /grid5000/code/bin/g5k-setup-docker', 'stdout': 'curl -sSL https://get.docker.com/ | sh\nsudo mkdir -p /etc/docker\necho "{ \\"registry-mirrors\\": [\\"http://docker-cache.grid5000.fr\\"] }" | sudo tee /etc/docker/daemon.json\nsudo systemctl restart docker\nsudo chmod o+rw /var/run/docker.sock', 'stderr': '+ sh -c apt-get update -qq >/dev/null\nW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Some index files failed to download. They have been ignored, or old ones used instead.\n+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null\n+ sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /usr/share/keyrings/docker-archive-keyring.gpg\n+ sh -c echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian buster stable" > /etc/apt/sources.list.d/docker.list\n+ sh -c apt-get update -qq >/dev/null\nW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Some index files failed to download. They have been ignored, or old ones used instead.\n+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce-cli docker-scan-plugin docker-ce >/dev/null\n+ version_gte 20.10\n+ [ -z ]\n+ return 0\n+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null\n+ sh -c docker version', 'rc': 0, 'start': '2021-08-24 17:14:35.800313', 'end': '2021-08-24 17:15:05.871384', 'delta': '0:00:30.071071', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'which docker || /grid5000/code/bin/g5k-setup-docker', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['curl -sSL https://get.docker.com/ | sh', 'sudo mkdir -p /etc/docker', 'echo "{ \\"registry-mirrors\\": [\\"http://docker-cache.grid5000.fr\\"] }" | sudo tee /etc/docker/daemon.json', 'sudo systemctl restart docker', 'sudo chmod o+rw /var/run/docker.sock'], 'stderr_lines': ['+ sh -c apt-get update -qq >/dev/null', 'W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Some index files failed to download. They have been ignored, or old ones used instead.', '+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null', '+ sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /usr/share/keyrings/docker-archive-keyring.gpg', '+ sh -c echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian buster stable" > /etc/apt/sources.list.d/docker.list', '+ sh -c apt-get update -qq >/dev/null', 'W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Some index files failed to download. They have been ignored, or old ones used instead.', '+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce-cli docker-scan-plugin docker-ce >/dev/null', '+ version_gte 20.10', '+ [ -z ]', '+ return 0', '+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null', '+ sh -c docker version'], '_ansible_no_log': False})
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='which docker || /grid5000/code/bin/g5k-setup-docker', status='OK', payload={'cmd': 'which docker || /grid5000/code/bin/g5k-setup-docker', 'stdout': 'curl -sSL https://get.docker.com/ | sh\nsudo mkdir -p /etc/docker\necho "{ \\"registry-mirrors\\": [\\"http://docker-cache.grid5000.fr\\"] }" | sudo tee /etc/docker/daemon.json\nsudo systemctl restart docker\nsudo chmod o+rw /var/run/docker.sock', 'stderr': '+ sh -c apt-get update -qq >/dev/null\nW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Some index files failed to download. They have been ignored, or old ones used instead.\n+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null\n+ sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /usr/share/keyrings/docker-archive-keyring.gpg\n+ sh -c echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian buster stable" > /etc/apt/sources.list.d/docker.list\n+ sh -c apt-get update -qq >/dev/null\nW: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>\nW: Some index files failed to download. They have been ignored, or old ones used instead.\n+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce-cli docker-scan-plugin docker-ce >/dev/null\n+ version_gte 20.10\n+ [ -z ]\n+ return 0\n+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null\n+ sh -c docker version', 'rc': 0, 'start': '2021-08-24 17:14:35.755730', 'end': '2021-08-24 17:15:06.346393', 'delta': '0:00:30.590663', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'which docker || /grid5000/code/bin/g5k-setup-docker', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['curl -sSL https://get.docker.com/ | sh', 'sudo mkdir -p /etc/docker', 'echo "{ \\"registry-mirrors\\": [\\"http://docker-cache.grid5000.fr\\"] }" | sudo tee /etc/docker/daemon.json', 'sudo systemctl restart docker', 'sudo chmod o+rw /var/run/docker.sock'], 'stderr_lines': ['+ sh -c apt-get update -qq >/dev/null', 'W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Some index files failed to download. They have been ignored, or old ones used instead.', '+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null', '+ sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /usr/share/keyrings/docker-archive-keyring.gpg', '+ sh -c echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian buster stable" > /etc/apt/sources.list.d/docker.list', '+ sh -c apt-get update -qq >/dev/null', 'W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repo.radeon.com/rocm/apt/debian xenial InRelease: The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Failed to fetch https://repo.radeon.com/rocm/apt/debian/dists/xenial/InRelease The following signatures were invalid: EXPKEYSIG 9386B48A1A693C5C James Adrian Edwards (ROCm Release Manager) <JamesAdrian.Edwards@amd.com>', 'W: Some index files failed to download. They have been ignored, or old ones used instead.', '+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce-cli docker-scan-plugin docker-ce >/dev/null', '+ version_gte 20.10', '+ [ -z ]', '+ return 0', '+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null', '+ sh -c docker version'], '_ansible_no_log': False})
CustomCommandResult(host='parasilo-24.rennes.grid5000.fr', task='pip', status='OK', payload={'changed': True, 'cmd': ['/usr/bin/pip3', 'install', 'docker'], 'name': ['docker'], 'version': None, 'state': 'present', 'requirements': None, 'virtualenv': None, 'stdout': 'Collecting docker\n Downloading https://files.pythonhosted.org/packages/b2/5a/f988909dfed18c1ac42ad8d9e611e6c5657e270aa6eb68559985dbb69c13/docker-5.0.0-py2.py3-none-any.whl (146kB)\nCollecting websocket-client>=0.32.0 (from docker)\n Downloading https://files.pythonhosted.org/packages/55/44/030ea47390896c8d6dc9995c8e9a4c5df3a161cd45416d88119036c73eda/websocket_client-1.2.1-py2.py3-none-any.whl (52kB)\nRequirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.21.0)\nInstalling collected packages: websocket-client, docker\nSuccessfully installed docker-5.0.0 websocket-client-1.2.1\n', 'stderr': '', 'invocation': {'module_args': {'name': ['docker'], 'state': 'present', 'virtualenv_site_packages': False, 'virtualenv_command': 'virtualenv', 'editable': False, 'version': None, 'requirements': None, 'virtualenv': None, 'virtualenv_python': None, 'extra_args': None, 'chdir': None, 'executable': None, 'umask': None}}, 'stdout_lines': ['Collecting docker', ' Downloading https://files.pythonhosted.org/packages/b2/5a/f988909dfed18c1ac42ad8d9e611e6c5657e270aa6eb68559985dbb69c13/docker-5.0.0-py2.py3-none-any.whl (146kB)', 'Collecting websocket-client>=0.32.0 (from docker)', ' Downloading https://files.pythonhosted.org/packages/55/44/030ea47390896c8d6dc9995c8e9a4c5df3a161cd45416d88119036c73eda/websocket_client-1.2.1-py2.py3-none-any.whl (52kB)', 'Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.21.0)', 'Installing collected packages: websocket-client, docker', 'Successfully installed docker-5.0.0 websocket-client-1.2.1'], 'stderr_lines': [], '_ansible_no_log': False})
CustomCommandResult(host='parasilo-17.rennes.grid5000.fr', task='pip', status='OK', payload={'changed': True, 'cmd': ['/usr/bin/pip3', 'install', 'docker'], 'name': ['docker'], 'version': None, 'state': 'present', 'requirements': None, 'virtualenv': None, 'stdout': 'Collecting docker\n Downloading https://files.pythonhosted.org/packages/b2/5a/f988909dfed18c1ac42ad8d9e611e6c5657e270aa6eb68559985dbb69c13/docker-5.0.0-py2.py3-none-any.whl (146kB)\nCollecting websocket-client>=0.32.0 (from docker)\n Downloading https://files.pythonhosted.org/packages/55/44/030ea47390896c8d6dc9995c8e9a4c5df3a161cd45416d88119036c73eda/websocket_client-1.2.1-py2.py3-none-any.whl (52kB)\nRequirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.21.0)\nInstalling collected packages: websocket-client, docker\nSuccessfully installed docker-5.0.0 websocket-client-1.2.1\n', 'stderr': '', 'invocation': {'module_args': {'name': ['docker'], 'state': 'present', 'virtualenv_site_packages': False, 'virtualenv_command': 'virtualenv', 'editable': False, 'version': None, 'requirements': None, 'virtualenv': None, 'virtualenv_python': None, 'extra_args': None, 'chdir': None, 'executable': None, 'umask': None}}, 'stdout_lines': ['Collecting docker', ' Downloading https://files.pythonhosted.org/packages/b2/5a/f988909dfed18c1ac42ad8d9e611e6c5657e270aa6eb68559985dbb69c13/docker-5.0.0-py2.py3-none-any.whl (146kB)', 'Collecting websocket-client>=0.32.0 (from docker)', ' Downloading https://files.pythonhosted.org/packages/55/44/030ea47390896c8d6dc9995c8e9a4c5df3a161cd45416d88119036c73eda/websocket_client-1.2.1-py2.py3-none-any.whl (52kB)', 'Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.21.0)', 'Installing collected packages: websocket-client, docker', 'Successfully installed docker-5.0.0 websocket-client-1.2.1'], 'stderr_lines': [], '_ansible_no_log': False})
CustomCommandResult(host='parasilo-17.rennes.grid5000.fr', task='docker_container', status='OK', payload={'changed': True, 'container': {'Id': 'b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce', 'Created': '2021-08-24T15:15:16.653171818Z', 'Path': '/docker-entrypoint.sh', 'Args': ['nginx', '-g', 'daemon off;'], 'State': {'Status': 'running', 'Running': True, 'Paused': False, 'Restarting': False, 'OOMKilled': False, 'Dead': False, 'Pid': 14720, 'ExitCode': 0, 'Error': '', 'StartedAt': '2021-08-24T15:15:19.434872036Z', 'FinishedAt': '0001-01-01T00:00:00Z'}, 'Image': 'sha256:dd34e67e3371dc2d1328790c3157ee42dfcae74afffd86b297459ed87a98c0fb', 'ResolvConfPath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/resolv.conf', 'HostnamePath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/hostname', 'HostsPath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/hosts', 'LogPath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce-json.log', 'Name': '/my_webserver', 'RestartCount': 0, 'Driver': 'overlay2', 'Platform': 'linux', 'MountLabel': '', 'ProcessLabel': '', 'AppArmorProfile': 'docker-default', 'ExecIDs': None, 'HostConfig': {'Binds': [], 'ContainerIDFile': '', 'LogConfig': {'Type': 'json-file', 'Config': {}}, 'NetworkMode': 'default', 'PortBindings': None, 'RestartPolicy': {'Name': '', 'MaximumRetryCount': 0}, 'AutoRemove': False, 'VolumeDriver': '', 'VolumesFrom': None, 'CapAdd': None, 'CapDrop': None, 'CgroupnsMode': 'host', 'Dns': None, 'DnsOptions': None, 'DnsSearch': None, 'ExtraHosts': None, 'GroupAdd': None, 'IpcMode': 'private', 'Cgroup': '', 'Links': None, 'OomScoreAdj': 0, 'PidMode': '', 'Privileged': False, 'PublishAllPorts': False, 'ReadonlyRootfs': False, 'SecurityOpt': None, 'UTSMode': '', 'UsernsMode': '', 'ShmSize': 67108864, 'Runtime': 'runc', 'ConsoleSize': [0, 0], 'Isolation': '', 'CpuShares': 0, 'Memory': 0, 'NanoCpus': 0, 'CgroupParent': '', 'BlkioWeight': 0, 'BlkioWeightDevice': None, 'BlkioDeviceReadBps': None, 'BlkioDeviceWriteBps': None, 'BlkioDeviceReadIOps': None, 'BlkioDeviceWriteIOps': None, 'CpuPeriod': 0, 'CpuQuota': 0, 'CpuRealtimePeriod': 0, 'CpuRealtimeRuntime': 0, 'CpusetCpus': '', 'CpusetMems': '', 'Devices': None, 'DeviceCgroupRules': None, 'DeviceRequests': None, 'KernelMemory': 0, 'KernelMemoryTCP': 0, 'MemoryReservation': 0, 'MemorySwap': 0, 'MemorySwappiness': None, 'OomKillDisable': False, 'PidsLimit': None, 'Ulimits': None, 'CpuCount': 0, 'CpuPercent': 0, 'IOMaximumIOps': 0, 'IOMaximumBandwidth': 0, 'MaskedPaths': ['/proc/asound', '/proc/acpi', '/proc/kcore', '/proc/keys', '/proc/latency_stats', '/proc/timer_list', '/proc/timer_stats', '/proc/sched_debug', '/proc/scsi', '/sys/firmware'], 'ReadonlyPaths': ['/proc/bus', '/proc/fs', '/proc/irq', '/proc/sys', '/proc/sysrq-trigger'], 'Init': False}, 'GraphDriver': {'Data': {'LowerDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b-init/diff:/var/lib/docker/overlay2/b309b755d9ccbd962d1fa569ac24305182e792c0d856b382baf1148ea8f0d8cd/diff:/var/lib/docker/overlay2/c10519ef34f5c0ff8d8358d9bcf5ef5a6311f03b2903532759787e485450428a/diff:/var/lib/docker/overlay2/e8d94c857640f7ce2ec2214516b00bf53cb05ddfca8e19047dd87109c78880af/diff:/var/lib/docker/overlay2/2fcc44c10530f9c0b2616600ccb40c654fc355c834e1d812d2d0492f0ec97ddc/diff:/var/lib/docker/overlay2/8d4d0df6e602c7ade1c64221c19bbe38082176c42e45a38cab14de32cdfe141d/diff:/var/lib/docker/overlay2/1ada2f61beac7647403e96de6e0842b5e07ec8c8b8a9be8e017e126fe0a7d050/diff', 'MergedDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b/merged', 'UpperDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b/diff', 'WorkDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b/work'}, 'Name': 'overlay2'}, 'Mounts': [], 'Config': {'Hostname': 'b2bf90f48f4f', 'Domainname': '', 'User': '', 'AttachStdin': False, 'AttachStdout': False, 'AttachStderr': False, 'ExposedPorts': {'80/tcp': {}}, 'Tty': False, 'OpenStdin': False, 'StdinOnce': False, 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'NGINX_VERSION=1.21.1', 'NJS_VERSION=0.6.1', 'PKG_RELEASE=1~buster'], 'Cmd': ['nginx', '-g', 'daemon off;'], 'Image': 'nginx', 'Volumes': None, 'WorkingDir': '', 'Entrypoint': ['/docker-entrypoint.sh'], 'OnBuild': None, 'Labels': {'maintainer': 'NGINX Docker Maintainers <docker-maint@nginx.com>'}, 'StopSignal': 'SIGQUIT'}, 'NetworkSettings': {'Bridge': '', 'SandboxID': 'c951d1b9e2ff3763235f519c5aa4a9a3dba30978071061bea5d55b6a1acb8893', 'HairpinMode': False, 'LinkLocalIPv6Address': '', 'LinkLocalIPv6PrefixLen': 0, 'Ports': {'80/tcp': None}, 'SandboxKey': '/var/run/docker/netns/c951d1b9e2ff', 'SecondaryIPAddresses': None, 'SecondaryIPv6Addresses': None, 'EndpointID': '0cd99a2f0e7d08629ce92b636189b1df338c24ec71751ac02148d5bb4148db83', 'Gateway': '172.17.0.1', 'GlobalIPv6Address': '', 'GlobalIPv6PrefixLen': 0, 'IPAddress': '172.17.0.2', 'IPPrefixLen': 16, 'IPv6Gateway': '', 'MacAddress': '02:42:ac:11:00:02', 'Networks': {'bridge': {'IPAMConfig': None, 'Links': None, 'Aliases': None, 'NetworkID': '68d355b984b0dfee3529070077588f3932d5700f5eafc4eb295b8cac7aea062e', 'EndpointID': '0cd99a2f0e7d08629ce92b636189b1df338c24ec71751ac02148d5bb4148db83', 'Gateway': '172.17.0.1', 'IPAddress': '172.17.0.2', 'IPPrefixLen': 16, 'IPv6Gateway': '', 'GlobalIPv6Address': '', 'GlobalIPv6PrefixLen': 0, 'MacAddress': '02:42:ac:11:00:02', 'DriverOpts': None}}}}, 'invocation': {'module_args': {'image': 'nginx', 'state': 'started', 'name': 'my_webserver', 'docker_host': 'unix://var/run/docker.sock', 'tls_hostname': 'localhost', 'api_version': 'auto', 'timeout': 60, 'tls': False, 'use_ssh_client': False, 'validate_certs': False, 'debug': False, 'cleanup': False, 'force_kill': False, 'ignore_image': False, 'keep_volumes': True, 'networks_cli_compatible': True, 'output_logs': False, 'pull': False, 'purge_networks': False, 'recreate': False, 'restart': False, 'ca_cert': None, 'client_cert': None, 'client_key': None, 'ssl_version': None, 'auto_remove': False, 'blkio_weight': None, 'capabilities': None, 'cap_drop': None, 'cgroup_parent': None, 'command': None, 'comparisons': None, 'container_default_behavior': 'compatibility', 'cpu_period': None, 'cpu_quota': None, 'cpus': None, 'cpuset_cpus': None, 'cpuset_mems': None, 'cpu_shares': None, 'default_host_ip': None, 'detach': True, 'devices': None, 'device_read_bps': None, 'device_write_bps': None, 'device_read_iops': None, 'device_write_iops': None, 'device_requests': None, 'dns_servers': None, 'dns_opts': None, 'dns_search_domains': None, 'domainname': None, 'entrypoint': None, 'env': None, 'env_file': None, 'etc_hosts': None, 'exposed_ports': None, 'groups': None, 'healthcheck': None, 'hostname': None, 'init': False, 'interactive': False, 'ipc_mode': None, 'kernel_memory': None, 'kill_signal': None, 'labels': None, 'links': None, 'log_driver': None, 'log_options': None, 'mac_address': None, 'memory': '0', 'memory_reservation': None, 'memory_swap': None, 'memory_swappiness': None, 'mounts': None, 'network_mode': None, 'networks': None, 'oom_killer': None, 'oom_score_adj': None, 'paused': False, 'pid_mode': None, 'pids_limit': None, 'privileged': False, 'published_ports': None, 'read_only': False, 'removal_wait_timeout': None, 'restart_policy': None, 'restart_retries': None, 'runtime': None, 'security_opts': None, 'shm_size': None, 'stop_signal': None, 'stop_timeout': None, 'storage_opts': None, 'sysctls': None, 'tmpfs': None, 'tty': False, 'ulimits': None, 'user': None, 'userns_mode': None, 'uts': None, 'volume_driver': None, 'volumes': None, 'volumes_from': None, 'working_dir': None}}, '_ansible_no_log': False})
CustomCommandResult(host='parasilo-24.rennes.grid5000.fr', task='docker_container', status='OK', payload={'changed': True, 'container': {'Id': 'aea95e6d98990687c6125642d7d9208e1787e105e226db6e1a6f313e9c66c613', 'Created': '2021-08-24T15:15:16.833689448Z', 'Path': '/docker-entrypoint.sh', 'Args': ['nginx', '-g', 'daemon off;'], 'State': {'Status': 'running', 'Running': True, 'Paused': False, 'Restarting': False, 'OOMKilled': False, 'Dead': False, 'Pid': 27196, 'ExitCode': 0, 'Error': '', 'StartedAt': '2021-08-24T15:15:19.617202697Z', 'FinishedAt': '0001-01-01T00:00:00Z'}, 'Image': 'sha256:dd34e67e3371dc2d1328790c3157ee42dfcae74afffd86b297459ed87a98c0fb', 'ResolvConfPath': '/var/lib/docker/containers/aea95e6d98990687c6125642d7d9208e1787e105e226db6e1a6f313e9c66c613/resolv.conf', 'HostnamePath': '/var/lib/docker/containers/aea95e6d98990687c6125642d7d9208e1787e105e226db6e1a6f313e9c66c613/hostname', 'HostsPath': '/var/lib/docker/containers/aea95e6d98990687c6125642d7d9208e1787e105e226db6e1a6f313e9c66c613/hosts', 'LogPath': '/var/lib/docker/containers/aea95e6d98990687c6125642d7d9208e1787e105e226db6e1a6f313e9c66c613/aea95e6d98990687c6125642d7d9208e1787e105e226db6e1a6f313e9c66c613-json.log', 'Name': '/my_webserver', 'RestartCount': 0, 'Driver': 'overlay2', 'Platform': 'linux', 'MountLabel': '', 'ProcessLabel': '', 'AppArmorProfile': 'docker-default', 'ExecIDs': None, 'HostConfig': {'Binds': [], 'ContainerIDFile': '', 'LogConfig': {'Type': 'json-file', 'Config': {}}, 'NetworkMode': 'default', 'PortBindings': None, 'RestartPolicy': {'Name': '', 'MaximumRetryCount': 0}, 'AutoRemove': False, 'VolumeDriver': '', 'VolumesFrom': None, 'CapAdd': None, 'CapDrop': None, 'CgroupnsMode': 'host', 'Dns': None, 'DnsOptions': None, 'DnsSearch': None, 'ExtraHosts': None, 'GroupAdd': None, 'IpcMode': 'private', 'Cgroup': '', 'Links': None, 'OomScoreAdj': 0, 'PidMode': '', 'Privileged': False, 'PublishAllPorts': False, 'ReadonlyRootfs': False, 'SecurityOpt': None, 'UTSMode': '', 'UsernsMode': '', 'ShmSize': 67108864, 'Runtime': 'runc', 'ConsoleSize': [0, 0], 'Isolation': '', 'CpuShares': 0, 'Memory': 0, 'NanoCpus': 0, 'CgroupParent': '', 'BlkioWeight': 0, 'BlkioWeightDevice': None, 'BlkioDeviceReadBps': None, 'BlkioDeviceWriteBps': None, 'BlkioDeviceReadIOps': None, 'BlkioDeviceWriteIOps': None, 'CpuPeriod': 0, 'CpuQuota': 0, 'CpuRealtimePeriod': 0, 'CpuRealtimeRuntime': 0, 'CpusetCpus': '', 'CpusetMems': '', 'Devices': None, 'DeviceCgroupRules': None, 'DeviceRequests': None, 'KernelMemory': 0, 'KernelMemoryTCP': 0, 'MemoryReservation': 0, 'MemorySwap': 0, 'MemorySwappiness': None, 'OomKillDisable': False, 'PidsLimit': None, 'Ulimits': None, 'CpuCount': 0, 'CpuPercent': 0, 'IOMaximumIOps': 0, 'IOMaximumBandwidth': 0, 'MaskedPaths': ['/proc/asound', '/proc/acpi', '/proc/kcore', '/proc/keys', '/proc/latency_stats', '/proc/timer_list', '/proc/timer_stats', '/proc/sched_debug', '/proc/scsi', '/sys/firmware'], 'ReadonlyPaths': ['/proc/bus', '/proc/fs', '/proc/irq', '/proc/sys', '/proc/sysrq-trigger'], 'Init': False}, 'GraphDriver': {'Data': {'LowerDir': '/var/lib/docker/overlay2/a3bfdac9dabb966f7774c6907243038d9908b834ca93e098988e280972892f57-init/diff:/var/lib/docker/overlay2/ce92a27637ce8692edafd055a5eebf7e8618ddc1528421a5bdbe8eed03bcf868/diff:/var/lib/docker/overlay2/be0a6d396b028ca5931d9597375696746e4c604f7d85d3786d5b97fb8d6cd006/diff:/var/lib/docker/overlay2/987b11143b517784988819a75dea2fc1eb0f2d9218b8716cb3e6b5ea5f1a02f0/diff:/var/lib/docker/overlay2/035ab8632f71ba7dd22ef0b9a632901e35a6bab3d2be4cbe641f2223533e5c53/diff:/var/lib/docker/overlay2/a6f02bff3f0c51285375778f628991f90d837f3b56227c236297f1bd67e6f1bb/diff:/var/lib/docker/overlay2/220e9657d72892444611eff1df7a71b8afb45b730362e27a5f5f73f5a6328e3d/diff', 'MergedDir': '/var/lib/docker/overlay2/a3bfdac9dabb966f7774c6907243038d9908b834ca93e098988e280972892f57/merged', 'UpperDir': '/var/lib/docker/overlay2/a3bfdac9dabb966f7774c6907243038d9908b834ca93e098988e280972892f57/diff', 'WorkDir': '/var/lib/docker/overlay2/a3bfdac9dabb966f7774c6907243038d9908b834ca93e098988e280972892f57/work'}, 'Name': 'overlay2'}, 'Mounts': [], 'Config': {'Hostname': 'aea95e6d9899', 'Domainname': '', 'User': '', 'AttachStdin': False, 'AttachStdout': False, 'AttachStderr': False, 'ExposedPorts': {'80/tcp': {}}, 'Tty': False, 'OpenStdin': False, 'StdinOnce': False, 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'NGINX_VERSION=1.21.1', 'NJS_VERSION=0.6.1', 'PKG_RELEASE=1~buster'], 'Cmd': ['nginx', '-g', 'daemon off;'], 'Image': 'nginx', 'Volumes': None, 'WorkingDir': '', 'Entrypoint': ['/docker-entrypoint.sh'], 'OnBuild': None, 'Labels': {'maintainer': 'NGINX Docker Maintainers <docker-maint@nginx.com>'}, 'StopSignal': 'SIGQUIT'}, 'NetworkSettings': {'Bridge': '', 'SandboxID': '26c5548c4ecb0a437866844a73816231f1ece237273f8587bbdc2e095fbda025', 'HairpinMode': False, 'LinkLocalIPv6Address': '', 'LinkLocalIPv6PrefixLen': 0, 'Ports': {'80/tcp': None}, 'SandboxKey': '/var/run/docker/netns/26c5548c4ecb', 'SecondaryIPAddresses': None, 'SecondaryIPv6Addresses': None, 'EndpointID': '4c01de6268c5bab9577072491f208e663b0063580290d699f00e84423d208c2c', 'Gateway': '172.17.0.1', 'GlobalIPv6Address': '', 'GlobalIPv6PrefixLen': 0, 'IPAddress': '172.17.0.2', 'IPPrefixLen': 16, 'IPv6Gateway': '', 'MacAddress': '02:42:ac:11:00:02', 'Networks': {'bridge': {'IPAMConfig': None, 'Links': None, 'Aliases': None, 'NetworkID': 'b4b2f3dfea9675965a9ef011da20bfd444df4df5098da9a81e490eb9810c4150', 'EndpointID': '4c01de6268c5bab9577072491f208e663b0063580290d699f00e84423d208c2c', 'Gateway': '172.17.0.1', 'IPAddress': '172.17.0.2', 'IPPrefixLen': 16, 'IPv6Gateway': '', 'GlobalIPv6Address': '', 'GlobalIPv6PrefixLen': 0, 'MacAddress': '02:42:ac:11:00:02', 'DriverOpts': None}}}}, 'invocation': {'module_args': {'image': 'nginx', 'state': 'started', 'name': 'my_webserver', 'docker_host': 'unix://var/run/docker.sock', 'tls_hostname': 'localhost', 'api_version': 'auto', 'timeout': 60, 'tls': False, 'use_ssh_client': False, 'validate_certs': False, 'debug': False, 'cleanup': False, 'force_kill': False, 'ignore_image': False, 'keep_volumes': True, 'networks_cli_compatible': True, 'output_logs': False, 'pull': False, 'purge_networks': False, 'recreate': False, 'restart': False, 'ca_cert': None, 'client_cert': None, 'client_key': None, 'ssl_version': None, 'auto_remove': False, 'blkio_weight': None, 'capabilities': None, 'cap_drop': None, 'cgroup_parent': None, 'command': None, 'comparisons': None, 'container_default_behavior': 'compatibility', 'cpu_period': None, 'cpu_quota': None, 'cpus': None, 'cpuset_cpus': None, 'cpuset_mems': None, 'cpu_shares': None, 'default_host_ip': None, 'detach': True, 'devices': None, 'device_read_bps': None, 'device_write_bps': None, 'device_read_iops': None, 'device_write_iops': None, 'device_requests': None, 'dns_servers': None, 'dns_opts': None, 'dns_search_domains': None, 'domainname': None, 'entrypoint': None, 'env': None, 'env_file': None, 'etc_hosts': None, 'exposed_ports': None, 'groups': None, 'healthcheck': None, 'hostname': None, 'init': False, 'interactive': False, 'ipc_mode': None, 'kernel_memory': None, 'kill_signal': None, 'labels': None, 'links': None, 'log_driver': None, 'log_options': None, 'mac_address': None, 'memory': '0', 'memory_reservation': None, 'memory_swap': None, 'memory_swappiness': None, 'mounts': None, 'network_mode': None, 'networks': None, 'oom_killer': None, 'oom_score_adj': None, 'paused': False, 'pid_mode': None, 'pids_limit': None, 'privileged': False, 'published_ports': None, 'read_only': False, 'removal_wait_timeout': None, 'restart_policy': None, 'restart_retries': None, 'runtime': None, 'security_opts': None, 'shm_size': None, 'stop_signal': None, 'stop_timeout': None, 'storage_opts': None, 'sysctls': None, 'tmpfs': None, 'tty': False, 'ulimits': None, 'user': None, 'userns_mode': None, 'uts': None, 'volume_driver': None, 'volumes': None, 'volumes_from': None, 'working_dir': None}}, '_ansible_no_log': False})
%% Cell type:code id:affected-zealand tags:
``` python
results.filter(task="docker_container")[0]
```
%% Output
CustomCommandResult(host='parasilo-17.rennes.grid5000.fr', task='docker_container', status='OK', payload={'changed': True, 'container': {'Id': 'b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce', 'Created': '2021-08-24T15:15:16.653171818Z', 'Path': '/docker-entrypoint.sh', 'Args': ['nginx', '-g', 'daemon off;'], 'State': {'Status': 'running', 'Running': True, 'Paused': False, 'Restarting': False, 'OOMKilled': False, 'Dead': False, 'Pid': 14720, 'ExitCode': 0, 'Error': '', 'StartedAt': '2021-08-24T15:15:19.434872036Z', 'FinishedAt': '0001-01-01T00:00:00Z'}, 'Image': 'sha256:dd34e67e3371dc2d1328790c3157ee42dfcae74afffd86b297459ed87a98c0fb', 'ResolvConfPath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/resolv.conf', 'HostnamePath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/hostname', 'HostsPath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/hosts', 'LogPath': '/var/lib/docker/containers/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce/b2bf90f48f4f20a75b124c6f03c724a999d07c9db6a19443f3fa2cf85516fdce-json.log', 'Name': '/my_webserver', 'RestartCount': 0, 'Driver': 'overlay2', 'Platform': 'linux', 'MountLabel': '', 'ProcessLabel': '', 'AppArmorProfile': 'docker-default', 'ExecIDs': None, 'HostConfig': {'Binds': [], 'ContainerIDFile': '', 'LogConfig': {'Type': 'json-file', 'Config': {}}, 'NetworkMode': 'default', 'PortBindings': None, 'RestartPolicy': {'Name': '', 'MaximumRetryCount': 0}, 'AutoRemove': False, 'VolumeDriver': '', 'VolumesFrom': None, 'CapAdd': None, 'CapDrop': None, 'CgroupnsMode': 'host', 'Dns': None, 'DnsOptions': None, 'DnsSearch': None, 'ExtraHosts': None, 'GroupAdd': None, 'IpcMode': 'private', 'Cgroup': '', 'Links': None, 'OomScoreAdj': 0, 'PidMode': '', 'Privileged': False, 'PublishAllPorts': False, 'ReadonlyRootfs': False, 'SecurityOpt': None, 'UTSMode': '', 'UsernsMode': '', 'ShmSize': 67108864, 'Runtime': 'runc', 'ConsoleSize': [0, 0], 'Isolation': '', 'CpuShares': 0, 'Memory': 0, 'NanoCpus': 0, 'CgroupParent': '', 'BlkioWeight': 0, 'BlkioWeightDevice': None, 'BlkioDeviceReadBps': None, 'BlkioDeviceWriteBps': None, 'BlkioDeviceReadIOps': None, 'BlkioDeviceWriteIOps': None, 'CpuPeriod': 0, 'CpuQuota': 0, 'CpuRealtimePeriod': 0, 'CpuRealtimeRuntime': 0, 'CpusetCpus': '', 'CpusetMems': '', 'Devices': None, 'DeviceCgroupRules': None, 'DeviceRequests': None, 'KernelMemory': 0, 'KernelMemoryTCP': 0, 'MemoryReservation': 0, 'MemorySwap': 0, 'MemorySwappiness': None, 'OomKillDisable': False, 'PidsLimit': None, 'Ulimits': None, 'CpuCount': 0, 'CpuPercent': 0, 'IOMaximumIOps': 0, 'IOMaximumBandwidth': 0, 'MaskedPaths': ['/proc/asound', '/proc/acpi', '/proc/kcore', '/proc/keys', '/proc/latency_stats', '/proc/timer_list', '/proc/timer_stats', '/proc/sched_debug', '/proc/scsi', '/sys/firmware'], 'ReadonlyPaths': ['/proc/bus', '/proc/fs', '/proc/irq', '/proc/sys', '/proc/sysrq-trigger'], 'Init': False}, 'GraphDriver': {'Data': {'LowerDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b-init/diff:/var/lib/docker/overlay2/b309b755d9ccbd962d1fa569ac24305182e792c0d856b382baf1148ea8f0d8cd/diff:/var/lib/docker/overlay2/c10519ef34f5c0ff8d8358d9bcf5ef5a6311f03b2903532759787e485450428a/diff:/var/lib/docker/overlay2/e8d94c857640f7ce2ec2214516b00bf53cb05ddfca8e19047dd87109c78880af/diff:/var/lib/docker/overlay2/2fcc44c10530f9c0b2616600ccb40c654fc355c834e1d812d2d0492f0ec97ddc/diff:/var/lib/docker/overlay2/8d4d0df6e602c7ade1c64221c19bbe38082176c42e45a38cab14de32cdfe141d/diff:/var/lib/docker/overlay2/1ada2f61beac7647403e96de6e0842b5e07ec8c8b8a9be8e017e126fe0a7d050/diff', 'MergedDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b/merged', 'UpperDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b/diff', 'WorkDir': '/var/lib/docker/overlay2/c2262636665c2019ef552f8a15e5d10bcd3269103a7cf56591bd66c9683f4d1b/work'}, 'Name': 'overlay2'}, 'Mounts': [], 'Config': {'Hostname': 'b2bf90f48f4f', 'Domainname': '', 'User': '', 'AttachStdin': False, 'AttachStdout': False, 'AttachStderr': False, 'ExposedPorts': {'80/tcp': {}}, 'Tty': False, 'OpenStdin': False, 'StdinOnce': False, 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'NGINX_VERSION=1.21.1', 'NJS_VERSION=0.6.1', 'PKG_RELEASE=1~buster'], 'Cmd': ['nginx', '-g', 'daemon off;'], 'Image': 'nginx', 'Volumes': None, 'WorkingDir': '', 'Entrypoint': ['/docker-entrypoint.sh'], 'OnBuild': None, 'Labels': {'maintainer': 'NGINX Docker Maintainers <docker-maint@nginx.com>'}, 'StopSignal': 'SIGQUIT'}, 'NetworkSettings': {'Bridge': '', 'SandboxID': 'c951d1b9e2ff3763235f519c5aa4a9a3dba30978071061bea5d55b6a1acb8893', 'HairpinMode': False, 'LinkLocalIPv6Address': '', 'LinkLocalIPv6PrefixLen': 0, 'Ports': {'80/tcp': None}, 'SandboxKey': '/var/run/docker/netns/c951d1b9e2ff', 'SecondaryIPAddresses': None, 'SecondaryIPv6Addresses': None, 'EndpointID': '0cd99a2f0e7d08629ce92b636189b1df338c24ec71751ac02148d5bb4148db83', 'Gateway': '172.17.0.1', 'GlobalIPv6Address': '', 'GlobalIPv6PrefixLen': 0, 'IPAddress': '172.17.0.2', 'IPPrefixLen': 16, 'IPv6Gateway': '', 'MacAddress': '02:42:ac:11:00:02', 'Networks': {'bridge': {'IPAMConfig': None, 'Links': None, 'Aliases': None, 'NetworkID': '68d355b984b0dfee3529070077588f3932d5700f5eafc4eb295b8cac7aea062e', 'EndpointID': '0cd99a2f0e7d08629ce92b636189b1df338c24ec71751ac02148d5bb4148db83', 'Gateway': '172.17.0.1', 'IPAddress': '172.17.0.2', 'IPPrefixLen': 16, 'IPv6Gateway': '', 'GlobalIPv6Address': '', 'GlobalIPv6PrefixLen': 0, 'MacAddress': '02:42:ac:11:00:02', 'DriverOpts': None}}}}, 'invocation': {'module_args': {'image': 'nginx', 'state': 'started', 'name': 'my_webserver', 'docker_host': 'unix://var/run/docker.sock', 'tls_hostname': 'localhost', 'api_version': 'auto', 'timeout': 60, 'tls': False, 'use_ssh_client': False, 'validate_certs': False, 'debug': False, 'cleanup': False, 'force_kill': False, 'ignore_image': False, 'keep_volumes': True, 'networks_cli_compatible': True, 'output_logs': False, 'pull': False, 'purge_networks': False, 'recreate': False, 'restart': False, 'ca_cert': None, 'client_cert': None, 'client_key': None, 'ssl_version': None, 'auto_remove': False, 'blkio_weight': None, 'capabilities': None, 'cap_drop': None, 'cgroup_parent': None, 'command': None, 'comparisons': None, 'container_default_behavior': 'compatibility', 'cpu_period': None, 'cpu_quota': None, 'cpus': None, 'cpuset_cpus': None, 'cpuset_mems': None, 'cpu_shares': None, 'default_host_ip': None, 'detach': True, 'devices': None, 'device_read_bps': None, 'device_write_bps': None, 'device_read_iops': None, 'device_write_iops': None, 'device_requests': None, 'dns_servers': None, 'dns_opts': None, 'dns_search_domains': None, 'domainname': None, 'entrypoint': None, 'env': None, 'env_file': None, 'etc_hosts': None, 'exposed_ports': None, 'groups': None, 'healthcheck': None, 'hostname': None, 'init': False, 'interactive': False, 'ipc_mode': None, 'kernel_memory': None, 'kill_signal': None, 'labels': None, 'links': None, 'log_driver': None, 'log_options': None, 'mac_address': None, 'memory': '0', 'memory_reservation': None, 'memory_swap': None, 'memory_swappiness': None, 'mounts': None, 'network_mode': None, 'networks': None, 'oom_killer': None, 'oom_score_adj': None, 'paused': False, 'pid_mode': None, 'pids_limit': None, 'privileged': False, 'published_ports': None, 'read_only': False, 'removal_wait_timeout': None, 'restart_policy': None, 'restart_retries': None, 'runtime': None, 'security_opts': None, 'shm_size': None, 'stop_signal': None, 'stop_timeout': None, 'storage_opts': None, 'sysctls': None, 'tmpfs': None, 'tty': False, 'ulimits': None, 'user': None, 'userns_mode': None, 'uts': None, 'volume_driver': None, 'volumes': None, 'volumes_from': None, 'working_dir': None}}, '_ansible_no_log': False})
%% Cell type:markdown id:existing-istanbul tags:
### Background actions
Sometime you need to fire a process on some remote machines that needs to survive the remote connection that started it. EnOSlib provides a `keyword` argument for this purpose and can be used when calling modules (when supported).
%% Cell type:code id:prepared-needle tags:
``` python
# synchronous execution, will wait until the end of the shell command
results = en.run_command("for i in $(seq 1 10); do sleep 1; echo toto; done", roles=roles)
results
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [for i in $(seq 1 10); do sleep 1; echo toto; done] **************************************************************************************************
[started TASK: for i in $(seq 1 10); do sleep 1; echo toto; done on parasilo-17.rennes.grid5000.fr]
[started TASK: for i in $(seq 1 10); do sleep 1; echo toto; done on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='for i in $(seq 1 10); do sleep 1; echo toto; done', status='OK', payload={'cmd': 'for i in $(seq 1 10); do sleep 1; echo toto; done', 'stdout': 'toto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:15:50.628843', 'end': '2021-08-24 17:16:00.648062', 'delta': '0:00:10.019219', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'for i in $(seq 1 10); do sleep 1; echo toto; done', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto'], 'stderr_lines': [], '_ansible_no_log': False})
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='for i in $(seq 1 10); do sleep 1; echo toto; done', status='OK', payload={'cmd': 'for i in $(seq 1 10); do sleep 1; echo toto; done', 'stdout': 'toto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto\ntoto', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:15:50.654538', 'end': '2021-08-24 17:16:00.672614', 'delta': '0:00:10.018076', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'for i in $(seq 1 10); do sleep 1; echo toto; done', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto', 'toto'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:driving-frederick tags:
``` python
# The remote command will be daemonize on the remote machines
results = en.run_command("for i in $(seq 1 10); do sleep 1; echo toto; done", roles=roles, background=True)
results
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [for i in $(seq 1 10); do sleep 1; echo toto; done] **************************************************************************************************
[started TASK: for i in $(seq 1 10); do sleep 1; echo toto; done on parasilo-17.rennes.grid5000.fr]
[started TASK: for i in $(seq 1 10); do sleep 1; echo toto; done on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
AsyncCommandResult(host='parasilo-24.rennes.grid5000.fr', task='for i in $(seq 1 10); do sleep 1; echo toto; done', status='OK', payload={'started': 1, 'finished': 0, 'ansible_job_id': '22525073620.27377', 'results_file': '/root/.ansible_async/22525073620.27377', 'changed': True, '_ansible_no_log': False})
AsyncCommandResult(host='parasilo-17.rennes.grid5000.fr', task='for i in $(seq 1 10); do sleep 1; echo toto; done', status='OK', payload={'started': 1, 'finished': 0, 'ansible_job_id': '944605924460.14911', 'results_file': '/root/.ansible_async/944605924460.14911', 'changed': True, '_ansible_no_log': False})
%% Cell type:code id:middle-ozone tags:
``` python
# you can get back the status of the daemonized process by reading the remote results_file
h = roles["control"][0]
result_file = results.filter(host=h.alias)[0].results_file
cat_result = en.run_command(f"cat {result_file}",roles=h)
cat_result
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [cat /root/.ansible_async/944605924460.14911] ********************************************************************************************************
[started TASK: cat /root/.ansible_async/944605924460.14911 on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='cat /root/.ansible_async/944605924460.14911', status='OK', payload={'cmd': 'cat /root/.ansible_async/944605924460.14911', 'stdout': '{"cmd": "for i in $(seq 1 10); do sleep 1; echo toto; done", "stdout": "toto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto", "stderr": "", "rc": 0, "start": "2021-08-24 17:16:16.207795", "end": "2021-08-24 17:16:26.218495", "delta": "0:00:10.010700", "changed": true, "invocation": {"module_args": {"_raw_params": "for i in $(seq 1 10); do sleep 1; echo toto; done", "_uses_shell": true, "warn": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:16:46.747535', 'end': '2021-08-24 17:16:46.750084', 'delta': '0:00:00.002549', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'cat /root/.ansible_async/944605924460.14911', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{"cmd": "for i in $(seq 1 10); do sleep 1; echo toto; done", "stdout": "toto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto\\ntoto", "stderr": "", "rc": 0, "start": "2021-08-24 17:16:16.207795", "end": "2021-08-24 17:16:26.218495", "delta": "0:00:10.010700", "changed": true, "invocation": {"module_args": {"_raw_params": "for i in $(seq 1 10); do sleep 1; echo toto; done", "_uses_shell": true, "warn": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:cognitive-travel tags:
``` python
# the result_file content is json encoded so decoding it
import json
print(json.loads(cat_result[0].stdout)["stdout"])
```
%% Output
toto
toto
toto
toto
toto
toto
toto
toto
toto
toto
%% Cell type:markdown id:involved-screening tags:
## Using variables
%% Cell type:markdown id:impressive-gibraltar tags:
### Same variable value for everyone
Nothing surprising here, you can use regular python interpolation (e.g a `f-string`).
String are interpolated by the interpreter before being manipulated.
%% Cell type:code id:ideal-transparency tags:
``` python
host_to_ping = roles["control"][0].alias
host_to_ping
results = en.run_command(f"ping -c 5 {host_to_ping}", roles=roles)
results
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [ping -c 5 parasilo-17.rennes.grid5000.fr] ***********************************************************************************************************
[started TASK: ping -c 5 parasilo-17.rennes.grid5000.fr on parasilo-17.rennes.grid5000.fr]
[started TASK: ping -c 5 parasilo-17.rennes.grid5000.fr on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='ping -c 5 parasilo-17.rennes.grid5000.fr', status='OK', payload={'cmd': 'ping -c 5 parasilo-17.rennes.grid5000.fr', 'stdout': 'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.035 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.014 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.031 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.032 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.031 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 87ms\nrtt min/avg/max/mdev = 0.014/0.028/0.035/0.009 ms', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:21:12.043346', 'end': '2021-08-24 17:21:16.131011', 'delta': '0:00:04.087665', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'ping -c 5 parasilo-17.rennes.grid5000.fr', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.035 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.014 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.031 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.032 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.031 ms', '', '--- parasilo-17.rennes.grid5000.fr ping statistics ---', '5 packets transmitted, 5 received, 0% packet loss, time 87ms', 'rtt min/avg/max/mdev = 0.014/0.028/0.035/0.009 ms'], 'stderr_lines': [], '_ansible_no_log': False})
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='ping -c 5 parasilo-17.rennes.grid5000.fr', status='OK', payload={'cmd': 'ping -c 5 parasilo-17.rennes.grid5000.fr', 'stdout': 'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.188 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.192 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.203 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.128 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.146 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 95ms\nrtt min/avg/max/mdev = 0.128/0.171/0.203/0.031 ms', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:21:12.076227', 'end': '2021-08-24 17:21:16.172052', 'delta': '0:00:04.095825', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'ping -c 5 parasilo-17.rennes.grid5000.fr', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.188 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.192 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.203 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.128 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.146 ms', '', '--- parasilo-17.rennes.grid5000.fr ping statistics ---', '5 packets transmitted, 5 received, 0% packet loss, time 95ms', 'rtt min/avg/max/mdev = 0.128/0.171/0.203/0.031 ms'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:bored-cardiff tags:
``` python
[(r.host, r.stdout) for r in results]
```
%% Output
[('parasilo-17.rennes.grid5000.fr',
'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.035 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.014 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.031 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.032 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.031 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 87ms\nrtt min/avg/max/mdev = 0.014/0.028/0.035/0.009 ms'),
('parasilo-24.rennes.grid5000.fr',
'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.188 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.192 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.203 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.128 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.146 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 95ms\nrtt min/avg/max/mdev = 0.128/0.171/0.203/0.031 ms')]
%% Cell type:markdown id:disciplinary-participation tags:
### Using templates / Ansible variables
There's an alternative way to pass a variable to a task: using `extra_vars`.
The difference with the previous case (python interpreted variables) is the fact that the variable is interpolated right before execution happens on the remote node.
One could imagine the the value is broadcasted to all nodes and replaced right before the execution.
To indicate that we want to use this kind of variables, we need to pass its value using the `extra_vars` dictionnary and use a template (`{{ ... }}`) in the task description.
%% Cell type:code id:solar-eleven tags:
``` python
host_to_ping = roles["control"][0].alias
host_to_ping
results = en.run_command("ping -c 5 {{ my_template_variable }}", roles=roles, extra_vars=dict(my_template_variable=host_to_ping))
results
```
%% Output
INFO:enoslib.api:extra_vars = {'my_template_variable': 'parasilo-17.rennes.grid5000.fr', 'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [ping -c 5 parasilo-17.rennes.grid5000.fr] ***********************************************************************************************************
[started TASK: ping -c 5 {{ my_template_variable }} on parasilo-17.rennes.grid5000.fr]
[started TASK: ping -c 5 {{ my_template_variable }} on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='ping -c 5 parasilo-17.rennes.grid5000.fr', status='OK', payload={'cmd': 'ping -c 5 parasilo-17.rennes.grid5000.fr', 'stdout': 'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.143 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.188 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.188 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.158 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.091 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 89ms\nrtt min/avg/max/mdev = 0.091/0.153/0.188/0.038 ms', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:30:34.258643', 'end': '2021-08-24 17:30:38.347948', 'delta': '0:00:04.089305', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'ping -c 5 parasilo-17.rennes.grid5000.fr', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.143 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.188 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.188 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.158 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.091 ms', '', '--- parasilo-17.rennes.grid5000.fr ping statistics ---', '5 packets transmitted, 5 received, 0% packet loss, time 89ms', 'rtt min/avg/max/mdev = 0.091/0.153/0.188/0.038 ms'], 'stderr_lines': [], '_ansible_no_log': False})
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='ping -c 5 parasilo-17.rennes.grid5000.fr', status='OK', payload={'cmd': 'ping -c 5 parasilo-17.rennes.grid5000.fr', 'stdout': 'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.022 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.032 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.031 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.032 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.014 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 94ms\nrtt min/avg/max/mdev = 0.014/0.026/0.032/0.007 ms', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:30:34.308295', 'end': '2021-08-24 17:30:38.402891', 'delta': '0:00:04.094596', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'ping -c 5 parasilo-17.rennes.grid5000.fr', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.022 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.032 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.031 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.032 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.014 ms', '', '--- parasilo-17.rennes.grid5000.fr ping statistics ---', '5 packets transmitted, 5 received, 0% packet loss, time 94ms', 'rtt min/avg/max/mdev = 0.014/0.026/0.032/0.007 ms'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:markdown id:psychological-burns tags:
### Host specific variables
In the above, we've seen how a common value can be broadcasted to all remote nodes. What if we want host specific value ?
For instance in our case we'd like `host 1` to ping `host 2` and `host 2` to ping `host 1`. That make the `host_to_ping` variable host-specific.
For this purpose you can use the `extra` attribute of the `Host` objects and use a template as before.
%% Cell type:code id:irish-syria tags:
``` python
control_host = roles["control"][0]
compute_host = roles["compute"][0]
control_host.extra.update(host_to_ping=compute_host.address)
compute_host.extra.update(host_to_ping=control_host.address)
```
%% Cell type:markdown id:cooperative-market tags:
> Note that the `extra` attribute is mutable :(
%% Cell type:code id:joined-stand tags:
``` python
results = en.run_command("ping -c 5 {{ host_to_ping }}", roles=roles)
results
```
%% Output
INFO:enoslib.api:extra_vars = {'ansible_python_interpreter': 'python3'}
PLAY [all] ************************************************************************************************************************************************
TASK [ping -c 5 parasilo-24.rennes.grid5000.fr] ***********************************************************************************************************
[started TASK: ping -c 5 {{ host_to_ping }} on parasilo-17.rennes.grid5000.fr]
[started TASK: ping -c 5 {{ host_to_ping }} on parasilo-24.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-24.rennes.grid5000.fr]
CommandResult(host='parasilo-17.rennes.grid5000.fr', task='ping -c 5 parasilo-24.rennes.grid5000.fr', status='OK', payload={'cmd': 'ping -c 5 parasilo-24.rennes.grid5000.fr', 'stdout': 'PING parasilo-24.rennes.grid5000.fr (172.16.97.24) 56(84) bytes of data.\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=1 ttl=64 time=0.172 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=2 ttl=64 time=0.113 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=3 ttl=64 time=0.159 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=4 ttl=64 time=0.193 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=5 ttl=64 time=0.166 ms\n\n--- parasilo-24.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 91ms\nrtt min/avg/max/mdev = 0.113/0.160/0.193/0.029 ms', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:38:44.999426', 'end': '2021-08-24 17:38:49.091165', 'delta': '0:00:04.091739', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'ping -c 5 parasilo-24.rennes.grid5000.fr', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['PING parasilo-24.rennes.grid5000.fr (172.16.97.24) 56(84) bytes of data.', '64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=1 ttl=64 time=0.172 ms', '64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=2 ttl=64 time=0.113 ms', '64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=3 ttl=64 time=0.159 ms', '64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=4 ttl=64 time=0.193 ms', '64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=5 ttl=64 time=0.166 ms', '', '--- parasilo-24.rennes.grid5000.fr ping statistics ---', '5 packets transmitted, 5 received, 0% packet loss, time 91ms', 'rtt min/avg/max/mdev = 0.113/0.160/0.193/0.029 ms'], 'stderr_lines': [], '_ansible_no_log': False})
CommandResult(host='parasilo-24.rennes.grid5000.fr', task='ping -c 5 parasilo-17.rennes.grid5000.fr', status='OK', payload={'cmd': 'ping -c 5 parasilo-17.rennes.grid5000.fr', 'stdout': 'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.105 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.154 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.139 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.189 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.092 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 99ms\nrtt min/avg/max/mdev = 0.092/0.135/0.189/0.037 ms', 'stderr': '', 'rc': 0, 'start': '2021-08-24 17:38:45.000970', 'end': '2021-08-24 17:38:49.100012', 'delta': '0:00:04.099042', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'ping -c 5 parasilo-17.rennes.grid5000.fr', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.105 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.154 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.139 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.189 ms', '64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.092 ms', '', '--- parasilo-17.rennes.grid5000.fr ping statistics ---', '5 packets transmitted, 5 received, 0% packet loss, time 99ms', 'rtt min/avg/max/mdev = 0.092/0.135/0.189/0.037 ms'], 'stderr_lines': [], '_ansible_no_log': False})
%% Cell type:code id:nervous-difference tags:
``` python
[(r.host, r.stdout) for r in results]
```
%% Output
[('parasilo-17.rennes.grid5000.fr',
'PING parasilo-24.rennes.grid5000.fr (172.16.97.24) 56(84) bytes of data.\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=1 ttl=64 time=0.172 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=2 ttl=64 time=0.113 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=3 ttl=64 time=0.159 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=4 ttl=64 time=0.193 ms\n64 bytes from parasilo-24.rennes.grid5000.fr (172.16.97.24): icmp_seq=5 ttl=64 time=0.166 ms\n\n--- parasilo-24.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 91ms\nrtt min/avg/max/mdev = 0.113/0.160/0.193/0.029 ms'),
('parasilo-24.rennes.grid5000.fr',
'PING parasilo-17.rennes.grid5000.fr (172.16.97.17) 56(84) bytes of data.\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=1 ttl=64 time=0.105 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=2 ttl=64 time=0.154 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=3 ttl=64 time=0.139 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=4 ttl=64 time=0.189 ms\n64 bytes from parasilo-17.rennes.grid5000.fr (172.16.97.17): icmp_seq=5 ttl=64 time=0.092 ms\n\n--- parasilo-17.rennes.grid5000.fr ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 99ms\nrtt min/avg/max/mdev = 0.092/0.135/0.189/0.037 ms')]
%% Cell type:markdown id:practical-arnold tags:
## Cleaning
%% Cell type:code id:decent-syracuse tags:
``` python
provider.destroy()
```
%% Output
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from lille
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from luxembourg
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from lyon
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from nancy
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from nantes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading 1815745 from rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading rsd-01 from sophia
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Killing the job (rennes, 1815745)
%% Cell type:code id:hawaiian-crash tags:
``` python
```
......
%% Cell type:markdown id:regular-opening tags:
# Observability facilities
Third party software stack to keep an eye on your experiment or gather some metrics.
---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib
---
More generally, useful service for the experimentation are consolidated through `Services`.
In particular, EnOSlib has some some Services which deal with the problem of getting some insight on what's running on remote nodes.
A Service is a python object which exposes three main methods:
- `deploy`: which deploy the service
- `destroy`: remove stop the service
- `backup`: retrieve some states of the services (e.g monitoring information)
Usually a service is used as follow:
```python
service = Service(*args, **kwargs)
service.deploy()
...
# do stuffs
...
service.backup()
service.destroy()
```
But it's sometime useful to use a Context Manager when working with module:
```python
with Service(*args, **kwargs) as service:
...
# do stuffs
...
```
This allows for
- running the service for some time depending on what's inside the context manager
- cleaning (and backuping) stuffs automatically at the end
---
**Prerequisites:**
- Grid'5000 account
- A Grid'5000 account
- A working EnOSlib environment and Jupyter (not included in EnOSlib dependencies, but `pip install jupyterlab` will install it)
- Some visualization tools: `matplotlib`, `seaborn`
- Packet manipulation tool: `scapy`
---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib
---
%% Cell type:markdown id:3c2970ab-cc25-4147-bee4-d9af3fff5e64 tags:
## Common setup
%% Cell type:code id:0f2cce82-cc4b-4540-9fa1-b0b85c8b0c53 tags:
``` python
import enoslib as en
# get some logging info
import logging
logging.basicConfig(level=logging.INFO)
```
%% Cell type:code id:8635ce64-4978-499a-84b3-1151489c1c14 tags:
``` python
# claim the resources
network = en.G5kNetworkConf(id="n1", type="prod", roles=["my_network"], site="rennes")
conf = (
en.G5kConf.from_settings(job_type="allow_classic_ssh", job_name="enoslib_observability")
.add_network_conf(network)
.add_machine(
roles=["control", "xp"], cluster="parasilo", nodes=1, primary_network=network
)
.add_machine(
roles=["agent", "xp"], cluster="parasilo", nodes=1, primary_network=network
)
.finalize()
)
conf
```
%% Output
Conf@0x7fab4ed3edd0
{
"dhcp": true,
"force_deploy": false,
"env_name": "debian10-x64-nfs",
"job_name": "enoslib_observability",
"job_type": "allow_classic_ssh",
"key": "/home/msimonin/.ssh/id_rsa.pub",
"queue": "default",
"walltime": "02:00:00",
"resources": {
"machines": [
{
"roles": [
"control",
"xp"
],
"primary_network": "n1",
"secondary_networks": [],
"cluster": "parasilo",
"nodes": 1
},
{
"roles": [
"agent",
"xp"
],
"primary_network": "n1",
"secondary_networks": [],
"cluster": "parasilo",
"nodes": 1
}
],
"networks": [
{
"id": "n1",
"type": "prod",
"roles": [
"my_network"
],
"site": "rennes"
}
]
}
}
%% Cell type:code id:b847c50d-862f-4b78-be82-78e5aab5bfb8 tags:
``` python
provider = en.G5k(conf)
roles, networks = provider.init()
roles
```
%% Output
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from lille
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from luxembourg
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from lyon
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from nancy
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from nantes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from sophia
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Submitting {'name': 'enoslib_observability', 'types': ['allow_classic_ssh'], 'resources': "{cluster='parasilo'}/nodes=1+{cluster='parasilo'}/nodes=1,walltime=02:00:00", 'command': 'sleep 31536000', 'queue': 'default'} on rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Waiting for 1815850 on rennes [2021-08-25 13:41:35]
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Waiting for 1815850 on rennes [2021-08-25 13:41:35]
INFO:enoslib.infra.enos_g5k.g5k_api_utils:All jobs are Running !
{'control': [Host(address='parasilo-15.rennes.grid5000.fr', alias='parasilo-15.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())], 'xp': [Host(address='parasilo-15.rennes.grid5000.fr', alias='parasilo-15.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set()), Host(address='parasilo-17.rennes.grid5000.fr', alias='parasilo-17.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())], 'agent': [Host(address='parasilo-17.rennes.grid5000.fr', alias='parasilo-17.rennes.grid5000.fr', user='root', keyfile=None, port=None, extra={}, net_devices=set())]}
%% Cell type:markdown id:ab28c593-7ae4-40bb-a584-774e0b37d70c tags:
### A simple load generator
We'll install a simple load generator: `stress` available in the debian packages.
%% Cell type:code id:fcaca8e8-a6a3-43ef-8fc6-e1ddf86eac9f tags:
``` python
with en.actions(roles=roles["agent"]) as a:
a.apt(name="stress", state="present")
```
%% Output
[WARNING]: No inventory was parsed, only implicit localhost is available
PLAY [all] ************************************************************************************************************************************************
TASK [apt] ************************************************************************************************************************************************
[started TASK: apt on parasilo-17.rennes.grid5000.fr]
[WARNING]: Updating cache and auto-installing missing dependency: python3-apt
ok: [parasilo-17.rennes.grid5000.fr]
%% Cell type:markdown id:581b0b4e-427e-4cf2-957a-1fe2dcdcf737 tags:
## Monitoring with dstat
Dstat is a simple monitoring tool: https://github.com/dstat-real/dstat#information
It runs as a single process and collect metrics from various sources.
That makes it a good candidate for getting a quick insight on the resources consumptions during an experiment.
The EnOSlib implementation lets you easily
- start Dstat processes on remote machine and start dumping the metrics into a csv file( it's the purpose `deploy` method of the Dstat service)
- retrieve all the csvs file (one per remote node) on your local machine ( that's the purpose of the `backup` method)
- stop every remote Dstat processes (that's the purpose of the `destroy` method)
%% Cell type:markdown id:naughty-damages tags:
### Capture
Let's start with a single capture implemented using a context manager.
%% Cell type:code id:hairy-grade tags:
``` python
# Start a capture on all nodes
# - stress on some nodes
import time
with en.Dstat(nodes=roles["xp"]) as d:
time.sleep(5)
en.run_command("stress --cpu 4 --timeout 30", roles=roles["agent"])
time.sleep(5)
backup_dir = d.backup_dir
```
%% Output
PLAY [all] ************************************************************************************************************************************************
TASK [(tmux list-panes -t __enoslib_dstat__ -F '#{pane_pid}' | xargs -n1 kill -2) || true] ****************************************************************
[started TASK: (tmux list-panes -t __enoslib_dstat__ -F '#{pane_pid}' | xargs -n1 kill -2) || true on parasilo-15.rennes.grid5000.fr]
[started TASK: (tmux list-panes -t __enoslib_dstat__ -F '#{pane_pid}' | xargs -n1 kill -2) || true on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [apt] ************************************************************************************************************************************************
[started TASK: apt on parasilo-15.rennes.grid5000.fr]
[started TASK: apt on parasilo-17.rennes.grid5000.fr]
ok: [parasilo-17.rennes.grid5000.fr]
ok: [parasilo-15.rennes.grid5000.fr]
TASK [file] ***********************************************************************************************************************************************
[started TASK: file on parasilo-15.rennes.grid5000.fr]
[started TASK: file on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [get_url] ********************************************************************************************************************************************
[started TASK: get_url on parasilo-15.rennes.grid5000.fr]
[started TASK: get_url on parasilo-17.rennes.grid5000.fr]
[WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To
avoid this, create the remote_tmp dir with the correct permissions manually
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
TASK [Running dstat with the options -aT -o 1629891716952297243-dstat.csv] ********************************************************************************
[started TASK: Running dstat with the options -aT -o 1629891716952297243-dstat.csv on parasilo-15.rennes.grid5000.fr]
[started TASK: Running dstat with the options -aT -o 1629891716952297243-dstat.csv on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [stress --cpu 4 --timeout 30] ************************************************************************************************************************
[started TASK: stress --cpu 4 --timeout 30 on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [(tmux list-panes -t __enoslib_dstat__ -F '#{pane_pid}' | xargs -n1 kill -2) || true] ****************************************************************
[started TASK: (tmux list-panes -t __enoslib_dstat__ -F '#{pane_pid}' | xargs -n1 kill -2) || true on parasilo-15.rennes.grid5000.fr]
[started TASK: (tmux list-panes -t __enoslib_dstat__ -F '#{pane_pid}' | xargs -n1 kill -2) || true on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [Fetching the dstat output] **************************************************************************************************************************
[started TASK: Fetching the dstat output on parasilo-15.rennes.grid5000.fr]
[started TASK: Fetching the dstat output on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
%% Cell type:markdown id:73813968-8e9c-4a22-8e2a-ffcc5b0d6b44 tags:
### Visualization
All the CSVs files are available under the `backup_dir` inside subdirectories named after the corresponding remote host alias:
```bash
<backup_sir> / host1 / ... / <metrics>.csv
/ host2 / ..../ <metrics>.csv
```
The following bunch of python lines will recursively look for any csv file inside these directories and build a DataFrame and a visualization
%% Cell type:code id:e4211395-0d90-434a-8795-557d7dd79c22 tags:
``` python
import pandas as pd
import seaborn as sns
#Create a dictionnary of (alias) -> list of pandas df
result = pd.DataFrame()
for host in roles["xp"]:
host_dir = backup_dir / host.alias
csvs = host_dir.rglob("*.csv")
for csv in csvs:
df = pd.read_csv(csv, skiprows=5, index_col=False)
df["host"] = host.alias
df["csv"] = csv
result = pd.concat([result, df], axis=0)
result
```
%% Output
usr sys idl wai stl read writ recv send in \
0 1.124 0.373 98.430 0.073 0 55802.746 875486.983 0 0 0
1 0.031 0.000 99.969 0.000 0 0.000 0.000 106 132 0
2 0.000 0.000 99.969 0.031 0 0.000 143360.000 0 0 0
3 0.031 0.000 99.969 0.000 0 0.000 0.000 106 2020 0
4 0.000 0.000 100.000 0.000 0 0.000 0.000 296 228 0
.. ... ... ... ... ... ... ... ... ... ..
36 4.614 0.220 95.166 0.000 0 0.000 0.000 1800 47096 0
37 0.031 0.000 99.969 0.000 0 0.000 4096.000 106 0 0
38 0.000 0.000 100.000 0.000 0 0.000 0.000 0 0 0
39 0.000 0.000 100.000 0.000 0 0.000 0.000 288 0 0
40 0.000 0.031 99.969 0.000 0 0.000 0.000 182 0 0
... csw run blk new 1m 5m 15m epoch \
0 ... 5187.45 0 0 8.971 0.18 0.05 0.01 1.629892e+09
1 ... 72.00 0 0 0.000 0.18 0.05 0.01 1.629892e+09
2 ... 92.00 0 0 1.000 0.18 0.05 0.01 1.629892e+09
3 ... 169.00 0 0 0.000 0.16 0.04 0.01 1.629892e+09
4 ... 83.00 0 0 0.000 0.16 0.04 0.01 1.629892e+09
.. ... ... ... ... ... ... ... ... ...
36 ... 22462.00 0 0 6.000 1.65 0.41 0.14 1.629892e+09
37 ... 1322.00 0 0 0.000 1.65 0.41 0.14 1.629892e+09
38 ... 74.00 0 0 0.000 1.65 0.41 0.14 1.629892e+09
39 ... 159.00 0 0 2.000 1.65 0.41 0.14 1.629892e+09
40 ... 75.00 0 0 0.000 1.65 0.41 0.14 1.629892e+09
host \
0 parasilo-15.rennes.grid5000.fr
1 parasilo-15.rennes.grid5000.fr
2 parasilo-15.rennes.grid5000.fr
3 parasilo-15.rennes.grid5000.fr
4 parasilo-15.rennes.grid5000.fr
.. ...
36 parasilo-17.rennes.grid5000.fr
37 parasilo-17.rennes.grid5000.fr
38 parasilo-17.rennes.grid5000.fr
39 parasilo-17.rennes.grid5000.fr
40 parasilo-17.rennes.grid5000.fr
csv
0 /home/msimonin/workspace/repos/enoslib/docs/ju...
1 /home/msimonin/workspace/repos/enoslib/docs/ju...
2 /home/msimonin/workspace/repos/enoslib/docs/ju...
3 /home/msimonin/workspace/repos/enoslib/docs/ju...
4 /home/msimonin/workspace/repos/enoslib/docs/ju...
.. ...
36 /home/msimonin/workspace/repos/enoslib/docs/ju...
37 /home/msimonin/workspace/repos/enoslib/docs/ju...
38 /home/msimonin/workspace/repos/enoslib/docs/ju...
39 /home/msimonin/workspace/repos/enoslib/docs/ju...
40 /home/msimonin/workspace/repos/enoslib/docs/ju...
[82 rows x 26 columns]
%% Cell type:code id:01063388-546a-4615-8141-9b6366e9b1d4 tags:
``` python
# let's show the metrics !
sns.lineplot(data=result, x="epoch", y="usr", hue="host", markers=True, style="host")
```
%% Output
<AxesSubplot:xlabel='epoch', ylabel='usr'>
%% Cell type:markdown id:0bf1d176-a24d-42f8-b258-7b9e81bf8f6e tags:
## Packet sniffing with tcpdump
### Capture
%% Cell type:code id:9d309239-febb-4f37-9680-a9ff344d089a tags:
``` python
# start a capture
# - on all the interface configured on the my_network network
# - we dump icmp traffic only
# - for the duration of the commands (here a client is pigging the server)
with en.TCPDump(
hosts=roles["xp"], ifnames=["any"], options="icmp"
) as t:
backup_dir = t.backup_dir
_ = en.run(f"ping -c10 {roles['control'][0].address}", roles["agent"])
```
%% Output
PLAY [all] ************************************************************************************************************************************************
TASK [Stopping tcpdump on any] ****************************************************************************************************************************
[started TASK: Stopping tcpdump on any on parasilo-15.rennes.grid5000.fr]
[started TASK: Stopping tcpdump on any on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [debug] **********************************************************************************************************************************************
[started TASK: debug on parasilo-15.rennes.grid5000.fr]
[started TASK: debug on parasilo-17.rennes.grid5000.fr]
ok: [parasilo-15.rennes.grid5000.fr] => {
"tcpdump_ifs": []
}
ok: [parasilo-17.rennes.grid5000.fr] => {
"tcpdump_ifs": []
}
TASK [Stopping some tcpdumps] *****************************************************************************************************************************
[started TASK: Stopping some tcpdumps on parasilo-15.rennes.grid5000.fr]
[started TASK: Stopping some tcpdumps on parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [Install dependencies (tcpdump, tmux ...)] ***********************************************************************************************************
[started TASK: Install dependencies (tcpdump, tmux ...) on parasilo-15.rennes.grid5000.fr]
[started TASK: Install dependencies (tcpdump, tmux ...) on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [Create output directory] ****************************************************************************************************************************
[started TASK: Create output directory on parasilo-15.rennes.grid5000.fr]
[started TASK: Create output directory on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [tcpdump for any] ************************************************************************************************************************************
[started TASK: tcpdump for any on parasilo-15.rennes.grid5000.fr]
[started TASK: tcpdump for any on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [debug] **********************************************************************************************************************************************
[started TASK: debug on parasilo-15.rennes.grid5000.fr]
[started TASK: debug on parasilo-17.rennes.grid5000.fr]
ok: [parasilo-15.rennes.grid5000.fr] => {
"tcpdump_ifs": []
}
ok: [parasilo-17.rennes.grid5000.fr] => {
"tcpdump_ifs": []
}
TASK [tcpdump on some interfaces] *************************************************************************************************************************
[started TASK: tcpdump on some interfaces on parasilo-15.rennes.grid5000.fr]
[started TASK: tcpdump on some interfaces on parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [ping -c10 parasilo-15.rennes.grid5000.fr] ***********************************************************************************************************
[started TASK: ping -c10 parasilo-15.rennes.grid5000.fr on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [Stopping tcpdump on any] ****************************************************************************************************************************
[started TASK: Stopping tcpdump on any on parasilo-15.rennes.grid5000.fr]
[started TASK: Stopping tcpdump on any on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [debug] **********************************************************************************************************************************************
[started TASK: debug on parasilo-15.rennes.grid5000.fr]
[started TASK: debug on parasilo-17.rennes.grid5000.fr]
ok: [parasilo-15.rennes.grid5000.fr] => {
"tcpdump_ifs": []
}
ok: [parasilo-17.rennes.grid5000.fr] => {
"tcpdump_ifs": []
}
TASK [Stopping some tcpdumps] *****************************************************************************************************************************
[started TASK: Stopping some tcpdumps on parasilo-15.rennes.grid5000.fr]
[started TASK: Stopping some tcpdumps on parasilo-17.rennes.grid5000.fr]
PLAY [all] ************************************************************************************************************************************************
TASK [tar -czf tcpdump.tar.gz /tmp/__enoslib_tcpdump__] ***************************************************************************************************
[started TASK: tar -czf tcpdump.tar.gz /tmp/__enoslib_tcpdump__ on parasilo-15.rennes.grid5000.fr]
[started TASK: tar -czf tcpdump.tar.gz /tmp/__enoslib_tcpdump__ on parasilo-17.rennes.grid5000.fr]
[WARNING]: Consider using the unarchive module rather than running 'tar'. If you need to use command because unarchive is insufficient you can add 'warn:
false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
TASK [fetch] **********************************************************************************************************************************************
[started TASK: fetch on parasilo-15.rennes.grid5000.fr]
[started TASK: fetch on parasilo-17.rennes.grid5000.fr]
changed: [parasilo-15.rennes.grid5000.fr]
changed: [parasilo-17.rennes.grid5000.fr]
%% Cell type:markdown id:b2bc5992-cf82-4286-860f-532576ebfb4d tags:
### Visualization
%% Cell type:code id:eb5c264e-cc51-4308-8107-bb439a05b2b0 tags:
``` python
from scapy.all import rdpcap
import tarfile
# Examples:
# create a dictionnary of (alias, if) -> list of decoded packets by scapy
decoded_pcaps = dict()
for host in roles["control"]:
host_dir = backup_dir / host.alias
t = tarfile.open(host_dir / "tcpdump.tar.gz")
t.extractall(host_dir / "extracted")
# get all extracted pcap for this host
pcaps = (host_dir / "extracted").rglob("*.pcap")
for pcap in pcaps:
decoded_pcaps.setdefault((host.alias, pcap.with_suffix("").name),
rdpcap(str(pcap)))
# Displaying some packets
for (host, ifs), packets in decoded_pcaps.items():
print(host, ifs)
packets[0].show()
packets[1].show()
```
%% Output
parasilo-15.rennes.grid5000.fr any
###[ cooked linux ]###
pkttype = unicast
lladdrtype= 0x1
lladdrlen = 6
src = '\\xec\\xf4\\xbb\\xd1\x01\\xb8'
proto = IPv4
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 84
id = 45451
flags = DF
frag = 0
ttl = 64
proto = icmp
chksum = 0x6edc
src = 172.16.97.17
dst = 172.16.97.15
\options \
###[ ICMP ]###
type = echo-request
code = 0
chksum = 0x5032
id = 0x762e
seq = 0x1
unused = ''
###[ Raw ]###
load = '\\xbf,&a\x00\x00\x00\x00\\x87=\x06\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'
###[ cooked linux ]###
pkttype = unicast
lladdrtype= 0x1
lladdrlen = 6
src = '\\xec\\xf4\\xbb\\xd1\x01\\xb8'
proto = IPv4
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 84
id = 45451
flags = DF
frag = 0
ttl = 64
proto = icmp
chksum = 0x6edc
src = 172.16.97.17
dst = 172.16.97.15
\options \
###[ ICMP ]###
type = echo-request
code = 0
chksum = 0x5032
id = 0x762e
seq = 0x1
unused = ''
###[ Raw ]###
load = '\\xbf,&a\x00\x00\x00\x00\\x87=\x06\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'
%% Cell type:markdown id:3f8e67c4-5e61-4a02-b57c-34de0d160825 tags:
## (TODO) Monitoring with Telegraf/[InfluxDB|promtheus]
%% Cell type:markdown id:5934e2e0-ce4f-41ce-8615-a7dd466a5b16 tags:
## (TODO) Network observability with Skydive
%% Cell type:markdown id:15b9b539-2c3f-476c-9e4c-2e10813ed9dc tags:
## Cleaning
%% Cell type:code id:0ba9c728-080a-4227-9882-971f85bce55e tags:
``` python
provider.destroy()
```
%% Output
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from lille
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from luxembourg
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from lyon
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from nancy
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from nantes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading 1815850 from rennes
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Reloading enoslib_observability from sophia
INFO:enoslib.infra.enos_g5k.g5k_api_utils:Killing the job (rennes, 1815850)
%% Cell type:code id:c5a5451f-e7cb-45cf-9dd4-78435ac4db7d tags:
``` python
```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment