Mentions légales du service

Skip to content
Snippets Groups Projects
Commit 91ac9195 authored by Martin QUINSON's avatar Martin QUINSON
Browse files

Typos in 00_setup_and_basics.ipynb

parent 9e3f9557
No related branches found
No related tags found
1 merge request!1Typos in 00_setup_and_basics.ipynb
%% Cell type:markdown id:direct-johnston tags: %% Cell type:markdown id:direct-johnston tags:
# Setup and basic objects # Setup and basic objects
Get started with EnOSlib on Grid'5000. Get started with EnOSlib on Grid'5000.
--- ---
- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html - Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
- Instant chat: https://framateam.org/enoslib - Instant chat: https://framateam.org/enoslib
- Source code: https://gitlab.inria.fr/discovery/enoslib - Source code: https://gitlab.inria.fr/discovery/enoslib
--- ---
This is the first notebooks of a series that will let you discover the main features of EnOSlib on Grid'5000. This is the first notebook in a serie that will let you discover the main features of EnOSlib on Grid'5000.
If you want to actually execute them you'll need to setup your environment properly. If you want to actually execute them you'll need to setup your environment properly.
We sum up here the different steps to achieve this process. We sum up here the different steps to achieve this process.
1. Get a Grid'5000 account 1. Get a Grid'5000 account
- Register using this [page](https://www.grid5000.fr/w/Grid5000:Get_an_account). - Register using this [page](https://www.grid5000.fr/w/Grid5000:Get_an_account).
Pay attention to the fact that uploading a SSH key (public part) is mandatory to perform any EnOSlib action from your local machine. Pay attention to the fact that uploading a SSH key (public part) is mandatory to perform any EnOSlib action from your local machine.
- Make sure the SSH connection is ready. You can follow this [tutorial](https://www.grid5000.fr/w/Getting_Started). - Make sure the SSH connection is ready. You can follow this [tutorial](https://www.grid5000.fr/w/Getting_Started).
2. Make sure EnOSlib is available in your notebook environment 2. Make sure EnOSlib is available in your notebook environment
- Follow the steps [here](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#installation). - Follow the steps [here](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#installation).
Using a virtualenv is the way to go, make sure to use one. Using a virtualenv is the way to go, make sure to use one.
Also adding the optional `jupyter` will improve your experience. (`pip install enoslib[jupyter]`) Also adding the optional `jupyter` will improve your experience. (`pip install enoslib[jupyter]`)
%% Cell type:markdown id:ceramic-burst tags: %% Cell type:markdown id:ceramic-burst tags:
## Testing the import ## Testing the import
%% Cell type:code id:thorough-maker tags: %% Cell type:code id:thorough-maker tags:
``` python ``` python
import enoslib as en import enoslib as en
``` ```
%% Cell type:markdown id:civilian-determination tags: %% Cell type:markdown id:civilian-determination tags:
## Resources abstractions ## Resources abstractions
In this notebook we won't execute anything remotely, instead we'll just cover some basic abstractions provided by the library. In this notebook, we won't execute anything remotely. Instead we'll just cover some basic abstractions provided by the library.
We start with the abstractions of the resources (machines and networks that are usually given by an infrastructure) We start with the abstractions of resources (machines and networks that are usually given by an infrastructure)
### Host ### Host
An host is anything we can connect to and act on. Most of the time it corresponds to a machine reachable through SSH. An host is anything we can connect to and act on. Most of the time it corresponds to a machine reachable through SSH.
The datastructure reflects this. The datastructure reflects this.
Usually you don't instantiate hosts manually, instead they are brought to you by EnOSlib (because most likely they depend on a scheduler decision like OAR on Grid'5000). Usually you don't instantiate hosts manually, instead they are brought to you by EnOSlib (because most likely they depend on a scheduler decision like OAR on Grid'5000).
%% Cell type:code id:cleared-presence tags: %% Cell type:code id:cleared-presence tags:
``` python ``` python
bare_host = en.Host("192.168.0.1") bare_host = en.Host("192.168.0.1")
host_with_alias = en.Host("192.168.0.2", alias="one_alias") host_with_alias = en.Host("192.168.0.2", alias="one_alias")
host_with_alias_and_username = en.Host("192.168.0.3", alias="one_alias", user="foo") host_with_alias_and_username = en.Host("192.168.0.3", alias="one_alias", user="foo")
``` ```
%% Cell type:code id:peripheral-campus tags: %% Cell type:code id:peripheral-campus tags:
``` python ``` python
bare_host bare_host
``` ```
%% Cell type:code id:unlikely-credits tags: %% Cell type:code id:unlikely-credits tags:
``` python ``` python
host_with_alias host_with_alias
``` ```
%% Cell type:code id:empty-privilege tags: %% Cell type:code id:empty-privilege tags:
``` python ``` python
host_with_alias_and_username host_with_alias_and_username
``` ```
%% Cell type:markdown id:attractive-stationery tags: %% Cell type:markdown id:attractive-stationery tags:
The local machine can be represented by an instance of the `LocalHost` object. This is a specialization of an `Host`, the connection to this host will be made using sub-processes (instead of SSH). We can see it in the `extra` attribute of the `LocalHost` object. This `extra` attribute is actually interpreted when a "remote" action is triggered on our hosts. The local machine can be represented by an instance of the `LocalHost` object. This is a specialization of an `Host`, the connection to this host will be made using sub-processes (instead of SSH). We can see it in the `extra` attribute of the `LocalHost` object. This `extra` attribute is actually interpreted when a "remote" action is triggered on our hosts.
%% Cell type:code id:dental-population tags: %% Cell type:code id:dental-population tags:
``` python ``` python
localhost = en.LocalHost() localhost = en.LocalHost()
localhost localhost
``` ```
%% Cell type:markdown id:temporal-adaptation tags: %% Cell type:markdown id:temporal-adaptation tags:
Other types of Hosts are possible. The library has a `DockerHost` which represents a docker container we want to reach using the docker TCP protocol. One needs to specify where this container is running by passing an host instance. Other types of Hosts are possible. The library has a `DockerHost` which represents a docker container we want to reach using the docker TCP protocol. One needs to specify where this container is running by passing an host instance.
%% Cell type:code id:educational-conviction tags: %% Cell type:code id:educational-conviction tags:
``` python ``` python
docker_host = en.DockerHost("alias", "container_name", host_with_alias_and_username) docker_host = en.DockerHost("alias", "container_name", host_with_alias_and_username)
docker_host docker_host
``` ```
%% Cell type:markdown id:executive-liberia tags: %% Cell type:markdown id:executive-liberia tags:
The above `extra` field suggest that the connection to this docker container will be made through an ssh jump to the remote host hosting the container. The above `extra` field suggest that the connection to this docker container will be made through an ssh jump to the remote host hosting the container.
This will be done transparently by the library anyway. This will be done transparently by the library anyway.
--- ---
%% Cell type:markdown id:exceptional-techno tags: %% Cell type:markdown id:exceptional-techno tags:
### Roles ### Roles
%% Cell type:markdown id:tight-request tags: %% Cell type:markdown id:tight-request tags:
A common pratice when experimenting, especially with distributed applications, is to form logical group of machines. A common pratice when experimenting, especially with distributed applications, is to form logical group of machines.
Indeed, during an experiment your hosts will serve different purposes: some will host the system you are studying while other will install third party tools to inject some load, observe ... Indeed, during an experiment your hosts will serve different purposes: some will host the system you are studying while other will install third party tools to inject some load, observe ...
A natural way of configuring differently several sets of hosts is to tag them and group them according to their tags. A natural way of configuring differently several sets of hosts is to tag them and group them according to their tags.
The `Roles` datastructure serves this purpose: it lets you group your hosts based on tags. It follow a `dict-like` interface. The `Roles` datastructure serves this purpose: it lets you group your hosts based on tags. It follow a `dict-like` interface.
%% Cell type:code id:spatial-probe tags: %% Cell type:code id:spatial-probe tags:
``` python ``` python
h1 = en.Host("10.0.0.1") h1 = en.Host("10.0.0.1")
h2 = en.Host("10.0.0.2") h2 = en.Host("10.0.0.2")
h3 = en.Host("10.0.0.3") h3 = en.Host("10.0.0.3")
roles = en.Roles() roles = en.Roles()
roles["tag1"] = [h1, h2] roles["tag1"] = [h1, h2]
roles["tag2"] = [h3] roles["tag2"] = [h3]
roles["tag3"] = [h2, h3] roles["tag3"] = [h2, h3]
roles roles
``` ```
%% Cell type:markdown id:adapted-apparatus tags: %% Cell type:markdown id:adapted-apparatus tags:
### Network and Networks ### Network and Networks
`Network` and `Networks` are the same as `Host` and `Roles` but for networks: `Network` and `Networks` are the same as `Host` and `Roles` but for networks:
- `Network` represent a single Network - `Network` represent a single Network
- `Networks` represent a "Roles" of Network: networks indexed by their tags - `Networks` represent a "Roles" of Network: networks indexed by their tags
. .
Networks are usually given by an infrastructure and thus you won't really instantiate `Network` nor `Networks` by yourself. Networks are usually given by an infrastructure and thus you won't really instantiate `Network` nor `Networks` by yourself.
More precisely there exists a specific subclass of `Network` per infrastructure which will be returned automatically by EnOSlib when needed. More precisely there exists a specific subclass of `Network` per infrastructure which will be returned automatically by EnOSlib when needed.
Moreover `Network` datastructure isn't exposed in EnOSlib at the top level, let's see however how a `DefaultNetwork` can look like. A `DefaultNetwork` is a very common abstraction of a network that allows to represent a basic network with optionnally a pool of free ips/macs address. For instance a subnet or a vlan on Grid5000 are represented by a specific `DefaultNetwork`. Moreover `Network` datastructure isn't exposed in EnOSlib at the top level, let's see however how a `DefaultNetwork` can look like. A `DefaultNetwork` is a very common abstraction of a network that allows to represent a basic network with optionnally a pool of free ips/macs address. For instance a subnet or a vlan on Grid5000 are represented by a specific `DefaultNetwork`.
%% Cell type:code id:distributed-ceramic tags: %% Cell type:code id:distributed-ceramic tags:
``` python ``` python
from enoslib.objects import DefaultNetwork from enoslib.objects import DefaultNetwork
``` ```
%% Cell type:code id:empirical-somewhere tags: %% Cell type:code id:empirical-somewhere tags:
``` python ``` python
one_network = DefaultNetwork("192.168.1.0/24") one_network = DefaultNetwork("192.168.1.0/24")
one_network_with_a_pool_of_ips = DefaultNetwork("192.168.1.0/24", ip_start="192.168.1.10", ip_end="192.168.1.100") one_network_with_a_pool_of_ips = DefaultNetwork("192.168.1.0/24", ip_start="192.168.1.10", ip_end="192.168.1.100")
``` ```
%% Cell type:code id:stone-boxing tags: %% Cell type:code id:stone-boxing tags:
``` python ``` python
one_network one_network
``` ```
%% Cell type:code id:aerial-coral tags: %% Cell type:code id:aerial-coral tags:
``` python ``` python
one_network_with_a_pool_of_ips one_network_with_a_pool_of_ips
``` ```
%% Cell type:code id:appreciated-preservation tags: %% Cell type:code id:appreciated-preservation tags:
``` python ``` python
# get one free ip # get one free ip
ip_gen = one_network_with_a_pool_of_ips.free_ips ip_gen = one_network_with_a_pool_of_ips.free_ips
next(ip_gen) next(ip_gen)
``` ```
%% Cell type:code id:determined-township tags: %% Cell type:code id:determined-township tags:
``` python ``` python
# get another one # get another one
next(ip_gen) next(ip_gen)
``` ```
%% Cell type:markdown id:9ab4929f-1c39-46ca-b6ac-ae0b907cc3ed tags: %% Cell type:markdown id:9ab4929f-1c39-46ca-b6ac-ae0b907cc3ed tags:
## Providers (and their configurations) ## Providers (and their configurations)
EnOSlib uses `Provider`s to ... provide resources. EnOSlib uses `Provider`s to ... provide resources.
`Provider`s let the user get ownership of some resources (for the time of the experiment) in good shape (e.g access granted, network configured ...). `Provider`s let the user get ownership of some resources (for the time of the experiment) in good shape (e.g access granted, network configured ...).
They transform an abstract `Configuration` to `Roles, Networks` : They transform an abstract `Configuration` to `Roles, Networks` :
$Configuration \xrightarrow{provider} Roles, Networks$ $Configuration \xrightarrow{provider} Roles, Networks$
There are different providers in EnOSlib: There are different providers in EnOSlib:
- **Vbox/KVM** to work with locally hosted virtual machines - **Vbox/KVM** to work with locally hosted virtual machines
- **Openstack/Chameleon** to work with bare-metal resources hosted in the Chameleon platform - **Openstack/Chameleon** to work with bare-metal resources hosted in the Chameleon platform
- **FiT/IOT lab** to work with sensors or low profile machines - **FiT/IOT lab** to work with sensors or low profile machines
- **Grid'5000** to get bare-metal resources from G5k.<br/> - **Grid'5000** to get bare-metal resources from G5k.<br/>
There are also some composite providers that sit on top of the Grid'5000 provider There are also some composite providers that sit on top of the Grid'5000 provider
- **VmonG5k** to work with virtual machines on Grid'5000** - **VmonG5k** to work with virtual machines on Grid'5000**
- **Distem** to work with lxc containers on Grid'5000** - **Distem** to work with lxc containers on Grid'5000**
%% Cell type:markdown id:67a8c47b-5b0e-4e51-b17c-2ba3fd3d0d12 tags: %% Cell type:markdown id:67a8c47b-5b0e-4e51-b17c-2ba3fd3d0d12 tags:
### Configurations ### Configurations
A `Provider` must be fed with a `Configuration`. `Configuration` objects are specific to each provider. A `Provider` must be fed with a `Configuration`. `Configuration` objects are specific to each provider.
You can build them from a dictionnary (e.g from a yaml/json file) or programmatically. For instance the schema for Grid'5000 is [here](https://discovery.gitlabpages.inria.fr/enoslib/apidoc/infra.html#g5k-schema). You can build them from a dictionnary (e.g from a yaml/json file) or programmatically. For instance the schema for Grid'5000 is [here](https://discovery.gitlabpages.inria.fr/enoslib/apidoc/infra.html#g5k-schema).
In this section, we'll only build some configurations (No resource will be reserved on Grid'5000) In this section, we'll only build some configurations (No resource will be reserved on Grid'5000)
%% Cell type:code id:fef3970b-5d7a-4e18-bc6f-59d944396e83 tags: %% Cell type:code id:fef3970b-5d7a-4e18-bc6f-59d944396e83 tags:
``` python ``` python
import enoslib as en import enoslib as en
# An empty configuration isn't really useful but let you see # An empty configuration isn't really useful but let you see
# some of the default parameters # some of the default parameters
# Note that by default the job_type is set to deploy == the env_name will be deployed # Note that by default the job_type is set to deploy == the env_name will be deployed
conf = en.G5kConf() conf = en.G5kConf()
conf conf
``` ```
%% Cell type:code id:ecdde599-41c3-420b-9ac9-88a1842cdfe4 tags: %% Cell type:code id:ecdde599-41c3-420b-9ac9-88a1842cdfe4 tags:
``` python ``` python
# changing the top level options is done by calling the classmethod `from_settings` # changing the top level options is done by calling the classmethod `from_settings`
en.G5kConf.from_settings(walltime="10:00:00", job_name="my awesome job") en.G5kConf.from_settings(walltime="10:00:00", job_name="my awesome job")
``` ```
%% Cell type:code id:98ecf1f3-027f-4754-baed-7a7188990868 tags: %% Cell type:code id:98ecf1f3-027f-4754-baed-7a7188990868 tags:
``` python ``` python
# the canonical way of getting some machines # the canonical way of getting some machines
``` ```
%% Cell type:code id:4e95e4ef-442f-4364-bfdf-f0c7f8271027 tags: %% Cell type:code id:4e95e4ef-442f-4364-bfdf-f0c7f8271027 tags:
``` python ``` python
prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod") prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod")
conf = ( conf = (
en.G5kConf() en.G5kConf()
.add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network) .add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network)
.add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network) .add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network)
.add_network_conf(prod_network) .add_network_conf(prod_network)
# optional, but do some sanity checks on the configuration # optional, but do some sanity checks on the configuration
.finalize() .finalize()
) )
conf conf
``` ```
%% Cell type:code id:08d69a20-a7f0-453c-a066-5b1304913d9e tags: %% Cell type:code id:08d69a20-a7f0-453c-a066-5b1304913d9e tags:
``` python ``` python
# changing to a non-deploy job # changing to a non-deploy job
# == no deployment will occur, the production environment will be used # == no deployment will occur, the production environment will be used
prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod") prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod")
conf = ( conf = (
en.G5kConf.from_settings(job_type=["allow_classic_ssh"]) en.G5kConf.from_settings(job_type=["allow_classic_ssh"])
.add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network) .add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network)
.add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network) .add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network)
.add_network_conf(prod_network) .add_network_conf(prod_network)
# optional, but do some sanity checks on the configuration # optional, but do some sanity checks on the configuration
.finalize() .finalize()
) )
conf conf
``` ```
%% Cell type:code id:a6eb4609-a873-4bd6-8cd4-06b56ff670a4 tags: %% Cell type:code id:a6eb4609-a873-4bd6-8cd4-06b56ff670a4 tags:
``` python ``` python
# Using a secondary networks # Using a secondary networks
prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod") prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod")
kavlan_network = en.G5kNetworkConf(roles=["myprivate"], site="rennes", type="kavlan") kavlan_network = en.G5kNetworkConf(roles=["myprivate"], site="rennes", type="kavlan")
conf = ( conf = (
en.G5kConf() en.G5kConf()
.add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network, secondary_networks=[kavlan_network]) .add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network, secondary_networks=[kavlan_network])
.add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network, secondary_networks=[kavlan_network]) .add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network, secondary_networks=[kavlan_network])
.add_network_conf(prod_network) .add_network_conf(prod_network)
.add_network_conf(kavlan_network) .add_network_conf(kavlan_network)
# optional, but do some sanity checks on the configuration # optional, but do some sanity checks on the configuration
.finalize() .finalize()
) )
conf conf
``` ```
%% Cell type:markdown id:ac4ab8b3-cdb2-46b8-be32-e709bc11fee0 tags: %% Cell type:markdown id:ac4ab8b3-cdb2-46b8-be32-e709bc11fee0 tags:
### Discussion and references ### Discussion and references
- Many configurations options are possible. [The documentation](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html) will show you some more. - Many configurations options are possible. [The documentation](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html) will show you some more.
- In EnOSlib `Roles` and `Networks` don't really depend on the provider that produced them. In other words you can substitute one provider's configuration to another one easily without changing the artifact code. - In EnOSlib `Roles` and `Networks` don't really depend on the provider that produced them. In other words you can substitute one provider's configuration to another one easily without changing the artifact code.
%% Cell type:code id:ce78182e-562a-46a1-b6f9-17ffade1d851 tags: %% Cell type:code id:ce78182e-562a-46a1-b6f9-17ffade1d851 tags:
``` python ``` python
``` ```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment