Distributed experiments on Grid'5000 … and beyond !

Table of Contents

1 Foreword

1.1 Existing tools (Grid'5000)

  • EnOSlib falls under the Experiment management tools of the following list: https://www.grid5000.fr/w/Grid5000:Software
  • EnOSlib can target Grid'5000 but also other testbeds (Chameleon, local machines…)
  • EnOSlib provides high level constructs to help you with your experiments

1.3 Contributing

Before experimenting

  • Tell us what your plans are:
    • There might be already users doing similar thing
    • There might be some missing/hidden pieces in the library you might need

While experimenting

  • Write bug reports / ask questions
  • Fix bugs / add your features

After experimenting

2 Before you start

Make sure you are familiar with the Grid'5000 architecture. see section 1 & 2 of https://www.grid5000.fr/w/Getting_Started. Note that we won't do this tutorial we'll prefer to use higher level tools for now.

3 Setup on Grid'5000

Connect to a Grid'5000 frontend of your choice (e.g rennes, nancy …)

  • create a new directory to host all the scripts of the session
  • bootstrap a new python3 virtualenv
  • install EnOSlib and configure the access to the API
$frontend: mkdir enoslib_seminar
$frontend: cd enoslib_seminar
$frontend: virtualenv --python=python3 venv
$frontend: source venv/bin/activate
$frontend(venv): pip install enoslib
$frontend(venv): echo '
verify_ssl: False
' > ~/.python-grid5000.yaml

4 Your first experiment on Grid'5000

Let's experiment with iperf3: a network bandwidth measuring tool. The goal is to deploy a simple benchmark between two hosts.

We'll also instrument the deployment in order to visualize in real-time the network traffic between the hosts. Since this is super common, EnOSlib exposes a monitoring service that lets you deploy very quickly what is needed.

4.1 First iteration

We consider the following script 

from enoslib.api import run_command, wait_ssh
from enoslib.infra.enos_g5k.provider import G5k
from enoslib.infra.enos_g5k.configuration import Configuration, NetworkConfiguration
from enoslib.service import Monitoring

import logging


def pprint(d):
    """Utils fonction to pretty print the results"""
    for k, v in d["ok"].items():
        print(f"Result for {k}")
        print("-" * 70)
        print("STDOUT:")
        print(v.get("stdout", ""))
        print("STDERR:")
        print(v.get("stderr", ""))


logging.basicConfig(level=logging.INFO)

######################################################################
##
## Some parameters.
## Note 1: that you don't need to be on rennes frontend to use nodes
##         from rennes
## Note 2: Adapt the site/cluster according to the availibility
##         see the Gantt in https://www.grid5000.fr/w/Status
######################################################################

SITE = "rennes"
CLUSTER = "paravance"

######################################################################
##
## Configuration object describes the resource we want
## here: 2 machines on the same cluster using the production network
##
######################################################################
network = NetworkConfiguration(id="n1",
                               type="prod",
                               roles=["my_network"],
                               site=SITE)

conf = Configuration.from_settings(job_name="enoslib_tutorial",
                                   job_type="allow_classic_ssh")\
    .add_network_conf(network)\
    .add_machine(roles=["server"],
                 cluster=CLUSTER,
                 nodes=1,
                 primary_network=network)\
    .add_machine(roles=["client"],
                 cluster=CLUSTER,
                 nodes=1,
                 primary_network=network)\
    .finalize()

######################################################################
##
## Reserve the ressources corresponding to the configuration
##  you'll get two **physical machine** (not virtual)
## the roles object is a dictionnary of the concrete compute resources
## roles = {"server": [host1], "client": [host2] }
##
######################################################################

provider = G5k(conf)
roles, networks =  provider.init()
wait_ssh(roles)

######################################################################
##
## Below is the experimentation logic
##  - It installs the bare minimum to run iperf3
##  - The machine with the role 'server' is used to run a iperf3 server
##    started in the background (using tmux)
##  - The machine with the role 'client' connects to that server and initiate a
##    transfer for 30s (duration variable)
##  - Report is printed in stdout
######################################################################

server = roles["server"][0]
duration = 30
run_command("apt update && apt install -y iperf3 tmux", roles=roles)
run_command("tmux new-session -d 'exec iperf3 -s'", pattern_hosts="server", roles=roles)
result = run_command(f"iperf3 -c {server.address} -t {duration}", pattern_hosts="client", roles=roles)
pprint(result)

######################################################################
##
## Destroy the reservation, uncomment when needed
##
######################################################################

# provider.destroy()

How fast is the network between the nodes you have chosen ?

Before moving to next questions, you'll need to clean the reservation. you can either uncomment the line provider.destroy() at the end of the script. You can also do it manually using the low-level oarstat / oardel tools. 

# get you reservation id
$frontend: oarstat -u
# release the resources / kill the reservation
$frontend: oardel <the id of the reservation goes here>

Can you adapt the script so that:

  1. The two nodes are in two different cluster in the same site ?
  2. The two nodes are in two different sites ?

4.2 Let's observe in real-time what is happening

Make sure you have cleaned your previous reservations.

The following script installs a monitoring stack on your node. This is almost the same script as before except the lines corresponding to the configuration of the monitoring stack 

from enoslib.api import run_command, wait_ssh
from enoslib.infra.enos_g5k.provider import G5k
from enoslib.infra.enos_g5k.configuration import Configuration, NetworkConfiguration
from enoslib.service import Monitoring

import logging


def pprint(d):
    """Utils fonction to pretty print the results"""
    for k, v in d["ok"].items():
        print(f"Result for {k}")
        print("-" * 70)
        print("STDOUT:")
        print(v.get("stdout", ""))
        print("STDERR:")
        print(v.get("stderr", ""))


logging.basicConfig(level=logging.INFO)

######################################################################
##
## Some parameters.
## Note 1: that you don't need to be on rennes frontend to use nodes
##         from rennes
## Note 2: Adapt the site/cluster according to the availibility
##         see the Gantt in https://www.grid5000.fr/w/Status
######################################################################

SITE = "rennes"
CLUSTER = "paravance"

######################################################################
##
## Configuration object describes the resource we want
## here: 2 machines on the same cluster using the production network
##
######################################################################
network = NetworkConfiguration(id="n1",
                               type="prod",
                               roles=["my_network"],
                               site=SITE)

conf = Configuration.from_settings(job_name="enoslib_tutorial",
                                   job_type="allow_classic_ssh")\
    .add_network_conf(network)\
    .add_machine(roles=["server"],
                 cluster=CLUSTER,
                 nodes=1,
                 primary_network=network)\
    .add_machine(roles=["client"],
                 cluster=CLUSTER,
                 nodes=1,
                 primary_network=network)\
    .finalize()

######################################################################
##
## Reserve the ressources corresponding to the configuration
##  you'll get two **physical machine** (not virtual)
## the roles object is a dictionnary of the concrete compute resources
## roles = {"server": [host1], "client": [host2] }
##
######################################################################

provider = G5k(conf)
roles, networks =  provider.init()
wait_ssh(roles)


######################################################################
##
## This deploys a monitoring stack. It is composed of
## - some agents on each monitored nodes
## - one collector that collects the metrics from the agents
## - one UI to visualize
##
######################################################################

m = Monitoring(collector=roles["server"],
               agent=roles["server"] + roles["client"],
               ui=roles["server"])
m.deploy()


######################################################################
##
## Below is the experimentation logic
##  - It installs the bare minimum to run iperf3
##  - The machine with the role 'server' is used to run a iperf3 server
##    started in the background (using tmux)
##  - The machine with the role 'client' connects to that server and initiate a
##    transfer for 600s (duration variable)
##  - Report is printed in stdout
######################################################################

server = roles["server"][0]
duration = 600
run_command("apt update && apt install -y iperf3 tmux", roles=roles)
run_command("tmux new-session -d 'exec iperf3 -s'", pattern_hosts="server", roles=roles)
result = run_command(f"iperf3 -c {server.address} -t {duration}", pattern_hosts="client", roles=roles)
pprint(result)

######################################################################
##
## Destroy the reservation, uncomment when needed
##
######################################################################

# provider.destroy()

Now, let's visualize the network traffic in real-time !

Usually I follow this to access services running inside Grid'5000: https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#accessing-http-services-inside-grid-5000. Today you can just create a tunnel like this (from your local machine). 

# Adapt the node names with the node where grafana (the UI) has been installed
# Replace <login> by your Grid'5000 login
$yourmachine: ssh -NL 3000:paravance-16.rennes.grid5000.fr:3000 <login>@access.grid5000.fr

# point your browser to localhost:3000
# username/mdp: admin/admin

Part of the experimenter work also consists in analysing the data, write the right request to monitor the traffic (check the Fig. 1) You should be able to visualize such a thing (after a bit of point and clicks)

iperf3.png

Figure 1: iperf3 / monitoring

4.3 Discussion

So, far this seems (hopefully for you) very handy. But there might be some problems in our setup:

  • we aren't isolated from the other users
  • we aren't isolated from ourself in the sense that the monitoring stack generates its own network traffic (yes, this is negligible in our case)

Sometimes it's desirable to have the following setup (see Fig. 2).

skydive_enoslib.png

Figure 2: nodes are using two network interfaces. Monitoring traffic and benchmark traffic are separated.

4.4 A better approach (maybe)

Access the full file: exercices/iperf3_better.py

On Grid'5000, using the secondary interfaces requires to deploy the nodes: an new OS will be installed on your nodes. This will give you full control on the physical machine (root access). This might be longer to run the experiment due to this deployment phase.

4.5 Ninja level

Add the Skydive service to your deployment. It should be accessible on the port 8082 of the analyzer node. You should get something like Fig. 2.

5 Providers: to replicate your experiment

The resources that are used for your experiment are acquired through a provider. Providers are a mean to decouple the infrastructure code (the code that gets the resources) from the code that runs the experiment. Changing the provider allows to replicate the experiment on another testbed.

Originally it was used to iterate on the code locally (using the Vagrant provider) and to only test on Grid'5000 when necessary.

We now have couple of providers that you may picked or mixed.

5.1 iperf3 on virtual machines on Grid'5000

We'll adapt the initial iperf3 example to use virtual machines instead of bare-metal machine.

Note that:

  • The configuration object is different
  • The experimentation logic is the same
  • Some part have been rewritten using modules (see later in the dedicated section).
from enoslib.api import play_on, wait_ssh
from enoslib.infra.enos_vmong5k.provider import VMonG5k
from enoslib.infra.enos_vmong5k.configuration import Configuration

import logging
import os

logging.basicConfig(level=logging.DEBUG)

CLUSTER = "paravance"

# path to the inventory
inventory = os.path.join(os.getcwd(), "hosts")

# claim the resources
conf = Configuration.from_settings(job_name="enoslib_tutorial", gateway=True)\
                    .add_machine(roles=["server"],
                                 cluster=CLUSTER,
                                 number=1,
                                 flavour="large")\
                    .add_machine(roles=["client"],
                                 cluster=CLUSTER,
                                 number=1,
                                 flavour="medium")\
                    .finalize()

provider = VMonG5k(conf)

roles, networks = provider.init()
wait_ssh(roles)

# Below is the experimentation logic
# It installs the bare minimum to run iperf3
# The machine with the role 'server' is used to run a iperf3 server
#     started in the background in a tmux
# The machine with the role 'client' connects to that server
# Report is printed in stdout
server = roles["server"][0]

with play_on(roles=roles) as p:
    p.apt(name=["iperf3", "tmux"], state="present")

with play_on(pattern_hosts="server", roles=roles) as p:
    p.shell("tmux new-session -d 'exec iperf3 -s'")

with play_on(pattern_hosts="client", roles=roles) as p:
    p.shell(f"iperf3 -c {server.address} -t 30")

with play_on(pattern_hosts="client", roles=roles) as p:
    p.shell(f"iperf3 -c {server.address} -t 30 --logfile iperf3.out")
    p.fetch(src="iperf3.out", dest="iperf3.out")

Using module using the play_on context manager does not bring back the results of the commands. Iperf3 let's you write the result of the command on a file. We just need to scp the file back to our local machine using the fetch module.

6 Variables in EnOSlib

Learn how to get 2 nodes from Grid'5000 and start launching remote commands.

6.1 Discover the run command and its variants

Before proceeding you can add this util function to your code. It is only used to pretty print a python dictionnary. 

def pprint(d):
    import json
    print(json.dumps(d, indent=4))

And use the enoslib.api.run function 

server = roles["server"][0]
# ---
# Using run
# --------------------------------------------------------------------
result = run(f"ping -c 5 {server.address}", roles["client"])
pprint(result)

Or the enoslib.api.run_command function 

# ---
# Using run_command 1/2
# --------------------------------------------------------------------
result = run_command(f"ping -c 5 {server.address}",
                     pattern_hosts="client",
                     roles=roles)
pprint(result)

enoslib.api.run is a specialisation of enoslib.api.run_command. The latter let's you use some fancy patterns to determine the list of hosts to run the command on.

And yes, it uses Ansible behind the scene.

6.2 Advanced usages

For all the remote interactions, EnOSlib relies on Ansible. Ansible has it own variables management system. For instance the task Gather Facts at the beginning of the previous tasks gathers informations about all/some remote hosts and store them in the Ansible management system.

Let's see what Ansible is gathering about the hosts: 

# ---
# Gather facts
# --------------------------------------------------------------------
result = gather_facts(roles=roles)
pprint(result)

EnOSlib sits in between two worlds: the Python world and the Ansible world. One common need is to pass a variables from one world to another.

  • enoslib.api.gather_facts is a way to get, in Python, the variables known by Ansible about each host.
  • extra_vars keyword argument of enoslib.api.run or enoslib.api.run_command will pass variables from Python world to Ansible world (global variable)
  • Injecting a key/value in a Host.extra attribute will make the variable key available to Ansible. This makes the variables Host specific.

The following inject a global variable in the Ansible world 

# ---
# Passing a variable to the Ansible World using a global level variable
# --------------------------------------------------------------------
server = roles["server"][0]
extra_vars={"server_ip": server.address}
result = run("ping -c 5 {{ server_ip }}", roles["client"], extra_vars=extra_vars)

6.3 Ninja level

The following is valid and inject in the client host a specific variable to keep track of the server IP. 

# ---
# Passing a variable to the Ansible World using a host level variable
# --------------------------------------------------------------------
server = roles["server"][0]
client = roles["client"][0]
client.extra.update(server_ip=server.address)
result = run("ping -c 5 {{ server_ip }}", roles["client"])

Host level variables are interesting to introduce some dissymetry between hosts while still using one single command to reach all of them.

How to perform simultaneously the ping to the other machine in calling only once run or run_command and using host level variables?

We'd like to create 5 server machines and 5 client machines and start 5 parallel streams of data using iperf3. To answer this we'll need to learn a bit more on how variables are handled in EnOSlib.

6.4 Putting all together

Access the full file: exercices/run.py

7 Modules: for safer remote actions

In this section we'll discover the idiomatic way of managing resources on the remote hosts. A resource can be anything: a user, a file, a line in a file, a repo on Gitlab, a firewall rule …

7.1 Idempotency

Let's assume you want to create a user (foo). With the run_command this would look like: 

run_command("useradd -m foo", roles=role)

The main issue with this code is that it is not idempotent. Running it once will applied the effect (create the user). But, as soon as the user exist in the system, this will raise an error.

7.2 One reason why idempotency is important

Let's consider the following snippet (mispelling the second command is intentional) 

run_command("useradd -m foo", roles=role)
run_command("mkdirz plop")

Executing the above leads the system with the user foo created but the the directory plop not created since the second command fails.

So what you want to do is to fix the second command and re-run the snippet again. But, you can't do that because useradd isn't idempotent.

7.3 Idempotency trick

One easy solution is to protect your call to non idempotent commands with some ad'hoc tricks

Here it can look like this: 

run_command("id foo || useradd -m foo", roles=role)
run_command("mkdir -p plop")

What's wrong with that

  • The trick depends on the command
  • Re-reading the code is more complex: the code focus on the how not the what

7.4 General idempotency

The idiomatic solution is to use modules (inherited from the Ansible Modules). The modules are specified in a declarative way and they ensure idempotency for most of them.

So rewriting the example with modules looks like: 

with play_on(roles=roles) as p:
    p.user(name="foo", state="present", create_home="yes")
    p.file(name="plop", state="directory")

enoslib.api.play_on is the entry point to the module system.

You can run this code as many times as you want without any error. You'll eventually find one user foo and one directory plop in your target systems.

They are more than 2500 modules: https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html

If you can't find what you want you must know that:

  • Writing your own module is possible
  • Falling back to the idempotency trick is reasonable

8 Tasks: to organize your experiment

To discover the Task API, head to https://discovery.gitlabpages.inria.fr/enoslib/tutorials/using-tasks.html.

The examples are written for Vagrant but may be changed to whatever provider you like/have.

Adapt the iperf3 example to provide a command line

  • Either using G5k physical machines: 

    # deploy the dependencies of the experimentation using the G5k provider
myperf g5k

# launch a performance measurement
# ideally exposes all the iperf3 client options there ;)
myperf bench -t 120

# Backup the reports / influxdb database
myperf backup

    myperf destroy

  • Either using the virtual machines on Grid'5000: 

    # deploy the dependencies of the experimentation using the G5k provider
myperf vm5k

# Subsequent command line should be the same as above
# enjoy :)

Author: Matthieu Simonin

Created: 2019-11-15 ven. 00:49

Validate