SIMONIN Matthieu
python-grid5000

Repository



python-grid5000
python-grid5000 is a python package wrapping the Grid’5000 REST API. You can
  use it as a library in your python project or you can explore the Grid’5000
  resources interactively using the embedded shell.
Thanks
The core code is borrowed from python-gitlab with small adaptations to
  conform with the Grid5000 API models (with an ‘s’!)
Contributing

  To contribute, you can drop me an email or open an issue for a bug report, or feature request.
  There are many areas where this can be improved some of them are listed here:
    
      The complete coverage of the API isn’t finished (yet) but this should be fairly easy to reach.
        Most of the logic go in ~grid5000.objects~. And to be honnest I only
        implemented the feature that I needed the most.
      Returned status code aren’t yet well treated.
    
  
Comparison with …

  
RESTfully:
    It consumes REST API following the HATEOAS principles. This allows the client
    to fully discover the resources and actions available. Most of the G5K API
    follow theses principles but, for instance the Storage API don’t. Thus
    RESTfully isn’t compatible with all the features offered by the Grid’5000 API.
    It’s a ruby library. Python-grid5000 borrows the friendly syntax for resource
    browsing, but in python.
  
Execo:
    Written in Python. The api module gathers a lot of utils functions leveraging
    the Grid’5000 API. Resources aren’t exposed in a syntax friendly manner,
    instead functions for some classical operations are exposed (mainly getters).
    It has a convenient way of caching the reference API. Python-grid5000 is a
    wrapper around the Grid’5000 that seeks 100% coverage. Python-grid5000 is
    resource oriented.
  
Raw requests:
    The reference for HTTP library in python. Good for prototyping but low-level.
    python-grid5000 encapsulates this library.

Installation and examples

  Please refer to https://api.grid5000.fr/doc/4.0/reference/spec.html for
    the complete specification.
  All the examples are exported in the examples subdirectory so you can
    easily test and adapt them.
  The configuration is read from a configuration file located in the home
    directory (should be compatible with the restfully one).
    It can be created with the following:

echo '
username: MYLOGIN
password: MYPASSWORD
' > ~/.python-grid5000.yaml


  Using a virtualenv is recommended (python 3.5+ is required)

virtualenv -p python3 venv
source venv/bin/activate
pip install python-grid5000

Grid’5000 shell
If you call grid5000 on the command line you should land in a ipython shell.
  Before starting, the file $HOME/.python-grid5000.yaml will be loaded.
$) grid5000

Python 3.6.5 (default, Jun 17 2018, 21:32:15) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: gk.sites.list()                                                                                                                                                                                            
Out[1]: 
[<Site uid:grenoble>,
 <Site uid:lille>,
 <Site uid:luxembourg>,
 <Site uid:lyon>,
 <Site uid:nancy>,
 <Site uid:nantes>,
 <Site uid:rennes>,
 <Site uid:sophia>]

In [2]: # gk is your entry point    

Reference API
Get node information
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

node_info = gk.sites["nancy"].clusters["grisou"].nodes["grisou-1"]
print("grisou-1 has {threads} threads and has {ram} bytes of RAM".format(
    threads=node_info.architecture["nb_threads"],
    ram=node_info.main_memory["ram_size"]))

Get Versions of resources
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

root_versions = gk.root.get().versions.list()
print(root_versions)

rennes = gk.sites["rennes"]
site_versions = rennes.versions.list()
print(site_versions)

cluster = rennes.clusters["paravance"]
cluster_versions = cluster.versions.list()
print(cluster_versions)

node_versions = cluster.nodes["paravance-1"]
print(node_versions)


Monitoring API
Get Statuses of resources
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

rennes = gk.sites["rennes"]
site_statuses = rennes.status.list()
print(site_statuses)

cluster = rennes.clusters["paravance"]
cluster_statuses = cluster.status.list()


Job API
Job filtering
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

# state=running will be placed in the query params
running_jobs = gk.sites["rennes"].jobs.list(state="running")
print(running_jobs)

# get a specific job by its uid
job = gk.sites["rennes"].jobs.get("424242")
print(job)
# or using the bracket notation
job = gk.sites["rennes"].jobs["424242"]
print(job)


Submit a job
import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

# This is equivalent to gk.sites.get("rennes")
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
                        "command": "sleep 3600"})

while job.state != "running":
    job.refresh()
    print("Waiting for the job [%s] to be running" % job.uid)
    time.sleep(10)

print(job)
print("Assigned nodes : %s" % job.assigned_nodes)

Deployment API
Deploy an environment
import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

# This is equivalent to gk.sites.get("rennes")
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
                        "command": "sleep 3600",
                        "types": ["deploy"]})

while job.state != "running":
    job.refresh()
    print("Waiting the job [%s] to be running" % job.uid)
    time.sleep(10)

print("Assigned nodes : %s" % job.assigned_nodes)

deployment = site.deployments.create({"nodes": job.assigned_nodes,
                                      "environment": "debian9-x64-min"})

while deployment.status != "terminated":
    deployment.refresh()
    print("Waiting for the deployment [%s] to be finished" % deployment.uid)
    time.sleep(10)

print(deployment.result)

Storage API
Get Storage accesses
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

print(gk.sites["rennes"].storage["msimonin"].access.list())

Set storage accesses (e.g for vms)
from netaddr import IPNetwork
import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
                        "command": "sleep 3600",
                        "resources": "slash_22=1+nodes=1"})

while job.state != "running":
    job.refresh()
    print("Waiting the job [%s] to be running" % job.uid)
    time.sleep(5)

subnet = job.resources_by_type['subnets'][0]
ip_network = [str(ip) for ip in IPNetwork(subnet)]

# create acces for all ips in the subnet
access = site.storage["msimonin"].access.create({"ipv4": ip_network,
                                                  "termination": {"job": job.uid,
                                                                  "site": site.uid}})

Vlan API
Get vlan(s)
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

site = gk.sites["rennes"]

# Get all vlans
vlans = site.vlans.list()
print(vlans)

# Get on specific
vlan = site.vlans.get("4")
print(vlan)

vlan = site.vlans["4"]
print(vlan)

# Get vlan of some nodes
print(site.vlansnodes.submit({"nodes": ["paravance-1.rennes.grid5000.fr", "paravance-2.rennes.grid5000.fr"]}))


# Get nodes in vlan
print(site.vlans["4"].nodes.list())


Set nodes in vlan

  Putting primary interface in a vlan
    import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
                        "command": "sleep 3600",
                        "resources": "{type='kavlan'}/vlan=1+nodes=1",
                        "types": ["deploy"]})

while job.state != "running":
    job.refresh()
    print("Waiting the job [%s] to be runnning" % job.uid)
    time.sleep(5)

deployment = site.deployments.create({"nodes": job.assigned_nodes,
                                      "environment": "debian9-x64-min",
                                      "vlan": job.resources_by_type["vlans"][0]})

while deployment.status != "terminated":
    deployment.refresh()
    print("Waiting for the deployment [%s] to be finished" % deployment.uid)
    time.sleep(10)

print(deployment.result)
    
  
  Putting the secondary interface in a vlan
    import logging
import os
import time

from grid5000 import Grid5000

  
logging.basicConfig(level=logging.DEBUG)


def _to_network_address(host, interface):
    """Translate a host to a network address
    e.g:
    paranoia-20.rennes.grid5000.fr -> paranoia-20-eth2.rennes.grid5000.fr
    """
    splitted = host.split('.')
    splitted[0] = splitted[0] + "-" + interface

    return ".".join(splitted)


conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
                        "command": "sleep 3600",
                        "resources": "{type='kavlan'}/vlan=1+{cluster='paranoia'}nodes=1",
                        "types": ["deploy"]
})

while job.state != "running":
    job.refresh()
    print("Waiting the job [%s] to be runnning" % job.uid)
    time.sleep(5)
   
vlanid = job.resources_by_type["vlans"][0]

# we hard code the interface but this can be discovered in the node info
# TODO: write the code here to discover
nodes = [_to_network_address(n, "eth2") for n in job.assigned_nodes]
print(nodes)

# set in vlan
site.vlans[vlanid].submit({"nodes": nodes})
    
  
More snippets
Site of a cluster
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

clusters = ["dahu", "parasilo", "chetemi"]

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

sites = gk.sites.list()
matches = []
for site in sites:
    candidates = site.clusters.list()
    matching = [c.uid for c in candidates if c.uid in clusters]
    if len(matching) == 1:
        matches.append((site, matching[0]))
        clusters.remove(matching[0])
print("We found the following matches %s" % matches)

Get all job with a given name on all the sites
import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

NAME = "pyg5k"
  
conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

sites = gk.sites.list()
site = gk.sites["rennes"]
sites = [gk.sites["rennes"], gk.sites["nancy"], gk.sites["grenoble"]]
  
# creates some jobs
jobs = []
for site in sites:
    job = site.jobs.create({"name": "pyg5k",
                            "command": "sleep 3600"})
    jobs.append(job)

_jobs = []
for site in sites:
    _jobs.append((site.uid, site.jobs.list(name=NAME,
                                           state="waiting,launching,running")))

print("We found %s" % _jobs)

# deleting the jobs
for job in jobs:
    job.delete()

Appendix
How to export this file

  Produce README.rst.
    To generate the rst file, load the ox-rst.el file from
    https://github.com/msnoigrs/ox-rst into emacs. Then do, C-c C-e r r or M-x
    org-rst-export-to-rst.
  Produce python example scripts.
    Do C-c C-v t or M-x
    org-babel-tangle. The scripts are available under available under examples.