README.rst 17.6 KB
Newer Older
1 2 3
===============
python-grid5000
===============
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
4

5

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
6 7 8
``python-grid5000`` is a python package wrapping the Grid’5000 REST API. You can
use it as a library in your python project or you can explore the Grid’5000
resources interactively using the embedded shell.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
9 10 11

.. warning::

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
12 13
    The code is currently being developed heavily. Jump to the contributing section
    if you want to be involved.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
14

15 16
1 Thanks
--------
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
17

18
The core code is borrowed from `python-gitlab <https://github.com/python-gitlab/python-gitlab>`_ with small adaptations to
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
19
conform with the Grid5000 API models (with an ’s’!)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
20

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
21 22 23 24 25 26 27 28 29 30 31 32 33
2 Contributing
--------------

- To contribute, you can drop me an email or open an issue for a bug report, or feature request.

- There are many areas where this can be improved some of them are listed here:

  - The complete coverage of the API isn’t finished (yet) but this should be fairly easy to reach.
    Most of the logic go in ```grid5000.objects`` <https://gitlab.inria.fr/msimonin/python-grid5000/blob/master/grid5000/objects.py>`_. And to be honnest I only
    implemented the feature that I needed the most.

  - Returned `status code <https://www.grid5000.fr/mediawiki/index.php/API#Status_Codes>`_ aren’t yet well treated.

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
34 35 36 37
3 Comparison with ...
---------------------

- `RESTfully <https://api.grid5000.fr/doc/4.0/tools/restfully.html>`_: 
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
38 39 40 41 42 43
  It consumes REST API following the `HATEOAS <https://en.m.wikipedia.org/wiki/HATEOAS>`_ principles. This allows the client
  to fully discover the resources and actions available. Most of the G5K API
  follow theses principles but, for instance the `Storage API <https://www.grid5000.fr/mediawiki/index.php/Storage_Manager>`_ don’t. Thus
  RESTfully isn’t compatible with all the features offered by the Grid’5000 API.
  It’s a ruby library. Python-grid5000 borrows the friendly syntax for resource
  browsing, but in python.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
44 45 46 47 48

- `Execo <http://execo.gforge.inria.fr>`_: 
  Written in Python. The api module gathers a lot of utils functions leveraging
  the Grid’5000 API. Resources aren’t exposed in a syntax friendly manner,
  instead functions for some classical operations are exposed (mainly getters).
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
49
  It has a convenient way of caching the reference API. Python-grid5000 is a
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
50 51
  wrapper around the Grid’5000 that seeks 100% coverage. Python-grid5000 is
  resource oriented.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
52 53 54

- `Raw requests <http://docs.python-requests.org>`_: 
  **The** reference for HTTP library in python. Good for prototyping but low-level.
55
  python-grid5000 encapsulates this library.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
56

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
57 58
4 Installation and examples
---------------------------
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
59

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
60 61
- Please refer to `https://api.grid5000.fr/doc/4.0/reference/spec.html <https://api.grid5000.fr/doc/4.0/reference/spec.html>`_ for 
  the complete specification.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
62

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
63 64 65 66 67 68 69 70 71 72 73 74 75
- All the examples are exported in the examples subdirectory so you can
  easily test and adapt them.

- The configuration is read from a configuration file located in the home
  directory (should be compatible with the restfully one). 
  It can be created with the following:

::

    echo '
    username: MYLOGIN
    password: MYPASSWORD
    ' > ~/.python-grid5000.yaml
76

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
77
.. hint::
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
78

79 80
    From a Grid’5000 frontend this file is optionnal. In all other cases the
    configuration file is mandatory.
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
81

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
82 83 84 85 86 87 88 89
- Using a virtualenv is recommended (python 3.5+ is required)

::

    virtualenv -p python3 venv
    source venv/bin/activate
    pip install python-grid5000

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117
4.1 Grid’5000 shell
~~~~~~~~~~~~~~~~~~~

If you call ``grid5000`` on the command line you should land in a ipython shell.
Before starting, the file ``$HOME/.python-grid5000.yaml`` will be loaded.

::

    $) grid5000

    Python 3.6.5 (default, Jun 17 2018, 21:32:15) 
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

    In [1]: gk.sites.list()                                                                                                                                                                                            
    Out[1]: 
    [<Site uid:grenoble>,
     <Site uid:lille>,
     <Site uid:luxembourg>,
     <Site uid:lyon>,
     <Site uid:nancy>,
     <Site uid:nantes>,
     <Site uid:rennes>,
     <Site uid:sophia>]

    In [2]: # gk is your entry point    

4.2 Reference API
118 119
~~~~~~~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
120
4.2.1 Get node information
121
^^^^^^^^^^^^^^^^^^^^^^^^^^
122 123 124

.. code:: python

125
    import logging
126 127 128 129
    import os

    from grid5000 import Grid5000

130 131 132

    logging.basicConfig(level=logging.DEBUG)

133 134 135 136 137 138 139 140
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)

    node_info = gk.sites["nancy"].clusters["grisou"].nodes["grisou-1"]
    print("grisou-1 has {threads} threads and has {ram} bytes of RAM".format(
        threads=node_info.architecture["nb_threads"],
        ram=node_info.main_memory["ram_size"]))

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
141
4.2.2 Get Versions of resources
142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

    import logging
    import os

    from grid5000 import Grid5000


    logging.basicConfig(level=logging.DEBUG)

    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)

    root_versions = gk.root.get().versions.list()
    print(root_versions)

    rennes = gk.sites["rennes"]
    site_versions = rennes.versions.list()
    print(site_versions)

    cluster = rennes.clusters["paravance"]
    cluster_versions = cluster.versions.list()
    print(cluster_versions)

    node_versions = cluster.nodes["paravance-1"]
    print(node_versions)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
171
4.3 Monitoring API
172 173
~~~~~~~~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
174
4.3.1 Get Statuses of resources
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

    import logging
    import os

    from grid5000 import Grid5000


    logging.basicConfig(level=logging.DEBUG)

    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)

    rennes = gk.sites["rennes"]
    site_statuses = rennes.status.list()
    print(site_statuses)

    cluster = rennes.clusters["paravance"]
    cluster_statuses = cluster.status.list()

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
197
4.4 Job API
198 199
~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
200
4.4.1 Job filtering
201
^^^^^^^^^^^^^^^^^^^
202 203 204

.. code:: python

205
    import logging
206 207 208 209
    import os

    from grid5000 import Grid5000

210 211 212

    logging.basicConfig(level=logging.DEBUG)

213 214 215 216 217 218 219 220 221 222
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)

    # state=running will be placed in the query params
    running_jobs = gk.sites["rennes"].jobs.list(state="running")
    print(running_jobs)

    # get a specific job by its uid
    job = gk.sites["rennes"].jobs.get("424242")
    print(job)
223 224 225
    # or using the bracket notation
    job = gk.sites["rennes"].jobs["424242"]
    print(job)
226

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
227
4.4.2 Submit a job
228
^^^^^^^^^^^^^^^^^^
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
229

230
.. code:: python
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
231

232
    import logging
233
    import os
234
    import time
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
235

236
    from grid5000 import Grid5000
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
237

238 239 240

    logging.basicConfig(level=logging.DEBUG)

241 242
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
243

244
    # This is equivalent to gk.sites.get("rennes")
245
    site = gk.sites["rennes"]
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
246

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
247 248
    job = site.jobs.create({"name": "pyg5k",
                            "command": "sleep 3600"})
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
249

250
    while job.state != "running":
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
251 252 253
        job.refresh()
        print("Waiting for the job [%s] to be running" % job.uid)
        time.sleep(10)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
254

255 256
    print(job)
    print("Assigned nodes : %s" % job.assigned_nodes)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
257

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
258
4.5 Deployment API
259 260
~~~~~~~~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
261
4.5.1 Deploy an environment
262
^^^^^^^^^^^^^^^^^^^^^^^^^^^
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
263

264
.. code:: python
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
265

266
    import logging
267
    import os
268
    import time
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
269 270 271

    from grid5000 import Grid5000

272 273 274

    logging.basicConfig(level=logging.DEBUG)

275 276
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
277

278 279 280
    # This is equivalent to gk.sites.get("rennes")
    site = gk.sites["rennes"]

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
281 282 283
    job = site.jobs.create({"name": "pyg5k",
                            "command": "sleep 3600",
                            "types": ["deploy"]})
284 285 286 287 288 289 290

    while job.state != "running":
        job.refresh()
        print("Waiting the job [%s] to be running" % job.uid)
        time.sleep(10)

    print("Assigned nodes : %s" % job.assigned_nodes)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
291

292 293
    deployment = site.deployments.create({"nodes": job.assigned_nodes,
                                          "environment": "debian9-x64-min"})
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
294 295 296 297 298 299 300 301 302 303
    # To get SSH access to your nodes you can pass your public key
    #
    # from pathlib import Path
    #
    # key_path = Path.home().joinpath(".ssh", "id_rsa.pub")
    #
    #
    # deployment = site.deployments.create({"nodes": job.assigned_nodes,
    #                                       "environment": "debian9-x64-min"
    #                                       "key": key_path.read_text()})
304 305 306 307 308 309 310 311

    while deployment.status != "terminated":
        deployment.refresh()
        print("Waiting for the deployment [%s] to be finished" % deployment.uid)
        time.sleep(10)

    print(deployment.result)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
312
4.6 Storage API
313 314
~~~~~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
315
4.6.1 Get Storage accesses
316
^^^^^^^^^^^^^^^^^^^^^^^^^^
317 318 319

.. code:: python

320
    import logging
321 322 323 324
    import os

    from grid5000 import Grid5000

325

326 327
    logging.basicConfig(level=logging.DEBUG)

328 329
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
330

331
    print(gk.sites["rennes"].storage["msimonin"].access.list())
332

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
333
4.6.2 Set storage accesses (e.g for vms)
334
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
335 336 337 338

.. code:: python

    from netaddr import IPNetwork
339
    import logging
340 341 342 343 344 345
    import os
    import time

    from grid5000 import Grid5000


346 347
    logging.basicConfig(level=logging.DEBUG)

348 349 350 351
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)
    site = gk.sites["rennes"]

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
352 353 354
    job = site.jobs.create({"name": "pyg5k",
                            "command": "sleep 3600",
                            "resources": "slash_22=1+nodes=1"})
355 356 357

    while job.state != "running":
        job.refresh()
358
        print("Waiting the job [%s] to be running" % job.uid)
359 360 361 362 363 364
        time.sleep(5)

    subnet = job.resources_by_type['subnets'][0]
    ip_network = [str(ip) for ip in IPNetwork(subnet)]

    # create acces for all ips in the subnet
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
365
    access = site.storage["msimonin"].access.create({"ipv4": ip_network,
366
                                                      "termination": {"job": job.uid,
367
                                                                      "site": site.uid}})
368

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
369
4.7 Vlan API
370 371
~~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
372
4.7.1 Get vlan(s)
373
^^^^^^^^^^^^^^^^^
374 375 376

.. code:: python

377
    import logging
378 379 380 381 382
    import os

    from grid5000 import Grid5000


383 384
    logging.basicConfig(level=logging.DEBUG)

385 386 387
    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
388 389
    site = gk.sites["rennes"]

390
    # Get all vlans
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
391
    vlans = site.vlans.list()
392 393 394
    print(vlans)

    # Get on specific
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
395
    vlan = site.vlans.get("4")
396 397
    print(vlan)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
398
    vlan = site.vlans["4"]
399 400
    print(vlan)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
401 402
    # Get vlan of some nodes
    print(site.vlansnodes.submit({"nodes": ["paravance-1.rennes.grid5000.fr", "paravance-2.rennes.grid5000.fr"]}))
403 404


SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
405 406
    # Get nodes in vlan
    print(site.vlans["4"].nodes.list())
407

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
408
4.7.2 Set nodes in vlan
409
^^^^^^^^^^^^^^^^^^^^^^^
410

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
411
- Putting primary interface in a vlan
412

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
413
  .. code:: python
414

415
      import logging
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
416 417
      import os
      import time
418

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
419
      from grid5000 import Grid5000
420 421


422 423
      logging.basicConfig(level=logging.DEBUG)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
424 425 426
      conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
      gk = Grid5000.from_yaml(conf_file)
      site = gk.sites["rennes"]
427

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
428 429 430 431
      job = site.jobs.create({"name": "pyg5k",
                              "command": "sleep 3600",
                              "resources": "{type='kavlan'}/vlan=1+nodes=1",
                              "types": ["deploy"]})
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
432 433 434 435 436 437 438

      while job.state != "running":
          job.refresh()
          print("Waiting the job [%s] to be runnning" % job.uid)
          time.sleep(5)

      deployment = site.deployments.create({"nodes": job.assigned_nodes,
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
439
                                            "environment": "debian9-x64-min",
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
440 441 442 443 444 445 446 447 448 449 450 451 452
                                            "vlan": job.resources_by_type["vlans"][0]})

      while deployment.status != "terminated":
          deployment.refresh()
          print("Waiting for the deployment [%s] to be finished" % deployment.uid)
          time.sleep(10)

      print(deployment.result)

- Putting the secondary interface in a vlan

  .. code:: python

453
      import logging
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
454 455 456 457 458
      import os
      import time

      from grid5000 import Grid5000

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
459 460 461 462

      logging.basicConfig(level=logging.DEBUG)


SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
463
      def _to_network_address(host, interface):
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
464 465 466 467 468 469 470 471
          """Translate a host to a network address
          e.g:
          paranoia-20.rennes.grid5000.fr -> paranoia-20-eth2.rennes.grid5000.fr
          """
          splitted = host.split('.')
          splitted[0] = splitted[0] + "-" + interface

          return ".".join(splitted)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
472 473 474 475


      conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
      gk = Grid5000.from_yaml(conf_file)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
476

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
477 478
      site = gk.sites["rennes"]

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
479 480 481 482
      job = site.jobs.create({"name": "pyg5k",
                              "command": "sleep 3600",
                              "resources": "{type='kavlan'}/vlan=1+{cluster='paranoia'}nodes=1",
                              "types": ["deploy"]
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498
      })

      while job.state != "running":
          job.refresh()
          print("Waiting the job [%s] to be runnning" % job.uid)
          time.sleep(5)

      vlanid = job.resources_by_type["vlans"][0]

      # we hard code the interface but this can be discovered in the node info
      # TODO: write the code here to discover
      nodes = [_to_network_address(n, "eth2") for n in job.assigned_nodes]
      print(nodes)

      # set in vlan
      site.vlans[vlanid].submit({"nodes": nodes})
499

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
500
4.8 More snippets
501 502
~~~~~~~~~~~~~~~~~

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
503
4.8.1 Site of a cluster
504
^^^^^^^^^^^^^^^^^^^^^^^
505 506 507 508 509 510 511 512 513 514 515 516 517 518 519

.. code:: python

    import logging
    import os

    from grid5000 import Grid5000


    logging.basicConfig(level=logging.DEBUG)

    clusters = ["dahu", "parasilo", "chetemi"]

    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
520

521 522 523 524 525 526 527 528 529 530
    sites = gk.sites.list()
    matches = []
    for site in sites:
        candidates = site.clusters.list()
        matching = [c.uid for c in candidates if c.uid in clusters]
        if len(matching) == 1:
            matches.append((site, matching[0]))
            clusters.remove(matching[0])
    print("We found the following matches %s" % matches)

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
531
4.8.2 Get all job with a given name on all the sites
532
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
533 534 535

.. code:: python

SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
536

537 538 539 540 541 542 543 544 545 546 547 548
    import logging
    import os

    from grid5000 import Grid5000


    logging.basicConfig(level=logging.DEBUG)

    NAME = "pyg5k"

    conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
    gk = Grid5000.from_yaml(conf_file)
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
549

550 551 552 553 554 555 556
    sites = gk.sites.list()
    site = gk.sites["rennes"]
    sites = [gk.sites["rennes"], gk.sites["nancy"], gk.sites["grenoble"]]

    # creates some jobs
    jobs = []
    for site in sites:
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
557 558 559
        job = site.jobs.create({"name": "pyg5k",
                                "command": "sleep 3600"})
        jobs.append(job)
560 561 562

    _jobs = []
    for site in sites:
SIMONIN Matthieu's avatar
SIMONIN Matthieu committed
563 564
        _jobs.append((site.uid, site.jobs.list(name=NAME,
                                               state="waiting,launching,running")))
565 566 567 568 569 570

    print("We found %s" % _jobs)

    # deleting the jobs
    for job in jobs:
        job.delete()
571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642

4.8.3 Caching API responses
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Grid’5000 reference API is static. In this situation to speed up the
requests, one could leverage heavily on caching. Currently
``python-grid5000`` doesn’t do caching out-of the box but defers that to the
consuming application. There are many solutions to implement a cache.
Amongst them LRU cache
(`https://docs.python.org/3/library/functools.html#functools.lru_cache <https://docs.python.org/3/library/functools.html#functools.lru_cache>`_)
provides an in-memory caching facilities but doesn’t give you control on the
cache. The ring library (`https://ring-cache.readthedocs.io/en/stable/ <https://ring-cache.readthedocs.io/en/stable/>`_) is
great as it implements different backends for your cache (esp.
cross-processes cache) and give you control on the cached object. Enough talking:


.. code:: python

    import logging
    import threading
    import os

    import diskcache
    from grid5000 import Grid5000
    import ring


    _api_lock = threading.Lock()
    # Keep track of the api client
    _api_client = None

    storage = diskcache.Cache('cachedir')

    def get_api_client():
        """Gets the reference to the API cient (singleton)."""
        with _api_lock:
            global _api_client
            if not _api_client:
                conf_file = os.path.join(os.environ.get("HOME"),
                                        ".python-grid5000.yaml")
                _api_client = Grid5000.from_yaml(conf_file)

            return _api_client


    @ring.disk(storage)
    def get_sites_obj():
        """Get all the sites."""
        gk = get_api_client()
        return gk.sites.list()


    @ring.disk(storage)
    def get_all_clusters_obj():
        """Get all the clusters."""
        sites = get_sites_obj()
        clusters = []
        for site in sites:
            # should we cache the list aswell ?
            clusters.extend(site.clusters.list())
        return clusters


    if __name__ == "__main__":
        logging.basicConfig(level=logging.DEBUG)
        clusters = get_all_clusters_obj()
        print(clusters)
        print("Known key in the cache")
        print(get_all_clusters_obj.get())
        print("Calling again the function is now faster")
        clusters = get_all_clusters_obj()
        print(clusters)