Commit b77f04b9 authored by CHARDET Maverick's avatar CHARDET Maverick
Browse files

Merge branch 'unstable' into 'master'

README + simplified server_client examples

See merge request mchardet/madpp!1
parents fbef4eb9 dbc783cb
# MAD
# Concerto
The implementation of the Madeus model in Python.
This is a preliminary implementation in Python 3 of the Concerto reconfiguration model.
Authors: Maverick Chardet, Hélène Coullon, Christian Perez {first}.{last}@inria.fr
Licence: GNU GPL v3
## Documentation
## Setup
A complete documentation, including a getting started is available online at
https://mad.readthedocs.io/en/latest/
### Option 1: installing Concerto (permanent)
TODO: packaging
### Option 2: adding Concerto to PYTHONPATH (temporary)
If using bash, go to the Concerto root directory using `cd`, and then run `source source_dir.sh`, which will append the
current directory to your `PYTHONPATH` during this session only. You can then run any Concerto script inside this
terminal.
## Getting started
TODO: this introduction needs to be reworked to explain everything in more detail.
Concerto is a reconfiguration model which allows to describe distributed software as an evolving assembly of components.
### Component
A component usually represents a module of a distributed application, but can actually represent anything, as long as it
can express its interface (the services and data it needs to work properly and the services and data it provides)
explicitely. This is done by using ports. There are four types of ports: service use (using a service), service provide
(providing a service), data use (using a piece of data) and data provide (providing a piece of data). Each component also
defines a life-cycle represented as a state-machine (possibly presenting parallel transitions like Petri-nets). A place
(a state of the state-machine) represents a milestone in the life-cycle of the component. A transition between two places
allows a token (which marks a currently active place) to go from one place to another by executing an action tied to the
transition. In this implementation, an action is a Python function. Transitions are labeled with a behavior. A transition
can only be executed if it is labeled by the current active behavior of the component. The ports of a component are bound
to places, transitions or groups of places to indicate that a service/data is used/provided when the corresponding place,
transition or group is active. A group is active if there is at least one token in one of the places it contains or one
of the transitions going from one place it contains to another place it also contains.
TODO: input and output docks, input docks sets
To define a component type, declare a new class extending `concerto.component.Component` (tip:
import `Component` either from `concerto.component` or from `concerto.all`). Declare a `create` method (with no
arguments), which must initialize five properties:
- `self.places`: list of names of places
- `self.groups`: dictionary mapping a group name to a list of the names of the places it contains
- `self.transitions`: dictionary mapping a transition name to a tuple `(init, dest, bhv, set, f [, args])` where:
* `init` is the name of the initial place of the transition
* `dest` is the name of the destination place of the transition
* `bhv` is the name of the behavior the transition is labelled with
* `set` is the id of the input docks set relative to the behavior (usually 0)
* `f` is a Python callable (usually a reference to a method declared within the class itself)
* `args` is an optional list of parameters to pass to `f`
- `self.dependencies`: dictionary mapping a port name to a tuple `(type, bindings)` where:
* `type` is a value of the enumeration `concerto.dependency.DepType` (USE, PROVIDE, DATA_USE or DATA_PROVIDE)
* `bindings` is a list of names of places, groups or transitions the port is bound to.
- `self.initial_place`: name of the place which should hold a token when the component is created
Example (from `/examples/server_clients/client.py`):
```python
from concerto.all import *
class Client(Component):
def create(self):
self.places = [
'off',
'installed',
'configured',
'running',
'paused'
]
self.groups = {
'using_service': ['running', 'paused']
}
self.transitions = {
'install1': ('off', 'installed', 'install_start', 0, self.install1),
'install2': ('off', 'configured', 'install_start', 0, self.install2),
'configure': ('installed', 'configured', 'install_start', 0, self.configure),
'start': ('configured', 'running', 'install_start', 0, self.start),
'suspend1': ('running', 'paused', 'stop', 0, self.suspend1),
'suspend2': ('paused', 'configured', 'stop', 0, self.suspend2)
}
self.dependencies = {
'server_ip': (DepType.DATA_USE, ['configure']),
'service': (DepType.USE, ['using_service'])
}
self.initial_place = 'off'
```
The constructor of the `concerto.component.Component` class will call `create`. Keep in mind that if you override the
`__init__` function, you must call explicitely the constructor of the parent class. If the `create` function uses anything
this is initialized in the `__init__` function, call `Component.__init__` after their initialization.
For complete examples, please look at the components declared in the `examples` directory, such as
`/examples/server_clients/client.py`.
### Assembly
An assembly is a possibly continuously evolving collection of component instances interconnected by their ports. Initially,
an assembly is empty. Reconfiguration operations, written using the ScoreL language, can be applied to it to add or remove
components, connect or disconnect their ports, push a behavior to the queue of behaviors of a component or wait until a
component's queue is empty.
To define an assembly, declare a new class extending `concerto.assembly.Assembly` (tip: import `Assembly` either from
`concerto.assembly` or from `concerto.all`). To define a new reconfiguration, create a new method ending with those two
instructions (you can do otherwise if you know what you're doing):
```python
# Here goes your ScoreL program
self.wait_all() # Waits for all the components to finish their queued behaviors (ensures the reconfiguration is finished)
self.synchronize() # Synchronizes with the execution thread, returning when the reconfiguration is actually finished
```
Before those, you can write any ScoreL program using the following `Assembly` methods:
- `add_component(c, ref)`: adds a component instance to the assembly, where `c` is an arbitrary string identifier for the component and `ref` is a reference to an instance of the Python class of the desired component
- `del_component(c)`: deletes a component instance from the assembly, where `c` is the identifier of the component to remove
- `connect(c1, p1, c2, p2)`: connects the port `p1` of component `c1` and the port `p2` of component `c2`
- `disconnect(c1, p1, c2, p2)`: disconnects the port `p1` of component `c1` and the port `p2` of component `c2`
- `push_b(c, behavior)`: adds the behavior `behavior` to the queue of component `c`
- `wait(c)`: waits until component `c` has an empty queue before executing the next instructions
- `wait_all()`: waits until all the components of the assembly have empty queues before executing the next instructions
Note that these methods add the corresponding ScoreL instructions to a queue of instructions to apply to the assembly. As
a consequence, none of them are blocking. In order to wait until the last instruction was actually applied, you can call
the `synchronize()` instruction.
Example (from `examples/server_clients/server_client_assembly.py`):
```python
from concerto.all import *
from client import Client
from server import Server
class ServerClient(Assembly):
def __init__(self):
self.server = Server()
self.client = Client()
Assembly.__init__(self)
def deploy(self):
self.print("### DEPLOYING ####")
self.add_component('client', self.client)
self.add_component('server', self.server)
self.connect('client', 'server_ip',
'server', 'ip')
self.connect('client', 'service',
'server', 'service')
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait_all()
self.synchronize()
```
Note that here the Python instances of the Client and Server classes were declared in `__init__` as properties of the assembly.
This is only one of many ways to do it. One could for instance construct the instances directly inside the `add_component`
calls.
For complete examples, please look at the assemblies declared in the `examples` directory, such as
`/examples/server_clients/server_client_assembly.py`.
### Implementation-specific use of Assembly objects
The simpliest way to execute a Concerto reconfiguration is to only declare an instance of an assembly and execute the first
reconfiguration (usually deployment), possibly followed by others. Finally, a call to the `terminate()` method waits for
all the reconfigurations to be finished and kills the execution thread of the assembly.
Example (modified extract from from `examples/server_clients/test_server_client.py`):
```python
from server_client_assembly import ServerClient
sca = ServerClient()
sca.deploy()
sca.suspend()
sca.restart()
sca.terminate()
```
Our implementation however provides additional functionality. These `Assembly` methods can be called before the first reconfiguration
is called:
- `set_use_gantt_chart(b)`: if `b` is `True`, records the start and end time of transitions and behaviors' execections
so that a Gantt Chart can be exported at the end (in JSON or GNUplot format). (default: `False`)
- `set_verbosity(verbosity)`: if `verbosity` is `-1`, any call that components make to the `Component.t_print` function will
be skipped, and no debug information will be printed. If `verbosity` is `0`, only calls to `Component.t_print` will print. If
`verbosity` is `1`, limited debug information will be printed. If `verbosity` is `2` or more, full debug information will be printed.
(default: `0`)
- `set_print_time(b)`: if `b` is `True`, `Component.t_print` and debug printing functions will print the time before the message. If
`b` is false, they will not. (default: `True`)
- `set_dryrun(b)`: if `b` is `True`, the transitions executed by the components will all be replaces by `pass` and do nothing. This
allows to check for the well-formedness of the assembly. (default: `False`)
These `Assembly` methods can be called after `terminate` is called:
- `get_gantt_chart()`: if `set_use_gantt_chart(True)` was called, returns a `concerto.gantt_chart.GanttChart` object. The GanttChart
methods `export_json(file_name)` and `export_gnuplot(file_name, title='')` can then be used to export the Gantt chart.
Note that it is possible to customize the verbosity of a specific component, or hide this component from the resulting Gantt chart
by calling the following `Component` methods on a `Component` Python object:
- `force_vebosity(forced_verobisty)`
- `force_hide_from_gantt_chart()`
Example (modified extract from from `examples/server_clients/test_server_client.py`):
```python
from server_client_assembly import ServerClient
sca = ServerClient()
sca.set_use_gantt_chart(True)
sca.set_verbosity(-1)
sca.set_print_time(False)
# The following is possible because server and client are declared as properties in ServerClient.__init__()
sca.server.force_vebosity(2)
sca.client.force_hide_from_gantt_chart()
sca.deploy()
sca.suspend()
sca.restart()
sca.terminate()
gc : GanttChart = sca.get_gantt_chart()
gc.export_gnuplot("results_server.gpl", "Gantt chart with the Server component only")
gc.export_json("results_server.json")
```
......@@ -78,7 +78,7 @@ class Assembly (object):
for c in self.components:
self.components[c].set_print_time(value)
def set_dryrun(self, value : int):
def set_dryrun(self, value : bool):
self.dryrun = value
for c in self.components:
self.components[c].set_dryrun(value)
......
......@@ -34,41 +34,35 @@ class Client(Component):
self.initial_place = 'off'
def __init__(self, t_ci1=1., t_ci2=1., t_cc=1., t_cr=1., t_cs1=2., t_cs2=0.):
def __init__(self):
self.server_ip = None
self.t_ci1 = t_ci1
self.t_ci2 = t_ci2
self.t_cc = t_cc
self.t_cr = t_cr
self.t_cs1 = t_cs1
self.t_cs2 = t_cs2
Component.__init__(self)
def install1(self):
self.print_color("installing (1/2)")
time.sleep(self.t_ci1)
time.sleep(2.)
self.print_color("installed (1/2)")
def install2(self):
self.print_color("installing (2/2)")
time.sleep(self.t_ci2)
time.sleep(3.)
self.print_color("installed (2/2)")
def configure(self):
self.server_ip = self.read('server_ip')
self.print_color("configuring [server IP: %s]" % self.server_ip)
time.sleep(self.t_cc)
time.sleep(1.)
self.print_color("configured")
def start(self):
self.print_color("starting")
time.sleep(self.t_cr)
time.sleep(1.)
self.print_color("running")
def suspend1(self):
self.print_color("suspending")
time.sleep(self.t_cs1)
time.sleep(0.)
def suspend2(self):
time.sleep(1.)
self.print_color("suspended")
time.sleep(self.t_cs2)
......@@ -26,32 +26,28 @@ class Server(Component):
self.initial_place = 'undeployed'
def __init__(self, t_sa=4., t_sr=4., t_su=1., t_sc=0.5):
def __init__(self):
self.my_ip = None
self.t_sa = t_sa
self.t_sr = t_sr
self.t_su = t_su
self.t_sc = t_sc
Component.__init__(self)
def allocate(self):
self.print_color("allocating resources")
time.sleep(self.t_sa)
time.sleep(6.)
self.my_ip = "123.124.1.2"
self.write('ip', self.my_ip)
self.print_color("finished allocation (IP: %s)" % self.my_ip)
def run(self):
self.print_color("preparing to run")
time.sleep(self.t_sr)
time.sleep(4.)
self.print_color("running")
def update(self):
self.print_color("updating")
time.sleep(self.t_su)
time.sleep(3.)
self.print_color("updated")
def cleanup(self):
self.print_color("cleaning up")
time.sleep(self.t_sc)
time.sleep(2.)
self.print_color("cleaned up")
#!/usr/bin/python3
from concerto.all import *
from client import Client
from server import Server
class ServerClient(Assembly):
def __init__(self):
self.server = Server()
self.client = Client()
Assembly.__init__(self)
def deploy(self):
self.print("### DEPLOYING ####")
self.add_component('client', self.client)
self.add_component('server', self.server)
self.connect('client', 'server_ip',
'server', 'ip')
self.connect('client', 'service',
'server', 'service')
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait_all()
self.synchronize()
def suspend(self):
self.print("### SUSPENDING ###")
self.push_b('client', 'stop')
self.push_b('server', 'stop')
self.wait_all()
self.synchronize()
def restart(self):
self.print("### RESTARTING ###")
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait_all()
self.synchronize()
def maintain(self):
self.print("### MAINTAINING ###")
self.push_b('client', 'stop')
self.push_b('server', 'stop')
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait_all()
self.synchronize()
#!/usr/bin/python3
from concerto.all import *
from client import Client
from server import Server
class ServerClients(Assembly):
def __init__(self, nb_clients):
self.nb_clients = nb_clients
Assembly.__init__(self)
self.clients = []
for i in range(self.nb_clients):
self.clients.append(Client())
self.add_component(self.client_name(i), self.clients[i])
self.server = Server()
self.add_component('server', self.server)
self.synchronize()
@staticmethod
def client_name(id : int):
return 'client%d'%id
def deploy(self):
self.print("### DEPLOYING ####")
for i in range(self.nb_clients):
self.connect(self.client_name(i), 'server_ip',
'server', 'ip')
self.connect(self.client_name(i), 'service',
'server', 'service')
self.push_b(self.client_name(i), 'install_start')
self.push_b('server', 'deploy')
self.wait_all()
self.synchronize()
def suspend(self):
self.print("### SUSPENDING ###")
for i in range(self.nb_clients):
self.push_b(self.client_name(i), 'stop')
self.push_b('server', 'stop')
self.wait_all()
self.synchronize()
def restart(self):
self.print("### RESTARTING ###")
for i in range(self.nb_clients):
self.push_b(self.client_name(i), 'install_start')
self.push_b('server', 'deploy')
self.wait_all()
self.synchronize()
......@@ -4,213 +4,31 @@ from concerto.all import *
from concerto.utility import Printer
import time, datetime
from client import Client
from server import Server
from examples.utils import *
class ServerClient(Assembly):
def __init__(self, t_sa=4., t_sr=4., t_su=1., t_sc=0.5, t_ci1=1., t_ci2=1., t_cc=1., t_cr=1., t_cs1=2., t_cs2=0.):
self.t_sa = t_sa
self.t_sr = t_sr
self.t_su = t_su
self.t_sc = t_sc
self.t_ci1 = t_ci1
self.t_ci2 = t_ci2
self.t_cc = t_cc
self.t_cr = t_cr
self.t_cs1 = t_cs1
self.t_cs2 = t_cs2
self.server = Server(t_sa=self.t_sa, t_sr=self.t_sr, t_su=self.t_su, t_sc=self.t_sc)
self.client = Client(t_ci1=self.t_ci1, t_ci2=self.t_ci2, t_cc=self.t_cc, t_cr=self.t_cr, t_cs1=self.t_cs1, t_cs2=self.t_cs2)
Assembly.__init__(self)
def deploy(self):
self.print("### DEPLOYING ####")
self.add_component('client', self.client)
self.add_component('server', self.server)
self.connect('client', 'server_ip',
'server', 'ip')
self.connect('client', 'service',
'server', 'service')
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait('client')
self.synchronize()
def suspend(self):
self.print("### SUSPENDING ###")
self.push_b('client', 'stop')
self.push_b('server', 'stop')
self.wait('server')
self.synchronize()
def restart(self):
self.print("### RESTARTING ###")
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait('client')
self.synchronize()
def maintain(self):
self.print("### MAINTAINING ###")
self.push_b('client', 'stop')
self.push_b('server', 'stop')
self.push_b('client', 'install_start')
self.push_b('server', 'deploy')
self.wait('client')
self.synchronize()
from server_client_assembly import ServerClient
# Creating an instance of the ServerClient assembly and setting some parameters
sca = ServerClient()
sca.set_use_gantt_chart(True)
sca.set_verbosity(1)
sca.set_print_time(True)
def time_test(verbosity : int = 0, printing : bool = False, print_time : bool = False) -> float:
start_time : float = time.perf_counter()
if printing: Printer.st_tprint("Main: creating the assembly")
sca = ServerClient()
sca.set_use_gantt_chart(True)
sca.set_verbosity(verbosity)
sca.set_print_time(print_time)
if printing: Printer.st_tprint("Main: deploying the assembly")
sca.deploy()
if printing: Printer.st_tprint("Main: waiting a little before reconfiguring")
time.sleep(3)
sca.suspend()
if printing: Printer.st_tprint("Main: waiting a little before restarting")
time.sleep(5)
sca.restart()
end_time : float = time.perf_counter()
total_time = end_time-start_time
if printing: Printer.st_tprint("Total time in seconds: %f"%total_time)
sca.terminate()
gc : GanttChart = sca.get_gantt_chart()
gc.export_gnuplot("results.gpl")
gc.export_json("results.json")
return total_time
Printer.st_tprint("Main: deploying the assembly")
sca.deploy() # First reconfiguration
Printer.st_tprint("Main: waiting a little before reconfiguring")
time.sleep(3)
def time_test_params(verbosity : int = -1, printing : bool = True, print_time : bool = False,
t_sa=4., t_sr=4., t_su=1., t_sc=0.5, t_ci1=1., t_ci2=1., t_cc=1., t_cr=1., t_cs1=2., t_cs2=0.):
if printing: Printer.st_tprint("Main: creating the assembly")
sca = ServerClient(t_sa, t_sr, t_su, t_sc, t_ci1, t_ci2, t_cc, t_cr, t_cs1, t_cs2)
sca.set_use_gantt_chart(False)
sca.set_verbosity(verbosity)
sca.set_print_time(print_time)
if printing: Printer.st_tprint("Main: deploying the assembly")
deploy_start_time : float = time.perf_counter()
sca.deploy()
deploy_end_time : float = time.perf_counter()
total_deploy_time = deploy_end_time-deploy_start_time
if printing: Printer.st_tprint("Main: waiting a little before reconfiguring")
time.sleep(1)
if printing: Printer.st_tprint("Main: reconfiguring the assembly")
reconf_start_time : float = time.perf_counter()
sca.maintain()
reconf_end_time : float = time.perf_counter()
total_reconf_time = reconf_end_time-reconf_start_time
total_time = total_deploy_time+total_reconf_time
Printer.st_tprint("Total time in seconds: %f\n deploy: %f\n reconf: %f"%(total_time,total_deploy_time,total_reconf_time))
deploy_theoretical_time = max(t_sa+t_sr,t_cr+max(t_ci2,t_cc+max(t_ci1,t_sa)))
reconf_theoretical_time = max(t_cs1+t_cs2+t_cr,t_sr+t_cs1+max(t_su,t_sc))
total_theoretical_time = deploy_theoretical_time + reconf_theoretical_time
Printer.st_tprint("Theoretical time in seconds: %f\n deploy: %f\n reconf: %f"%(total_theoretical_time,deploy_theoretical_time,reconf_theoretical_time))
sca.terminate()
return total_deploy_time, total_reconf_time, deploy_theoretical_time, reconf_theoretical_time
sca.suspend() # Second reconfiguration
Printer.st_tprint("Main: waiting a little before restarting")
time.sleep(5)
def random_tests(nb_trials, min_value=0., max_value=10.):
from random import uniform
from json import dumps
from sys import stderr
results = {
"trials": [],
"max_distance_deploy": 0.,
"max_distance_reconf": 0.
}
for i in range(nb_trials):
print("Trial %d/%d"%(i+1,nb_trials), file=stderr)
t_sa = uniform(min_value,max_value)
t_sr = uniform(min_value,max_value)
t_su = uniform(min_value,max_value)
t_sc = uniform(min_value,max_value)
t_ci1 = uniform(min_value,max_value)
t_ci2 = uniform(min_value,max_value)
t_cc = uniform(min_value,max_value)
t_cr = uniform(min_value,max_value)