Mentions légales du service

Skip to content
Snippets Groups Projects
Commit fe039288 authored by Guillaume Pallez (Aupy)'s avatar Guillaume Pallez (Aupy)
Browse files

Merge branch 'python_package' into 'master'

Python package

See merge request !1
parents 9861d56d ba9c5e51
Branches
No related tags found
1 merge request!1Python package
Showing with 1093 additions and 393 deletions
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
......@@ -23,6 +23,12 @@ These works are available in the folder "references".
## Requirements
TODO
## Installation
```
pip install git+https://gitlab.inria.fr/adjoint-computation/H-Revolve.git
```
## Main Algorithms:
### INPUTS
......@@ -44,10 +50,10 @@ The algorithms take into input:
#Single Level algorithms
- [*Revolve*] Revolve is the implementation of the single-level dynamic programming presented in [3], where the costs of read and write are considered equal to 0. It provides the user a schedule that minimizes the execution time to reverse the graph.
Utilisation is:
`$ python Revolve.py [l] [cm] --uf [uf] --ub [ub] `
`$ Revolve [l] [cm] --uf [uf] --ub [ub] `
Examples:
`$ python Revolve.py 100 8 --uf 1.2 --ub 3`
`$ python Revolve.py 20 2`
`$ Revolve 100 8 --uf 1.2 --ub 3`
`$ Revolve 20 2`
#Two Level algorithms
The implementations of this section considered a specific model with two levels of storage:
......@@ -56,25 +62,25 @@ The implementations of this section considered a specific model with two levels
- [*Disk-Revolve*] Disk-Revolve is the implementation of the two-level dynamic programming presented in [1]. It provides the user a schedule that minimizes the execution time to reverse the graph.
Utilisation is:
`$ python Disk-Revolve.py [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd]`
`$ Disk-Revolve [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd]`
Examples:
`$ python Disk-Revolve.py 100 8 --uf 1.2 --ub 3 --wd 8 --rd 2.5`
`$ python Disk-Revolve.py 20 2`
`$ Disk-Revolve 100 8 --uf 1.2 --ub 3 --wd 8 --rd 2.5`
`$ Disk-Revolve 20 2`
- [*Periodic-Disk-Revolve*] Periodic-Disk-Revolve returns a schedule where the number of forward steps between two consecutive disk checkpoints is constant. The schedule provided is shown to be asymptotically optimal in [2]. In addition, a conjecture was made w.r.t the value of the optimal period that Periodic-Disk-Revolve should use. The `--fast` option allows to compute this value with a much lower complexity than [Periodic-Disk-Revolve] (constant time instead of O(l^3)).
Utilisation is:
`$ python Periodic-Disk-Revolve.py [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd] --fast`
`$ Periodic-Disk-Revolve [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd] --fast`
Examples:
`$ python Periodic-Disk-Revolve.py 100 3 --uf 1.2 --ub 3 --wd 8 --rd 2.5`
`$ python Periodic-Disk-Revolve.py 200 8`
`$ Periodic-Disk-Revolve 100 3 --uf 1.2 --ub 3 --wd 8 --rd 2.5`
`$ Periodic-Disk-Revolve 200 8`
- [*Rev-Revolve*] Rev-Revolve is a slight modification of Disk-Revolve, when during the reverse sweep, one uses Revolve instead of 1D-Revolve (see [1]). It returns the optimal strategy when each Disk checkpoint can only be read once (see [0] for the proof).
Utilisation is:
`$ python Rev-Revolve.py [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd]`
`$ Rev-Revolve [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd]`
Examples:
`$ python Rev-Revolve.py 100 3 --uf 1.2 --ub 3 --wd 8 --rd 2.5`
`$ python Rev-Revolve.py 200 8`
`$ Rev-Revolve 100 3 --uf 1.2 --ub 3 --wd 8 --rd 2.5`
`$ Rev-Revolve 200 8`
......@@ -83,7 +89,6 @@ The implementations of this section considered a specific model with two levels
- [*H-revolve*] Description and usage to write.
### OUTPUT
The following outputs are provided:
......@@ -110,42 +115,70 @@ The following outputs are provided:
There is an option to give more compact format of the Sequence output. The option is `--concat` and its value [val] can take the following values: 0 (default), 1, 2 or 3.
Utilisation is:
`$ python Rev-Revolve [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd] --concat [val]`
`$ Rev-Revolve [l] [cm] --uf [uf] --ub [ub] --wd [wd] --rd [rd] --concat [val]`
Examples:
`$ python Disk-Revolve.py 100 3 --uf 1.2 --ub 3 --wd 8 --rd 2.5 --concat 3`
`$ python Disk-Revolve.py 200 8 --concat 1`
`$ Disk-Revolve 100 3 --uf 1.2 --ub 3 --wd 8 --rd 2.5 --concat 3`
`$ Disk-Revolve 200 8 --concat 1`
- [--concat 0]: The default option gives the entire schedule.
`$ python Disk-Revolve.py 15 2 --concat 0` returns:
`$ Disk-Revolve 15 2 --concat 0` returns:
> Sequence: [WD_0, F_0->5, WM_6, F_6->11, WM_12, F_12, F_13, F_14, B_15, RM_12, F_12, F_13, B_14, RM_12, F_12, B_13, RM_12, B_12, DM_12, RM_6, F_6->8, WM_9, F_9, F_10, B_11, RM_9, F_9, B_10, RM_9, B_9, DM_9, RM_6, F_6->6, WM_7, F_7, B_8, RM_7, B_7, DM_7, RM_6, B_6, DM_6, RD_0, WM_0, F_0->2, WM_3, F_3, F_4, B_5, RM_3, F_3, B_4, RM_3, B_3, DM_3, RM_0, F_0->0, WM_1, F_1, B_2, RM_1, B_1, DM_1, RM_0, B_0, DM_0]
- [--concat 1]: Factorizes the subschedule that are computed by a Revolve algorithm.
`$ python Disk-Revolve.py 15 2 --concat 0` returns:
`$ Disk-Revolve 15 2 --concat 0` returns:
> Sequence: [WD_0, F_0->5, Revolve(9, 2), RD_0, Revolve(5, 2)]
Note that to obtain a _defactorized_ version of Revolve(9,2) (for instance), one can call the subprogram `$ python Revolve.py 9 2`
Note that to obtain a _defactorized_ version of Revolve(9,2) (for instance), one can call the subprogram `$ Revolve 9 2`
- [--concat 2]: Factorizes the subschedules that are computed by a 1D-Revolve algorithm (see [1]).
`$ python Disk-Revolve.py 15 2 --concat 0` returns:
`$ Disk-Revolve 15 2 --concat 0` returns:
> Sequence: [WD_0, F_0->5, Revolve(9, 2), RD_0, 1D-Revolve(5, 2)]
Note that to obtain a _defactorized_ version of 1D-Revolve(5,2) (for instance), one can call the subprogram `$ python 1D-Revolve.py 5 2`
Note that to obtain a _defactorized_ version of 1D-Revolve(5,2) (for instance), one can call the subprogram `$ 1D-Revolve 5 2`
- [--concat 3]: To understand this last option, one needs to go back to what a schedule looks like [1,2]. A schedule provided by Disk-Revolve is always of the form: *Forward sweep; Turn; Backward Sweep*.
To explain this schedule, we will use the example of `Disk-Revolve.py 100 4` which under the option `--concat 2` returns:
To explain this schedule, we will use the example of `Disk-Revolve 100 4` which under the option `--concat 2` returns:
> [WD_0, F_0->14, WD_15, F_15->29, WD_30, F_30->44, WD_45, F_45->59, WD_60, F_60->74, Revolve(25, 4), RD_60, 1D-Revolve(14, 4), RD_45, 1D-Revolve(14, 4), RD_30, 1D-Revolve(14, 4), RD_15, 1D-Revolve(14, 4), RD_0, 1D-Revolve(14, 4)]
- The *forward sweep* is a sequence of Disk writes followed by forward operations.
Eg: for `Disk-Revolve.py 100 4` -> (WD_0, F_0->14, WD_15, F_15->29, WD_30, F_30->44, WD_45, F_45->59, WD_60, F_60->74)
Eg: for `Disk-Revolve 100 4` -> (WD_0, F_0->14, WD_15, F_15->29, WD_30, F_30->44, WD_45, F_45->59, WD_60, F_60->74)
- The *Turn* is a Revolve function.
Eg: for `Disk-Revolve.py 100 4` -> Revolve(25, 4)
Eg: for `Disk-Revolve 100 4` -> Revolve(25, 4)
- The *backward sweep* is a sequence of read disks followed by 1D-Revolve operations.
Eg: for `Disk-Revolve.py 100 4` -> RD_60, 1D-Revolve(14, 4), RD_45, 1D-Revolve(14, 4), RD_30, 1D-Revolve(14, 4), RD_15, 1D-Revolve(14, 4), RD_0, 1D-Revolve(14, 4)
Eg: for `Disk-Revolve 100 4` -> RD_60, 1D-Revolve(14, 4), RD_45, 1D-Revolve(14, 4), RD_30, 1D-Revolve(14, 4), RD_15, 1D-Revolve(14, 4), RD_0, 1D-Revolve(14, 4)
In [2], we denote by the term "periods", the number of forward operations between two writes on disks. Finally the idea of the last option is to give: the sizes of the different periods [m1], ..., [mk] and the length of the turn [tn], and is returned under the form: ([m1], [m2], .., [mk]; [tn]). To obtain a _defactorized_ version of it, we start by a WD_0, then F_0->[m1-1], then WD_[m1] etc.
`$ python Disk-Revolve.py 100 4 --concat 3` returns:
`$ Disk-Revolve 100 4 --concat 3` returns:
> Sequence: (15, 15, 15, 15, 15; 26)
## Library interface
The various checkpointing algorithms are also available as library calls in the
`hrevolve` package. With the exception of HRevolve, these take the positional
arguments `l` and `cm` which are integers with the same meaning as above.
Additional configuration parameters can be set using keyword arguments. The
result is a `Sequence` object encoding the schedule. For example:
```python
In [1]: from hrevolve import disk_revolve
In [2]: print(disk_revolve(15, 2, concat=2))
[WD_0, F_0->5, Revolve(9, 2), RD_0, 1D-Revolve(5, 2)]
```
The library interface to HRevolve takes the positional arguments `l`, `cvect`,
`wvect`, and `rvect`. `l` is the length of the graph. The other arguments are
vectors indicating, respectively, the number of slots at each level of memory
and the cost of writing and reading a slot at each level of memory. For
example:
```python
In [1]: from hrevolve import hrevolve
In [2]: print(hrevolve(20, [1, 2, 10], [0, 2, 3], [0, 2, 3], concat=2))
[W^2_0, F_0->6, HRevolve_1(13, 2), R^2_0, F_0->2, HRevolve_1(3, 2), R^2_0, HRevolve_1(2, 2)]
```
BIBLIOGRAPHY
......
from .revolve import revolve # NOQA F401
from .revolve_1d import revolve_1d # NOQA F401
from .disk_revolve import disk_revolve # NOQA F401
from .periodic_disk_revolve import periodic_disk_revolve # NOQA F401
from .hrevolve import hrevolve # NOQA F401
#!/usr/bin/python
import parameters
import math
import os
import sys
import time
import numpy as np
import functools
official_names = {
"Forward": "F",
......@@ -26,23 +19,24 @@ official_names = {
"Forward_branch": "F",
"Forwards_branch": "F",
"Backward_branch": "B",
"Turn" : "T",
"Turn": "T",
"Discard_branch": "DB",
"Checkpoint_branch": "C"
}
def my_buddy(index, last_index):
return index + 1
def beta(x,y):
def beta(x, y):
if y < 0:
return 0
return math.factorial(x+y) / (math.factorial(x) * math.factorial(y))
def argmin(l):
#Return the last argmin (1-based)
# Return the last argmin (1-based)
index = 0
m = l[0]
for i in range(len(l)):
......@@ -51,8 +45,9 @@ def argmin(l):
m = l[i]
return 1 + index
def argmin0(l):
#Return the first argmin (0-based)
# Return the first argmin (0-based)
index = 0
m = l[0]
for i in range(len(l)):
......@@ -61,6 +56,7 @@ def argmin0(l):
m = l[i]
return index
def argmin_list(l):
index_list = [0]
m = l[0]
......@@ -82,7 +78,7 @@ def from_list_to_string(l):
class Operation:
def __init__(self, operation_type, operation_index):
def __init__(self, operation_type, operation_index, params):
if operation_type not in official_names:
raise ValueError("Unreconized operation name: " + operation_type)
self.type = operation_type
......@@ -93,6 +89,7 @@ class Operation:
if self.type == "Forwards_branch" and self.index[1] == self.index[2]:
self.type = "Forward_branch"
self.index = [self.index[0], self.index[1]]
self.params = params
def __repr__(self):
if self.index is None:
......@@ -108,24 +105,18 @@ class Operation:
return official_names[self.type] + "^" + str(self.index[0]) + "_" + str(self.index[1])
def cost(self):
if os.path.basename(sys.argv[0]) == "HeteRevolve.py":
return self.cost_heterogeneous()
else:
return self.cost_homogeneous()
def cost_homogeneous(self):
if self.type == "Forward":
return parameters.uf
return self.params["uf"]
if self.type == "Forwards":
return (self.index[1] - self.index[0] + 1) * parameters.uf
return (self.index[1] - self.index[0] + 1) * self.params["uf"]
if self.type == "Backward":
return parameters.ub
return self.params["ub"]
if self.type == "Checkpoint":
return 0
if self.type == "Read_disk":
return parameters.rd
return self.params["rd"]
if self.type == "Write_disk":
return parameters.wd
return self.params["wd"]
if self.type == "Read_memory":
return 0
if self.type == "Write_memory":
......@@ -135,37 +126,26 @@ class Operation:
if self.type == "Discard_memory":
return 0
if self.type == "Read":
return parameters.architecture.rd[self.index[0]]
return self.params["rd"][self.index[0]]
if self.type == "Write":
return parameters.architecture.wd[self.index[0]]
return self.params["wd"][self.index[0]]
if self.type == "Discard":
return 0
if self.type == "Forward_branch":
return parameters.uf
return self.params["uf"]
if self.type == "Forwards_branch":
return (self.index[2] - self.index[1] + 1) * parameters.uf
return (self.index[2] - self.index[1] + 1) * self.params["uf"]
if self.type == "Backward_branch":
return parameters.ub
return self.params["ub"]
if self.type == "Turn":
return parameters.up
return self.params["up"]
if self.type == "Discard_branch":
return 0
if self.type == "Checkpoint_branch":
return 0
raise ValueError("Unknown cost for operation type " + self.type)
def cost_heterogeneous(self):
if self.type == "Forward":
return parameters.chain.fweigth[self.index]
if self.type == "Forwards":
return sum(parameters.chain.fweigth[self.index[0]:self.index[1]+1])
if self.type == "Backward":
return parameters.chain.bweigth[self.index]
if self.type == "Checkpoint":
return 0
raise ValueError("Unknown cost for operation type " + self.type)
def shift(self, size, branch = -1):
def shift(self, size, branch=-1):
if type(self.index) is int:
self.index += size
elif type(self.index) is list:
......@@ -197,71 +177,73 @@ class Function:
class Sequence:
def __init__(self, function):
self.sequence = [] #List of Operation and Sequence
self.function = function #Description the function (name and parameters)
self.makespan = 0 #Makespan to be updated
def __init__(self, function, levels=None, concat=0):
self.sequence = [] # List of Operation and Sequence
self.function = function # Description the function (name and parameters)
self.levels = levels
self.concat = concat
self.makespan = 0 # Makespan to be updated
if self.function.name == "HRevolve" or self.function.name == "HRevolve_aux":
self.storage = [[] for _ in range(parameters.architecture.nblevels)] # List of list of checkpoints in hierarchical storage
self.storage = [[] for _ in range(self.levels)] # List of list of checkpoints in hierarchical storage
else:
self.memory = [] #List of memory checkpoints
self.disk = [] #List of disk checkpoints
self.memory = [] # List of memory checkpoints
self.disk = [] # List of disk checkpoints
self.type = "Function"
def __repr__(self):
if self.function.name == "HRevolve" or self.function.name == "HRevolve_aux":
return self.concat_sequence_hierarchic(concat=parameters.concat).__repr__()
return self.concat_sequence_hierarchic(self.concat).__repr__()
else:
if parameters.concat == 3:
if self.concat == 3:
return from_list_to_string(self.canonical())
else:
return self.concat_sequence(concat = parameters.concat).__repr__()
return self.concat_sequence(self.concat).__repr__()
def canonical(self):
if self.function.name == "Disk-Revolve":
concat = 2
if self.function.name == "1D-Revolve" or self.function.name == "Revolve":
concat = 1
l = [x.l + 1 for x in self.concat_sequence(concat = concat) if x.__class__.__name__ == "Function"]
l = [x.l + 1 for x in self.concat_sequence(concat=concat) if x.__class__.__name__ == "Function"]
l.reverse()
return l
def concat_sequence(self, concat = parameters.concat):
def concat_sequence(self, concat):
l = []
for x in self.sequence:
if x.__class__.__name__ == "Operation":
l.append(x)
elif x.__class__.__name__ == "Sequence":
if concat == 0:
l += x.concat_sequence(concat = concat)
if self.concat == 0:
l += x.concat_sequence(concat=concat)
elif concat == 1:
if x.function.name == "Revolve":
l.append(x.function)
else:
l += x.concat_sequence(concat = concat)
l += x.concat_sequence(concat=concat)
elif concat == 2:
if x.function.name == "Revolve" or x.function.name == "1D-Revolve":
l.append(x.function)
else:
l += x.concat_sequence(concat = concat)
l += x.concat_sequence(concat=concat)
else:
raise ValueError("Unknown concat value: " + str(concat))
else:
raise ValueError("Unknown class name: " + x.__class__.__name__)
return l
def concat_sequence_hierarchic(self, concat = parameters.concat):
def concat_sequence_hierarchic(self, concat):
l = []
for x in self.sequence:
if x.__class__.__name__ == "Operation":
l.append(x)
elif x.__class__.__name__ == "Sequence":
if concat == 0:
l += x.concat_sequence_hierarchic(concat = concat)
l += x.concat_sequence_hierarchic(concat=concat)
elif x.function.name == "HRevolve" and x.function.index[0] <= concat-1:
l.append(x.function)
l.append(x.function)
else:
l += x.concat_sequence_hierarchic(concat = concat)
l += x.concat_sequence_hierarchic(concat=concat)
else:
raise ValueError("Unknown class name: " + x.__class__.__name__)
return l
......@@ -302,7 +284,7 @@ class Sequence:
self.memory += sequence.memory
self.disk += sequence.disk
def shift(self, size, branch = -1):
def shift(self, size, branch=-1):
for x in self.sequence:
x.shift(size, branch=branch)
if self.function.name == "HRevolve" or self.function.name == "HRevolve_aux":
......@@ -313,7 +295,7 @@ class Sequence:
self.disk = [x + size if type(x) is int else (x[0], x[1] + size) if x[0] == branch else (x[0], x[1]) for x in self.disk]
return self
def remove_useless_wm(self, K = -1):
def remove_useless_wm(self, K=-1):
if len(self.sequence) > 0:
if self.sequence[0].type == "Write_memory" or self.sequence[0].type == "Checkpoint":
self.remove(0)
......@@ -344,7 +326,7 @@ class Sequence:
return self.sequence[i+1]
def convert_old_to_branch(self, index):
for (i,x) in enumerate(self.memory):
for (i, x) in enumerate(self.memory):
if type(x) is int:
self.memory[i] = (index, x)
to_remove = []
......@@ -385,13 +367,13 @@ class Sequence:
elif op.type in ["Read_disk", "Write_disk", "Discard_disk"]:
ValueError("Cannot use convert_old_to_branch on sequences from two-memory architecture")
else:
ValueError("Unknown data type %s in convert_old_to_branch"%op.type)
ValueError("Unknown data type %s in convert_old_to_branch" % op.type)
for (i, index) in enumerate(to_remove):
self.remove(index-i)
return self
def convert_new_to_branch(self, index):
for (i,x) in enumerate(self.memory):
for (i, x) in enumerate(self.memory):
if type(x) is int:
self.memory[i] = (index, x)
to_remove = []
......@@ -415,14 +397,14 @@ class Sequence:
elif op.type in ["Read_disk", "Write_disk", "Discard_disk"]:
ValueError("Cannot use convert_new_to_branch on sequences from two-memory architecture")
else:
ValueError("Unknown data type %s in convert_new_to_branch"%op.type)
ValueError("Unknown data type %s in convert_new_to_branch" % op.type)
for (i, index) in enumerate(to_remove):
self.remove(index-i)
return self
class Table:
def __init__(self, n = 0, x = float("inf")):
def __init__(self, n=0, x=float("inf")):
self.content = [x for _ in range(n)]
self.size = n
self.print_table = 0
......@@ -436,7 +418,7 @@ class Table:
self.content.append(x)
self.size += 1
if self.print_table:
self.file.write("%d\t%d\n"%(self.size-1, x))
self.file.write("%d\t%d\n" % (self.size-1, x))
if self.size % 10 == 0:
self.file.flush()
......@@ -448,7 +430,7 @@ class Table:
try:
return self.content[i]
except IndexError:
raise IndexError("In table of length %d, index %d does not exist."%(len(self), i))
raise IndexError("In table of length %d, index %d does not exist." % (len(self), i))
def __repr__(self):
return self.content.__repr__()
......@@ -460,206 +442,3 @@ class Table:
def __len__(self):
return self.content.__len__()
############# Useless functions so far ############################
def compute_lmax(cmem):
""" Compute the maximum possible value for
l_X^(1) as defined in the paper """
td = 0
while beta(cmem, td) < 2*(parameters.wd + parameters.rd) / parameters.uf:
continue
return beta(cmem, td+3) - beta(cmem, td+2) / 2
def compute_l1X(cmem, opt_0 = None, opt_1d = None, opt_inf = None):
""" Compute l_X^(1) as defined in the paper """
lmax = compute_lmax(cmem)
if opt_0 == None or len(opt_0) < lmax:
opt_0 = get_opt_0_table(lmax, cmem)
if opt_1d == None or len(opt_1d) < lmax:
opt_1d = get_opt_1d_table(lmax, cmem, opt_0 = opt_0)
if opt_inf == None:
opt_inf = get_opt_inf_table(l, cmem, opt_0=opt_0, opt_1d=opt_1d)
l1X = 1
while opt_0[l1X][cmem] < min([parameters.wd + j * parameters.uf + opt_inf[l1X - 1 -j] + parameters.rd + opt_1d[j-1] for j in range(1, l1X)]):
l1X = l1X + 1
return l1X
def check_sequence_multi_revolve(seq, ls):
k = len(ls)
op_list = seq.concat_sequence()
memory = ["x^%d_0"%(i) for i in range(k)]
mem_usage = len(memory)
for op in op_list:
print(memory)
memory.reverse()
if op.type == "Forward_branch":
m = op.index[0]
i = op.index[1]
input = "x^%d_%d"%(m,i)
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
memory.append("x^%d_%d"%(m,i+1))
if op.type == "Forwards_branch":
m = op.index[0]
i = op.index[1]
j = op.index[2]
input = "x^%d_%d"%(m,i)
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
memory.append("x^%d_%d"%(m,j+1))
if op.type == "Backward_branch":
m = op.index[0]
i = op.index[1]
input1 = "x^%d_%d"%(m,i)
input2 = "b^%d_%d"%(m,i+1)
if input1 not in memory:
raise ValueError
if input2 not in memory:
raise ValueError
memory.remove(input1)
memory.remove(input2)
memory.reverse()
memory.append("b^%d_%d"%(m,i))
if op.type == "Discard_branch":
m = op.index[0]
i = op.index[1]
input = "b^%d_%d"%(m,i)
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
if op.type == "Turn":
for m in range(k):
input = "x^%d_%d"%(m,ls[m])
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
for m in range(k):
memory.append("b^%d_%d"%(m,ls[m]))
if op.type == "Checkpoint_branch":
m = op.index[0]
i = op.index[1]
input = "x^%d_%d"%(m,i)
if input not in memory:
raise ValueError
memory.reverse()
memory.append(input)
mem_usage = max(mem_usage, len(memory))
print(memory)
return mem_usage
def check_sequence_new_revolve(seq, l):
op_list = seq.concat_sequence()
memory = ["x_0", "b_%d"%(l+1)]
mem_usage = len(memory)
for op in op_list:
print(memory)
memory.reverse()
if op.type == "Forward":
i = op.index
input = "x_%d"%i
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
memory.append("x_%d" % (i + 1))
if op.type == "Forwards":
i = op.index[0]
j = op.index[1]
input = "x_%d" % (i)
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
memory.append("x_%d" % (j + 1))
if op.type == "Backward":
i = op.index
input1 = "x_%d" % (i)
input2 = "b_%d" % (i + 1)
if input1 not in memory:
raise ValueError
if input2 not in memory:
raise ValueError
memory.remove(input1)
memory.remove(input2)
memory.reverse()
memory.append("b_%d" % (i))
if op.type == "Checkpoint":
i = op.index
input = "x_%d" % (i)
if input not in memory:
raise ValueError
memory.reverse()
memory.append(input)
mem_usage = max(mem_usage, len(memory))
print(memory)
return mem_usage
def mem_size(memory, chain):
size = 0
for ele in memory:
index = int(ele.split('_')[-1])
type = ele.split('_')[0]
if type == "x":
size += chain.cweigth[index]
else:
size += chain.cbweigth[index]
return size
def check_sequence_heterevolve(seq, chain):
op_list = seq.concat_sequence()
l = chain.length
memory = ["x_0", "b_%d"%(l+1)]
mem_usage = mem_size(memory, chain)
for op in op_list:
print(memory, " -> ", mem_size(memory, chain))
memory.reverse()
if op.type == "Forward":
i = op.index
input = "x_%d"%i
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
memory.append("x_%d" % (i + 1))
if op.type == "Forwards":
i = op.index[0]
j = op.index[1]
input = "x_%d" % (i)
if input not in memory:
raise ValueError
memory.remove(input)
memory.reverse()
memory.append("x_%d" % (j + 1))
if op.type == "Backward":
i = op.index
input1 = "x_%d" % (i)
input2 = "b_%d" % (i + 1)
if input1 not in memory:
raise ValueError
if input2 not in memory:
raise ValueError
memory.remove(input1)
memory.remove(input2)
memory.reverse()
memory.append("b_%d" % (i))
if op.type == "Checkpoint":
i = op.index
input = "x_%d" % (i)
if input not in memory:
raise ValueError
memory.reverse()
memory.append(input)
mem_usage = max(mem_usage, mem_size(memory,chain))
print(memory, " -> ", mem_size(memory, chain))
return mem_usage
#!/usr/bin/python
import parameters
from basic_functions import *
from Revolve import *
Module1D = __import__('1D-Revolve')
OneD_Revolve = Module1D.OneD_Revolve
get_opt_1d_table = Module1D.get_opt_1d_table
from .parameters import defaults
from .basic_functions import (Operation as Op, Table, Sequence, Function, my_buddy,
argmin)
from .revolve import get_opt_0_table, revolve
from .revolve_1d import revolve_1d, get_opt_1d_table
from functools import partial
def get_opt_inf_table(lmax, cmem, opt_0 = None, opt_1d = None):
""" Compute the opt_inf table for cmem and l=0...lmax
def get_opt_inf_table(lmax, cm, uf, ub, rd, wd, one_read_disk, print_table,
opt_0=None, opt_1d=None, **params):
""" Compute the opt_inf table for cm and l=0...lmax
This computation uses a dynamic program"""
if opt_0 == None:
opt_0 = get_opt_0_table(lmax, cmem)
if opt_1d == None and not parameters.one_read_disk:
opt_1d = get_opt_1d_table(lmax, cmem, opt_0 = opt_0)
if opt_0 is None:
opt_0 = get_opt_0_table(lmax, cm, **params)
if opt_1d is None and not one_read_disk:
opt_1d = get_opt_1d_table(lmax, cm, opt_0=opt_0, **params)
opt_inf = Table()
if __name__ == '__main__' and parameters.print_table:
opt_inf.set_to_print(parameters.print_table)
opt_inf.append(parameters.ub)
#Opt_inf[1] for cmem
if cmem == 0:
opt_inf.append(parameters.wd + parameters.uf + 2 * parameters.ub + parameters.rd)
if __name__ == '__main__' and print_table:
opt_inf.set_to_print(print_table)
opt_inf.append(ub)
# Opt_inf[1] for cm
if cm == 0:
opt_inf.append(wd + uf + 2 * ub + rd)
else:
opt_inf.append(parameters.uf + 2 * parameters.ub)
# Opt_inf[2...lmax] for cmem
opt_inf.append(uf + 2 * ub)
# Opt_inf[2...lmax] for cm
for l in range(2, lmax + 1):
if parameters.one_read_disk:
opt_inf.append(min(opt_0[cmem][l], min([parameters.wd + j * parameters.uf + opt_inf[l - j] + parameters.rd + opt_0[cmem][j-1] for j in range(1, l)])))
if one_read_disk:
opt_inf.append(min(opt_0[cm][l], min([wd + j * uf + opt_inf[l - j] + rd + opt_0[cm][j-1] for j in range(1, l)])))
else:
opt_inf.append(min(opt_0[cmem][l], min([parameters.wd + j * parameters.uf + opt_inf[l - j] + parameters.rd + opt_1d[j-1] for j in range(1, l)])))
opt_inf.append(min(opt_0[cm][l], min([wd + j * uf + opt_inf[l - j] + rd + opt_1d[j-1] for j in range(1, l)])))
return opt_inf
def Disk_Revolve(l, cmem, opt_0 = None, opt_1d = None, opt_inf = None):
def disk_revolve(l, cm, opt_0=None, opt_1d=None,
opt_inf=None, **params):
""" l : number of forward step to execute in the AC graph
cmem : number of available memory slots
Return the optimal sequence of makespan Opt_inf(l, cmem)"""
if opt_0 == None:
opt_0 = get_opt_0_table(l, cmem)
if opt_1d == None and not parameters.one_read_disk:
opt_1d = get_opt_1d_table(l, cmem, opt_0 = opt_0)
if opt_inf == None:
opt_inf = get_opt_inf_table(l, cmem, opt_0 = opt_0, opt_1d = opt_1d)
sequence = Sequence(Function("Disk-Revolve", l, cmem))
cm : number of available memory slots
Return the optimal sequence of makespan Opt_inf(l, cm)"""
parameters = dict(defaults)
parameters.update(params)
uf = parameters["uf"]
rd = parameters["rd"]
wd = parameters["wd"]
one_read_disk = parameters["one_read_disk"]
if opt_0 is None:
opt_0 = get_opt_0_table(l, cm, **parameters)
if opt_1d is None and not one_read_disk:
opt_1d = get_opt_1d_table(l, cm, opt_0=opt_0, **parameters)
if opt_inf is None:
opt_inf = get_opt_inf_table(l, cm, opt_0=opt_0, opt_1d=opt_1d,
**parameters)
sequence = Sequence(Function("Disk-Revolve", l, cm), concat=parameters["concat"])
Operation = partial(Op, params=parameters)
if l == 0:
sequence.insert(Operation("Backward", my_buddy(-1, l-1)))
return sequence
if l == 1:
if cmem == 0:
if cm == 0:
sequence.insert(Operation("Write_disk", 0))
sequence.insert(Operation("Forward", 0))
sequence.insert(Operation("Backward", my_buddy(0, l - 1)))
......@@ -64,32 +74,27 @@ def Disk_Revolve(l, cmem, opt_0 = None, opt_1d = None, opt_inf = None):
sequence.insert(Operation("Backward", my_buddy(-1, l - 1)))
sequence.insert(Operation("Discard_memory", 0))
return sequence
if parameters.one_read_disk:
list_mem = [parameters.wd + j * parameters.uf + opt_inf[l - j] + parameters.rd + opt_0[cmem][j-1] for j in range(1, l)]
if one_read_disk:
list_mem = [wd + j * uf + opt_inf[l - j] + rd + opt_0[cm][j-1] for j in range(1, l)]
else:
list_mem = [parameters.wd + j * parameters.uf + opt_inf[l - j] + parameters.rd + opt_1d[j-1] for j in range(1, l)]
if min(list_mem) < opt_0[cmem][l]:
list_mem = [wd + j * uf + opt_inf[l - j] + rd + opt_1d[j-1] for j in range(1, l)]
if min(list_mem) < opt_0[cm][l]:
jmin = argmin(list_mem)
sequence.insert(Operation("Write_disk", 0))
sequence.insert(Operation("Forwards", [0, jmin - 1]))
sequence.insert_sequence(Disk_Revolve(l - jmin, cmem, opt_0 = opt_0, opt_1d = opt_1d, opt_inf = opt_inf).shift(jmin))
sequence.insert_sequence(
disk_revolve(l - jmin, cm, opt_0=opt_0, opt_1d=opt_1d, opt_inf=opt_inf, **parameters).shift(jmin)
)
sequence.insert(Operation("Read_disk", 0))
if parameters.one_read_disk:
sequence.insert_sequence(Revolve(jmin - 1, cmem, opt_0 = opt_0))
if one_read_disk:
sequence.insert_sequence(
revolve(jmin - 1, cm, opt_0=opt_0, **parameters)
)
else:
sequence.insert_sequence(OneD_Revolve(jmin - 1, cmem, opt_0 = opt_0, opt_1d = opt_1d))
sequence.insert_sequence(
revolve_1d(jmin - 1, cm, opt_0=opt_0, opt_1d=opt_1d, **parameters)
)
return sequence
else:
sequence.insert_sequence(Revolve(l, cmem, opt_0 = opt_0))
sequence.insert_sequence(revolve(l, cm, opt_0=opt_0, **parameters))
return sequence
if __name__ == '__main__':
start_time = time.time()
seq = Disk_Revolve(parameters.l, parameters.cm)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Disk: ", seq.disk)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms"%(1000*(end_time - start_time)))
#!/usr/bin/python
import parameters
from basic_functions import *
from .parameters import defaults
from .basic_functions import (Operation as Op, Sequence, Function, my_buddy,
argmin)
from functools import partial
def get_hopt_table(lmax, architecture):
def get_hopt_table(lmax, cvect, wvect, rvect, ub, uf, **params):
""" Compute the HOpt table for architecture and l=0...lmax
This computation uses a dynamic program"""
K = architecture.nblevels
cvect = architecture.sizes
wvect = architecture.wd
rvect = architecture.rd
assert(len(wvect) == K and len(rvect) == K and len(cvect) == K)
uf = parameters.uf
ub = parameters.ub
K = len(cvect)
assert len(wvect) == len(rvect) == len(cvect)
opt = [[[float("inf")] * (cvect[i] + 1) for _ in range(lmax + 1)] for i in range(K)]
optp = [[[float("inf")] * (cvect[i] + 1) for _ in range(lmax + 1)] for i in range(K)]
# Initialize borders of the table
......@@ -23,7 +20,7 @@ def get_hopt_table(lmax, architecture):
opt[k][0][m] = ub
optp[k][0][m] = ub
for m in range(mmax + 1):
if (m == 0)and(k == 0):
if (m == 0) and (k == 0):
continue
optp[k][1][m] = uf + 2 * ub + rvect[0]
opt[k][1][m] = wvect[0] + optp[k][1][m]
......@@ -35,11 +32,11 @@ def get_hopt_table(lmax, architecture):
for m in range(2, mmax + 1):
for l in range(2, lmax + 1):
optp[0][l][m] = min([j * uf + opt[0][l - j][m - 1] + rvect[0] + optp[0][j - 1][m] for j in range(1, l)] + [optp[0][l][1]])
opt[0][l][m] = wvect[0] + optp[0][l][m]
opt[0][l][m] = wvect[0] + optp[0][l][m]
# Fill K > 0
for k in range(1, K):
mmax = cvect[k]
for l in range(2,lmax+1):
for l in range(2, lmax+1):
opt[k][l][0] = opt[k-1][l][cvect[k-1]]
for m in range(1, mmax + 1):
for l in range(1, lmax + 1):
......@@ -48,17 +45,17 @@ def get_hopt_table(lmax, architecture):
return (optp, opt)
def HRevolve_aux(l, K, cmem, hoptp = None, hopt = None):
""" l : number of forward step to execute in the AC graph
def HRevolve_aux(l, K, cmem, cvect, wvect, rvect, hoptp=None, hopt=None, **params):
r""" l : number of forward step to execute in the AC graph
K: the level of memory
cmem: number of available slots in the K-th level of memory
Return the optimal sequence of makespan \overline{HOpt}(l, architecture)"""
cvect = parameters.architecture.sizes
wvect = parameters.architecture.wd
rvect = parameters.architecture.rd
if (hoptp == None) or (hopt == None):
(hoptp, hopt) = get_hopt_table(l, parameters.architecture)
sequence = Sequence(Function("HRevolve_aux", l, [K,cmem]))
uf = params["uf"]
if (hoptp is None) or (hopt is None):
(hoptp, hopt) = get_hopt_table(l, cvect, wvect, rvect, **params)
sequence = Sequence(Function("HRevolve_aux", l, [K, cmem]),
levels=len(cvect), concat=params["concat"])
Operation = partial(Op, params=params)
if cmem == 0:
raise KeyError("HRevolve_aux should not be call with cmem = 0. Contact developers.")
if l == 0:
......@@ -70,11 +67,11 @@ def HRevolve_aux(l, K, cmem, hoptp = None, hopt = None):
sequence.insert(Operation("Forward", 0))
sequence.insert(Operation("Backward", my_buddy(0, l - 1)))
if wvect[0] + rvect[0] < rvect[K]:
sequence.insert(Operation("Read", [0,0]))
sequence.insert(Operation("Read", [0, 0]))
else:
sequence.insert(Operation("Read", [K,0]))
sequence.insert(Operation("Read", [K, 0]))
sequence.insert(Operation("Backward", my_buddy(-1, l - 1)))
sequence.insert(Operation("Discard", [0,0]))
sequence.insert(Operation("Discard", [0, 0]))
return sequence
if K == 0 and cmem == 1:
for index in range(l - 1, -1, -1):
......@@ -87,72 +84,104 @@ def HRevolve_aux(l, K, cmem, hoptp = None, hopt = None):
sequence.insert(Operation("Discard", [0, 0]))
return sequence
if K == 0:
list_mem = [j * parameters.uf + hopt[0][l - j][cmem - 1] + rvect[0] + hoptp[0][j - 1][cmem] for j in range(1, l)]
list_mem = [j * uf + hopt[0][l - j][cmem - 1] + rvect[0] + hoptp[0][j - 1][cmem] for j in range(1, l)]
if min(list_mem) < hoptp[0][l][1]:
jmin = argmin(list_mem)
sequence.insert(Operation("Forwards", [0, jmin - 1]))
sequence.insert_sequence(HRevolve(l - jmin, 0, cmem - 1, hoptp=hoptp, hopt=hopt).shift(jmin))
sequence.insert_sequence(
hrevolve_recurse(l - jmin, 0, cmem - 1, cvect, wvect, rvect, hoptp=hoptp, hopt=hopt).shift(jmin)
)
sequence.insert(Operation("Read", [0, 0]))
sequence.insert_sequence(HRevolve_aux(jmin - 1, 0, cmem, hoptp=hoptp, hopt=hopt))
sequence.insert_sequence(
HRevolve_aux(jmin - 1, 0, cmem, cvect, wvect, rvect, uf,
hoptp=hoptp, hopt=hopt, **params)
)
return sequence
else:
sequence.insert_sequence(HRevolve_aux(l, 0, 1, hoptp=hoptp, hopt=hopt))
sequence.insert_sequence(
HRevolve_aux(l, 0, 1, cvect, wvect, rvect, uf,
hoptp=hoptp, hopt=hopt, **params)
)
return sequence
list_mem = [j * parameters.uf + hopt[K][l - j][cmem - 1] + rvect[K] + hoptp[K][j - 1][cmem] for j in range(1, l)]
list_mem = [j * uf + hopt[K][l - j][cmem - 1] + rvect[K] + hoptp[K][j - 1][cmem] for j in range(1, l)]
if min(list_mem) < hopt[K-1][l][cvect[K-1]]:
jmin = argmin(list_mem)
sequence.insert(Operation("Forwards", [0, jmin - 1]))
sequence.insert_sequence(HRevolve(l - jmin, K, cmem - 1, hoptp=hoptp, hopt=hopt).shift(jmin))
sequence.insert_sequence(
hrevolve_recurse(l - jmin, K, cmem - 1, cvect, wvect, rvect,
hoptp=hoptp, hopt=hopt, **params).shift(jmin)
)
sequence.insert(Operation("Read", [K, 0]))
sequence.insert_sequence(HRevolve_aux(jmin - 1, K, cmem, hoptp=hoptp, hopt=hopt))
sequence.insert_sequence(
HRevolve_aux(jmin - 1, K, cmem, cvect, wvect, rvect,
hoptp=hoptp, hopt=hopt, **params)
)
return sequence
else:
sequence.insert_sequence(HRevolve(l, K-1, cvect[K-1], hoptp = hoptp, hopt = hopt))
sequence.insert_sequence(
hrevolve_recurse(l, K-1, cvect[K-1], cvect, wvect, rvect,
hoptp=hoptp, hopt=hopt, **params)
)
return sequence
def HRevolve(l, K, cmem, hoptp = None, hopt = None):
def hrevolve(l, cvect, wvect, rvect, **params):
""" l : number of forward step to execute in the AC graph
cvect: the number of slots in each level of memory
wvect: the cost of writing to each level of memory
rvect: the cost of reading from each level of memory
Return the optimal sequence of makespan HOpt(l, architecture)"""
params["wd"] = wvect
params["rd"] = rvect
return hrevolve_recurse(l, len(cvect)-1, cvect[-1], cvect, wvect, rvect,
hoptp=None, hopt=None, **params)
def hrevolve_recurse(l, K, cmem, cvect, wvect, rvect, hoptp=None, hopt=None, **params):
""" l : number of forward step to execute in the AC graph
K: the level of memory
cmem: number of available slots in the K-th level of memory
cvect: the number of slots in each level of memory
wvect: the cost of writing to each level of memory
rvect: the cost of reading from each level of memory
Return the optimal sequence of makespan HOpt(l, architecture)"""
cvect = parameters.architecture.sizes
wvect = parameters.architecture.wd
if (hoptp == None) or (hopt == None):
(hoptp, hopt) = get_hopt_table(l, parameters.architecture)
sequence = Sequence(Function("HRevolve", l, [K,cmem]))
parameters = dict(defaults)
parameters.update(params)
if (hoptp is None) or (hopt is None):
(hoptp, hopt) = get_hopt_table(l, cvect, wvect, rvect, **parameters)
sequence = Sequence(Function("HRevolve", l, [K, cmem]),
levels=len(cvect), concat=parameters["concat"])
Operation = partial(Op, params=parameters)
if l == 0:
sequence.insert(Operation("Backward", my_buddy(-1, l-1)))
return sequence
if K == 0 and cmem == 0:
raise KeyError("It's impossible to execute an AC graph of size > 0 with no memory.")
raise KeyError("It's impossible to execute an AC graph of size > 0 with no memory.")
if l == 1:
sequence.insert(Operation("Write", [0, 0]))
sequence.insert(Operation("Forward", 0))
sequence.insert(Operation("Backward", my_buddy(0, l - 1)))
sequence.insert(Operation("Read", [0,0]))
sequence.insert(Operation("Read", [0, 0]))
sequence.insert(Operation("Backward", my_buddy(-1, l - 1)))
sequence.insert(Operation("Discard", [0,0]))
sequence.insert(Operation("Discard", [0, 0]))
return sequence
if K ==0:
if K == 0:
sequence.insert(Operation("Write", [0, 0]))
sequence.insert_sequence(HRevolve_aux(l, 0, cmem, hoptp=hoptp, hopt=hopt))
sequence.insert_sequence(
HRevolve_aux(l, 0, cmem, cvect, wvect, rvect,
hoptp=hoptp, hopt=hopt, **parameters)
)
return sequence
if wvect[K] + hoptp[K][l][cmem] < hopt[K-1][l][cvect[K-1]]:
sequence.insert(Operation("Write", [K, 0]))
sequence.insert_sequence(HRevolve_aux(l, K, cmem, hoptp=hoptp, hopt=hopt))
sequence.insert_sequence(
HRevolve_aux(l, K, cmem, cvect, wvect, rvect,
hoptp=hoptp, hopt=hopt, **parameters)
)
return sequence
else:
sequence.insert_sequence(HRevolve(l, K-1, cvect[K-1], hoptp=hoptp, hopt=hopt))
sequence.insert_sequence(
hrevolve_recurse(l, K-1, cvect[K-1], cvect, wvect, rvect,
hoptp=hoptp, hopt=hopt, **parameters)
)
return sequence
if __name__ == '__main__':
start_time = time.time()
seq = HRevolve(parameters.l, parameters.architecture.nblevels-1, parameters.architecture.sizes[-1])
end_time = time.time()
print("Sequence: ", seq)
for i in range(parameters.architecture.nblevels):
print("Storage level ", i, ": ", seq.storage[i])
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms"%(1000*(end_time - start_time)))
\ No newline at end of file
#!/usr/bin/python
import argparse
import sys
import os
# Default parameter values. Not all parameters are used by all algorithms.
defaults = {
"uf": 1, # Cost of a forward step.
"ub": 1, # Cost of a backward step.
"up": 1, # Cost of the loss function.
"wd": 5, # Cost of writing to disk.
"rd": 5, # Cost of reading from disk.
"mx": None, # Size of the period (defaults to the optimal).
"one_read_disk": False, # Disk checkpoints are only read once.
"fast": False, # Use the clode formula for mx.
"concat": 0, # Level of sequence concatenation.
"print_table": None, # File to which to print the results table.
}
class Architecture:
......@@ -17,9 +29,9 @@ class Architecture:
self.rd = []
self.nblevels = -1
self.read_file()
if (len(self.sizes) != self.nblevels)or(len(self.wd) != self.nblevels)or(len(self.rd) != self.nblevels):
if (len(self.sizes) != self.nblevels) or (len(self.wd) != self.nblevels) or (len(self.rd) != self.nblevels):
raise ImportError("The level in the architecture file does not correspond to the number of line.")
if (sorted(self.wd) != self.wd)or(sorted(self.rd) != self.rd):
if (sorted(self.wd) != self.wd) or (sorted(self.rd) != self.rd):
print("WARNING!!! This code is optimal only if the costs of writing and reading of the architecture are in the increasing order for the levels.")
def __del__(self):
......@@ -46,274 +58,92 @@ class Architecture:
if self.nblevels < 0:
try:
self.nblevels = int(line_list[0])
except:
except: # NOQA E722
raise SyntaxError("The first line of the architecture file should be an integer.")
continue
if len(line_list) != 3:
raise SyntaxError("Every line of the architecture file should be a triplet (c: integer, wd: float, rd: float).")
try:
self.sizes.append(int(line_list[0]))
except:
except: # NOQA E722
raise SyntaxError("Every line of the architecture file should be a triplet (c: integer, wd: float, rd: float).")
try:
self.wd.append(int(line_list[1]))
except:
except: # NOQA E722
try:
self.wd.append(float(line_list[1]))
except:
except: # NOQA E722
raise SyntaxError("Every line of the architecture file should be a triplet (c: integer, wd: float, rd: float).")
try:
self.rd.append(int(line_list[2]))
except:
except: # NOQA E722
try:
self.rd.append(float(line_list[2]))
except:
except: # NOQA E722
raise SyntaxError("Every line of the architecture file should be a triplet (c: integer, wd: float, rd: float).")
self.file.seek(0)
class Chain:
def __init__(self, file_name):
self.file_name = file_name
try:
self.file = open(file_name, "r")
except FileNotFoundError:
raise FileNotFoundError("The file "+file_name+" describing the heterogeneous chain is not found.")
self.fweigth = []
self.bweigth = []
self.cweigth = []
self.cbweigth = []
self.length = -1
print("file: ", file_name)
self.read_file()
if (len(self.fweigth) != self.length)or(len(self.bweigth) != self.length+1)or(len(self.cweigth) != self.length+1)or(len(self.cbweigth) != self.length+1):
raise ImportError("The length in the chain file does not correspond to the number of element per lines.")
self.cbweigth.append(1)
def __del__(self):
try:
self.file.close()
except FileNotFoundError:
pass
except AttributeError:
pass
def __repr__(self):
l = []
for i in range(self.length):
l.append((self.fweigth[i], self.bweigth[i], self.cweigth[i]))
return l.__repr__()
def read_file(self):
real_line = 0
for line in self.file:
if line[0] == "#":
continue
line_list = [x for x in line.split()]
if len(line_list) > 1 and real_line == 0:
raise SyntaxError("The first line of the architecture file should be a single integer.")
if real_line == 0:
try:
self.length = int(line_list[0])
except:
raise SyntaxError("The first line of the architecture file should be an integer.")
if real_line == 1:
for x in line_list:
try:
if int(x) < 0:
raise IndexError
self.fweigth.append(int(x))
except:
raise SyntaxError("The forward values should be positive integers")
if real_line == 2:
for x in line_list:
try:
if int(x) < 0:
raise IndexError
self.bweigth.append(int(x))
except:
raise SyntaxError("The backward values should be positive integers")
if real_line == 3:
for x in line_list:
try:
if int(x) < 0:
raise IndexError
self.cweigth.append(int(x))
except:
raise SyntaxError("The memory sizes should be positive integers")
if real_line == 4:
for x in line_list:
try:
if int(x) < 0:
raise IndexError
self.cbweigth.append(int(x))
except:
raise SyntaxError("The memory sizes should be positive integers")
real_line += 1
self.file.seek(0)
def parse_arguments_Revolve():
parser = argparse.ArgumentParser(description='Compute Revolve with l and cm')
parser.add_argument("l", help = "Size of the Adjoint Computation problem (number of foward steps)", type = int)
parser.add_argument("cm", help = "Memory Size", type = int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "ub")
parser.add_argument("--concat", help="Level of concatenation between 0 and 2? (default: 0)", default = 0, type = int, metavar = "int", dest = "concat")
parser.add_argument("--print_table", help = "Name of the file to print the table of results", default = None, type = str, metavar = "str", dest = "print_table")
global l
global cm
global uf
global ub
global concat
global print_table
parameters = parser.parse_args()
l = parameters.l
cm = parameters.cm
uf = parameters.uf
ub = parameters.ub
concat = parameters.concat
print_table = parameters.print_table
parser.add_argument("l", help="Size of the Adjoint Computation problem (number of foward steps)", type=int)
parser.add_argument("cm", help="Memory Size", type=int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default=defaults["uf"], type=float, metavar="float", dest="uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default=defaults["ub"], type=float, metavar="float", dest="ub")
parser.add_argument("--concat", help="Level of concatenation between 0 and 2? (default: 0)", default=defaults["concat"], type=int, metavar="int", dest="concat")
parser.add_argument("--print_table", help="Name of the file to print the table of results", default=defaults["print_table"], type=str, metavar="str", dest="print_table")
return vars(parser.parse_args())
def parse_arguments_1D_Revolve():
parser = argparse.ArgumentParser(description='Compute 1D-Revolve with l and cm')
parser.add_argument("l", help = "Size of the Adjoint Computation problem (number of foward steps)", type = int)
parser.add_argument("cm", help = "Memory Size", type = int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "ub")
parser.add_argument("--wd", help="Cost of writting in the disk (default: 5)", default = 5, type = float, metavar = "float", dest = "wd")
parser.add_argument("--rd", help="Cost of reading in the disk (default: 5)", default = 5, type = float, metavar = "float", dest = "rd")
parser.add_argument("l", help="Size of the Adjoint Computation problem (number of foward steps)", type=int)
parser.add_argument("cm", help="Memory Size", type=int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default=defaults["uf"], type=float, metavar="float", dest="uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default=defaults["ub"], type=float, metavar="float", dest="ub")
parser.add_argument("--wd", help="Cost of writting in the disk (default: 5)", default=defaults["wd"], type=float, metavar="float", dest="wd")
parser.add_argument("--rd", help="Cost of reading in the disk (default: 5)", default=defaults["rd"], type=float, metavar="float", dest="rd")
parser.add_argument("--one_read_disk", help="Option to force one read maximum for disk checkpoints", action='store_true')
parser.add_argument("--concat", help="Level of concatenation between 0 and 3? (default: 0)", default = 0, type = int, metavar = "int", dest = "concat")
parser.add_argument("--print_table", help = "Name of the file to print the table of results", default = None, type = str, metavar = "str", dest = "print_table")
global l
global cm
global rd
global wd
global uf
global ub
global one_read_disk
global concat
global print_table
parameters = parser.parse_args()
l = parameters.l
cm = parameters.cm
rd = parameters.rd
wd = parameters.wd
uf = parameters.uf
ub = parameters.ub
one_read_disk = parameters.one_read_disk
concat = parameters.concat
print_table = parameters.print_table
parser.add_argument("--concat", help="Level of concatenation between 0 and 3? (default: 0)", default=defaults["concat"], type=int, metavar="int", dest="concat")
parser.add_argument("--print_table", help="Name of the file to print the table of results", default=defaults["print_table"], type=str, metavar="str", dest="print_table")
return vars(parser.parse_args())
def parse_arguments_Disk_Revolve():
parser = argparse.ArgumentParser(description='Compute Disk-Revolve with l and cm')
parser.add_argument("l", help = "Size of the Adjoint Computation problem (number of foward steps)", type = int)
parser.add_argument("cm", help = "Memory Size", type = int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "ub")
parser.add_argument("--wd", help="Cost of writting in the disk (default: 5)", default = 5, type = float, metavar = "float", dest = "wd")
parser.add_argument("--rd", help="Cost of reading in the disk (default: 5)", default = 5, type = float, metavar = "float", dest = "rd")
parser.add_argument("l", help="Size of the Adjoint Computation problem (number of foward steps)", type=int)
parser.add_argument("cm", help="Memory Size", type=int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default=defaults["uf"], type=float, metavar="float", dest="uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default=defaults["ub"], type=float, metavar="float", dest="ub")
parser.add_argument("--wd", help="Cost of writting in the disk (default: 5)", default=defaults["wd"], type=float, metavar="float", dest="wd")
parser.add_argument("--rd", help="Cost of reading in the disk (default: 5)", default=defaults["rd"], type=float, metavar="float", dest="rd")
parser.add_argument("--one_read_disk", help="Option to force one read maximum for disk checkpoints", action='store_true')
parser.add_argument("--concat", help="Level of concatenation between 0 and 3? (default: 0)", default = 0, type = int, metavar = "int", dest = "concat")
parser.add_argument("--print_table", help = "Name of the file to print the table of results", default = None, type = str, metavar = "str", dest = "print_table")
global l
global cm
global rd
global wd
global uf
global ub
global one_read_disk
global concat
global print_table
parameters = parser.parse_args()
l = parameters.l
cm = parameters.cm
rd = parameters.rd
wd = parameters.wd
uf = parameters.uf
ub = parameters.ub
one_read_disk = parameters.one_read_disk
concat = parameters.concat
print_table = parameters.print_table
parser.add_argument("--concat", help="Level of concatenation between 0 and 3? (default: 0)", default=defaults["concat"], type=int, metavar="int", dest="concat")
parser.add_argument("--print_table", help="Name of the file to print the table of results", default=defaults["one_read_disk"], type=str, metavar="str", dest="print_table")
return vars(parser.parse_args())
def parse_arguments_Periodic_Disk_Revolve():
parser = argparse.ArgumentParser(description='Compute Disk-Revolve with l and cm')
parser.add_argument("l", help = "Size of the Adjoint Computation problem (number of foward steps)", type = int)
parser.add_argument("cm", help = "Memory Size", type = int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "ub")
parser.add_argument("--wd", help="Cost of writting in the disk (default: 5)", default = 5, type = float, metavar = "float", dest = "wd")
parser.add_argument("--rd", help="Cost of reading in the disk (default: 5)", default = 5, type = float, metavar = "float", dest = "rd")
parser.add_argument("--mx", help="Size of the period (default: the optimal one)", default = None, type = int, metavar = "int", dest = "mx")
parser.add_argument("l", help="Size of the Adjoint Computation problem (number of foward steps)", type=int)
parser.add_argument("cm", help="Memory Size", type=int)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default=defaults["uf"], type=float, metavar="float", dest="uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default=defaults["ub"], type=float, metavar="float", dest="ub")
parser.add_argument("--wd", help="Cost of writting in the disk (default: 5)", default=defaults["wd"], type=float, metavar="float", dest="wd")
parser.add_argument("--rd", help="Cost of reading in the disk (default: 5)", default=defaults["rd"], type=float, metavar="float", dest="rd")
parser.add_argument("--mx", help="Size of the period (default: the optimal one)", default=defaults["mx"], type=int, metavar="int", dest="mx")
parser.add_argument("--one_read_disk", help="Option to force one read maximum for disk checkpoints", action='store_true')
parser.add_argument("--fast", help="Option to use the clode formula for mx (faster)", action='store_true')
parser.add_argument("--concat", help="Level of concatenation between 0 and 3? (default: 0)", default = 0, type = int, metavar = "int", dest = "concat")
parser.add_argument("--print_table", help = "Name of the file to print the table of results", default = None, type = str, metavar = "str", dest = "print_table")
global l
global cm
global rd
global wd
global uf
global ub
global mx
global one_read_disk
global fast
global concat
global print_table
parameters = parser.parse_args()
l = parameters.l
cm = parameters.cm
rd = parameters.rd
wd = parameters.wd
uf = parameters.uf
ub = parameters.ub
mx = parameters.mx
one_read_disk = parameters.one_read_disk
fast = parameters.fast
concat = parameters.concat
print_table = parameters.print_table
parser.add_argument("--concat", help="Level of concatenation between 0 and 3? (default: 0)", default=defaults["concat"], type=int, metavar="int", dest="concat")
parser.add_argument("--print_table", help="Name of the file to print the table of results", default=defaults["print_table"], type=str, metavar="str", dest="print_table")
return vars(parser.parse_args())
def parse_arguments_HRevolve():
parser = argparse.ArgumentParser(description='Compute HRevolve with l and the architecture described in file_name')
parser.add_argument("l", help = "Size of the Adjoint Computation problem (number of foward steps)", type = int)
parser.add_argument("file_name", help = "File describing the architecture", type = str)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default = 1, type = float, metavar = "float", dest = "ub")
parser.add_argument("--concat", help="Level of concatenation between 0 and K? (default: 0)", default = 0, type = int, metavar = "int", dest = "concat")
global l
global file_name
global architecture
global uf
global ub
global concat
parameters = parser.parse_args()
l = parameters.l
file_name = parameters.file_name
architecture = Architecture(parameters.file_name)
uf = parameters.uf
ub = parameters.ub
concat = parameters.concat
if os.path.basename(sys.argv[0]) == "Revolve.py":
parse_arguments_Revolve()
if os.path.basename(sys.argv[0]) == "1D-Revolve.py":
parse_arguments_1D_Revolve()
if os.path.basename(sys.argv[0]) == "Disk-Revolve.py":
parse_arguments_Disk_Revolve()
if os.path.basename(sys.argv[0]) == "Periodic-Disk-Revolve.py":
parse_arguments_Periodic_Disk_Revolve()
if os.path.basename(sys.argv[0]) == "HRevolve.py":
parse_arguments_HRevolve()
parser.add_argument("l", help="Size of the Adjoint Computation problem (number of foward steps)", type=int)
parser.add_argument("file_name", help="File describing the architecture", type=str)
parser.add_argument("--uf", help="Cost of the forward steps (default: 1)", default=defaults["uf"], type=float, metavar="float", dest="uf")
parser.add_argument("--ub", help="Cost of the backward steps (default: 1)", default=defaults["ub"], type=float, metavar="float", dest="ub")
parser.add_argument("--concat", help="Level of concatenation between 0 and K? (default: 0)", default=defaults["concat"], type=int, metavar="int", dest="concat")
return vars(parser.parse_args())
#!/usr/bin/python
from .parameters import defaults
from .basic_functions import (Operation as Op, Sequence, Function, beta)
from .revolve import revolve, get_opt_0_table
import math
from functools import partial
import parameters
from basic_functions import *
from Revolve import *
Module1D = __import__('1D-Revolve')
OneD_Revolve = Module1D.OneD_Revolve
get_opt_1d_table = Module1D.get_opt_1d_table
from .revolve_1d import get_opt_1d_table, revolve_1d
def compute_mmax(cmem):
def compute_mmax(cm, wd, rd, uf, **params):
""" Compute m_max, the bound on the optimal
period mX, as defined in the paper """
td1 = 0
while beta(cmem, td1) <= (parameters.wd + parameters.rd) / parameters.uf:
while beta(cm, td1) <= (wd + rd) / uf:
td1 += 1
td2 = 0
while beta(cmem, td2) <= parameters.wd / parameters.uf:
while beta(cm, td2) <= wd / uf:
td2 += 1
return int(max(beta(cmem, td1+1), 2*beta(cmem, td2) + 1))
return int(max(beta(cm, td1+1), 2*beta(cm, td2) + 1))
def RelCostX(m, opt_1d_m_moins_1):
def RelCostX(m, opt_1d_m_moins_1, wd, rd, **params):
""" The RelCost_X function, as defined in the paper """
return 1.0*(parameters.wd + parameters.rd + opt_1d_m_moins_1) / m
return 1.0*(wd + rd + opt_1d_m_moins_1) / m
def compute_mx(cmem, opt_0 = None, opt_1d = None, mmax = None):
def compute_mx(cm, opt_0=None, opt_1d=None, mmax=None, **params):
""" Compute the optimal period mX, as defined in the paper """
if mmax == None:
mmax = compute_mmax(cmem)
if opt_0 == None or len(opt_0) < mmax:
opt_0 = get_opt_0_table(mmax, cmem)
if opt_1d == None or len(opt_1d) < mmax:
opt_1d = get_opt_1d_table(mmax, cmem, opt_0=opt_0)
if mmax is None:
mmax = compute_mmax(cm, **params)
if opt_0 is None or len(opt_0) < mmax:
opt_0 = get_opt_0_table(mmax, cm, **params)
if opt_1d is None or len(opt_1d) < mmax:
opt_1d = get_opt_1d_table(mmax, cm, opt_0=opt_0, **params)
mx = 1
objbest = RelCostX(1, opt_1d[0])
objbest = RelCostX(1, opt_1d[0], **params)
for mxi in range(2, mmax+1):
obj = RelCostX(mxi, opt_1d[mxi-1])
#print("mxi", mxi, "opt", opt_1d[mxi-1], "wd", parameters.wd, "rd", parameters.rd, "obj", obj)
obj = RelCostX(mxi, opt_1d[mxi-1], **params)
# print("mxi", mxi, "opt", opt_1d[mxi-1], "wd", wd, "rd", rd, "obj", obj)
if obj <= objbest:
objbest = obj
mx = mxi
return mx
def mx_close_formula(cmem, opt_0 = None, opt_1d = None):
def mx_close_formula(cm, rd, wd, opt_0=None, opt_1d=None, **params):
""" Compute mX using the close formula in the paper
It's not proven yet, but it's been verified and it's faster """
def f(x, y, c):
return int(beta(c + 1, x + y - 1) - sum([beta(c, k) for k in range(0,y)]))
return int(beta(c + 1, x + y - 1) - sum([beta(c, k) for k in range(0, y)]))
x = 0
while (parameters.rd >= beta(cmem+1, x)):
while (rd >= beta(cm+1, x)):
x += 1
y = 0
while (parameters.wd > sum([f(j,x,cmem) for j in range(1,y+1)])):
while (wd > sum([f(j, x, cm) for j in range(1, y+1)])):
y += 1
mx = f(y, x, cmem)
mx = f(y, x, cm)
x += 1
y = 0
while (parameters.wd > sum([f(j, x, cmem) for j in range(1, y+1)])):
while (wd > sum([f(j, x, cm) for j in range(1, y+1)])):
y += 1
mxalt = f(y, x, cmem)
mxalt = f(y, x, cm)
mmax = max(mx, mxalt)
if opt_0 == None or len(opt_0) < mmax:
opt_0 = get_opt_0_table(mmax, cmem)
if opt_1d == None or len(opt_1d) < mmax:
opt_1d = get_opt_1d_table(mmax, cmem, opt_0=opt_0)
if opt_0 is None or len(opt_0) < mmax:
opt_0 = get_opt_0_table(mmax, cm, **params)
if opt_1d is None or len(opt_1d) < mmax:
opt_1d = get_opt_1d_table(mmax, cm, opt_0=opt_0, **params)
if (RelCostX(mx, opt_1d[mx-1]) < RelCostX(mxalt, opt_1d[mxalt-1])):
return int(mx)
else:
return int(mxalt)
def mxrr_close_formula(cmem):
def mxrr_close_formula(cm, uf, rd, wd, **params):
""" Compute mXrr using the close formula in the paper"""
t = 0
while beta(cmem +1, t) <= (parameters.wd + parameters.rd) / parameters.uf:
while beta(cm+1, t) <= (wd + rd) / uf:
t += 1
#t -= 1
return int(beta(cmem, t))
# t -= 1
return int(beta(cm, t))
def combin(k,n):
def combin(k, n):
return int(math.factorial(n)/(math.factorial(k)*math.factorial(n-k)))
def Periodic_Disk_Revolve(l, cmem, opt_0 = None, opt_1d = None, mmax = None, mx = None):
def periodic_disk_revolve(l, cm, opt_0=None, opt_1d=None, mmax=None, **params):
""" l : number of forward step to execute in the AC graph
cmem : number of available memory slots
cm : number of available memory slots
Return the periodic sequence with optimal period"""
if mmax == None:
if parameters.one_read_disk:
mmax = mxrr_close_formula(cmem)
if mx == None:
parameters = dict(defaults)
parameters.update(params)
mx = parameters["mx"]
one_read_disk = parameters["one_read_disk"]
fast = parameters["fast"]
if mmax is None:
if one_read_disk:
mmax = mxrr_close_formula(cm, **parameters)
if mx is None:
mx = mmax
else:
mmax = compute_mmax(cmem)
if mx != None:
mmax = compute_mmax(cm, **parameters)
if mx is not None:
mmax = max(mmax, mx) + 1
if opt_0 == None:
opt_0 = get_opt_0_table(mmax, cmem)
if opt_1d == None and not parameters.one_read_disk:
opt_1d = get_opt_1d_table(mmax, cmem, opt_0=opt_0)
sequence = Sequence(Function("Periodic-Disk-Revolve", l, cmem))
if mx == None:
if parameters.one_read_disk:
mx = mxrr_close_formula(cmem)
elif parameters.fast == True:
mx = mx_close_formula(cmem, opt_0=opt_0, opt_1d=opt_1d)
if opt_0 is None:
opt_0 = get_opt_0_table(mmax, cm, **parameters)
if opt_1d is None and not one_read_disk:
opt_1d = get_opt_1d_table(mmax, cm, opt_0=opt_0, **parameters)
sequence = Sequence(Function("Periodic-Disk-Revolve", l, cm), concat=parameters["concat"])
Operation = partial(Op, params=parameters)
if mx is None:
if one_read_disk:
mx = mxrr_close_formula(cm, **parameters)
elif fast:
mx = mx_close_formula(cm, opt_0=opt_0, opt_1d=opt_1d, **parameters)
else:
mx = compute_mx(cmem, opt_0=opt_0, opt_1d=opt_1d)
mx = compute_mx(cm, opt_0=opt_0, opt_1d=opt_1d, **parameters)
print("We use periods of size ", mx)
current_task = 0
while l - current_task > mx:
sequence.insert(Operation("Write_disk", current_task))
sequence.insert(Operation("Forwards", [current_task, current_task + mx - 1]))
current_task += mx
if parameters.one_read_disk or opt_1d[l - current_task] == opt_0[cmem][l - current_task]:
sequence.insert_sequence(Revolve(l - current_task, cmem, opt_0=opt_0).shift(current_task))
if one_read_disk or opt_1d[l - current_task] == opt_0[cm][l - current_task]:
sequence.insert_sequence(
revolve(l - current_task, cm, opt_0=opt_0, **parameters).shift(current_task)
)
else:
sequence.insert(Operation("Write_disk", current_task))
sequence.insert_sequence(OneD_Revolve(l - current_task, cmem, opt_0=opt_0, opt_1d=opt_1d).shift(current_task))
sequence.insert_sequence(
revolve_1d(l - current_task, cm, opt_0=opt_0, opt_1d=opt_1d, **parameters).shift(current_task)
)
while current_task > 0:
current_task -= mx
sequence.insert(Operation("Read_disk", current_task))
if parameters.one_read_disk:
sequence.insert_sequence(Revolve(mx - 1, cmem, opt_0=opt_0).shift(current_task))
if one_read_disk:
sequence.insert_sequence(
revolve(mx - 1, cm, opt_0=opt_0, **parameters).shift(current_task)
)
else:
sequence.insert_sequence(OneD_Revolve(mx-1, cmem, opt_0=opt_0, opt_1d=opt_1d).shift(current_task))
sequence.insert_sequence(
revolve_1d(mx-1, cm, opt_0=opt_0, opt_1d=opt_1d, **parameters).shift(current_task)
)
return sequence
if __name__ == '__main__':
start_time = time.time()
seq = Periodic_Disk_Revolve(parameters.l, parameters.cm, mx=parameters.mx)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Disk: ", seq.disk)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms"%(1000*(end_time - start_time)))
#!/usr/bin/python
from .parameters import defaults
from .basic_functions import (Operation as Op, Sequence, Function, Table, beta,
my_buddy, argmin)
from functools import partial
import parameters
from basic_functions import *
def get_t(l, cmem):
def get_t(l, cm):
t = 0
while (beta(cmem, t) <= l):
while (beta(cm, t) <= l):
t += 1
return t-1
def opt_0_closed_formula(l, cmem):
def opt_0_closed_formula(l, cm, uf, ub, **params):
""" Fast computation of Opt_0 based on the closed formula """
if l == 0:
return parameters.ub
t = get_t(l, cmem)
return ((l+1) * (t+1) - beta(cmem+1, t)) * parameters.uf + (l+1) * parameters.ub
return ub
t = get_t(l, cm)
return ((l+1) * (t+1) - beta(cm+1, t)) * uf + (l+1) * ub
def get_opt_0_table(lmax, mmax):
def get_opt_0_table(lmax, mmax, uf, ub, print_table, **params):
""" Return the Opt_0 tables
for every Opt_0[m][l] with l = 0...lmax and m = 0...mmax
The computation uses a dynamic program"""
uf = parameters.uf
ub = parameters.ub
# Build table
opt = [Table() for _ in range(mmax + 1)]
if __name__ == '__main__' and parameters.print_table:
opt[mmax].set_to_print(parameters.print_table)
if __name__ == '__main__' and print_table:
opt[mmax].set_to_print(print_table)
# Initialize borders of the tables
for m in range(mmax + 1):
opt[m].append(ub)
......@@ -43,18 +43,21 @@ def get_opt_0_table(lmax, mmax):
return opt
def Revolve(l, cmem, opt_0 = None):
def revolve(l, cm, opt_0=None, **params):
""" l : number of forward step to execute in the AC graph
cmem : number of available memory slots
Return the optimal sequence of makespan Opt_0(l, cmem)"""
if opt_0 == None:
opt_0 = get_opt_0_table(l, cmem)
sequence = Sequence(Function("Revolve", l, cmem))
cm : number of available memory slots
Return the optimal sequence of makespan Opt_0(l, cm)"""
parameters = dict(defaults)
parameters.update(params)
if opt_0 is None:
opt_0 = get_opt_0_table(l, cm, **params)
sequence = Sequence(Function("Revolve", l, cm), concat=parameters["concat"])
Operation = partial(Op, params=parameters)
if l == 0:
sequence.insert(Operation("Backward", my_buddy(-1, l-1)))
sequence.insert(Operation("Discard_memory", 0))
return sequence
elif cmem == 0:
elif cm == 0:
raise ValueError("It's impossible to execute an AC graph without memory")
elif l == 1:
sequence.insert(Operation("Write_memory", 0))
......@@ -64,33 +67,26 @@ def Revolve(l, cmem, opt_0 = None):
sequence.insert(Operation("Backward", my_buddy(-1, l-1)))
sequence.insert(Operation("Discard_memory", 0))
return sequence
elif cmem == 1:
elif cm == 1:
sequence.insert(Operation("Write_memory", 0))
for index in range(l - 1, -1, -1):
if index != l - 1:
sequence.insert(Operation("Read_memory", 0))
sequence.insert(Operation("Forwards", [0,index]))
sequence.insert(Operation("Forwards", [0, index]))
sequence.insert(Operation("Backward", my_buddy(index, l-1)))
sequence.insert(Operation("Read_memory", 0))
sequence.insert(Operation("Backward", my_buddy(-1, l-1)))
sequence.insert(Operation("Discard_memory", 0))
return sequence
list_mem = [j*parameters.uf + opt_0[cmem-1][l-j] + opt_0[cmem][j-1] for j in range(1,l)]
list_mem = [j*parameters["uf"] + opt_0[cm-1][l-j] + opt_0[cm][j-1] for j in range(1, l)]
jmin = argmin(list_mem)
sequence.insert(Operation("Write_memory", 0))
sequence.insert(Operation("Forwards", [0,jmin-1]))
sequence.insert_sequence(Revolve(l - jmin, cmem - 1, opt_0 = opt_0).shift(jmin))
sequence.insert(Operation("Forwards", [0, jmin-1]))
sequence.insert_sequence(
revolve(l - jmin, cm - 1, opt_0=opt_0, **parameters).shift(jmin)
)
sequence.insert(Operation("Read_memory", 0))
sequence.insert_sequence(Revolve(jmin-1, cmem, opt_0 = opt_0).remove_useless_wm())
sequence.insert_sequence(
revolve(jmin-1, cm, opt_0=opt_0, **parameters).remove_useless_wm()
)
return sequence
if __name__ == '__main__':
start_time = time.time()
seq = Revolve(parameters.l, parameters.cm)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms"%(1000*(end_time - start_time)))
#print("Check closed formula: ", opt_0_closed_formula(parameters.l, parameters.cm))
\ No newline at end of file
#!/usr/bin/python
import parameters
from basic_functions import *
from Revolve import *
from .parameters import defaults
from .basic_functions import (Operation as Op, Sequence, Function, Table,
my_buddy, argmin)
from .revolve import revolve, get_opt_0_table
from functools import partial
def get_opt_1d_table(lmax, cmem, opt_0=None):
""" Compute the opt_1d table for cmem and l=0...lmax
def get_opt_1d_table(lmax, cm, print_table, ub, uf, rd, one_read_disk,
opt_0=None, **params):
""" Compute the opt_1d table for cm and l=0...lmax
This computation uses a dynamic program
We consider that x_0 is already stored on the disk"""
if opt_0 == None:
opt_0 = get_opt_0_table(lmax, cmem)
if opt_0 is None:
opt_0 = get_opt_0_table(lmax, cm)
opt_1d = Table()
if __name__ == '__main__' and parameters.print_table:
opt_1d.set_to_print(parameters.print_table)
opt_1d.append(parameters.ub)
#Opt_1d[1] for cmem
if cmem == 0:
opt_1d.append(parameters.uf + 2 * parameters.ub + parameters.rd)
if __name__ == '__main__' and print_table:
opt_1d.set_to_print(print_table)
opt_1d.append(ub)
# Opt_1d[1] for cm
if cm == 0:
opt_1d.append(uf + 2 * ub + rd)
else:
opt_1d.append(parameters.uf + 2 * parameters.ub)
# Opt_1d[2...lmax] for cmem
opt_1d.append(uf + 2 * ub)
# Opt_1d[2...lmax] for cm
for l in range(2, lmax + 1):
if parameters.one_read_disk:
opt_1d.append(min(opt_0[cmem][l], min([j * parameters.uf + opt_0[cmem][l - j] + parameters.rd + opt_0[cmem][j-1] for j in range(1, l)])))
if one_read_disk:
opt_1d.append(min(opt_0[cm][l], min([j * uf + opt_0[cm][l - j] + rd + opt_0[cm][j-1] for j in range(1, l)])))
else:
opt_1d.append(min(opt_0[cmem][l], min([j * parameters.uf + opt_0[cmem][l - j] + parameters.rd + opt_1d[j-1] for j in range(1, l)])))
opt_1d.append(min(opt_0[cm][l], min([j * uf + opt_0[cm][l - j] + rd + opt_1d[j-1] for j in range(1, l)])))
return opt_1d
def OneD_Revolve(l, cmem, opt_0 = None, opt_1d = None):
def revolve_1d(l, cm, opt_0=None, opt_1d=None, **params):
""" l : number of forward step to execute in the AC graph
cmem : number of available memory slots
Return the optimal sequence of makespan Opt_1d(l, cmem)
cm : number of available memory slots
Return the optimal sequence of makespan Opt_1d(l, cm)
We consider that x_0 is already stored on the disk"""
if opt_0 == None:
opt_0 = get_opt_0_table(l, cmem)
if opt_1d == None:
opt_1d = get_opt_1d_table(l, cmem, opt_0 = opt_0)
sequence = Sequence(Function("1D-Revolve", l, cmem))
parameters = dict(defaults)
parameters.update(params)
rd = parameters["rd"]
uf = parameters["uf"]
one_read_disk = parameters["one_read_disk"]
if opt_0 is None:
opt_0 = get_opt_0_table(l, cm, **parameters)
if opt_1d is None:
opt_1d = get_opt_1d_table(l, cm, opt_0=opt_0, **parameters)
sequence = Sequence(Function("1D-Revolve", l, cm), concat=parameters["concat"])
Operation = partial(Op, params=parameters)
if l == 0:
sequence.insert(Operation("Backward", my_buddy(-1, l-1)))
return sequence
if l ==1:
if cmem == 0:
if l == 1:
if cm == 0:
sequence.insert(Operation("Forward", 0))
sequence.insert(Operation("Backward", my_buddy(0, l - 1)))
sequence.insert(Operation("Read_disk", 0))
......@@ -57,30 +66,27 @@ def OneD_Revolve(l, cmem, opt_0 = None, opt_1d = None):
sequence.insert(Operation("Backward", my_buddy(-1, l - 1)))
sequence.insert(Operation("Discard_memory", 0))
return sequence
if parameters.one_read_disk:
list_mem = [j * parameters.uf + opt_0[cmem][l - j] + parameters.rd + opt_0[cmem][j-1] for j in range(1, l)]
if one_read_disk:
list_mem = [j * uf + opt_0[cm][l - j] + rd + opt_0[cm][j-1] for j in range(1, l)]
else:
list_mem = [j * parameters.uf + opt_0[cmem][l - j] + parameters.rd + opt_1d[j-1] for j in range(1, l)]
if min(list_mem) < opt_0[cmem][l]:
list_mem = [j * uf + opt_0[cm][l - j] + rd + opt_1d[j-1] for j in range(1, l)]
if min(list_mem) < opt_0[cm][l]:
jmin = argmin(list_mem)
sequence.insert(Operation("Forwards", [0, jmin - 1]))
sequence.insert_sequence(Revolve(l - jmin, cmem, opt_0 = opt_0).shift(jmin))
sequence.insert_sequence(
revolve(l - jmin, cm, opt_0=opt_0, **parameters).shift(jmin)
)
sequence.insert(Operation("Read_disk", 0))
if parameters.one_read_disk:
sequence.insert_sequence(Revolve(jmin - 1, cmem, opt_0 = opt_0))
if one_read_disk:
sequence.insert_sequence(
revolve(jmin - 1, cm, opt_0=opt_0, **parameters)
)
else:
sequence.insert_sequence(OneD_Revolve(jmin - 1, cmem, opt_0 = opt_0, opt_1d = opt_1d))
sequence.insert_sequence(
revolve_1d(jmin - 1, cm, opt_0=opt_0, opt_1d=opt_1d,
**parameters)
)
return sequence
else:
sequence.insert_sequence(Revolve(l, cmem, opt_0 = opt_0))
sequence.insert_sequence(revolve(l, cm, opt_0=opt_0, **parameters))
return sequence
if __name__ == '__main__':
start_time = time.time()
seq = OneD_Revolve(parameters.l, parameters.cm)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms"%(1000*(end_time - start_time)))
\ No newline at end of file
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "hrevolve"
version = "0.9.0"
authors = [
{ name="Guillaume Pallez (Aupy)", email="guillaume.pallez@inria.fr" },
{ name="Julien Herrmann", email="julien.herrmann@inria.fr"}
]
maintainers = [
{ name="David A. Ham", email="david.ham@imperial.ac.uk"}
]
description = "An implementation of HRevolve and related checkpointing algorithms."
readme = "Readme.md"
requires-python = ">=3.7"
classifiers = [
"Programming Language :: Python :: 3",
"License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)",
"Operating System :: OS Independent",
]
[project.urls]
"Homepage" = "https://gitlab.inria.fr/adjoint-computation/H-Revolve/"
"Bug Tracker" = "https://gitlab.inria.fr/adjoint-computation/H-Revolve/-/issues"
\ No newline at end of file
#! /usr/bin/env python
from hrevolve.parameters import parse_arguments_1D_Revolve
from hrevolve import revolve_1d
import time
parameters = parse_arguments_1D_Revolve()
start_time = time.time()
seq = revolve_1d(**parameters)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms" % (1000*(end_time - start_time)))
#! /usr/bin/env python
from hrevolve.parameters import parse_arguments_Disk_Revolve
from hrevolve import disk_revolve
import time
parameters = parse_arguments_Disk_Revolve()
start_time = time.time()
seq = disk_revolve(**parameters)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Disk: ", seq.disk)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms" % (1000*(end_time - start_time)))
#! /usr/bin/env python
from hrevolve.parameters import parse_arguments_HRevolve, Architecture
from hrevolve import hrevolve
import time
parameters = parse_arguments_HRevolve()
architecture = Architecture(parameters["file_name"])
start_time = time.time()
seq = hrevolve(cvect=architecture.sizes, wvect=architecture.wd,
rvect=architecture.rd, **parameters)
end_time = time.time()
print("Sequence: ", seq)
for i in range(architecture.nblevels):
print("Storage level ", i, ": ", seq.storage[i])
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms" % (1000*(end_time - start_time)))
#! /usr/bin/env python
from hrevolve.parameters import parse_arguments_Periodic_Disk_Revolve
from hrevolve import periodic_disk_revolve
import time
parameters = parse_arguments_Periodic_Disk_Revolve()
start_time = time.time()
seq = periodic_disk_revolve(**parameters)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Disk: ", seq.disk)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms" % (1000*(end_time - start_time)))
#! /usr/bin/env python
from hrevolve.parameters import parse_arguments_Revolve
from hrevolve import revolve
import time
parameters = parse_arguments_Revolve()
start_time = time.time()
seq = revolve(**parameters)
end_time = time.time()
print("Sequence: ", seq)
print("Memory: ", seq.memory)
print("Makespan: ", seq.makespan)
print("Compute time: %.3f ms" % (1000*(end_time - start_time)))
[flake8]
extend-ignore = E741,E501
[options]
packages = hrevolve
scripts = scripts/Revolve
scripts/HRevolve
scripts/1D-Revolve
scripts/Disk-Revolve
scripts/Periodic-Disk-Revolve
\ No newline at end of file
from setuptools import setup
setup()
from hrevolve import disk_revolve, revolve_1d, periodic_disk_revolve, hrevolve
def compare_sequences(sequence1, sequence2):
for i, (char1, char2) in enumerate(zip(sequence1, sequence2)):
if char1 != char2:
raise ValueError(
f"Sequences differ at pos {i}: {char1} is not {char2}"
)
def test_hrevolve_l2():
# Check that we reproduce Listing 2 from Herrmann and Pallez (2020).
sequence = hrevolve(20, [1, 2, 10], [0, 2, 3], [0, 2, 3])
assert sequence.storage[0] == [17, 14, 12, 9, 7, 3, 0]
assert sequence.storage[1] == [7, 12]
assert sequence.storage[2] == [0]
assert sequence.makespan == 89
compare_sequences(
str(sequence), '[W^2_0, F_0->6, W^1_7, F_7->11, W^1_12,'
' F_12->16, W^0_17, F_17->19, B_20, R^0_17, F_17->18, B_19, R^0_17,'
' F_17, B_18, R^0_17, B_17, D^0_17, R^1_12, F_12->13, W^0_14,'
' F_14->15, B_16, R^0_14, F_14, B_15, R^0_14, B_14, D^0_14, R^1_12,'
' W^0_12, F_12, B_13, R^0_12, B_12, D^0_12, R^1_7, F_7->8, W^0_9,'
' F_9->10, B_11, R^0_9, F_9, B_10, R^0_9, B_9, D^0_9, R^1_7, W^0_7,'
' F_7, B_8, R^0_7, B_7, D^0_7, R^2_0, F_0->2, W^0_3, F_3->5, B_6,'
' R^0_3, F_3->4, B_5, R^0_3, F_3, B_4, R^0_3, B_3, D^0_3, R^2_0,'
' W^0_0, F_0->1, B_2, R^0_0, F_0, B_1, R^0_0, B_0, D^0_0]'
)
def test_disk_revolve_l3():
# Check that we reproduce Listing 3 from Herrmann and Pallez (2020).
sequence = disk_revolve(l=10, cm=2, wd=2, rd=1, ub=0)
assert sequence.memory == [5, 8, 6, 0, 3, 1]
assert sequence.disk == [0]
compare_sequences(
str(sequence), '[WD_0, F_0->4, WM_5, F_5->7, WM_8, F_8->9, B_10,'
' RM_8, F_8, B_9, RM_8, B_8, DM_8, RM_5, F_5, WM_6, F_6, B_7, RM_6,'
' B_6, DM_6, RM_5, B_5, DM_5, RD_0, WM_0, F_0->2, WM_3, F_3, B_4,'
' RM_3, B_3, DM_3, RM_0, F_0, WM_1, F_1, B_2, RM_1, B_1, DM_1, RM_0,'
' B_0, DM_0]'
)
assert sequence.makespan == 22
def test_disk_revolve_l4():
# Check that we reproduce Listing 4 from Herrmann and Pallez (2020).
sequence = disk_revolve(l=100, cm=2, wd=10, rd=2, ub=0, concat=2)
compare_sequences(
str(sequence), '[WD_0, F_0->15, WD_16, F_16->31, WD_32,'
' F_32->47, WD_48, F_48->63, WD_64, F_64->79, WD_80, F_80->90,'
' Revolve(9, 2), RD_80, 1D-Revolve(10, 2), RD_64, 1D-Revolve(15, 2),'
' RD_48, 1D-Revolve(15, 2), RD_32, 1D-Revolve(15, 2), RD_16,'
' 1D-Revolve(15, 2), RD_0, 1D-Revolve(15, 2)]'
)
def test_revolve_1d_l4():
# Check that we reproduce Listing 4 from Herrmann and Pallez (2020).
sequence = revolve_1d(l=15, cm=2, rd=2, concat=1)
compare_sequences(
str(sequence), '[F_0->5, Revolve(9, 2), RD_0, Revolve(5, 2)]'
)
def test_disk_revolve_l5():
# Check that we reproduce Listing 5 from Herrmann and Pallez (2020).
sequence = disk_revolve(l=100, cm=2, wd=10, rd=2, ub=0, concat=3)
compare_sequences(
str(sequence), "(16, 16, 16, 16, 16, 11; 10)"
)
def test_periodic_disk_revolve_l6():
# Check that we reproduce Listing 6 from Herrmann and Pallez (2020).
sequence = periodic_disk_revolve(l=10, cm=2, wd=2, rd=1, ub=0)
compare_sequences(
str(sequence), '[WD_0, F_0->2, WD_3, F_3->5, WD_6, F_6->8, WM_9,'
' F_9, B_10, RM_9, B_9, DM_9, RD_6, WM_6, F_6, WM_7, F_7, B_8, RM_7,'
' B_7, DM_7, RM_6, B_6, DM_6, RD_3, WM_3, F_3, WM_4, F_4, B_5, RM_4,'
' B_4, DM_4, RM_3, B_3, DM_3, RD_0, WM_0, F_0, WM_1, F_1, B_2, RM_1,'
' B_1, DM_1, RM_0, B_0, DM_0]'
)
def test_periodic_disk_revolve_l7():
# Check that we reproduce Listing 7 from Herrmann and Pallez (2020).
sequence = periodic_disk_revolve(l=10, cm=2, wd=2, rd=1, ub=0,
one_read_disk=True)
compare_sequences(
str(sequence), '[WD_0, F_0->2, WD_3, F_3->5, WD_6, F_6->8, WM_9,'
' F_9, B_10, RM_9, B_9, DM_9, RD_6, WM_6, F_6, WM_7, F_7, B_8, RM_7,'
' B_7, DM_7, RM_6, B_6, DM_6, RD_3, WM_3, F_3, WM_4, F_4, B_5, RM_4,'
' B_4, DM_4, RM_3, B_3, DM_3, RD_0, WM_0, F_0, WM_1, F_1, B_2, RM_1,'
' B_1, DM_1, RM_0, B_0, DM_0]'
)
assert sequence.memory == [9, 6, 7, 3, 4, 0, 1]
assert sequence.disk == [0, 3, 6]
assert sequence.makespan == 25
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment