# H-REVOLVE: Program reversals with multiple checkpoint levels
Authors: Julien Herrmann, Guillaume Aupy
Licence: See file LICENSE
## Abstract:
This is the library described in the paper "H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms" by Herrmann and Pallez [0].
In this code we propose the code for the implementation of a checkpointing strategy for adjoint computation in the presence of multiple levels of storage:
- a fast and limited in size storage (e.g. memory),
- a slow and larger in size storage (e.g. disks),
- etc where the larger the storage, the slower the access.
Details of the assumption taken (write/read costs etc), of what the code does/will do are available in:
- "H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms" by Herrmann and Pallez [0]
- "Optimal Multi-stage algorithm for Adjoint Computation" by Aupy, Herrmann, Hovland and Robert [1]
- "Periodicity in optimal hierarchical checkpointing schemes for adjoint computations" by Aupy and Herrmann [2].
These works are available in the folder "references".
| [l] | the length of the graph to be differentiated |
| [cm] | the number of forward time steps that can be stored in memory |
| [uf] | the time to perform a forward time step (Default: 1) |
| [ub] | the time to perform a backward time step (Default: 1) |
| [wd] | the time to store a forward time step in disks (Default: 5) |
| [rd] | the time to retrieve a forward time step from disks (Default: 5) |
### ALGORITHMS
#Single Level algorithms
- [*Revolve*] Revolve is the implementation of the single-level dynamic programming presented in [3], where the costs of read and write are considered equal to 0. It provides the user a schedule that minimizes the execution time to reverse the graph.
Utilisation is:
`$ Revolve [l] [cm] --uf [uf] --ub [ub] `
Examples:
`$ Revolve 100 8 --uf 1.2 --ub 3`
`$ Revolve 20 2`
#Two Level algorithms
The implementations of this section considered a specific model with two levels of storage:
- a fast (read and write = 0) and limited in size storage (e.g. memory),
- a slow and unbounded in size storage (e.g. disks),
- [*Disk-Revolve*] Disk-Revolve is the implementation of the two-level dynamic programming presented in [1]. It provides the user a schedule that minimizes the execution time to reverse the graph.
- [*Periodic-Disk-Revolve*] Periodic-Disk-Revolve returns a schedule where the number of forward steps between two consecutive disk checkpoints is constant. The schedule provided is shown to be asymptotically optimal in [2]. In addition, a conjecture was made w.r.t the value of the optimal period that Periodic-Disk-Revolve should use. The `--fast` option allows to compute this value with a much lower complexity than [Periodic-Disk-Revolve] (constant time instead of O(l^3)).
- [*Rev-Revolve*] Rev-Revolve is a slight modification of Disk-Revolve, when during the reverse sweep, one uses Revolve instead of 1D-Revolve (see [1]). It returns the optimal strategy when each Disk checkpoint can only be read once (see [0] for the proof).
- [*Online-Disks*] In [2], it was shown that one can compute an asymptotically optimal online solution (size of adjoint is not known before-hand) using the optimal period provided by [Periodic-Disk-Revolve].
- [*H-revolve*] Description and usage to write.
### OUTPUT
The following outputs are provided:
- [Sequence:] the full schedule following a grammar:
| Operation | Action |
|--:|---|
| [F_i] | Executes the i forward operation |
| [F_i->j] | Executes the i, i+1, ..., j-1, j forward operations |
| [B_i] | Executes the x backward operation |
| [WD_i] | Writes the output of the (i-1) forward operation to disk |
| [RD_i] | Reads the output of the (i-1) forward operation from disk |
| [WM_i] | Writes the output of the (i-1) forward operation to memory |
| [RM_i] | Reads the output of the (i-1) forward operation from memory |
| [DM_i] | Discards the output of the (i-1) forward operation from memory |
- [Memory:] lists all the outputs of forward operations that were stored on memory, in the order that they were saved to memory
- [Disk:] lists all the outputs of forward operations that were stored on memory, in the order that they were saved to memory
- [Makespan:] Provides the makespan in the schedule. Note that with the options `--uf 1 --ub 0` it gives the number of recomputations of forward steps (as discussed in [3]).
- [Period size:] ([Periodic-Disk-Revolve] only) the optimal number of forward steps there should be between two consecutive disk checkpoints. This is the period to use in the online case.
### Result Formats
There is an option to give more compact format of the Sequence output. The option is `--concat` and its value [val] can take the following values: 0 (default), 1, 2 or 3.
Note that to obtain a _defactorized_ version of 1D-Revolve(5,2) (for instance), one can call the subprogram `$ 1D-Revolve 5 2`
- [--concat 3]: To understand this last option, one needs to go back to what a schedule looks like [1,2]. A schedule provided by Disk-Revolve is always of the form: *Forward sweep; Turn; Backward Sweep*.
To explain this schedule, we will use the example of `Disk-Revolve 100 4` which under the option `--concat 2` returns:
In [2], we denote by the term "periods", the number of forward operations between two writes on disks. Finally the idea of the last option is to give: the sizes of the different periods [m1], ..., [mk] and the length of the turn [tn], and is returned under the form: ([m1], [m2], .., [mk]; [tn]). To obtain a _defactorized_ version of it, we start by a WD_0, then F_0->[m1-1], then WD_[m1] etc.
`$ Disk-Revolve 100 4 --concat 3` returns:
> Sequence: (15, 15, 15, 15, 15; 26)
## Library interface
The various checkpointing algorithms are also available as library calls in the
`hrevolve` package. With the exception of HRevolve, these take the positional
arguments `l` and `cm` which are integers with the same meaning as above.
Additional configuration parameters can be set using keyword arguments. The
result is a `Sequence` object encoding the schedule. For example:
```python
In[1]:fromhrevolveimportdisk_revolve
In[2]:print(disk_revolve(15,2,concat=2))
[WD_0,F_0->5,Revolve(9,2),RD_0,1D-Revolve(5,2)]
```
The library interface to HRevolve takes the positional arguments `l`, `cvect`,
`wvect`, and `rvect`. `l` is the length of the graph. The other arguments are
vectors indicating, respectively, the number of slots at each level of memory
and the cost of writing and reading a slot at each level of memory. For
[0] Herrmann, Pallez, "H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms", ACM Transactions on Mathematical Software 46(2), 2020.
[1] Aupy, Herrmann, Hovland and Robert, "Optimal Multi-stage algorithm for Adjoint Computation", Siam Journal on Scientific Computing, 38(3), 2016.
[2] Aupy and Herrmann, "Periodicity in optimal hierarchical checkpointing schemes for adjoint computations", Optimization Methods and Software, 32(3): 594-624 , 2017.
[3] Griewank and Walther, "Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation." ACM Transactions on Mathematical Software (TOMS) 26.1 (2000): 19-45.
