# H-REVOLVE: Program reversals with multiple checkpoint levels

Authors: Julien Herrmann, Guillaume Aupy

Licence: See file LICENSE

## Abstract:

This is the library described in the paper "H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms" by Herrmann and Pallez [0].

In this code we propose the code for the implementation of a checkpointing strategy for adjoint computation in the presence of multiple levels of storage:

- a fast and limited in size storage (e.g. memory),

- a slow and larger in size storage (e.g. disks),

- etc where the larger the storage, the slower the access.

Details of the assumption taken (write/read costs etc), of what the code does/will do are available in:

- "H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms" by Herrmann and Pallez [0]

- "Optimal Multi-stage algorithm for Adjoint Computation" by Aupy, Herrmann, Hovland and Robert [1]

- "Periodicity in optimal hierarchical checkpointing schemes for adjoint computations" by Aupy and Herrmann [2].

These works are available in the folder "references".

| [l] | the length of the graph to be differentiated |

| [cm] | the number of forward time steps that can be stored in memory |

| [uf] | the time to perform a forward time step (Default: 1) |

| [ub] | the time to perform a backward time step (Default: 1) |

| [wd] | the time to store a forward time step in disks (Default: 5) |

| [rd] | the time to retrieve a forward time step from disks (Default: 5) |

### ALGORITHMS

#Single Level algorithms

- [*Revolve*] Revolve is the implementation of the single-level dynamic programming presented in [3], where the costs of read and write are considered equal to 0. It provides the user a schedule that minimizes the execution time to reverse the graph.

Utilisation is:

`$ Revolve [l] [cm] --uf [uf] --ub [ub] `

Examples:

`$ Revolve 100 8 --uf 1.2 --ub 3`

`$ Revolve 20 2`

#Two Level algorithms

The implementations of this section considered a specific model with two levels of storage:

- a fast (read and write = 0) and limited in size storage (e.g. memory),

- a slow and unbounded in size storage (e.g. disks),

- [*Disk-Revolve*] Disk-Revolve is the implementation of the two-level dynamic programming presented in [1]. It provides the user a schedule that minimizes the execution time to reverse the graph.

- [*Periodic-Disk-Revolve*] Periodic-Disk-Revolve returns a schedule where the number of forward steps between two consecutive disk checkpoints is constant. The schedule provided is shown to be asymptotically optimal in [2]. In addition, a conjecture was made w.r.t the value of the optimal period that Periodic-Disk-Revolve should use. The `--fast` option allows to compute this value with a much lower complexity than [Periodic-Disk-Revolve] (constant time instead of O(l^3)).

- [*Rev-Revolve*] Rev-Revolve is a slight modification of Disk-Revolve, when during the reverse sweep, one uses Revolve instead of 1D-Revolve (see [1]). It returns the optimal strategy when each Disk checkpoint can only be read once (see [0] for the proof).

- [*Online-Disks*] In [2], it was shown that one can compute an asymptotically optimal online solution (size of adjoint is not known before-hand) using the optimal period provided by [Periodic-Disk-Revolve].

- [*H-revolve*] Description and usage to write.

### OUTPUT

The following outputs are provided:

- [Sequence:] the full schedule following a grammar:

| Operation | Action |

|--:|---|

| [F_i] | Executes the i forward operation |

| [F_i->j] | Executes the i, i+1, ..., j-1, j forward operations |

| [B_i] | Executes the x backward operation |

| [WD_i] | Writes the output of the (i-1) forward operation to disk |

| [RD_i] | Reads the output of the (i-1) forward operation from disk |

| [WM_i] | Writes the output of the (i-1) forward operation to memory |

| [RM_i] | Reads the output of the (i-1) forward operation from memory |

| [DM_i] | Discards the output of the (i-1) forward operation from memory |

- [Memory:] lists all the outputs of forward operations that were stored on memory, in the order that they were saved to memory

- [Disk:] lists all the outputs of forward operations that were stored on memory, in the order that they were saved to memory

- [Makespan:] Provides the makespan in the schedule. Note that with the options `--uf 1 --ub 0` it gives the number of recomputations of forward steps (as discussed in [3]).

- [Period size:] ([Periodic-Disk-Revolve] only) the optimal number of forward steps there should be between two consecutive disk checkpoints. This is the period to use in the online case.

### Result Formats

There is an option to give more compact format of the Sequence output. The option is `--concat` and its value [val] can take the following values: 0 (default), 1, 2 or 3.

Note that to obtain a _defactorized_ version of 1D-Revolve(5,2) (for instance), one can call the subprogram `$ 1D-Revolve 5 2`

- [--concat 3]: To understand this last option, one needs to go back to what a schedule looks like [1,2]. A schedule provided by Disk-Revolve is always of the form: *Forward sweep; Turn; Backward Sweep*.

To explain this schedule, we will use the example of `Disk-Revolve 100 4` which under the option `--concat 2` returns:

In [2], we denote by the term "periods", the number of forward operations between two writes on disks. Finally the idea of the last option is to give: the sizes of the different periods [m1], ..., [mk] and the length of the turn [tn], and is returned under the form: ([m1], [m2], .., [mk]; [tn]). To obtain a _defactorized_ version of it, we start by a WD_0, then F_0->[m1-1], then WD_[m1] etc.

`$ Disk-Revolve 100 4 --concat 3` returns:

> Sequence: (15, 15, 15, 15, 15; 26)

## Library interface

The various checkpointing algorithms are also available as library calls in the

`hrevolve` package. With the exception of HRevolve, these take the positional

arguments `l` and `cm` which are integers with the same meaning as above.

Additional configuration parameters can be set using keyword arguments. The

result is a `Sequence` object encoding the schedule. For example:

```python

In[1]:fromhrevolveimportdisk_revolve

In[2]:print(disk_revolve(15,2,concat=2))

[WD_0,F_0->5,Revolve(9,2),RD_0,1D-Revolve(5,2)]

```

The library interface to HRevolve takes the positional arguments `l`, `cvect`,

`wvect`, and `rvect`. `l` is the length of the graph. The other arguments are

vectors indicating, respectively, the number of slots at each level of memory

and the cost of writing and reading a slot at each level of memory. For

[0] Herrmann, Pallez, "H-Revolve: A Framework for Adjoint Computation on Synchronous Hierarchical Platforms", ACM Transactions on Mathematical Software 46(2), 2020.

[1] Aupy, Herrmann, Hovland and Robert, "Optimal Multi-stage algorithm for Adjoint Computation", Siam Journal on Scientific Computing, 38(3), 2016.

[2] Aupy and Herrmann, "Periodicity in optimal hierarchical checkpointing schemes for adjoint computations", Optimization Methods and Software, 32(3): 594-624 , 2017.

[3] Griewank and Walther, "Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation." ACM Transactions on Mathematical Software (TOMS) 26.1 (2000): 19-45.