Nitro: Geo-distributed storage system for Virtual Machine Images

Nitro is a storage system that is designed to work in geo-distributed cloud environments to efficiently manage Virtual Machine Images (VMIs).

The aim of Nitro is to improve VM provisioning in geo-distributed cloud environments by efficiently ensuring the availability of the corresponding VM image on the destination site. Nitro incorporates two technics to overcome the limitations of the wide-area networks (WANs), such as heterogeneity and low bandwidth. First, Nitro employs deduplication to store VMIs which allow fine-grained management of the images. Second, Nitro implements a network-aware scheduling algorithm (based on maximum flow algorithm) to compute the optimal chunk scheduling plan, in sub-second runtime, that results in minimal VM provisioning time.

More details can be found in our work: 'J. Darrous, S. Ibrahim, A.C. Zhou, C. Perez. "Nitro: Network-Aware Virtual Machine Images Management in Geo-Distributed Clouds." CCGrid, May 2018, Washington DC, USA.'

Architecture

Nitro operates in peer-to-peer way; each involved data center run a nitro process, which no one of them is a master. Each process is provided with a list of IP addresses of the other peers. On each site, Nitro relies on key-value store for local data storage.

API

Nitro exposes a REST API to communicate with other services. The two main functionalities are:

• Add a new image: Read an image from the local file system and returns the image ID.
• Checkout an image: Write the image of the given ID to the local file system.

Prerequisite

The code is compatible with python 2.7.

Redis is used as the local key-value store.

apt-get install -y software-properties-common
echo "deb http://ftp.utexas.edu/dotdeb/ stable all" | tee -a /etc/apt/sources.list.d/dotdeb.list
echo "deb-src http://ftp.utexas.edu/dotdeb/ stable all" | tee -a /etc/apt/sources.list.d/dotdeb.list
wget https://www.dotdeb.org/dotdeb.gpg -O - | apt-key add -
apt-get update && apt-get install -y redis-server

Dependencies

pip install docopt pyyaml xxhash pyzmq requests cherrypy redis networkx bitarray

Run

Start

First of all, a Redis server instance should be up and running in each site. Then, on each machine (one machine per site), and at the same time, run the two nitro daemons. Each site needs a custom configuration file specifying some network properties (e.g. peer ips, local Redis server ip..) and some configurations related to Nitro (e.g. chunk size, scheduling algorithm..)

cd nitro
# default configuration file path is /etc/nitro/nitro.yaml
python comm-driver.py [--config-file <config-file>] &
COMM_PID=$!
sleep 3
# default configuration file path is /etc/nitro/nitro.yaml
python driver.py [--config-file <config-file>] &
PROXY_PID=$!

Add/Get images

The api.py is just a python client (i.e., REST requests wrapper) for Nitro. The core functions are add and checkout.

# in one site, add an image
$ python api.py add <input-image-path>
<image-id>
# in another site, retrieve that image by ID
$ python api.py checkout <image-id> <ouput-image-path>
Image checkout done.

Release

At the end, kill nitro daemons

kill -2 $PROXY_PID
sleep 5
kill -2 $COMM_PID

echo waiting driver to terminate
wait $PROXY_PID
echo waiting comm-driver to terminate
wait $COMM_PID
echo done.

Development

Nitro can be run locally in a development environment using docker containers.

# go to dev dir
cd dev
# ensure that run_docker.sh is executable
chmod +x run_docker.sh
# first time initialization
# + create private network
# + build nitro image
# + create some directories
# + download a test image (cirros)
./run_docker.sh init
# prepare config files
./run_docker.sh up <nb_sites>
# run some test
./run_docker.sh workload
# remove containers
./run_docker.sh rm
# to clean what has been done by init
./run_docker.sh clean

Deployment

Nitro has be deployed and tested on Grid'5000; a French testbed for scientific experimentation. Alls the experiments have been performed on nova cluster in Lyon site. However, other clusters or set of (virtual) machines can be used to replicate the experiments.