Commit b9702588 authored by DURIF Ghislain's avatar DURIF Ghislain

Release of version 1.0

parent b32183d6
# Byte-compiled / optimized / DLL files
__pycache__/
build
dist
# C extensions
*.o
*.so
# output and logs
log*
output*
Copyright 2017 Inria
3-Clause BSD License
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
include README.md
include LICENSE.txt
include setup.py
recursive-include ckntf *.py
recursive-include ckntf *.h
recursive-include ckntf *.cc
recursive-include doc *.md
# CONVOLUTIONAL KERNEL NETWORK
---
Re-implementation of Convolutional Kernel Network (CKN) from Mairal (2016)
in Python based on the TensorFlow framework (<https://www.tensorflow.org/>).
Authors: Ghislain Durif <br>
Credits: Alberto Bietti, Dexiong Chen, Julien Mairal, Daan Wynen <br>
Copyright: 2017 Inria <br>
If you encounter any issue when installing or using the 'ckntf' package, you can
contact us at <ckn.dev@inria.fr> (for bug report, support, etc.).
---
The [ckntf](./ckntf) package is available under the BSD license. The additional
[miso_svm](./miso_svm), [spherical_kmeans](./spherical_kmeans) and
[whitening](./whitening) packages are available under the GPL-v3 license.
---
## Requirement:
* Python3.x
* Numpy, Scipy
* Tensorflow
* Python packages [miso_svm](./miso_svm), [spherical_kmeans](./spherical_kmeans)
and [whitening](./whitening) are required to run included tutorials and
examples. They are available in the corresponding sub-folders (and require
the MKL library that is available with the Anaconda Python distribution).
---
## Installation
You can check the dedicated [file](./doc/Install.md). <br>
Regarding packages `miso_svm`, `spherical_kmeans` and `whitening`, you can
check the installation instructions in [miso_svm/README.md](./miso_svm/README.md),
[spherical_kmeans/README.md](./spherical_kmeans/README.md) and
[whitening/README.md](./whitening/README.md) respectively.
---
## Tutorial
**NB:** To run the tutorial, you need to download the Cifar10 data set
(either or both version):
* the pre-whitened version `cifar10_whitened.pkl` from
<http://pascal.inrialpes.fr/data2/gdurif/cifar10_whitened.pkl>
* the standard version `cifar10_batches` from
<http://pascal.inrialpes.fr/data2/gdurif/cifar10_batches.tar.gz>
You can either directly put the file `cifar10_whitened.pkl` and
the directory `cifar10_batches` (after you extracted it
with the command `tar -zxvf cifar10_batches.tar.gz`) in
the `ckntf/data` sub-folder before installing the `ckntf` package
(with `python setup.py install`), or you can store them somewhere else on
your machine, and you will have to modify the tutorial examples (see
[unsupervised_example.py](./ckntf/tutorials/unsupervised_example.py) or
[supervised_example.py](./ckntf/tutorials/supervised_example.py))
to load the data with the correct path.
A tutorial is available as a sub-module [ckntf.tutorials](./ckntf/tutorials) and
a doc [page](./doc/Tutorials.md). To replicate the following results that
were presented in Mairal (2016), every details can be found on the dedicated
[page](./doc/Tutorials.md). <br>
### Results
Reproduction of the results from Mairal (2016) with the 'ckntf' python package.
The results from the original paper (Mairal, 2016) were achieved using
some `cudnn`-based Matlab code available at
<https://gitlab.inria.fr/mairal/ckn-cudnn-matlab>.
#### Unsupervised CKN
Here is a summary of the results regarding **unsupervised** CKN on
Cifar10 image data set, with online whitening but without data augmentation
and/or model averaging.
| Architectures | nb layers | nb filters | filter size | subsampling | sigma | Accuracy |
|:--------------|:---------:|:-------------:|:-------------:|:-----------:|:------------:|:--------:|
| (1) | 1 | 64 | 3x3 | 2 | 0.6 | ~68.0 |
| (2) | 2 | 64 <br> 256 | 3x3 <br> 2x2 | 2 <br> 6 | 0.6 <br> 0.6 | ~77.4 |
| (3) | 2 | 256 <br> 1024 | 3x3 <br> 2x2 | 2 <br> 6 | 0.6 <br> 0.6 | ~81.2 |
| (4) | 2 | 512 <br> 4096 | 3x3 <br> 2x2 | 2 <br> 6 | 0.6 <br> 0.6 | ~83.6 |
#### Supervised CKN
Regarding **supervised** CKN model training (without data augmentation),
the best results regarding prediction accuracy were achieved with a 14 layer
CKN model. Details about the architectures that were used can be found
[here](./ckn/tutorials/supervised_example.py).
**NB:** The Matlab-based code was able to reach an accuracy of 90.5% with
the 14 layer model on the pre-whitened Cifar10 data set. For unknow reasons
(we did run extensive tests), the Tensorflow-based package 'ckntf' was only
able to reach an accuracy of 88.50%. However, for smaller architectures (such
as the 5 layer architecture defined in the tutorial), the 'ckntf' package was
able to exactly reproduce the results from the Matlab-based code (accuracy
of 81.5%).
---
## For developpers
(on GNU/Linux and MacOs only)
* To build/install/test the package, see:
```bash
./dev_command.sh help
```
---
## References
Mairal, J., 2016. End-to-end kernel learning with supervised convolutional
kernel networks, in: Advances in Neural Information Processing
Systems. pp. 1399–1407.
# TODO
`****` sqrt_inv_kZtZ is computed only when Z is modified DONE
`****` store filter1 and filter_gaussian as attributes DONE
`****` create a test file that loads a matlab cifar-10 model, perform an encoding pass, train an svm with miso, and evaluate the performance on a grid of regularization parameters. DONE
`****` create a test file that learns a one-layer ckn unsupervised model, perform an encoding pass, train an svm with miso, and evaluate the performance on a grid of regularization parameters. DONE
`****` create a test file that learns a two-layer ckn unsupervised model with high performance, perform an encoding pass, train an svm with miso, and evaluate the performance on a grid of regularization parameters. DONE
`****` linear_pooling: if n_filters > 10 (to be replaced by 2 conv1d on each direction)
`****` create template for miso (on float or double, c.f. Daan's code)
`****` write bindings for k-means
`**` for big data sets: one run to extract batches + on each batches sample patches
`*` Mask for non-squared images
# from __future__ import absolute_import
# __all__ = ['core', 'data', 'utils']
#
# from ckn import core
# from ckn import data
# from ckn import utils
from __future__ import absolute_import
from ckntf.core.ckn_layer import CKNlayer
from ckntf.core.ckn_model import CKNmodel
This diff is collapsed.
"""running test on the CKN layer class"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
__author__ = "Ghislain Durif, THOTH TEAM INRIA Grenoble Alpes"
__copyright__ = "INRIA"
__credits__ = ["Alberto Bietti", "Dexiong Chen", "Ghislain Durif",
"Julien Mairal", "Daan Wynen"]
__license__ = "BSD"
__version__ = "1.0"
__maintainer__ = "Ghislain Durif"
__email__ = "ghislain.durif@inria.fr"
__status__ = "Development"
__date__ = "2017"
import logging
import os
import sys
logging.basicConfig(stream=sys.stdout,
format='%(levelname)s:%(message)s',
level=logging.DEBUG)
from timeit import default_timer as timer
from pprint import pprint
import pkg_resources
import tensorflow as tf
import numpy as np
import math
import ckntf.core
import ckntf.input
class args:
## data
sample_size = 1000
batch_size = 500
data_folder = pkg_resources.resource_filename('ckntf', 'data')
class param:
## CKN layer parameters
filter_height = 3
filter_width = 3
out_channels = 64
subsampling = 2
n_sampled_patches = 10000
kernel = 'exp0'
kernel_param = 0.7
dtype = tf.float32
padding = "SAME"
name = "ckn_layer"
epsilon_norm = 1e-5
epsilon_inv = 1e-3
class CKNlayerTest(tf.test.TestCase):
def testCKNlayer(self):
print("Running tests on the CKNlayer class")
pprint(vars(args))
pprint(vars(param))
### import data
logging.info("START: importing data")
(Xtr, Ytr,
Xte, Yte) = ckntf.input.load_cifar10(args.data_folder,
args.sample_size,
whitened=True)
(N, H, W, C) = Xtr.shape
logging.info("END: data imported")
# data spec
logging.info("dimensions of the data set ({})".format(Xtr.shape))
### run tensorflow
with self.test_session() as sess:
# input
input = tf.placeholder(dtype=param.dtype, shape=[None, H, W, C])
# define a layer
simple_layer = ckntf.core.CKNlayer.from_param(param)
# encode the layer
print("#### encode the layer ####")
output = simple_layer.encode(input)
##### unsupervised training of the filters
print("#### unsupervised training ####")
# batch nb
n_batch = Xtr.shape[0] // args.batch_size \
+ 1 * (args.sample_size % args.batch_size != 0)
# extract patches
logging.info("START: extract the patches")
logging.info("data separated in {} batch(es)".format(n_batch))
start = timer()
for i_batch in range(0,(n_batch)):
logging.debug("batch_index = %d / %d" %(i_batch+1, n_batch))
i_start = i_batch * args.batch_size
i_end = Xtr.shape[0] if i_batch == n_batch - 1 \
else (i_batch + 1) * args.batch_size
# logging.debug(" i_start = {} // i_end = {}"
# .format(i_start, i_end))
sub_data = Xtr[i_start:i_end,]
## feed placeholder
simple_layer.append_patches(
sess,
n_batch=n_batch,
feed_dict={input: sub_data})
simple_layer.concatenate_patches()
end = timer()
logging.info("END: extraction of the patches in {:.4f} sec"
.format(end - start))
# update the filters
logging.info("START: train the filters with k-means")
start = timer()
simple_layer.update_filters_kmeans()
simple_layer.compute_update_filters_kmeans(sess)
end = timer()
logging.info("END: training of the filters in {:.4f} sec"
.format(end - start))
# get the filters
logging.info("get the filters")
filters = simple_layer.get_filters(sess)
logging.info("filter shape = {}".format(filters.shape))
self.assertAllEqual(filters.shape,
(param.filter_height,
param.filter_width,
Xtr.shape[3],
param.out_channels))
# set the filters
logging.info("set the filters")
simple_layer.update_filters(filters)
simple_layer.compute_update_filters(sess)
# output of the layer
logging.info("START: computation of the layer output")
logging.info("data separated in {} batch(es)".format(n_batch))
output = []
start = timer()
for i_batch in range(0,(n_batch)):
logging.debug("batch_index = %d / %d" %(i_batch+1, n_batch))
i_start = i_batch * args.batch_size
i_end = Xtr.shape[0] if i_batch == n_batch - 1 \
else (i_batch + 1) * args.batch_size
# logging.debug(" i_start = {} // i_end = {}"
# .format(i_start, i_end))
sub_data = Xtr[i_start:i_end,]
## feed placeholder
run_metadata = tf.RunMetadata()
tmp = sess.run(simple_layer.output,
feed_dict={input: sub_data})
output.append(tmp)
output = np.concatenate(output, axis=0)
end = timer()
# logging.info("END: computation of the output in {:.4f} sec"
# .format(end - start))
logging.info("output shape = {}".format(output.shape))
self.assertAllEqual(output.shape,
(Xtr.shape[0],
Xtr.shape[1] // param.subsampling,
Xtr.shape[2] // param.subsampling,
param.out_channels))
print("#### tests over ####")
if __name__ == "__main__":
tf.test.main()
This diff is collapsed.
"""running test on the CKN model class in supervised mode"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
__author__ = "Ghislain Durif, THOTH TEAM INRIA Grenoble Alpes"
__copyright__ = "INRIA"
__credits__ = ["Alberto Bietti", "Dexiong Chen", "Ghislain Durif",
"Julien Mairal", "Daan Wynen"]
__license__ = "BSD"
__version__ = "1.0"
__maintainer__ = "Ghislain Durif"
__email__ = "ghislain.durif@inria.fr"
__status__ = "Development"
__date__ = "2017"
import logging
import sys
logging.basicConfig(stream=sys.stdout,
format='%(levelname)s:%(message)s',
level=logging.INFO)
from timeit import default_timer as timer
from pprint import pprint
import pkg_resources
import tensorflow as tf
import numpy as np
import math
import ckntf.core
import ckntf.input
from ckntf.core.utils import obj
class args:
## data
sample_size = 1000
batch_size = 64
data_folder = pkg_resources.resource_filename('ckntf', 'data')
## online whitening
online_whitening = False
## tensorboard
output_dir = 'test_supervised_mode'
## regularization parameter
l2_reg = 1e-1
## learning rate
learning_rate = 10
## tag
tag = "test"
## CKN model parameters
params = [
{
'filter_height': 3, 'filter_width': 3,
'out_channels': 512, 'subsampling': 2,
'n_sampled_patches': 1000, 'kernel': 'exp0', 'kernel_param': 0.6,
'dtype': tf.float32, 'padding': "SAME", 'name': "layer1",
'epsilon_norm': 1e-5, 'epsilon_inv': 1e-3
}
]
## for tensorboard
print("For tensorboard:\n run `tensorboard --logdir="
+ args.output_dir + "` in a shell"
+ "\n and go to"
+ "`localhost:6006` on your web-browser")
class CKNmodelTest(tf.test.TestCase):
def testCKNmodel(self):
print("Running tests on the CKNmodel class in supervised mode")
ckntf.core.ckn_layer.DEBUG = False
ckntf.core.ckn_model.DEBUG = False
pprint(vars(args))
pprint(params)
### import data
logging.info("importing data")
(Xtr, Ytr,
Xte, Yte) = ckntf.input.load_cifar10(args.data_folder,
args.sample_size,
whitened=not args.online_whitening)
(N, H, W, C) = Xtr.shape
n_class = len(set(Ytr))
logging.info("data imported")
# data spec
logging.info("dimensions of the data set ({})".format(Xtr.shape))
### run tensorflow
with self.test_session() as sess:
# define the model
model = ckntf.core.CKNmodel.from_param(params, sess, args.batch_size,
online_whitening=args.online_whitening)
# encode the model
print("#### encode the model ####")
model.encode(H, W, C)
model.encode_supervised(n_class, loss_type='squared_hinge',
l2_reg=args.l2_reg)
model.encode_optim(sample_size=Xtr.shape[0],
learning_rate=args.learning_rate)
# initialize tf variables
model.initialize_variables()
##### initialization of the filters with k-means
print("#### init filters with k-means ####")
logging.info("START: unsupervised training of the {} layer(s) "
.format(len(model.layer_list)) \
+ "in the model with k-means")
start = timer()
model.initialize_filters(input_data=Xtr,
output_dir=args.output_dir,
n_iter_kmeans=0)
end = timer()
logging.info("END: training of the {} layers in {:.4f} sec"
.format(len(model.layer_list), end - start))
##### supervised training
print("#### supervised training ####")
logging.info("START: supervised training of the {} layer(s) "
.format(len(model.layer_list)))
start = timer()
model.sup_train(output_dir=args.output_dir,
input_data=Xtr,
input_labels=Ytr,
test_data=Xte,
test_labels=Yte,
n_iter=9000,
batch_size_eval=200,
alternate_miso=False)
end = timer()
logging.info("END: training of the {} layers in {:.4f} sec"
.format(len(model.layer_list), end - start))
print("#### tests over ####")
if __name__ == "__main__":
tf.test.main()
"""running test on the CKN model class in unsupervised mode"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
__author__ = "Ghislain Durif, THOTH TEAM INRIA Grenoble Alpes"
__copyright__ = "INRIA"
__credits__ = ["Alberto Bietti", "Dexiong Chen", "Ghislain Durif",
"Julien Mairal", "Daan Wynen"]
__license__ = "BSD"
__version__ = "1.0"
__maintainer__ = "Ghislain Durif"
__email__ = "ghislain.durif@inria.fr"
__status__ = "Development"
__date__ = "2017"
import logging
import sys
logging.basicConfig(stream=sys.stdout,
format='%(levelname)s:%(message)s',
level=logging.DEBUG)
from timeit import default_timer as timer
from pprint import pprint
import pkg_resources
import tensorflow as tf
import numpy as np
import math
import ckntf.core
import ckntf.input
from ckntf.core.utils import obj
class args:
## data
sample_size = 1000
batch_size = 500
data_folder = pkg_resources.resource_filename('ckntf', 'data')
## online whitening
online_whitening = True
## tensorboard
output_dir = 'test_unsupervised_mode'
## CKN model parameters
# params = [
# {
# 'filter_height': 3, 'filter_width': 3,
# 'out_channels': 64, 'subsampling': 2,
# 'n_sampled_patches': 10000, 'kernel': 'exp0', 'kernel_param': 0.6,
# 'dtype': tf.float32, 'padding': "SAME", 'name': "layer1",
# 'epsilon_norm': 1e-5, 'epsilon_inv': 1e-3
# },
# {
# 'filter_height': 2, 'filter_width': 2,
# 'out_channels': 256, 'subsampling': 6,
# 'n_sampled_patches': 10000, 'kernel': 'exp0', 'kernel_param': 0.6,
# 'dtype': tf.float32, 'padding': "SAME", 'name': "layer2",
# 'epsilon_norm': 1e-5, 'epsilon_inv': 1e-3
# }
# ]
params = [
{
'filter_height': 3, 'filter_width': 3,
'out_channels': 64, 'subsampling': 2,
'n_sampled_patches': 10000, 'kernel': 'exp0', 'kernel_param': 0.6,
'dtype': tf.float32, 'padding': "SAME", 'name': "layer1",
'epsilon_norm': 1e-5, 'epsilon_inv': 1e-3
}
]
## for tensorboard
print("For tensorboard:\n run `tensorboard --logdir="
+ args.output_dir + "` in a shell"
+ "\n and go to"
+ "`localhost:6006` on your web-browser")
class CKNmodelTest(tf.test.TestCase):
def testCKNmodel(self):
print("Running tests on the CKNmodel class in unsupervised mode")
ckntf.core.ckn_layer.DEBUG = True
ckntf.core.ckn_model.DEBUG = True
pprint(vars(args))
pprint(params)
### import data
logging.info("importing data")
# (Xtr, Ytr,
# Xte, Yte) = ckntf.input.read_whitened_dataset_cifar10(args.data_file,
# args.sample_size)
(Xtr, Ytr,
Xte, Yte) = ckntf.input.load_cifar10(args.data_folder,
args.sample_size,
whitened=not args.online_whitening)
(N, H, W, C) = Xtr.shape
logging.info("data imported")
# data spec
logging.info("dimensions of the data set ({})".format(Xtr.shape))
### run tensorflow
with self.test_session() as sess:
# define the model
model = ckntf.core.CKNmodel.from_param(params, sess, args.batch_size,
online_whitening=args.online_whitening)
# encode the model
print("#### encode the model ####")
model.encode(H, W, C)
##### unsupervised training of the filters
print("#### unsupervised training ####")
## training of the model
logging.info("START: unsupervised training of the {} layer(s) "
.format(len(model.layer_list)) \
+ "in the model with k-means")
start = timer()
model.unsup_train(input_data=Xtr)
end = timer()
logging.info("END: training of the {} layers in {:.4f} sec"
.format(len(model.layer_list), end - start))
# get the filters
logging.info("get the filters")
filter_list = model.get_filters()
for ind, filters in enumerate(filter_list):
logging.info("layer {}: filter shape = {}"
.format(ind+1, filters.shape))
lparam = obj(params[ind])
if args.online_whitening and ind == 0:
# specific case for the first layer
# when doing online whitening
self.assertAllEqual(filters.shape,
(1,
1,
model.layer_list[ind]
.input.get_shape().as_list()[3],
lparam.out_channels))
else:
self.assertAllEqual(filters.shape,
(lparam.filter_height,
lparam.filter_width,
model.layer_list[ind]
.input.get_shape().as_list()[3],
lparam.out_channels))
# set the filters
logging.info("set the filters")
model.update_filters(filter_list)
# output of the model
logging.info("computation of the model output")