GLcv tool

This tool is an OpenGL shader host (ShaderToy-compatible) able to transfer the render output to an OpenCV matrix object for processing, e.g. to run feature detection or optical flow algorithms. It also includes supplementary features helpful for statistical calculations.

# Getting the project
git clone https://gitlab.inria.fr/vtaverni/GLcv.git

# Or
git pull

# Update modules
git submodule update --init --recursive

Table of contents

• Getting started
• Quickref
• Keybindings
• Description
• Implemented CV algorithms
• Unsupported features
• FAQ
  • How do I add more uniforms to be used by the shader buffers?

Getting started

The following tools and libraries are required:

• CMake
• C++14-enabled compiler (recent GCC or clang3.8+)
• Boost 1.58
• OpenGL 4.x
• GLEW
• nanogui (included, but needs dependencies libxinerama-dev , libxcursor-dev, libxi-dev)
• OpenCV 2.4.13.2 with CUDA support
• CUDA 8

If you want to build sample videos, ffmpeg is also needed.

To install the required dependencies on 16.04:

# Development (and OpenCV) package dependencies
sudo apt-get install -y cmake libxinerama-dev libxcursor-dev libxi-dev \
	libboost-all-dev libglew-dev libopencv-dev build-essential checkinstall \
	pkg-config yasm libtiff5-dev libjpeg-dev libjasper-dev libavcodec-dev \
	libavformat-dev libswscale-dev libdc1394-22-dev libtiff4-dev \
	libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev \
	python-dev python-numpy libtbb-dev libqt4-dev libgtk2.0-dev libfaac-dev \
	libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev \
	libvorbis-dev libxvidcore-dev x264

# CUDA 8 installation
wget -O cuda.deb https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64-deb
# For 14.04
#wget -O cuda.deb https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda-repo-ubuntu1404-8-0-local-ga2_8.0.61-1_amd64-deb

sudo dpkg -i cuda.deb
sudo apt-get update
sudo apt-get install -y cuda

# OpenCV compiled version (~ 5GB)
sudo tar -C /usr/local -xjvf /net/vendion/disc/opencv2.tar.bz2

# oglplus
git clone --branch 0.69.0 https://github.com/matus-chochlik/oglplus.git
cd oglplus
sudo ./configure.sh --no-docs --no-examples --use-gl-init-lib GLFW3 --build --install

Project Makefiles can be generated using the following command (once), from this directory:

# See script for details
./configure.sh

Then, from the build directory, generate the binaries using the following command:

make -j8

This generates a GLcv binary in the build directory.

Note: You can build the GLcv application without OpenCV support by appending the -DUSE_OPENCV=OFF when running configure.sh. However, you must have libpng-dev installed and all render and analysis features will be disabled.

Building the docs

When the Unix makefiles have been generated, you can generate the docs using the following commands:

# In the build/ directory
make doc

# Open the resulting web pages
firefox html/index.html

# Compile the doc as a PDF (requires pdflatex)
cd latex
make

# Open the resulting PDF
evince refman.pdf

Building the sample videos

If the ffmpeg package was installed when running the CMake command, you can generate the sample videos using the following command:

# In the build/ directory
make samples

Rendered videos will be stored in the build/render directory, or if a /disc directory is present, build/render will be symlinked to /disc/vtaverni/GLcv/render where the actual rendered files are stored.

Do not use any parallel options (-j) as this will only slow down the process (both GLcv and ffmpeg use all cores on the CPU).

Quickref

$ ./build/GLcv --help

GLcv ShaderToy OpenCV host:
  --help                         show a help message
  -c [ --config ] arg            path to toy config
  -f [ --fps ] arg (=60)         target framerate
  --cv arg                       OpenCV algorithm to apply
  -w [ --width ] arg (=800)      window width
  -h [ --height ] arg (=600)     window height
  -W [ --size ] arg (=-1)        window size (square, overrides -w and -h)
  -z [ --zoom ] arg (=1)         zoom factor, relative to specified size
  --render.dir arg (="frames")   screenshot output folder
  --render.start arg (=-1)       starting frame for render
  --render.end arg (=-1)         exclusive ending frame for render
  --render.format arg (=0)       screenshot format (PNG, TIFF or EXR)
  --accumulator.buffer arg       buffer to accumulate the frames from
  --accumulator.depth arg (=128) number of frames to retain in the accumulator
  -r [ --restore ] [=arg(="")]   restore the program state from the state file 
                                 associated with the config
  -s [ --store ] arg             path for the output program state storage
  -D [ --define ] arg            preprocessor defines for the shader code
  --vsync                        enable VSync
  --postprocess arg (="")        postprocessing pass, as a GLSL shader
  --no-gui                       hide the GUI when starting the application
  --image.shader arg             shader file for buffer image
  --image.0.type arg (=texture)  type of input for image.0
  --image.0.source arg           source for input image.0
  --image.0.wrap arg (=repeat)   wrapping mode for input image.0
  --image.0.filter arg (=mipmap) filtering mode for input image.0
  --a.shader arg                 shader file for buffer a
  --a.0.type arg (=texture)      type of input for a.0

Keybindings

Note: if keybindings don't work, use an alternate (AZERTY/QWERTY) keyboard layout.

Description

This tool uses OpenGL to host ShaderToy programs, while enabling the use of OpenCV to anlayze and process the generated frames. The wrapper.fsh file declares the supported uniforms for hosted programs.

The Boost program_options library is used to support configuring the shader being tested, which means all options in the output from ./GLcv --help can be specified in an INI-formatted configuration file.

This is an example of a basic ShaderToy program config:

// gradient.ini
[image]
# program file for the image buffer
shader = gradient.fsh

// gradient.fsh
// Default shader from the ShaderToy 'New' page
void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
	vec2 uv = fragCoord.xy / iResolution.xy;
	fragColor = vec4(uv,0.5+0.5*sin(iGlobalTime),1.0);
}

// Running the example
./GLcv gradient.ini

Note: options from the command line override options in the config file.

Internally, there are 5 rendering buffers corresponding respectively to the "Image", a, b, c and d buffers from ShaderToy. They are all rendered in sequence, as long as they have a shader program associated, to a dedicated framebuffer. The output of the Image buffer is moved to GPU global memory for processing by the OpenCV GPU (CUDA) module. The output of the chosen processing algorithm is then rendered to the screen using OpenGL. Postprocessing of this result may include vertical split rendering to compare the algorithm output and its input. Vsync is set to the driver default.

The tool also supports a batch render mode (enabled by specifying the wanted number of frames using --render.frames), which render frames to the frames/ subdirectory, using the specified CV algorithm. No output is shown during rendering to speed up the process.

The tool also implements an image accumulator, which is a 3D texture whose layers (r texture coordinates) are the previously n rendered frames of a chosen buffer. It is a circular buffer, meaning the oldest frames will be replaced by newer frames once enough frames have been generated. The number of stored frames is given by --accumulator.depth. The output is taken from the buffer given by --accumulator.buffer. This texture is available to all shaders as the iAccumulator 3D sampler, and iAccumulatorDepth corresponding to the number of layers in the accumulator. Its resolution is the same as the rendering context (ie. as available in iResolution).

Implemented CV algorithms

• copy: just copies input to its output (default)
• spectrum: Fourier spectrum magnitude visualization. Magnitude is scaled based on its histogram to feature a good contrast for viewing.
• optical_flow: LK optical flow as implemented by OpenCV. Visualization is WIP.

Unsupported features

The following features from ShaderToy are not supported by this tool yet:

• Video inputs (and related uniforms)
• Sound (and related uniforms)
• Wrapping and filtering options for render buffer as inputs

FAQ

How do I add more uniforms to be used by the shader buffers?

The state of uniform parameters is kept in an instance of the ShaderInputsType class, which is a member of the GLcvApp class. This type is declared in UniformState.hpp towards the end of the file.

If you want to declare, let's say a vec3 uniform, insert at line 267 a declaration for this uniform, as follows:

/// oglplus::Vec3f is the OpenGL wrapper 3 float component vector type
DECLARE_UNIFORM(oglplus::Vec3f, iMyCustomUniform);

Then, register this uniform as a part of the program state in the ShaderInputsType definition, as follows:

/// Default program state typedef
typedef ShaderInputs<
	iResolution,
	iGlobalTime,
	iTimeDelta,
	iFrame,
	iFrameRate,
	iMouse,
	iChannel0,
	iChannel1,
	iChannel2,
	iChannel3,
	iDate,
	iChannelResolution,
	iAccumulator,
	iAccumulatorDepth,
	iMyCustomUniform    // <-- HERE
> ShaderInputsType;

You can then update this uniform from other applicative code, using the following syntax (assuming state is of type GLcv::ShaderInputsType&):

state.V<iMyCustomUniform>() = oglplus::Vec3f(1.0f, 2.0f, 3.0f);

The corresponding uniform in shader code is uniform vec3 iMyCustomUniform;. You can add it to your own code or to the wrapper file. It must be declared for the GLSL compilation to succeed.