Mentions légales du service

Skip to content
Snippets Groups Projects
Select Git revision
  • main default protected
  • kinovis
2 results
Name Last commit Last update
readme_images
resources
scripts
src
.gitignore
CMakeLists.txt
LICENSE.md
README.md
imgui.ini
install_script.sh
install_script_win.ps1
params.txt
volumetricRendererParams.txt
README.md

alt text

Project

Welcome to the repository for our paper called "ProbeSDF", available here. Please cite our work if you find it useful:

@misc{toussaint2024probesdf,
title={ProbeSDF: Light Field Probes for Neural Surface Reconstruction},
author={Briac Toussaint and Diego Thomas and Jean-Sébastien Franco},
year={2024},
eprint={2412.10084},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.10084},
}

License

This code is subject to a non-commerical license.

Issues

Issues, questions or suggestions can be posted on this github repository.

Installation Requirements

  • Nvidia GPU Series 3000 or 4000 (compute capability 8.6)
  • A minimum of 6GB of VRAM is required (DTU uses a bit more than 4GB, successfully tested on an rtx 3060 laptop GPU).
  • cmake > 3.20
  • linux or windows
  • gcc/g++ (tested with 11.4) or visual studio (tested with version 22)
  • cuda toolkit (tested with 11.7 and 12.6)

Intallation on Linux

Assuming gcc-11 and cuda 11.7 are installed:

  • Intall the following packages:

    sudo apt-get install -y xorg-dev

  • Clone the repo:

git clone -b main https://gitlab.inria.fr/projects-morpheo/ProbeSDF.git
cd ProbeSDF/

  • Define your compiler and cuda path (make sure to adapt it to your specific cuda installation):

export CC=/usr/bin/gcc-11
export CXX=/usr/bin/g++-11
export CUDACXX=/usr/local/cuda-11.7/bin/nvcc
export PATH="/usr/local/cuda-11.7/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-11.7/lib64:$LD_LIBRARY_PATH"

  • Run the install script to download and install the required libraries:

./install_script.sh

  • Compile the main program:

mkdir Release
cd Release
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=$(pwd)/../libs_install -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-11 -DCMAKE_CUDA_FLAGS="-ccbin /usr/bin/g++-11"
make -j
cd ../

Intallation on Windows

Assuming an appropriate version of Visual Studio and cuda is installed (tested with visual studio 22 and cuda 12.6), open a developper powershell command prompt and type the following commands:

  • Clone the repo:

git clone -b main https://gitlab.inria.fr/projects-morpheo/ProbeSDF.git
cd ProbeSDF/

  • Run the install script to download and install the required libraries:

$install_dir = (Join-Path $pwd "/libs_install")
./install_script_win.ps1

  • Compile the main program:

mkdir build
cd build
cmake .. -DCMAKE_PREFIX_PATH="$install_dir" -DCMAKE_GENERATOR_TOOLSET="cuda=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6"
cmake --build . --target INSTALL --config Release -j
cd ../

  • Copy the dlls in the Release folder:

cp ./libs_install/bin/assimp-vc143-mt.dll ./Release/
cp ./libs_install/bin/glfw3.dll ./Release/

For laptops having both integrated and discrete gpus, make sure that the discrete gpu will be selected using the Windows graphics settings panel.

Installing the python environment

conda env create -f ./scripts/environment.yml
conda activate millimetric_humans

Running the reconstruction

MVMannequins Dataset

Refer to the instructions given on the Millimetric Humans webpage.

DTU Dataset

Download the DTU preprocessed dataset from Voxurf's project page here. Generate the configuration files with:

python ./scripts/run_dtu.py ./Release path/to/DTU/

Run the reconstruction in headless mode on all the examples with:

python ./scripts/run_dtu.py ./Release path/to/DTU -run_ours

Or use the gui to run one reconstruction at a time. You can run the reconstruction then evaluate with:

python ./scripts/run_dtu.py ./Release path/to/DTU -run_ours -eval_psnr -eval_chamfer -eval_ours -gen_tables

BlendedMVS Dataset

Download the BMVS preprocessed dataset from neus' project page here. The ground truth meshes can be downloaded from the dataset's project page here. Generate the configuration files with:

python ./scripts/run_BlendedMVS.py ./Release path/to/BMVS/

Run the reconstruction in headless mode on all the examples with:

python ./scripts/run_BlendedMVS.py ./Release path/to/BMVS -run_ours

Or use the gui to run one reconstruction at a time. You can run the reconstruction then evaluate with:

python ./scripts/run_BlendedMVS.py ./Release path/to/BMVS/ -run_ours -eval_psnr -eval_chamfer -eval_ours -gen_tables

ActorsHQ Dataset

The dataset can be downloaded using the script provided on the project page here, after requesting access tokens. Here we only use the 1000th frame of each actor. Run the following to prepare a subset of the dataset for our method:

python ./scripts/run_actorshq.py ./Release path/to/ActorsHQ path/to/ActorsHQ_Subset -prepare_files

Run the reconstruction in headless mode and evaluate on all the examples with:

python ./scripts/run_actorshq.py ./Release path/to/ActorsHQ path/to/ActorsHQ_Subset -run_ours -eval_psnr -eval_ours -gen_tables

4D Human Outfit Dataset

You can request access to the dataset here. The following command will extract 3 sequences from the dataset to "/local_path/to/4DHO" and then compute the reconstruction on each frame.

python ./scripts/run_4dhumanoutfit.py ./Release /path/to/4DHO /local_path/to/4DHO --names-list kim-jea-dance bob-jea-boxe mia-opt-cartwheel

Headless reconstruction

The python scripts above will compute the reconstructions using the headless program (i.e. no user interface). On linux, the headless program uses EGL to create an opengl context so that it works even on server GPUs that are not plugged to a screen (or when running the code through ssh). Furthermore, the gpu can be selected using the --gpu flag. On Windows, the context will just be created with a hidden window since EGL is not available. Selecting the GPU is not feasible programmatically so you have to do so in the graphics settings panel.

Using the GUI

A graphical user interface is provided for convenience. It is particularly useful to visualize the cameras and to make sure that the reference frame is correctly defined. The optimization can be seen in real time which is also useful to quickly finetune hyperparameters.

Using DTU as an example, start by generating the configuration files with:

python ./scripts/run_dtu.py ./Release path/to/DTU/

Launch the gui program with:

./Release/ProbeSDF

Go to the scene folder of your choice (here we use dtu_scan63) and drag and drop the file named "scene_config.txt" on the "Scene config file" button, then click "Load scene".

alt text

Click "Prepare volume" once, and then "Optimize" to run the reconstruction code. alt text

Key bindings:

  • "C" to show / hide the mesh colors.
  • "P" to show / hide a planar cut of the volume.
  • "V" to enable / disable volumetric rendering.

Note: The state of almost all the user interface (sliders, checkboxes, windows) is saved into the "params.txt" file when the program is closed. These parameters are shared with the headless program so be careful to restore this file after experimenting with the sliders. The config file and the schedule file of the current scene will also be overwritten when closing the program.

Volumetric Renderer

The raw volumetric data will be exported by default, in addition to the colored meshes for each level of detail.

You can visualize the reconstruction afterwards using the volumetric renderer:

./Release/VolumetricRenderer

Drag and drop one of the exported files called "vol_0.zip" on the button "Choose Volumetric File". Do the same with the schedule.json file on the button "Choose Schedule File" and click "Load".

alt text

Mesh simplification and texture extraction

A python script is given (scripts/decimateMesh.py) to simplify and unwrap one or several meshes. Once the mesh is unwrapped, a texture can be extracted from the volume data instead of per-vertex colors using the TextureExtractor executable.

Data format

Information about the data format used for the inputs and outputs of ProbeSDF can be found in Schedule.py.

Internal dependencies

These libraries are packaged in src/

External dependencies

These libraries are downloaded by the installation script:

  • GLFW (Cross-platform window management library)
  • Assimp (Import / Export to various mesh formats)
  • Eigen (Linear algebra)

Project structure

Most of the GUI is orchestrated from Sequence.cpp while the optimization loop is in DifferentialRendering.cpp (in the fonction named step()). Part of the GPU code in written in cuda (src/sequence/cuda) while the rest is in GLSL shaders (resources/shaders). The shaders use syntax hacks so that they can be edited with a c++ editor, they are preprocessed into regular glsl at run time.

Changelog

  • 16/12/2024 Initial release.