README.md 13.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
# Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow               {#semantic_reflectionsPage}

This project is the reference implementation of [Rodriguez et al. 20] *Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow*, (http://www-sop.inria.fr/reves/Basilic/2020/RPHD20/).

If you use the code, we would greatly appreciate it if you could cite the corresponding paper:

```
@Article{RPHD20,
  author       = "Rodriguez, Simon and Prakash, Siddhant and Hedman, Peter and Drettakis, George",
  title        = "Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow",
  journal      = "Proceedings of the ACM on Computer Graphics and Interactive Techniques",
  number       = "1",
  volume       = "3",
  month        = "may",
  year         = "2020",
  keywords     = "image-based rendering, semantic labeling, reflection rendering",
  url          = "http://www-sop.inria.fr/reves/Basilic/2020/RPHD20"
}
```

and of course the actual *sibr* system:

```
@misc{sibr2020,
RODRIGUEZ Simon's avatar
RODRIGUEZ Simon committed
25
   author       = "Bonopera, Sebastien and Hedman, Peter and Esnault, Jerome and Prakash, Siddhant and Rodriguez, Simon and Thonat, Theo and Benadel, Mehdi and Chaurasia, Gaurav and Philip, Julien and Drettakis, George",
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
   title        = "sibr: A System for Image Based Rendering",
   year         = "2020",
   url          = "https://gitlab.inria.fr/sibr/sibr_core"
}
```

---

## Authors

Simon Rodriguez, Siddhant Prakash, Peter Hedman, and George Drettakis, presented at I3D 2020.

---

## How to use

DRETTAKIS George's avatar
READMe  
DRETTAKIS George committed
42 43 44 45 46 47 48 49
### Binary distribution

The easiest way to use *SIBR* is to download the binary distribution. All steps described below, including all preprocessing for your datasets will work using this code.

Download the precompiled distribution from the page: https://sibr.gitlabpages.inria.fr/download.html (Semantic Reflections  850Mb); unzip the file and rename the directory "install". 

You will need CUDA 10.1 and the latest NVIDIA driver.

50 51 52 53 54 55 56 57 58 59
### Build & Install

You will need:

- COLMAP (https://colmap.github.io/ , version with GPU support strongly recommended). This tutorial will use the GUI version, but it could be adapted to run on the command-line by passing the proper parameters.
- Meshlab will be required for two manual steps
- tensorflow >= 1.6.1, ideally with GPU support. (We suggest you follow the tensorflow virtual environment installation procedure described here: https://www.tensorflow.org/install/pip ). 
- the external repository for deeplabv3-resnet trained on the ADE20K with car labels: https://gitlab.inria.fr/sibr/projects/semantic-reflections/segmentation-network/
- RealityCapture (for texturing, required by Deep Blending only)

60
In SIBR, clone `sibr_core` (https://gitlab.inria.fr/sibr/sibr_core), `fribr_framework`, `inside_out_deep_blending`, `tfgl_interop`, and `semantic_reflections` (https://gitlab.inria.fr/sibr/projects/semantic-reflections/semantic_reflections). 
61 62 63

You can do this using the following commands:

Mehdi Benadel's avatar
Mehdi Benadel committed
64 65
```bash
## through HTTPS
66
git clone https://gitlab.inria.fr/sibr/sibr_core.git
Mehdi Benadel's avatar
Mehdi Benadel committed
67 68
## through SSH
git clone git@gitlab.inria.fr:sibr/sibr_core.git
69 70 71 72
```

Then go to *src/projects* and clone the other projects:

Mehdi Benadel's avatar
Mehdi Benadel committed
73 74 75 76 77 78 79 80 81 82 83
```bash
## through HTTPS
git clone https://gitlab.inria.fr/sibr/projects/fribr_framework.git
git clone https://gitlab.inria.fr/sibr/projects/tfgl_interop.git
git clone https://gitlab.inria.fr/sibr/projects/inside_out_deep_blending.git
git clone https://gitlab.inria.fr/sibr/projects/semantic-reflections/semantic_reflection.git
## through SSH
git clone git@gitlab.inria.fr:sibr/projects/fribr_framework.git
git clone git@gitlab.inria.fr:sibr/projects/tfgl_interop.git
git clone git@gitlab.inria.fr:sibr/projects/inside_out_deep_blending.git
git clone git@gitlab.inria.fr:sibr/projects/semantic-reflections/semantic_reflection.git
84 85 86 87
```

Enable projects in Cmake (`BUILD_IBR_FRIBR_FRAMEWORK`, `BUILD_IBR_DEEP_BLENDING`, `BUILD_IBR_TFGL_INTEROP`, `BUILD_IBR_SEMANTIC_REFLECTIONS`) before running the configuration (see [Configuring](https://sibr.gitlabpages.inria.fr/docs/develop/index.html#sibr_configure_cmake)), then INSTALL everything (see [Compiling](https://sibr.gitlabpages.inria.fr/docs/develop/index.html#sibr_compile)).

88 89
See also the general installation [instructions](https://sibr.gitlabpages.inria.fr/docs/develop/index.html#install_sec) for *sibr*. 

90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208
## How to preprocess the dataset

All paths in this section are specified with respect to the dataset root `/path/to/dataset/`.

### Scene initial layout

Place your captured images in an `images/` directory.  

### Camera calibration and reconstruction
  
- Start Colmap. File > New project. 
- Create a new database at `colmap/database.db`. Select the `images` directory and save.

Perform cameras calibration:

- Processing > Feature extraction. If your images are fisheye-like (GoPro), select the OPENCV_FISHEYE camera model, else select the OPENCV model. If all images were taken with the same settings, you can check Shared for all images. Set the number of features to 20000.
- Processing > Feature matching. Exhaustive matching tab. Check "Guided matching". Depending on your VRAM you might want to decrease the block size.
- Reconstruction > Start reconstruction

Export the scene for RealityCapture texturing:

- Extras > Undistortion. CMP-MVS format. Export to `capreal/undistorted`.

Geometry reconstruction:

- Reconstruction > Dense reconstruction. Select `colmap/stereo/` as the workspace. Press Undistort.
- File > Export model as text. Select `colmap/stereo/sparse/` as an output. Check that the camera model in `colmap/stereo/sparse/cameras.txt` is PINHOLE, and that the given image size corresponds to the size of the files in `colmap/stereo/images/`.

If that's not the case:

- delete the `.txt` files in sparse (but keep the `.bin`)
- close and restart COLMAP
- re-open the project using File > New Project and selecting the existing `colmap/database.db` file and the images folder. 
- File > Import model from `colmap/stereo/sparse/`. 
- repeat the export step again choosing the same `colmap/stereo/sparse/` folder.
- you will also have to reselect the `colmap/stereo` workspace for the next steps

Dense reconstruction and Delaunay meshing:

- in Dense Reconstruction, press the Stereo button
- then the Fusion button
- and finally the Delaunay button

This will generate a `colmap/stereo/meshed_delaunay.ply` containing the geometry.

Using Meshlab, re-export `colmap/stereo/meshed_delaunay.ply` as `capreal/mesh.obj` and `capreal/mesh.ply` to fix the 'line endings' issue and get a RC-compatible mesh.

You should now have:

	- capreal/mesh.obj
	- capreal/undistorted/
	- colmap/
		stereo/
			images/
			sparse/
				cameras.txt
			stereo/


### Segmentation

> See section 3.1 of the article

Go to the `segmentation-network` repository on your machine, enable your virtual environment and run the segmentation network on the input images by calling:

	python inference.py path/to/dataset/

You should now have a `semantic_reflections/masks/` directory


### Geometry refinement

> See section 3.2 of the article

Run the shrinkwrap refinement (in `sibr/install/bin/`):
	
	./semanticReflectionsShrinkwrap_rwdi.exe --path path/to/dataset

You should now have a `semantic_reflections/shrinkwrap/` directory.

For each car, we need to perform alignment of the symmetrized mesh. Two options:

- Automatic ICP (in `sibr/install/bin/`):
	
		./semanticReflectionsDump_rwdi.exe --path path/to/dataset/ --icp

- Meshlab ICP (more efficient but cumbersome, their algorithm is more refined):  
	For each file named `smooth_mesh_XXX_final.ply` in `semantic_reflections/shrinkwrap/`:
		- Open Meshlab, import `smooth_mesh_XXX_final.ply` and `smooth_mesh_XXX_final_flip.ply`
		- Click on the Align button in the toolbar ("A" in a yellow circle)
		- Select `smooth_mesh_XXX_final` in the list on the right, click on "Glue here Mesh" and "Set as Base Mesh"
		- Select `smooth_mesh_XXX_final_flip` in the list, click on "Glue here Mesh"
		- Then click on Process and wait.
		- If the ICP fails, you can try to: 
			- Edit Defaults ICP parameters > Sample Number (increase)
			- Edit Defaults ICP parameters > Max Iteration Num (increase)
			- Edit Arc Creation parameters > Arc Area Thr. (decrease)
		- You can then close the Align tool
		- Right click the "flip" mesh in the list, "Freeze the current Matrix"
		- Export the new `smooth_mesh_XXX_final_flip` to `semantic_reflections/shrinkwrap/smooth_mesh_XXX_final_flip_aligned.ply`.
	
The next steps in the algorithm will use the Meshlab version if it's present and fallback to the automated version otherwise.
Generate the final mesh and additional data for the next steps (in `sibr/install/bin/`) .
	
	./semanticReflectionsDump_rwdi.exe --path path/to/dataset/ --geometry

You should now have a `capreal/mesh_final.obj` file.

### Object removal
We need to remove small leftover floaters from the global mesh:

- open the `capreal/mesh_final.obj` mesh in Meshlab
- select "Filters > Cleaning and Repairing > Remove isolated pieces wrt diameter".  
- set the "perc. on" field to 0.5, 
- click "Apply".   
- export the cleaned mesh to `capreal/mesh_final_cleaned.obj`

You should now have the cleaned mesh as `capreal/mesh_final_cleaned.obj` file.

209
### Retexture the mesh for Deep Blending
210

211 212 213 214 215 216 217 218 219
This will be needed for Deep Blending. If you don't want to use DB for the background, you can alternatively just copy `capreal/mesh_final_cleaned.obj` to `capreal/mesh_final_retextured_cleaned.obj`. Please refer to the `deep_blending` documentation for additional details on the simplification and texturing. Please note that the original datasets were textured using Reality Capture, using a functionality that is not available anymore.

From `sibr/install/bin/` run:

	python simplify_mesh.py -inputMesh path/to/dataset/capreal/mesh_final_cleaned.obj -outputMesh path/to/dataset/capreal/mesh_simplified.ply

	./unwrapMesh_rwdi.exe --path path/to/dataset/capreal/mesh_simplified.ply --output path/to/dataset/capreal/mesh_final_retextured_cleaned.obj --texture-name texture.png
    
	./textureMesh_rwdi.exe --path path/to/dataset/ --mesh path/to/dataset/capreal/mesh_final_retextured_cleaned.obj --output path/to/dataset/capreal/texture.png --size 8192 --flood
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301

You should now have the retextured full mesh as `capreal/mesh_final_retextured_cleaned.obj` file.
> Any other external unwrapping and texturing tool can be used.

### Min background generation

> See section 5 of the article, details on the background layer generation. It is performed before the dense fitting because of a legacy dependency between steps.

Run the following command to generate the interior min composites:

	./semanticReflectionsDump_rwdi.exe --path path/to/dataset/ --background

### Ellipsoid parameter estimation

> See section 4.1 and 4.2 of the article, and the supplemental document.

Run the features-based parameter estimation (in `sibr/install/bin/`):
	
	./semanticReflectionsFeatureMatcher_rwdi.exe --path path/to/dataset/

You should now have a `semantic_reflections/reflections/parameters_features.txt` file.

Run the dense parameter estimation (in `sibr`):

	./semanticReflectionsFitting_rwdi.exe --path path/to/dataset --ncount 25 --max_width 720

You should now have a `semantic_reflections/reflections/parameters.txt` file


### Layers separation

> See section 5.1 of the article

Generate flow data for the layers separation (in `sibr`) (slow, rendering heavy):

	./semanticReflectionsDump_rwdi.exe --path path/to/dataset/ --flows --ncount 25 [--max_width 1920 for crashy scenes, else don't specify]

Run the layers separation (memory heavy):

	./semanticReflectionsLayerSep_rwdi.exe --path path/to/dataset/


### Deep blending preprocess

> See section 6.1 of the article for details on our background rendering

Run the deep blending preprocess (in `deep_blending/preprocess`):

	python ibr_db_preprocess.py -i path/to/dataset/ -w 1920 -b -g 64 --bin path/to/sibr/install/bin/ -r

This will load `mesh_final_retextured_cleaned.obj` if it exists. Refer to the deep_blending documentation for additional details.

You should now have the `deep_blending` directory data generated.

### Final dataset layout

	- capreal 							Output of the texturing and mesh merging steps
	- colmap 							Output of the camera calibration and reconstruction process
	- deep_blending 					Output of the deep blending generation process
	- semantic_reflections 				Output of the reflection processing output
		- layers_bidir					Output of the flow data generation step
		- masks							Output of the segmentation step.
		- masks_per_window				Masks containing windows labeled with their geoemtric ID
		- masks_refined_parts 			Final window masks
		- min_backgrounds 				Interior min backgrounds, generated automatically.
		- min_foregrounds				Output of the layer separation step.
		- reflections/parameters.txt 	Output of the radii estimation step, WIN_ID X_RADIUS Y_RADIUS on each line.
		- shrinkwrap 					Output of the shrinkwrap step
	- clipping_planes.txt 				Input camera clipping planes, precomputed based on the geometry.


## Running

> See section 4.1 and 6.1 of the article

Run the viewer (in `sibr/install/bin`), you can lower the input data resolution for large scenes (`--max_width 1280`) and you can adjust the rendering resolution with `--width`, `--height`. You can use ULR instead of Deep Blending for the background by passing the `--nodb` argument.
	
	./SemanticReflectionsDemo_rwdi.exe --path path/to/dataset --max_width 1920 --width 1920 --height 1080 --model path/to/deep_model

Comparisons with the input textured mesh and the ULR method can be displayed directly in the app, by toggling them in the GUI.  
The final datasets can be found here: https://repo-sam.inria.fr/fungraph/ibr-cars-semantic/datasets/

302
## Training
303

304
To train Deep Blending on your new scenes and evaluate it offline, see the [DeepBlending page](@subpage semantic_reflectionsDeepPage) for this project.
305

RODRIGUEZ Simon's avatar
RODRIGUEZ Simon committed
306
To train the segmentation network, see the [Segmentation page](@subpage semantic_reflectionsLabelPage).