Multi-view relighting using a geometry-aware network {#outdoorRelightingPage}

Introduction

This Project contains the implementations of one paper:

[Philip et al. 19] Multi-view relighting using a geometry-aware network ｜Project Page

If you use the code, we would greatly appreciate it if you could cite the corresponding papers:

@Article{PGZED19,
  author       = "Philip, Julien and Gharbi, Micha{\"e}l and Zhou, Tinghui and Efros, Alexei and Drettakis, George",
  title        = "Multi-view Relighting Using a Geometry-Aware Network",
  journal      = "ACM Transactions on Graphics (SIGGRAPH Conference Proceedings)",
  number       = "4",
  volume       = "38",
  month        = "July",
  year         = "2019",
  url          = "http://www-sop.inria.fr/reves/Basilic/2019/PGZED19"
}

and of course the actual sibr system:

@misc{sibr2020,
   author       = "Bonopera, Sebastien and Hedman, Peter and Esnault, Jerome and Prakash, Siddhant and Rodriguez, Simon and Thonat, Theo and Benadel, Mehdi and Chaurasia, Gaurav and Philip, Julien and Drettakis, George",
   title        = "sibr: A System for Image Based Rendering",
   year         = "2020",
   url          = "https://sibr.gitlabpages.inria.fr/"

Authors

The code provided here is the original code developed by Julien Philip at inria under the overall supervision of George Drettakis.

What do we release?

For now we release the SIBR based code and application to run the our method on your own scenes. We release tests scenes used in the paper as well, that can be found on the project page. We also release the uncurrated python training code and hope to clean it and release the training data soon.

How to Install

The code has been developed for Windows (10), and we currently only support this platform. A Linux version will follow shortly.

This project is based on SIBR (Project Page) The first step is to clone SIBR_core. Then clone the tfgl_interop repo and outdoor_relighting (this repo) projects in src/projects of SIBR_core. See the sibr project documentation for more details.

Then run Cmake, making sure you check all three projects and also BUILD_IBR_ULR which is required to relight the output of an ULR rendering.

In VS2019, make sure the configuration is either release or releasewithdebuginfo. Build sibr_outdoor_relighting_app Build sibr_outdoor_relighting_app_install

See SIBR installation doc for more details on installation.

How to run a scene

1. Download a test scene from the Project Page 1b. Alternatively, create a dataset using reality capture you can follow SIBR documentation for this step LINK. /!\ color the vertices of your mesh
2. Launch the program! For an iteractive session use the following arguments:
   • --path path/to/sibr/scene
   • --interactive
   • --model path/to/core\sibr_core\src\projects\outdoor_relighting\renderer\network
3. The first time you run a scene you will to do two manual steps
   • First select the zenith position using two sliders. In the interface you will see an orthographic rendering of the mesh, this is the view from the light position. You want to put the camera so that you see the scene from the top.
   • Second select the input sun position. Using the same interface. Here you can also visualize the shadows live in the input images to help you. In the orthographic view you should not see any shadows when the direction is correctly set.

For some tests scenes the zenith and input sun direction are already provided. If you wish to set them yourself, you just have to delete the "outdoorRelighting" folder in the scene.

The UI contains 4 sliders:

• The first one select the input image to relight.
• The second one allows to display both outputs of the network (relit sunny and cloudy) plus the intermediate refined shadow masks.
• The Theta and Phi sliders control the target sun direction.

How to relight the output of ULR

1. Just add the following command line argument:
   • --ulr

Other arguments

You can setup the application to open on a specific image with the "--startIm 27", argument, which will start the app on the 27th image of the dataset. You can specify the output resolution with "--outH 720" or "--outW 1000", to specify an height of 720p (for instance) or a width of 1000p (for instance). Specifying both will overwrite the output ratio, which is by default equal to the one of the first image of the dataset. The --cloud flag allows the user to interpolate between the sunny and cloudy output.

Special case of internet based scenes (Manarola)

For the scenes that are created from a collection of images coming from the internet, the input sun direction is not the same over all the images. This is the case for instance of the manarola scene. Hence the input sun position selection must be done for a specific image in the dataset. Using the interface, make sure the image visualization correspond to the image you want to relight, if not change it with the image number slider.

For manarola, we provide the parameters used for the paper in the scene zip which correspond to the image 38.