Abstract by David Hart
Style Transfer for Light Field Photography
For many years, light fields have been a unique way of capturing a scene. By using a special set of optics, a light field camera is able to, in a single moment, take images of the same scene from multiple perspectives. These perspectives can be used to calculate the scene geometry and allow for effects not possible with standard photographs, such as refocus and the creation of novel views.
Neural style transfer is the process of training a neural network to render photographs in the style of a particular painting or piece of art. This is a simple process for a single photograph, but naively applying style transfer to each view in a light field generates inconsistencies in coloring between views. Because of these inconsistencies, common light field effects break down.
We propose a style transfer method for light fields that maintains consistencies between different views of the scene. This is done with warping techniques based on the depth estimation of the scene. These warped images are then used to compare areas of similarity between views and incorporate differences into the loss function of the style transfer network. This is done without requiring a light field training set.