Deep image compositing

Deep image compositing is a recently emerged way of compositing and rendering digital images. In addition to the usual color and opacity channels a notion of depth is created. This allows multiple samples in the depth of the image to make up the final resulting color. This technique produces high quality results and removes artifacts around edges that could not be dealt with otherwise.

Deep data

Deep data is an image that holds additional information about each pixel along the z axis like multi-layered images where each layer has a depth associated with. But unlike voxels, only at some arbitrary specified points in depth. The data can be evaluated through different types of interpolation.

Function-based (integrated)

The data is stored as a function of depth. This results in a function curve that can be used to look up the data at any arbitrary depth. Manipulating the data is harder.

Sample-based (deintegrated)

Each sample is considered as an independent piece and can so be manipulated easily. To make sure the data is representing the right detail, an additional expand value needs to be introduced.

Generating deep data

3D renderers produce the necessary data as a part of the rendering pipeline. Samples are gathered in depth and then combined. The deep data can be written out before this happens and so is nothing new to the process. Generating deep data from camera data needs a proper depth map. This is used in a couple of cases but still not accurate enough for detailed representation. For basic holdout task this can be sufficient though.

Compositing deep data images

Deep images can be composited like regular images. The depth component makes it easier to determine the layering order. Traditionally this had to be input by the user. Deep images have that information for themselves and need no user input. Edge artifacts are reduced as transparent pixels have more data to work with.

History

Deep Images have been around in 3D rendering packages for quite a while now. The use of them for holdouts was first done at several VFX houses in shaders. Holdout mattes can be generated at render time. Using them in a more interactive manner was started recently by several companies, SideFX integrated it in their Houdini software and facilities like Industrial Light & Magic, DreamWorks Animation, Weta, AnimalLogic and DRD studios have implemented interactive solutions.

In 2014 the Academy of Motion Picture Arts and Sciences honored the technology with its annual SciTech awards. Dr. Peter Hillman for the long-term development and continued advancement of innovative, robust and complete toolsets for deep compositing and to Colin Doncaster, Johannes Saam, Areito Echevarria, Janne Kontkanen and Chris Cooper for the development, prototyping and promotion of technologies and workflows for deep compositing.

Resources

• Pixar Paper
• Deep Image Paper
• Deepimg.com
• Video tutorial of Deep Imaging as used on 2012 film Rise of the Planet of the Apes, Nuke compositing software
• Deep Image Training
• Deep Image File Format at Google Code
• Academy Award for the Technology