DIST: Rendering Deep Implicit Signed Distance Function
with Differentiable Sphere Tracing


Shaohui Liu1,3      Yinda Zhang2      Songyou Peng1      Boxin Shi4      Marc Pollefeys1,5      Zhaopeng Cui1

1ETH Zurich         2Google         3Tsinghua University         4Peking University         5Microsoft




Abstract

We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward pass of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backward to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noise.



Download



Video




More Results


Texture Re-rendering
Shape Completion from Sparse Depths

Inverse Optimization over Camera Extrinsics

Shape Completion from Depth Image with Different Sparsity

Multi-view Reconstruction from Video Sequences
Synthetic data
Real data