Projects | Peek-a-Boo: Occlusion Reasoning in Indoor Scenes With Plane Representations

We tackle the complex task of understanding 3D indoor scenes in the presence of occlusions. Our approach represents these scenes using a set of planes, each defined by its normal, offset, and two masks: one delineating the visible portion and the other encompassing the entire region, including both visible and occluded parts of the plane. We extract these planes from a single input image using a novel neural network architecture.

Our architecture features a two-branch category-specific module designed to separately predict the layout and objects within the scene. This separation allows us to handle different types of planes more effectively. Additionally, we introduce a unique loss function based on plane warping, which leverages multiple views during training to enhance occlusion-aware reasoning.

To train and evaluate our occlusion-aware model, we employ the ScanNet dataset. We also introduce two key components: first, a method for automatically extracting ground truth information for both visible and hidden regions, and second, a novel evaluation metric that specifically assesses predictions in the hidden regions. Our empirical results demonstrate that our proposed approach outperforms competitive baselines on the ScanNet dataset, achieving a remarkable 42.65% relative improvement in accuracy for hidden regions.