When: Thursday 6th of March, 1:00pm AEDT
Where: This seminar will be partially presented at the ACFR seminar area, J04 lvl 2 (Rose St Building) and partially online via Zoom. RSVP
Speaker: Nikolai Goncharov
Title: Segment Anything in Light Fields for Real-Time Applications via Constrained Prompting
Abstract:
Segmented light field images can serve as a powerful representation in many computer vision tasks exploiting geometry and appearance of objects, such as object pose tracking. For those images, segmentation presents an additional objective of recognizing the same segment through all the views. Segment Anything Model 2 (SAM 2) allows for producing semantically meaningful segments for monocular images and videos. Using the video SAM 2 on less general 4D functions such as light fields is ineffective. In this work, we present a novel segmentation method that adapts SAM 2 to the light field domain without retraining or modifying the model. By utilizing epipolar constraints, our method produces high quality and view-consistent masks, outperforming the SAM 2 video tracking baseline and working 7 times faster, moving towards a real-time segmentation speed. We achieve this by exploiting the epipolar geometry cues to propagate the masks between the views, probing the SAM 2 latent space to estimate their occlusion, and further prompting SAM 2 for their refinement.
Bio:
Nikolai graduated from BSc Engineering, Moscow Institute of Electronics and Mathematics, Moscow, Russia in June 2020, and MSc Science, Skolkovo Institute of Science and Technology, Moscow, Russia in July 2022.