"SegmentThat" - Interactive Gaussian Splat Segmentation

 

 

 

3D Gaussian Splatting (3DGS) enables realistic rendering, but realistic rendering alone is not enough: We also need interactivity!

 

In the context of the Industrial Metaverse Reallabor project funded by the Ministry of Economics Baden-Württemberg, we investigate how extended reality (XR) technologies can support industrial use cases. One of our first exploring projects is the training of midwives in collaboration with the Labor and Delivery Unit at Tübingen University Hospital. For this, we use 3DGS to render realistic room environments in which trainees need to locate specific medications.

Point Clouds Gaussian Splats
↔
Gaussian Splats Point Clouds

For realistic training scenarios, users must be able to select, grab, and manipulate individual objects (e.g., open drawers and cabinets). However, 3DGS primarily represents scenes for rendering rather than object-level interaction. Enabling such interactions therefore requires a way to identify and isolate individual objects within the Gaussian representation.

 

How can we enable object-level interaction in 3D Gaussian Splatting scenes?

 

To address this, Dr. Sevinc Eroglu developed SegmentThat, a tool that provides an interactive workflow for segmenting and exporting individual objects directly from 3DGS scenes.

 

 

SegmentThat workflow and user interface, © 2026 Filmakademie Baden-Württemberg GmbH

SegmentThat workflow and user interface, © 2026 Filmakademie Baden-Württemberg GmbH
Hospital bed captured from multiple camera angles 1
Gaussian splats representing the hospital bed 2
Refined 3D segmentation of the hospital bed 3

Simplified visualisation of the steps (AI-supported images)

Multi-view mask generation

Through an interactive 2D interface, users select viewpoints that provide clear and complementary views of the target object. For each selected view, users identify the target object using text prompts, visual clicks, or a combination of both. SAM 3 uses this input to generate a 2D segmentation mask. Users can inspect the segmentation results and, if necessary, repeat the process or add more point prompts to refine the mask. This interactive verification helps prevent segmentation errors, such as missed or incorrectly included object regions, from propagating into the subsequent 3D segmentation.

Differentiable foreground attribution

Rather than directly back-projecting the 2D masks into 3D, differentiable rendering is used to estimate each Gaussian’s contribution to the rendered image and the masked foreground region. Gradients with respect to an auxiliary per-Gaussian signal are computed to quantify these contributions. The estimates are then aggregated across multiple views and normalized to derive a foreground confidence score for each Gaussian, which indicates its association with the target object.

Graph-cut refinement

To obtain a spatially coherent segmentation, graph-cut optimization is employed to combine the foreground confidence scores with relationships between neighbouring Gaussians. The confidence scores define the unary terms of the optimization. Pairwise terms encourage spatially neighboring and visually similar Gaussians to receive consistent labels, while preserving boundaries where the evidence indicates differences. The optimization produces the final 3D segmentation of the target object.

The segmentation results can be exported as separate PLY files, each containing an individual object with its pivot relocated to the origin. In addition, a binary file stores the object labels and associated information, which Unity can read to identify the corresponding objects and render them at their original positions in the scene. This provides a basis for object-level interaction in VR.

 

Segmented object in interactive application (Unity Game Engine),
© 2026 Filmakademie Baden-Württemberg GmbH

Beyond VR training scenarios, Gaussian Splatting is also being explored in filmmaking, particularly in virtual and hybrid production. By reconstructing real-world locations as 3DGS scenes, filmmakers can use these environments as virtual backdrops without needing to film on location. However, adapting a scene to a specific production often requires modifying its contents, such as removing or rearranging objects to create the desired setting. Our tool, SegmentThat, is therefore also being adapted for use at Carl Laemmle Institut of Filmakademie Baden-Wuerttemberg to support the design and modification of Gaussian Splatting environments for virtual production.

 

Our tool produces promising results for the interactive segmentation of individual objects in pre-trained 3DGS scenes. However, the segmentation is limited by the completeness of the underlying Gaussian representation. If parts of an object are missing from the reconstructed scene, they cannot be recovered through segmentation alone. Furthermore, removing an object can expose regions, such as the wall behind it, that are not represented by the remaining Gaussians. Future work will explore generative object completion and the reconstruction of missing scene regions to enable more comprehensive 3D scene editing.

 

Open-source implementation: https://github.com/FilmakademieRnd/SegmentThat

 

 

Note: This is an experimental, open-source project and has not yet been thoroughly tested or validated across a wide range of scenes and use cases. Feel free to use it as a starting point for integrating interactive 3D Gaussian Splatting segmentation into your own workflows. Adaptation and additional validation may be necessary.