3D Gaussian Splatting (3DGS) enables realistic rendering, but realistic rendering alone is not enough: We also need interactivity!
In the context of the Industrial Metaverse Reallabor project funded by the Ministry of Economics Baden-Württemberg, we investigate how extended reality (XR) technologies can support industrial use cases. One of our first exploring projects is the training of midwives in collaboration with the Labor and Delivery Unit at Tübingen University Hospital. For this, we use 3DGS to render realistic room environments in which trainees need to locate specific medications.
For realistic training scenarios, users must be able to select, grab, and manipulate individual objects (e.g., open drawers and cabinets). However, 3DGS primarily represents scenes for rendering rather than object-level interaction. Enabling such interactions therefore requires a way to identify and isolate individual objects within the Gaussian representation.
How can we enable object-level interaction in 3D Gaussian Splatting scenes?
To address this, Dr. Sevinc Eroglu developed SegmentThat, a tool that provides an interactive workflow for segmenting and exporting individual objects directly from 3DGS scenes.

SegmentThat workflow and user interface, © 2026 Filmakademie Baden-Württemberg GmbH
The segmentation results can be exported as separate PLY files, each containing an individual object with its pivot relocated to the origin. In addition, a binary file stores the object labels and associated information, which Unity can read to identify the corresponding objects and render them at their original positions in the scene. This provides a basis for object-level interaction in VR.

Segmented object in interactive application (Unity Game Engine),
© 2026 Filmakademie Baden-Württemberg GmbH
Beyond VR training scenarios, Gaussian Splatting is also being explored in filmmaking, particularly in virtual and hybrid production. By reconstructing real-world locations as 3DGS scenes, filmmakers can use these environments as virtual backdrops without needing to film on location. However, adapting a scene to a specific production often requires modifying its contents, such as removing or rearranging objects to create the desired setting. Our tool, SegmentThat, is therefore also being adapted for use at Carl Laemmle Institut of Filmakademie Baden-Wuerttemberg to support the design and modification of Gaussian Splatting environments for virtual production.
Our tool produces promising results for the interactive segmentation of individual objects in pre-trained 3DGS scenes. However, the segmentation is limited by the completeness of the underlying Gaussian representation. If parts of an object are missing from the reconstructed scene, they cannot be recovered through segmentation alone. Furthermore, removing an object can expose regions, such as the wall behind it, that are not represented by the remaining Gaussians. Future work will explore generative object completion and the reconstruction of missing scene regions to enable more comprehensive 3D scene editing.
Open-source implementation: https://github.com/FilmakademieRnd/SegmentThat
Note: This is an experimental, open-source project and has not yet been thoroughly tested or validated across a wide range of scenes and use cases. Feel free to use it as a starting point for integrating interactive 3D Gaussian Splatting segmentation into your own workflows. Adaptation and additional validation may be necessary.