[3DGS] A Survey on 3D Gaussian Splatting

 

• title : A Survey on 3D Guassian Splatting
• journal : IEEE Transaction on patern analysis and machine intelligence
• date : 22 Feb, 2025
• Author : Guikun Chen et al.

https://arxiv.org/abs/2401.03890

A Survey on 3D Gaussian Splatting

 

+) If you're interested in original paper of 3D GS, refer to below link

2025.02.11 - [Computer Science/AI] - [3DGS] 3D Gaussian Splatting for Real-Time Radiance Field Rendering : Paper Review

[3DGS] 3D Gaussian Splatting for Real-Time Radiance Field Rendering : Paper Review

 

 

Overview

 

Chapters 2 to 3 were omitted because this paper review was written with a focus on utilization.

 

Introduction

// development of 3D Scene reconstruction

early : focuson light field, basic scene reconstruction
-> structure-from -motion
-> multi-view stereo algorithm
-> NeRF

[limitation] high cost, modification is hard.

=>  The advent of 3D Guassian Splatting

 

 

3D Guassian Splatting : Directions

1. 3D GS for sparse Input

issue 

emergence of artifacts

existing method

1.  regularzation : additional constraints (e.g.DNGaussian, FSGS, ...)
   [limit] few of views => low efficiency of regularization
2. generalizability 
   - additional view ( augmentation)
   - feed-forward : [limit] struggle to represent high-frequency details.

Future work

• overfitting ↔ using learned prior (control trade-off)
• extending these to dynamic scenes

 

2. Memory-efficient 3DGS

issue

optimizing both training and storage

existing method

1. reducing number of Guassians : strategic pruning, neighboring guassian
2. compressing Guassian's properties
   (e.g. Neidermayr et al : compressed color)

Future work

current : achieved significant storage reduction ratio

• during training phase, memory efficiency ↑
• compression efficiency ↔ visual fidelity

 

3. Photorealistic 3DGS

issue

rendering pipeline of 3DGS has many drawbacks
 => too simple visibility algorithm

existing method

three main aspects of visual quality

1. aliasing : discrete sampling paradigm
   - training time improvements :  multi-scale Guassian, 2D map filter)
   - inference-time improvements : 2D scale adaptive filtering
2. reflection :  achieving realistic rendering of reflective materals
   - model reflective materials
   - enable relightable
3. blur : motion blur, defocus blur
   - incorporated blur modeling during training
   - coarse-to-fine kernel optimization
   - photometric bundle adjustment

Future Work

current : address individual problems ( detecting issues, then apply targeted optimization)

• developing an advanced reconstruction system that overcomes these limitations

 

4. Improved Optimization Algorithms

issue

raw optimization method : overreconstruction in some regions

existing method

1. additional regularization 
   => Geometry-aware approaches have been particularly successful & preserving scene structure
2. optimization procedure enhancement (e.g. GuassianPro)
3. constraint relaxation
   => COLMAP-free approaches

Future work

current: challenging yet promising solution which reconstructs scenes in few-shot manner

• meta-learning strategies that combine scene-specific and general knowlede

 

5. 3D Guassian with more properties

original purpose of 3D GS : novel-view synthesis =>  revolutionize various domain

some applications

1. Language Embedded Scene Representation
   Shi et al : quantization scheme that augmented 3D Gaussian
   => enhanced the precision of open-vocbulary querying
2. Scene Understanding and Editing
   feature field distillation
   => faster training & rendering speeds
3. Spatiotemporal Modeling
   complex spatial & temporal dynamics of 3D

 

6. Hybrid Representation

1. Facial Expression Modeling
2. Spatiotemporal Modeling
3. Style Transfer

 

7. New Rendering Algorithm for 3D Gaussians

issue

inefficient handling of hightly-distored cameras + guassian don't overlap

existing method(recent work)

ray tracing based rendering algorithms (e.g. GaussianTracer)

Future work

provide essential blocks in reverse rendering, physical material modeling, relighting, and complex scene reconstruction

 

Application Areas And Task

 

1. Robotics

In SLAM(Simulaneous localization and mapping), GS-based method bypasses the need for axxiliary pose estimation
=>  enable rapid single-stage tasks

Main categories

• adaptability in dynamic and unstructured environments
• scene-specific optimization processes
• 3D guassians' inherent lack of spatial order complicates feature aggregation

 

 

2. Dynamic Scene Reconstruction

The key to adapt 3D GS to dynamic scenes is modeling of temporal dimension

Main categories

• utilize additional fields or model deformation
• scene changes can be encoded into the 3D Gaussian representation

Future work

current : fine-grained primitives limits scalability and robustness

• prioritize object-centric frameworks that hierarchically group gaussians into persistent entities.
• enable efficient large-scale reconstruction

 

 

3. Generation and editing

generation : enable the synthesis of novel digital content from scratch or conditional inputs

Current three main approaches in generation

1. optimization : distill diffusion priors to guide 3D model updates with the score functions
2. reconstruction : reframethe generation problem as a multi-view reconstruction task
3. direct 3D generation : train diffusion models on 3D representations

 

editing : provides the crucial ability to refine

Current two main approaches in editing

1. 2D image-editing
2. direct manipulation 

 

 

4. Avatar

avatar : digital representations of users in virtual space

• full body avatars : anchor 3D Guassian & deform them / cage-based rigging to model
• head avatars : use deformable 3D Gaussians

5. Endoscopic Scene Reconstruction

endoscopic field exists distinct challenges : frequent tool occulsions, narrow cavities, single-view geometry ambiguities

strategies for endoscopic scene

• holistic Gaussian initialization for dense intialization
• spatiotemporal Gaussian tracking for modeling surface dynamics

but, suffer from under-reconstruction

 

Future work

produce ill-posed reconstruction problems

 

 

6. Large-scale Scene Reconstruction

field : autonomous driving, aerial surveying, AR/VR, demanding both photorealistic visual quality and real-time rendering capablilties, and so on..

current challenges

requires an immense number of Gaussians to maintain visual qualtiy
=> high GPU memory demands & considerable computational burdens

 

Existing Method & Future work

• training : divide-and-conquer strategy, distrubuted implementation of 3D GS training
• rendering : adoption of the Level of Details, integrating extra input modalities 

 

 

7. Physics

simulation of complex real-world dynamics ( seed dispersal, fluid motion .. )

existing method

3D Guassian are treated as discrete particles 
=> have to unify diverse physical behaviors

 

 

Performance Comparison

Datasets for 3D GS

 

1. Localization

1) SplaTAM 
2) GS-SLAM
3) Point-SLAM

 

2. Static Scenes

1) Gaussian-SLAM

 

3. Dynamic Scenes

 

4. Human Avatar

1) GART

 

5. Surgical Scenes

1) HFGS

 

Future Research directions

• Physics- and Semantics-aware Scene Representation
• Learning Physical Priors froml Large-scale Data
• Modeling Internal Structures of Objects with 3D GS
• 3D GS for Simulation in Autonomous Driving and beyond
• Empowering 3D GS with More Possibilities