Can You Turn a Moving Person Into 3D Video? — What 4D Gaussian Splatting Actually Needs

2026年8月20日

4D Gaussian Splatting (4DGS) is supposed to let you record a moving person along with the space around them. I looked into whether my own RTX 3090 can run it. My usual setup is an Ubuntu PC with an RTX 3090 and an RTX 3060 in it, which I use for local LLM and image generation work.

In an earlier article I covered photogrammetry: building a 3D model of a stationary object from photos taken on a phone. But that is strictly about things that hold still. Recording a person in motion — someone playing an instrument, dancing, a child at play — into a 3D space you can look around afterwards from any angle was fundamentally impossible with photogrammetry. If the subject moves during the shoot, the model falls apart.

In 2026 the technology to get past that wall has finally come close to practical. 4D Gaussian Splatting (4DGS) adds a time axis to 3D Gaussian Splatting, letting you reconstruct a moving scene as free-viewpoint video.

This article is a record of looking into whether 4DGS is feasible on the hardware I have. I have not shot or processed anything yet. What I did was work out what it takes and where the walls are, from what is publicly documented.

How far can turning a moving person into 3D video actually be pushed on a personal setup?

The previous article in this series:

3DGS (static) against 4DGS (dynamic)

3D Gaussian Splatting was presented at SIGGRAPH in 2023 and spread rapidly from 2024. It builds photorealistic 3D representations of static scenes from photos or video. 4DGS adds the time axis, so it can handle a 3D scene with movement in it.

Item	3DGS (static)	4DGS (dynamic)
Input	20–50 photos, or video	Synchronised multi-view video (one camera also possible)
Output	A static 3D scene	3D video with a time axis
Subjects	Buildings, rooms, still life	People moving, live action, moving objects
Processing (30 seconds of footage)	10–30 minutes	1 hour to half a day
GPU	RTX 3060 or better (12GB VRAM, the working memory on the GPU)	RTX 3090 recommended (24GB VRAM)
Playback	Web browser, VR headset	Dedicated viewer, WebXR

In 3DGS, photos taken from different angles at different moments are all treated as different views of the same static scene. The assumption is that the subject does not move as time passes.

In 4DGS, each frame represents “the 3D scene at a given instant." You reconstruct a 3D scene from the frames at time t, another from the frames at t+1, and that sequence is the moving 3D footage.

The maths of 3D Gaussian Splatting, intuitively

To follow 4DGS you first need how 3DGS works underneath. There are formulas, but I will say what each one is doing.

Representing space with Gaussian distributions

Conventional 3D models represent things as a collection of triangles (a mesh). 3DGS takes a completely different approach: it places countless “blurry balls of light" in 3D space and represents the scene as their superposition.

A single 3D Gaussian carries these parameters.

Parameter	Symbol	What it means intuitively
Position	μ (mu)	Where the centre of the ball is (x, y, z)
Covariance matrix	Σ (sigma)	Its size and orientation (which way and how far it is stretched into an ellipsoid)
Opacity	α (alpha)	How solid it is (transparent through to fully opaque)
Colour	SH coefficients	Colour that changes with viewing angle (expressed with spherical harmonics)

The influence of a given Gaussian at an arbitrary point x in 3D space is computed as:

G(x) = exp( -1/2 · (x - μ)ᵀ · Σ⁻¹ · (x - μ) )

Rendering: how the colour seen from the camera is computed

Rendering in 3DGS composites the Gaussians along the camera’s line of sight, front to back. The colour C of each pixel is:

C = Σᵢ cᵢ · αᵢ · Tᵢ    where Tᵢ = Π(j<i) (1 - αⱼ)

This computation parallelises well, so a GPU handles it quickly. That is precisely why 3DGS can render in real time (30fps and above).

Extending to 4DGS: adding the time axis

In 3DGS each Gaussian’s parameters (μ, Σ, α, colour) are fixed. 4DGS makes them functions of time t.

μ(t) = μ₀ + Δμ(t)    ← position changes over time
Σ(t) = Σ₀ + ΔΣ(t)    ← shape changes over time
α(t) = α₀ + Δα(t)    ← opacity changes over time

Several ways of expressing that time variation have been proposed.

Method	How time is represented	Advantage	Disadvantage
Deformable 3DGS	A deformation MLP predicts each Gaussian’s displacement	Smooth motion, memory efficient	Weak on sudden movement
4D Gaussian (CVPR 2024)	A 4D Gaussian distribution (3D space + 1D time)	Theoretically elegant, high quality	Computationally expensive
Per-frame optimisation	Train an independent 3DGS for each frame	Simple to implement	No consistency between frames

The mainstream today is the Deformable 3DGS family.

One phone and a cloud service — still only a dream

This was the route I had most hope for when I started looking. Upload a phone video, let a cloud service turn it into 4DGS. If that worked at no extra cost, there would be no better way in.

What was actually announced

The concept is that walking around a subject with a phone would be enough: the cloud converts the footage to 4DGS automatically, in tens of seconds, handling camera pose estimation (SfM) and 4DGS training in one pass. Up to an hour of material per run is mentioned. No date is given.

The viewer, however, is real

The same company released a free 4DGS viewer in October 2025. That one is not a concept. It runs in the browser on WebGL/WebGPU, on Windows, Mac, Linux and mobile.

You still have to generate the data yourself — but the means to show it already exists.

Intermediate: several phones and your own processing

Why more than one camera

With one phone you have to walk around the subject while filming, which causes two problems.

1. The operator appears in shot. Walking around the subject means the operator is moving too, and any other camera may catch them
2. You never have all directions at the same instant. At t=0 you have the front, at t=5 seconds the side, at t=10 the back. The same moment is never recorded from multiple angles

Fix several phones in place and film simultaneously, and you get multiple viewpoints of the same instant. That improves 4DGS quality substantially.

What you need

Item	Quantity	Rough cost	Notes
Phones	3–6	Ones you have, or used	1080p/30fps or better. Mixed models are fine
Tripods (cheap phone ones)	3–6	$3–7 each	The flexible clip type from a discount store is enough
PC	1	One you have (RTX 3090 recommended)	24GB of VRAM gives you room
ffmpeg	—	Free	Frame extraction and audio sync
COLMAP	—	Free	Camera calibration
A 4DGS implementation	—	Free (published on GitHub)	Training and rendering

Discount-store phone tripods with a clip are perfectly adequate. Three of them come to something like $10–20.

Synchronising: clap your hands

Professional production uses timecode sync, which phones cannot do. The practical substitute is clapping at the start of the take.

1. Start recording on every phone
2. Clap sharply in front of the subject
3. Shoot the take
4. Stop recording

Afterwards, analyse each video’s audio waveform in ffmpeg, find the peak of the clap, and align on that. This gets you frame-level accuracy (1/30 second, about 33ms).

# Detecting the clap position with ffmpeg
ffmpeg -i camera1.mp4 -af "silencedetect=noise=-30dB:d=0.1" -f null -

The workflow

1. Shoot: mount the phones on tripods and position them, start all recordings, clap, subject performs, stop recording
2. Extract frames: pull frames from each video with ffmpeg (5fps is a good target; 30fps is far too heavy to process)
3. Audio sync: align frame timings on the clap
4. COLMAP: estimate each camera’s position and intrinsics
5. 4DGS training: train the model on the synchronised frames
6. Check in a viewer: play the trained model in a dedicated or web viewer

Example camera layouts

Three cameras:

Front plus 60 degrees each side, in a fan around the subject. That covers about 120 degrees of the front. The back cannot be captured, so this assumes the subject is mostly facing the cameras.

Six cameras:

Six cameras at even 60-degree spacing around the subject. That covers the full 360 degrees, so there are far fewer holes whichever way you look. The obstacle is finding six phones; borrowing from family and friends is the realistic answer.

The serious tier: three mirrorless bodies

The cost, stated plainly first: bodies that shoot 5K or better run about $1,300 each, so three come to roughly $4,000. Unless cameras are already your trade, that is not a sum to spend on 4DGS alone. Treat this tier as an option for people who happen to own several bodies.

Three mirrorless bodies that shoot 5K or better come up as the highest-quality route within an individual’s reach. The input carries far more information than a phone’s 1080p.

What 5.7K buys you

• Detail: patterns in clothing, skin texture, the wood grain on an instrument. Things that collapse at 1080p
• Depth at distance: you can stand back from the subject and keep enough resolution. With a phone you have to get close, and lens distortion becomes a problem
• Room to crop: shoot 5.7K and you can still resize to 1080p or 2K later with information to spare, choosing resolution against processing time

Timecode sync (frame accurate)

Some bodies carry TC IN/OUT timecode ports. With BNC adapters you can synchronise timecode across three bodies and align the footage to the frame (1/24 second, about 42ms). That is an order of magnitude better than clap sync on phones.

The advantage of timecode is that drift does not accumulate over a long take. With audio sync on phones, small differences in frame rate can widen the gap as the recording goes on. Beyond about five minutes, timecode sync is effectively mandatory.

Camera placement: front plus 15 degrees each side

With three mirrorless bodies, a narrow fan of front plus 15 degrees each side looks like the sensible layout.

This is the exact opposite of aiming for 360 degrees with six phones. Concentrating three cameras at close angles reproduces “the parallax you get moving your head a little in the front row." For recording a concert or a dance, this layout produces immersive 6DoF footage (video where you can move back and forth, side to side, up and down, and change where you are looking).

Mixing focal lengths

Putting different lenses on the three bodies (25mm, 35mm, 50mm, say) is not a problem. COLMAP calibrates each camera’s intrinsics (focal length, distortion coefficients) individually. That said, wildly different fields of view reduce the overlap and hurt quality, so keeping the focal lengths close is more stable.

Resize before processing

Training 4DGS on full 5.7K makes the processing time balloon: even 30 seconds of footage looks like eight hours or more. In practice I would strongly recommend resizing to 1080p–2K with ffmpeg first.

# Resizing to 2K with ffmpeg
ffmpeg -i input_5.7k.mp4 -vf scale=2048:-1 -c:v libx264 -crf 18 output_2k.mp4

The point of shooting 5.7K is recording the maximum amount of information. You can resize down whenever you like; you cannot make low-resolution footage detailed after the fact.

How long does it take?

The first thing anyone wants to know about 4DGS is the processing time. Here are estimates by resolution and by clip length, all assuming an RTX 3090 (24GB VRAM).

By resolution (30-second clip, 5fps extraction, RTX 3090)

Input resolution	COLMAP	4DGS training	Total	VRAM
1080p	10–20 min	20–40 min	about 1 hour	8–12GB
2K	20–40 min	40–80 min	about 2 hours	12–16GB
4K	40–90 min	1–3 hours	about 4 hours	16–24GB
5.7K	1–2 hours	3–6 hours+	about 8 hours	24GB+ (right at the edge)

By clip length (1080p, 5fps extraction, RTX 3090)

Length	Frames	COLMAP	4DGS training	Total	Output size
5 seconds	25	3–5 min	5–10 min	about 15 min	tens of MB
30 seconds	150	10–20 min	30–60 min	about 1 hour	100–300MB
1 minute	300	20–40 min	1–2 hours	about 2.5 hours	300–600MB
5 minutes	1,500	1–2 hours	5–10 hours	about half a day	1–3GB
10 minutes	3,000	3–5 hours	12–24 hours	about a day	3–6GB

[kimono_bar title="VRAM by resolution (30-second clip, RTX 3090)" unit="GB" color="#e53935″ highlight="4″ max="28″] 1080p|10 2K|14 4K|20 5.7K|24 [/kimono_bar] [kimono_bar title="Total processing time by resolution (30-second clip, RTX 3090)" unit="hours" color="#1976d2″ highlight="1″ max="10″] 1080p|1 2K|2 4K|4 5.7K|8 [/kimono_bar]

How VRAM and time relate

4DGS processing time and VRAM usage scale roughly with input resolution and clip length. Organising the published figures, the relationship reads approximately as:

VRAM ≈ baseline(8GB) x (resolution/1080p)² x camera-count correction
Time  ≈ baseline(1h) x (resolution/1080p)² x (length/30s) x camera count

How to read these tables

Start with a five-second clip. If the estimates hold, you have a result in about 15 minutes, which lets you iterate quickly on settings and camera placement.

A caveat on the timings: these estimates are extrapolated from the 4DGS paper (CVPR 2024, about 8 minutes for a 5-second clip on an NVIDIA A6000). The RTX 3090 differs from the A6000 in VRAM (24GB against 48GB) and compute, so real times will differ, and scene complexity (how much motion, how fine the texture) and resolution move them a great deal. Measured figures on the RTX 3090 will be added once I have the environment built.

Around an hour for 30 seconds is what you should brace for. If one attempt takes an hour, changing parameters and redoing it five times is five hours. It is more efficient to tune on a short clip first and then commit.

Anything over five minutes assumes leaving it overnight. Start training before bed, check in the morning.

Training 4DGS at full 5.7K appears, from reported figures, to use up essentially all 24GB even on an RTX 3090. There is a real risk of running out of memory, so resizing to 1080p–2K is the practical route.

Cloud service against doing it yourself

Item	Letting a cloud do it	Your own machine (RTX 3090)
Processing time	20–30 minutes	1 hour to half a day
Cost	Service fee (check current terms)	Electricity only (about 0.35kW on an RTX 3090)
Maximum resolution	High resolution discouraged	Whatever your VRAM allows (24GB gets you to 4K)
Where the data goes	Uploaded to the cloud	Stays entirely local
Customisation	None (fixed service settings)	Parameters adjustable (learning rate, iterations, etc.)
Reproducibility	Depends on the service’s version	Code and model stay in your hands

The biggest advantage of doing it yourself is that the data never leaves. For footage of children, or performers whose consent is involved, if uploading gives you pause then local processing is the only option.

Conversely, if you want a result quickly and have no GPU, a cloud service is the sensible choice.

Where do you watch it? Playback platforms compared

Making a 4DGS is pointless without somewhere to play it. Here are the main platforms as of April 2026.

Platform	Device	3DGS	4DGS	Cost	Notes
Free web viewer	Browser	Yes	Best	Free	Streaming playback of 4DGS generated on the service
BDViewer	Quest 3	Best	Limited	Free	High-quality 3DGS playback. 4DGS support is experimental
Scaniverse	Quest 3/3S	Best	No	Free	Developed by Niantic. Focused on viewing 3DGS
WebXR (Gracia AI)	Quest / Pico / Vision Pro	Best	Best	Enquire	Supports various headsets over WebXR. Strongly business-oriented
Your own Unity build	PCVR / Quest	Best	Best	Free (your development time)	Maximum freedom, requires development skills

From what I found, the current pattern is to try it easily in the browser first, and build something in Unity if you want to see it in VR. Native 4DGS viewers for the Quest 3 were not mature as of April 2026, and BDViewer’s 4DGS support is at an experimental stage.

Gracia AI’s WebXR is mainly for business use, but technically its strength is supporting Quest, Pico and Vision Pro through a web browser. If platforms like that open up to individuals, device-agnostic 4DGS playback may become ordinary.

Cost against quality

Here is how the additional cost and the resulting quality line up.

Approach	Additional cost	Quality (out of 10)	Notes
3 phones + cheap tripods + RTX 3090 (owned)	about $10	6	Tripods only. The best value by far
6 phones + tripods + RTX 3090	about $20	7	360° coverage, fewer holes
3x mirrorless + RTX 3090	about $4,000	8	Roughly $1,300 a body, three bodies. $0 if you already own three
Professional studio (20+ cameras)	$3,300+	10	Broadcast and film quality

Technical limits

4DGS is not magic. It is worth knowing the main constraints as they stand in 2026.

There is a ceiling on how fast things can move

Frames with heavy motion blur cannot be handled correctly during training. At a shutter speed of 1/60 second, a subject that moves a lot within that window produces a blurred frame, and feature-point estimation degrades.

• Playing an instrument, singing: good. Little torso movement, and hand motion is relatively predictable
• Slow dance forms (classical Japanese dance, tai chi): good. Smooth motion, unlikely to break down
• Energetic dance (breaking, hip-hop): marginal. Fast limbs produce ghosting
• Sport (basketball, badminton): marginal to impossible. Ball and racket speeds exceed the shutter

The countermeasure is a faster shutter (1/250 or better), which means it gets darker and you need lighting.

Hair, transparent objects and reflective surfaces are hard

This weakness is inherited from 3DGS. Fine structures like hair, transparent things like glass and water, and reflective metal are all difficult for Gaussian Splatting to represent. It is not specific to 4DGS, but with a moving subject you add hair in motion, which makes it harder still than in a static scene.

Papers improving hair (GaussianHair and others) appeared in 2026, but the implementations are not yet at a stage where they are widely usable.

It is sensitive to changing light

Flashing stage spotlights, or sunlight shifting through a window, will substantially degrade quality. Gaussian Splatting learns “the colour of each Gaussian," and when the lighting changes that colour, consistency breaks down.

Even, stable lighting is the ideal.

Camera count and blind spots

With few cameras, the angles you did not record are literally holes. Three cameras covering 120 degrees at the front leaves 240 degrees behind with no information. The algorithm fills in to a degree, but quality at unseen angles clearly drops.

Scale in VR

There are reports that people appear the wrong size when 4DGS output is viewed in a VR headset. Without accurate camera calibration a subject can look like a giant or a doll. Building this into VR content in Unity or similar means matching against real-world scale.

What is 4DGS actually for?

Beyond being technically interesting, where does it help? Five uses that are visible today.

1. Recording a child growing up, spatially

Photos and video only ever let you look back from the angle you were standing at. With 4DGS you can revisit a child walking from any angle you like. Five or ten years later, stepping back into “that space" in a headset is qualitatively different from a flat video.

Three phones and cheap tripods is enough to start, so the cost barrier is low. If uploading footage of your children to a cloud service bothers you, local processing is the reassuring choice.

2. Checking form in dance and sport

You can see your own movement from the side and behind, not only head-on. For dance practice it closes the gap between “me in the mirror" and “me as others see me." In coaching, showing an athlete a view from over their shoulder communicates things that ordinary video analysis struggles with.

3. Property viewings with people in them

Static 3D tours built with Matterport or Polycam are increasingly common, but they cannot show a space with people living in it. 4DGS can turn a living room with a family relaxing in it into a virtual tour. For property viewing, conveying “what it feels like with someone actually living here" counts for a lot.

4. Recording live events

Record a gig, a school show or an event in 4DGS and you can revisit it not only from the seat you were in but from on stage, or from directly beside the performer. Consideration for performers’ image and publicity rights is essential here. Converting someone to 4DGS and publishing it without permission is more sensitive than ordinary filming, because 3D data includes the geometry of their face. Always get consent in advance.

5. Reference material for avatars

Motion data of a person generated in 4DGS can serve as reference for avatar animation in social VR. Using 4DGS data directly as an avatar is hard today (it needs converting to a mesh), but as a movement reference it is useful. If real-time streaming of 4DGS data matures, “the real you, in a VR space" comes into view.

In summary: is it feasible?

From what I found, the answer is that it looks feasible — though the assumptions change at each tier.

• One phone and a cloud service: experience 4DGS at no extra cost. Start here
• Three to six phones and your own processing: $10–20 of tripods raises the quality a step. With an RTX 3090 it stays entirely local
• Three mirrorless bodies with timecode sync: 5K or more of information and frame-accurate sync, approaching professional quality. About $4,000 if bought new, so this one is for people who already own them

Being able to climb this in stages is what is good about 4DGS. There is a zero-cost entry point, so you can confirm it works at all before adding hardware.

Starting with a five-second clip looks like the right move; if the estimates hold you get a result in about 15 minutes. When that works, stretch to 30 seconds. When 30 seconds is stable, go to a minute, then five. Try to process a full five-minute take from the start and a failure costs you half a day. Tune on short clips: that appears to be the golden rule of 4DGS.

Just as 3D scanning of static objects became an ordinary phone app, recording moving people in 3D looks set to become something anyone can do within a few years.

What I took away from looking into it

The surprise was that the heaviest part — the processing — is already covered by what I have. The 24GB on an RTX 3090 works out as enough through the intermediate tier.

What is missing is only the camera side: a few phones, and something to hold them still. That is a matter of a few thousand yen.

Going this route, it looks like I can do it with the equipment I already own. That is where the research lands. What remains is to actually shoot something — and when I do, I will write that up too.

Sources

• 4DGS.jp announcement (PR TIMES, 1 April 2026 — April Dream)
• April Dream — what the programme is
• IZUTSUYA — free 4DGS viewer (October 2025)
• Realize Innovations — 4DGS volumetric video studio (December 2025)

Hardware mentioned

[kimono_product id="16559″]