Depth-Assisted Video Relighting in the Browser with WebGPU and WebGL

I added an editable video-relighting effect to Timeline Studio, my open-source browser video editor. It lets you move a virtual light around a scene, adjust its color and softness, and compare the result with the original while keeping the effect editable on the timeline.

The useful part is the division of work: a depth model analyzes the footage, then a small GPU shader handles lighting adjustments. Moving the light does not require another inference pass.

Try Timeline Studio · Explore the source

Light-position artwork from the editor. The interactive control uses a sphere to make light direction easier to understand.

From cinematic depth to editable lighting

The editor already used depth estimation for its cinematic depth-of-field effect and grayscale depth-map videos. Relighting builds on that same scene-depth information.

The pipeline is:

1. Decode sampled frames from the selected clip.
2. Run Depth Anything V2 Small locally through Transformers.js and a WebGPU inference path.
3. Normalize and refine the depth maps using source-image edges.
4. Blend neighboring depth samples at the current source time.
5. Render the original frame and the depth textures through a WebGL lighting shader.

WebGPU runs the model; WebGL renders the relighting effect. Keeping those responsibilities separate lets the expensive analysis be reused while lighting parameters remain interactive.

Turning a depth map into an approximate surface

A grayscale depth map tells us which visible regions are nearer or farther away. Differences between neighboring depth pixels provide an approximate surface orientation.

The shader estimates a normal from those depth gradients. In simplified GLSL:

vec3 normal = normalize(vec3(-dx * 0.12, -dy * 0.12, 1.0));
vec3 lightDirection = normalize(lightPosition - surface);
float diffuse = max(0.0, dot(normal, lightDirection));

The implementation combines that directional response with a spatial falloff controlled by the light's range and distance. Color and strength determine the added illumination; softness changes the response and the neighborhood used to estimate the surface.

It also approximates a rear-light rim using foreground depth discontinuities. A single camera view does not provide the hidden geometry behind the subject, so this is a screen-space silhouette estimate.

The shader adds a restrained fill to the existing image rather than replacing its original shading. It uses an approximate gamma-to-linear conversion before adding light, then converts the result back for display.

A sphere instead of mysterious coordinates

The inspector exposes a draggable light-position sphere, horizontal and vertical angles, distance, strength, color, range, and softness. Warm, neutral, and cool presets provide quick starting points.

Internally, the sphere's angles become a three-dimensional light position:

const azimuth = light.x * Math.PI;
const elevation = -light.y * Math.PI / 2;
const position = [
  Math.sin(azimuth) * Math.cos(elevation) * light.distance,
  Math.sin(elevation) * light.distance,
  Math.cos(azimuth) * Math.cos(elevation) * light.distance,
];

The effect also has a hold-to-view-original button. Comparison is transient preview state: it does not change the saved effect, create an undo entry, or disable lighting in an export.

The controls are localized in all 13 interface languages supported by the editor.

Video needs source-time alignment

A still image needs one depth result. Video needs the right result at the right source time, including after trimming or changing playback speed.

The analysis signature includes the source, trim information, duration, playback rate, speed curve, quality, and model revision. Changing the source-time mapping invalidates reuse of an incompatible analysis.

At playback time, neighboring depth maps are blended. This softens sample-to-sample steps, although it is not motion-compensated interpolation and cannot guarantee flicker-free results on every clip.

Depth refinement uses a small edge-guided smoothing pass. Source-image edges discourage mixing across boundaries, while a depth-difference gate helps preserve foreground/background separation.

The seek bug that mattered for editing

A lighting shader can be correct and still display the wrong video frame.

Seeking a video element is asynchronous. Updating the playhead time does not mean the target frame has finished decoding. Painting too early can leave the effect canvas showing the previous frame after a drag.

The preview now avoids painting while the source is seeking and refreshes after decoded-frame events. The main video also uses video-frame callbacks where available. That connects repainting to the actual media frame rather than relying only on a UI state update.

Preview and export both call the shared relighting renderer. They use the same lighting math, though resolution, decoding, and encoding can still affect the final appearance.

Try it on your own footage

1. Open Timeline Studio and add a video to the timeline.
2. Select the clip and open Effects → Video lighting.
3. Analyze the depth, then drag the light on the sphere.
4. Adjust strength and color first; use range, softness, and distance to refine the look.
5. Hold the original-comparison button to judge the change, then export when you are happy with it.

For this effect, footage and depth inference stay in the browser. The model files need to download, and the current analysis route requires WebGPU support. Speed depends on the browser, GPU, clip length, and analysis quality.

What this approach can and cannot do

This is an editable approximation based on visible scene depth. It does not reconstruct a complete 3D scene, recover material properties, remove baked-in shadows, or simulate physically accurate cast shadows and reflections.

Its practical role is adding a directional fill, a color accent, or a restrained edge-light impression to existing footage. Strong motion, difficult depth boundaries, and reflective or transparent surfaces can expose the approximation.

The implementation is open source. Start with the relighting renderer, depth rendering integration, and analysis hook.

This article was prepared with AI writing assistance and checked against the feature's source code.