LlamaCasty/TASKS/task-51-webcam-bg-effects.md

TASK 51 — Webcam background effects: Remove + Blur (v1 requirement)

Catalog: TASKS.md — status and requirements live here. Booked 2026-10-08 by creator ruling — "we're doing the bg removal and blur for v1 - book it, Danno". This reverses the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).

Status: ✅ Built 2026-10-09 — all five slices shipped (engine, Blur, config + persistence, camera hookup, UI). Creator's real-camera check done 2026-10-09 and both findings addressed: 🎯 Remove "jittery" — FIXED (temporal mask smoothing, MaskSmoother — 0.75×new + 0.25×previous momentum, weights from the OBS/Jitsi consensus); 🎯 Blur "heavy" — FIXED (quadratic radius curve, max 12→6 small px, default 0.5 = 2 — StreamFX-consensus scale). Pending: creator's recheck of both on a real camera.

Goal: per-webcam Background Remove and Background Blur on the live webcam layer, selected per element in its properties panel, composited into stream + recording like any other layer. No external downloads, no hardware requirement, works on any machine. Replacement backgrounds are free via layers — drop any element behind the webcam.

---

Why this is now IN (and what changed)

The 2026-10-05 exclusion was based on: an in-app AI model = a runtime, a GPU/driver matrix, per-scene lighting tuning, and "a vcam app does it free." The ruling is reversed because the delivery shape changed:

1. Bundled small ONNX model (~2–5 MB) — MediaPipe Selfie-Segmentation family — shipped in the installer/fixture (TASK 48 MSIX), executed by onnxruntime (CPU default; DirectML optional later). No NVIDIA SDK, no GPU requirement, no per-client download.
2. Quality reality (creator): a 720p webcam displays at ~15% of a 1080p frame — premium segmentation quality is invisible at that size; commodity class is sufficient.
3. ⛔ Ruled out: NVIDIA Video Effects SDK (VFX / Maxine, AI Green Screen + Background Blur). Works on RTX 2000+/R570+ only, and needs a ~1 GB user-side redistributable on first use. Creator: "I don't think I'm putting a 1gb download requirement on a client box."

Constraints — non-negotiable

1. No external download at runtime. Model ships in the installer. (Store policy 10.2.2 — same posture as the ffmpeg bundling in TASK 48 item 1.)
2. No hardware requirement. CPU path must work; DirectML is an optional accelerator.
3. Per-element (Meld-style), not global. Each webcam element carries its own effect.
4. No feature gating. Background effects are in the free tier (2026-10-31 freemium keeps every feature free; the paid delta is output quality only — docs/ai/commercial.md). The old "only the branding flash is paid" line here was superseded in 2026-10-08, and the flash delta itself died on 2026-10-30.
5. Scene change / webcam add/remove must not crash or stall the pump — a failing model run skips to passthrough, never drops the layer.

Out of scope for v1 (same engine later)

• ❌ "Remove chair"-style object selection / quality tiers — not a picker engine
• ❌ Super resolution / denoise / prettify passes
• ❌ Virtual camera output — permanently out of product (no vcam)

1. Engine

• ✅ Model bundled (slice 1, 2026-10-09): Google's official MediaPipe Selfie Segmenter landscape (144×256, Apache-2.0 per the model card — license + changes recorded in THIRD-PARTY-NOTICES item 12). Converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op Convolution2DTransposeBias → ConvTranspose); verified against MediaPipe's own runtime by tools/selfie-seg/verify_against_mediapipe.py (max |diff| 5e-5). 430 KB, EmbeddedResource in the exe. Rejected sources: PINTO #109 (ML Kit-extracted, unclear license), obs-backgroundremoval code (GPL-3.0) — referenced for approach only (ONNX Runtime, CPU, 2 threads).
• ✅ Microsoft.ML.OnnxRuntime 1.31.0, CPU EP (OnnxPersonSegmenter). Failing load/run → logs once, returns null, caller passes through. DirectML still optional/later.
• ✅ Blur (slice 2, 2026-10-09): background box-averaged to ¼ res, 3 separable box-blur passes (≈ gaussian, radius by strength), mask-blended back; person pixels untouched (BackgroundEffectProcessor.ApplyBlur).
• ✅ Remove = alpha = mask outside the person (BackgroundEffectProcessor), IsOpaque = false. The compositor already alpha-blends per element — no compositor change expected (confirm in the hookup slice).
• ☐ Feed the VideoFrame seam (Services/VideoFrame.cs doc explicitly anticipates "every future capture source (screen, background-removed webcam) feeds the same seam"): the frame handed to BOTH the preview bitmap path and the compositor's _cameraManager.GetLatestFrame is the processed frame. One processing point, two consumers.

2. ☐ Config surface + persistence

• ✅ WebcamSceneConfig gains BackgroundEffect (None | Remove | Blur) + BackgroundStrength (slice 3).
• ✅ Persist via the existing WebcamSceneConfig table path in LayoutStore.Save.cs/ Load.cs (new columns + migration). Follow LayoutStore.Migrations.cs conventions.
• ✅ Properties UI (creator spec 2026-10-09, slice 5): in the webcam's properties in the Layers panel, two pill switches — "Remove background" and "Blur background" — using the existing PillToggle style (Themes/Controls.xaml, already used in Controls/LeftPanel.xaml). They act as radio buttons: turning one on turns the other off; at most one is active; both off = None. Bound to the element's BackgroundEffect, per webcam element. (Supersedes the old "effect dropdown" idea.) Plus a strength slider (creator: "keep", 2026-10-09) — Blur radius / Remove edge softness via Strength.
• ☐ Replace-background affordance: none in v1 — the creator drops any layer behind the webcam. Effect preview must not obscure the fact that layers below still show through.

3. ☐ Test seam (Good Dog — one integration test per unit)

• ✅ Model-runner interface seam (IPersonSegmenter) so tests inject a fake segmenter.
• ✅ The processed frame reaches both consumers (preview bitmap + compositor frame), raw stays raw (CameraManagerTests, slice 4).
• ✅ None = same-instance passthrough of the raw frame (BackgroundEffectEngineTests).
• ✅ Remove = alpha 0 outside the person, RGB untouched, non-opaque (BackgroundEffectEngineTests); ✅ Blur = background smoothed, person pixel-identical (BackgroundEffectEngineTests).
• ✅ Bundled-model contract: the embedded ONNX model through the C# preprocessing reproduces MediaPipe's reference mask on a fixture (≥99.5% agreement). This is a fixed, deterministic contract check of the shipped bytes — not a camera/lighting quality test.
• ✅ Config → persistence roundtrip per scene + legacy-DB migration (LayoutStorePersistenceTests).
• ☐ Engine failure → passthrough + single log, layer never dropped (code path exists; dedicated test lands with the hookup slice).
• ☐ Real inference: manual on the creator's box (camera + lighting), not in CI.
• ✅ Perf, synthetic 720p (Debug): segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms (after making the upsampling separable — naive was 46/80 ms). Runs on its own worker, off the raw path. ☐ Real numbers on the creator's box: startup.log → "background effects avg … ms/frame".

4. Sequencing

• ✅ Separate work unit after the zoom build (zoom shipped 2026-10-08).
• Slice order: 1 engine ✅ → 2 Blur ✅ → 3 config + persistence ✅ → 4 camera-path hookup ✅ (one processing point, both consumers) + perf check → 5 properties-panel UI ✅.

5. Creator's real-camera check (2026-10-09) — jitter FIXED, blur heavy deferred

• 🎯 Finding: "removal is jittery" (the edge flickers frame to frame). Root cause: the segmenter classifies each frame independently — 256×144 mask, bilinear upsample — zero temporal coherence. Fix shipped: MaskSmoother blends each raw mask with the previous smoothed one (CurrentWeight = 0.75 × new + 0.25 × previous), wired at the ONE production point (CameraManager.RunEffectWorker, per CameraSession). Why per-session and not in the segmenter: the OnnxPersonSegmenter is shared app-wide (MainViewModel:284-285, one processor), and the bundled-model contract test pins its raw per-frame output. Why 0.75/0.25: OBS's obs-backgroundremoval smoke-tests temporal_smooth_factor at 0.85 default; Jitsi's webcam background filter (jitsi/jitsi-meet#17080) found 0.7/0.3 best (0.5/0.5 smears on motion). Null mask resets history; first frame passes through; no per-frame allocation (two ping-pong buffers — a 256×144 mask is 147 KB > the 85 KB LOH). Integration test: CameraManagerTests.Temporal_Smoothing_… (two consecutive masks disagree at an edge → second frame keeps partial alpha where raw would cut).
• 🎯 Finding: "blur is heavy" — the OLD curve was linear 1→12 downscaled px (≈ 26–50 full-res px), far past any moderate background blur. Fix shipped: quadratic radius — BackgroundEffectProcessor.BlurRadius(strength) = 1 + 5·s²: default 0.5 = 2 small px (≈ a third of the old default's spread), max = 6 small px (≈ 24 full px). Why these numbers: StreamFX defaults its Box blur Size to 5 and its author stops downscale-based blurs around 8 px (cited). Integration test BackgroundEffectEngineTests.Blur_default_strength_is_gentle_and_full_strength_is_strong pins the mapping (BlurRadius(0)=1, 0.5→2, 1→6) and the step-edge behaviour: default smears light ~9px into the dark half vs ~26px at full strength. The existing slice-2 blur test (person sharp + checkerboard smoothed) passes unchanged.
• ✅ startup.log perf line present on the real box ("background effects avg … ms/frame").

Related (not duplicated here)

• task-48-distribution-msix.md — the model ships in the MSIX; bundled-binary posture established by the ffmpeg work.
• TASKS.md Webcam cadence section — the "⛔ no background removal" line is superseded by this ruling; the judder/compositor cost work is unaffected and still heads its lane.
• research-store-certification.md — 11.12/10.2.2 posture; nothing here sends data off-box.