LlamaCasty/TASKS/task-51-webcam-bg-effects.md

TASK 51 — Webcam background effects: Remove + Blur (v1 requirement)

Catalog: TASKS.md — status and requirements live here. Booked 2026-10-08 by creator ruling — "we're doing the bg removal and blur for v1 - book it, Danno". This reverses the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).

Status: ☐ Booked — v1 requirement. Queued behind the zoom build (current in-flight work unit).

Goal: per-webcam Background Remove and Background Blur on the live webcam layer, selected per element in its properties panel, composited into stream + recording like any other layer. No external downloads, no hardware requirement, works on any machine. Replacement backgrounds are free via layers — drop any element behind the webcam.

---

Why this is now IN (and what changed)

The 2026-10-05 exclusion was based on: an in-app AI model = a runtime, a GPU/driver matrix, per-scene lighting tuning, and "a vcam app does it free." The ruling is reversed because the delivery shape changed:

1. Bundled small ONNX model (~2–5 MB) — MediaPipe Selfie-Segmentation family — shipped in the installer/fixture (TASK 48 MSIX), executed by onnxruntime (CPU default; DirectML optional later). No NVIDIA SDK, no GPU requirement, no per-client download.
2. Quality reality (creator): a 720p webcam displays at ~15% of a 1080p frame — premium segmentation quality is invisible at that size; commodity class is sufficient.
3. ⛔ Ruled out: NVIDIA Video Effects SDK (VFX / Maxine, AI Green Screen + Background Blur). Works on RTX 2000+/R570+ only, and needs a ~1 GB user-side redistributable on first use. Creator: "I don't think I'm putting a 1gb download requirement on a client box."

Constraints — non-negotiable

1. No external download at runtime. Model ships in the installer. (Store policy 10.2.2 — same posture as the ffmpeg bundling in TASK 48 item 1.)
2. No hardware requirement. CPU path must work; DirectML is an optional accelerator.
3. Per-element (Meld-style), not global. Each webcam element carries its own effect.
4. No feature gating. Works for free users; only the branding flash is paid.
5. Scene change / webcam add/remove must not crash or stall the pump — a failing model run skips to passthrough, never drops the layer.

Out of scope for v1 (same engine later)

• ❌ "Remove chair"-style object selection / quality tiers — not a picker engine
• ❌ Super resolution / denoise / prettify passes
• ❌ Virtual camera output — permanently out of product (no vcam)

1. ☐ Engine

• ☐ Bundle an ONNX person-segmentation model (MediaPipe Selfie-Segmentation class, ~2–5 MB) into the app (csproj copy like ffmpeg/, or as an Asset). Verify license + THIRD-PARTY notices (TASK 36 item 6 gate).
• ☐ onnxruntime via NuGet. CPU EP by default; optionally DirectML EP later. Failing EP init → passthrough, log once, never throw from the pump.
• ☐ Blur = separable in-app blur (box/gaussian) over the masked background region. No model needed for blur once the mask exists.
• ☐ Remove = alpha 0 outside the person region. The compositor already alpha-blends per element (SceneCompositor, source-over) — no compositor change expected.
• ☐ Feed the VideoFrame seam (Services/VideoFrame.cs doc explicitly anticipates "every future capture source (screen, background-removed webcam) feeds the same seam"): the frame handed to BOTH the preview bitmap path and the compositor's _cameraManager.GetLatestFrame is the processed frame. One processing point, two consumers.

2. ☐ Config surface + persistence

• ☐ WebcamSceneConfig gains BackgroundEffect (None | Remove | Blur) + Strength.
• ☐ Persist via the existing WebcamSceneConfig table path in LayoutStore.Save.cs/ Load.cs (new columns + migration). Follow LayoutStore.Migrations.cs conventions.
• ☐ Properties panel: effect dropdown + strength slider, per webcam element.
• ☐ Replace-background affordance: none in v1 — the creator drops any layer behind the webcam. Effect preview must not obscure the fact that layers below still show through.

3. ☐ Test seam (Good Dog — one integration test per unit)

• ☐ Model-runner interface seam (like CameraEnumeratorOverride) so tests inject a fake segmenter — no real ONNX inference in the test suite.
• ☐ Assert the processed frame reaches both consumers (preview bitmap + compositor frame).
• ☐ None = byte-identical passthrough of the raw frame.
• ☐ Remove = alpha 0 outside person rect; Blur = background region blurred, person sharp.
• ☐ Config → persistence roundtrip (per webcam element).
• ☐ Engine failure → passthrough + single log, layer never dropped.
• ☐ Real inference: manual on the creator's box (camera + lighting), not in CI.

4. ☐ Sequencing

• ☐ Separate work unit AFTER the zoom build. This file is booked + scoped; do not co-commit it with the zoom work.

Related (not duplicated here)

• task-48-distribution-msix.md — the model ships in the MSIX; bundled-binary posture established by the ffmpeg work.
• TASKS.md Webcam cadence section — the "⛔ no background removal" line is superseded by this ruling; the judder/compositor cost work is unaffected and still heads its lane.
• research-store-certification.md — 11.12/10.2.2 posture; nothing here sends data off-box.