LlamaCasty/TASKS/task-51-webcam-bg-effects.md

# TASK 51 — Webcam background effects: Remove + Blur (v1 requirement)

> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
> **Booked 2026-10-08 by creator ruling** — *"we're doing the bg removal and blur for v1 - book it, Danno"*.
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).

**Status:** ✅ **Built 2026-10-09 — all five slices shipped** (engine, Blur, config + persistence,
camera hookup, UI). Remaining: the creator's manual check on a real camera (quality, lighting,
`startup.log` perf line).

**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
selected per element in its properties panel, composited into stream + recording like any
other layer. No external downloads, no hardware requirement, works on any machine.
Replacement backgrounds are **free via layers** — drop any element behind the webcam.

---

## Why this is now IN (and what changed)

The 2026-10-05 exclusion was based on: an in-app AI model = a runtime, a GPU/driver matrix,
per-scene lighting tuning, and "a vcam app does it free." The ruling is reversed because the
delivery shape changed:

1. **Bundled small ONNX model (~2–5 MB)** — MediaPipe Selfie-Segmentation family —
   shipped in the installer/fixture (TASK 48 MSIX), executed by **onnxruntime** (CPU default;
   DirectML optional later). No NVIDIA SDK, no GPU requirement, no per-client download.
2. **Quality reality (creator):** a 720p webcam displays at ~15% of a 1080p frame — premium
   segmentation quality is invisible at that size; commodity class is sufficient.
3. **⛔ Ruled out: NVIDIA Video Effects SDK (VFX / Maxine, AI Green Screen + Background Blur).**
   Works on RTX 2000+/R570+ only, and needs a ~1 GB user-side redistributable on first use.
   Creator: *"I don't think I'm putting a 1gb download requirement on a client box."*

## Constraints — non-negotiable

1. **No external download at runtime.** Model ships in the installer. (Store policy 10.2.2 —
   same posture as the ffmpeg bundling in TASK 48 item 1.)
2. **No hardware requirement.** CPU path must work; DirectML is an optional accelerator.
3. **Per-element (Meld-style), not global.** Each webcam element carries its own effect.
4. **No feature gating.** Everything is free (free-forever ruling 2026-10-08, TASK 54 — the old
   "only the branding flash is paid" line here was superseded).
5. **Scene change / webcam add/remove must not crash or stall the pump** — a failing model
   run skips to passthrough, never drops the layer.

## Out of scope for v1 (same engine later)

- ❌ **"Remove chair"-style object selection / quality tiers** — not a picker engine
- ❌ **Super resolution / denoise / prettify passes**
- ❌ **Virtual camera output** — permanently out of product (no vcam)

## 1. Engine

- ✅ **Model bundled (slice 1, 2026-10-09):** Google's official MediaPipe Selfie Segmenter
  **landscape** (144×256, Apache-2.0 per the model card — license + changes recorded in
  THIRD-PARTY-NOTICES item 12). Converted to ONNX by `tools/selfie-seg/convert_tflite_to_onnx.py`
  (custom op `Convolution2DTransposeBias` → `ConvTranspose`); verified against MediaPipe's own
  runtime by `tools/selfie-seg/verify_against_mediapipe.py` (max |diff| 5e-5). 430 KB,
  `EmbeddedResource` in the exe. Rejected sources: PINTO #109 (ML Kit-extracted, unclear license),
  obs-backgroundremoval code (GPL-3.0) — referenced for approach only (ONNX Runtime, CPU, 2 threads).
- ✅ `Microsoft.ML.OnnxRuntime` 1.31.0, CPU EP (`OnnxPersonSegmenter`). Failing load/run →
  logs once, returns null, caller passes through. DirectML still optional/later.
- ✅ **Blur (slice 2, 2026-10-09):** background box-averaged to ¼ res, 3 separable box-blur passes
  (≈ gaussian, radius by strength), mask-blended back; person pixels untouched (`BackgroundEffectProcessor.ApplyBlur`).
- ✅ Remove = alpha = mask outside the person (`BackgroundEffectProcessor`), `IsOpaque = false`.
  The compositor already alpha-blends per element — no compositor change expected (confirm in
  the hookup slice).
- ☐ Feed the **VideoFrame seam** (`Services/VideoFrame.cs` doc explicitly anticipates
  *"every future capture source (screen, background-removed webcam) feeds the same seam"*):
  the frame handed to BOTH the preview bitmap path and the compositor's
  `_cameraManager.GetLatestFrame` is the processed frame. One processing point, two consumers.

## 2. ☐ Config surface + persistence

- ✅ `WebcamSceneConfig` gains `BackgroundEffect` (`None | Remove | Blur`) + `BackgroundStrength` (slice 3).
- ✅ Persist via the existing `WebcamSceneConfig` table path in `LayoutStore.Save.cs`/
  `Load.cs` (new columns + migration). Follow `LayoutStore.Migrations.cs` conventions.
- ✅ **Properties UI (creator spec 2026-10-09, slice 5):** in the webcam's properties in the **Layers
  panel**, two **pill switches** — "Remove background" and "Blur background" — using the existing
  `PillToggle` style (`Themes/Controls.xaml`, already used in `Controls/LeftPanel.xaml`). They act
  as **radio buttons**: turning one on turns the other off; at most one is active; both off =
  `None`. Bound to the element's `BackgroundEffect`, per webcam element. (Supersedes the old
  "effect dropdown" idea.) **Plus a strength slider** (creator: "keep", 2026-10-09) — Blur
  radius / Remove edge softness via `Strength`.
- ☐ Replace-background affordance: **none in v1** — the creator drops any layer behind the
  webcam. Effect preview must not obscure the fact that layers below still show through.

## 3. ☐ Test seam (Good Dog — one integration test per unit)

- ✅ Model-runner interface seam (`IPersonSegmenter`) so tests inject a fake segmenter.
- ✅ The **processed frame reaches both consumers** (preview bitmap + compositor frame), raw stays raw (`CameraManagerTests`, slice 4).
- ✅ `None` = same-instance passthrough of the raw frame (`BackgroundEffectEngineTests`).
- ✅ `Remove` = alpha 0 outside the person, RGB untouched, non-opaque (`BackgroundEffectEngineTests`);
  ✅ `Blur` = background smoothed, person pixel-identical (`BackgroundEffectEngineTests`).
- ✅ **Bundled-model contract:** the embedded ONNX model through the C# preprocessing reproduces
  MediaPipe's reference mask on a fixture (≥99.5% agreement). This is a fixed, deterministic
  contract check of the shipped bytes — not a camera/lighting quality test.
- ✅ Config → persistence roundtrip per scene + legacy-DB migration (`LayoutStorePersistenceTests`).
- ☐ Engine failure → passthrough + single log, layer never dropped (code path exists; dedicated
  test lands with the hookup slice).
- ☐ Real inference: **manual on the creator's box** (camera + lighting), not in CI.
- ✅ **Perf, synthetic 720p (Debug):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms (after making the
  upsampling separable — naive was 46/80 ms). Runs on its own worker, off the raw path.
  ☐ **Real numbers on the creator's box:** `startup.log` → "background effects avg … ms/frame".

## 4. Sequencing

- ✅ Separate work unit after the zoom build (zoom shipped 2026-10-08).
- Slice order: 1 engine ✅ → 2 Blur ✅ → 3 config + persistence ✅ → 4 camera-path hookup ✅ (one
  processing point, both consumers) + perf check → 5 properties-panel UI ✅.

## Related (not duplicated here)

- [`task-48-distribution-msix.md`](task-48-distribution-msix.md) — the model ships in the
  MSIX; bundled-binary posture established by the ffmpeg work.
- `TASKS.md` Webcam cadence section — the "⛔ no background removal" line is superseded
  by this ruling; the **judder/compositor cost work is unaffected and still heads its lane.**
- `research-store-certification.md` — 11.12/10.2.2 posture; nothing here sends data off-box.