Zoom: compose the box as a real 2x layer above the background (Unit: zoom layer + magnification proof)

Right-click now creates a transient ZoomSceneElement at z-index 1 whose pixels are a
genuine 2x magnified sample of the backdrop (live capture under art), composited into
stream/recording like any layer while the output crop stays the tier. All shapes are
centered on the click and clamped to frame; healed on load/scene-create, never persisted,
hidden from the layers list + element template (IsZoomLayer). BuildCompositorOptions is
now internal = always tier crop (test seam). ZoomFrameBuilder is a pure static sampler
(box/2 window centered on the box center, single-slot cache) proven by a banded-backdrop
test; LayerReorderPersistenceTests drag resolves rows by item since the collapsed zoom
row shifts container indices. Also books TASK 51 (webcam background Remove + Blur) and
syncs memory docs (HANDOFF, MyMistakes, indexes). Full suite: 405 passed, 0 warnings.
This commit is contained in:
2026-10-07 23:39:54 -07:00
parent 966e718f1e
commit 4b3d82ff0f
17 changed files with 734 additions and 210 deletions
+94
View File
@@ -0,0 +1,94 @@
# TASK 51 — Webcam background effects: Remove + Blur (v1 requirement)
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
> **Booked 2026-10-08 by creator ruling** — *"we're doing the bg removal and blur for v1 - book it, Danno"*.
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).
**Status:** ☐ Booked — **v1 requirement**. Queued behind the zoom build (current in-flight work unit).
**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
selected per element in its properties panel, composited into stream + recording like any
other layer. No external downloads, no hardware requirement, works on any machine.
Replacement backgrounds are **free via layers** — drop any element behind the webcam.
---
## Why this is now IN (and what changed)
The 2026-10-05 exclusion was based on: an in-app AI model = a runtime, a GPU/driver matrix,
per-scene lighting tuning, and "a vcam app does it free." The ruling is reversed because the
delivery shape changed:
1. **Bundled small ONNX model (~2–5 MB)** — MediaPipe Selfie-Segmentation family —
shipped in the installer/fixture (TASK 48 MSIX), executed by **onnxruntime** (CPU default;
DirectML optional later). No NVIDIA SDK, no GPU requirement, no per-client download.
2. **Quality reality (creator):** a 720p webcam displays at ~15% of a 1080p frame — premium
segmentation quality is invisible at that size; commodity class is sufficient.
3. **⛔ Ruled out: NVIDIA Video Effects SDK (VFX / Maxine, AI Green Screen + Background Blur).**
Works on RTX 2000+/R570+ only, and needs a ~1 GB user-side redistributable on first use.
Creator: *"I don't think I'm putting a 1gb download requirement on a client box."*
## Constraints — non-negotiable
1. **No external download at runtime.** Model ships in the installer. (Store policy 10.2.2 —
same posture as the ffmpeg bundling in TASK 48 item 1.)
2. **No hardware requirement.** CPU path must work; DirectML is an optional accelerator.
3. **Per-element (Meld-style), not global.** Each webcam element carries its own effect.
4. **No feature gating.** Works for free users; only the branding flash is paid.
5. **Scene change / webcam add/remove must not crash or stall the pump** — a failing model
run skips to passthrough, never drops the layer.
## Out of scope for v1 (same engine later)
- ❌ **"Remove chair"-style object selection / quality tiers** — not a picker engine
- ❌ **Super resolution / denoise / prettify passes**
- ❌ **Virtual camera output** — permanently out of product (no vcam)
## 1. ☐ Engine
- ☐ Bundle an ONNX person-segmentation model (MediaPipe Selfie-Segmentation class, ~2–5 MB)
into the app (csproj copy like `ffmpeg/`, or as an `Asset`). Verify license + THIRD-PARTY
notices (TASK 36 item 6 gate).
- ☐ `onnxruntime` via NuGet. CPU EP by default; optionally DirectML EP later. Failing EP
init → passthrough, log once, never throw from the pump.
- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region. No
model needed for blur once the mask exists.
- ☐ Remove = **alpha 0 outside the person region.** The compositor already alpha-blends per
element (`SceneCompositor`, source-over) — no compositor change expected.
- ☐ Feed the **VideoFrame seam** (`Services/VideoFrame.cs` doc explicitly anticipates
*"every future capture source (screen, background-removed webcam) feeds the same seam"*):
the frame handed to BOTH the preview bitmap path and the compositor's
`_cameraManager.GetLatestFrame` is the processed frame. One processing point, two consumers.
## 2. ☐ Config surface + persistence
- ☐ `WebcamSceneConfig` gains `BackgroundEffect` (`None | Remove | Blur`) + `Strength`.
- ☐ Persist via the existing `WebcamSceneConfig` table path in `LayoutStore.Save.cs`/
`Load.cs` (new columns + migration). Follow `LayoutStore.Migrations.cs` conventions.
- ☐ Properties panel: effect dropdown + strength slider, per webcam element.
- ☐ Replace-background affordance: **none in v1** — the creator drops any layer behind the
webcam. Effect preview must not obscure the fact that layers below still show through.
## 3. ☐ Test seam (Good Dog — one integration test per unit)
- ☐ Model-runner interface seam (like `CameraEnumeratorOverride`) so tests inject a fake
segmenter — no real ONNX inference in the test suite.
- ☐ Assert the **processed frame reaches both consumers** (preview bitmap + compositor frame).
- ☐ `None` = byte-identical passthrough of the raw frame.
- ☐ `Remove` = alpha 0 outside person rect; `Blur` = background region blurred, person sharp.
- ☐ Config → persistence roundtrip (per webcam element).
- ☐ Engine failure → passthrough + single log, layer never dropped.
- ☐ Real inference: **manual on the creator's box** (camera + lighting), not in CI.
## 4. ☐ Sequencing
- ☐ **Separate work unit AFTER the zoom build.** This file is booked + scoped; do not
co-commit it with the zoom work.
## Related (not duplicated here)
- [`task-48-distribution-msix.md`](task-48-distribution-msix.md) — the model ships in the
MSIX; bundled-binary posture established by the ffmpeg work.
- `TASKS.md` Webcam cadence section — the "⛔ no background removal" line is superseded
by this ruling; the **judder/compositor cost work is unaffected and still heads its lane.**
- `research-store-certification.md` — 11.12/10.2.2 posture; nothing here sends data off-box.