TASK 51 slice 1: background-effects engine — bundled MediaPipe Selfie Segmenter (ONNX) + Remove (Unit: bg-effects engine)

- Model: Google's official MediaPipe Selfie Segmenter (landscape, Apache-2.0 per model card),
  converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op
  Convolution2DTransposeBias -> ConvTranspose) and verified against MediaPipe's own runtime
  (verify_against_mediapipe.py: max |diff| 5e-5, 100% binary agreement). Embedded resource, 430 KB.
- Services/BackgroundEffects: IPersonSegmenter seam, OnnxPersonSegmenter (onnxruntime 1.31.0 CPU,
  never throws), BackgroundEffectProcessor (None passthrough, Remove = alpha mask). Not wired yet.
- Tests: Remove integration test (fake segmenter) + bundled-model contract test vs MediaPipe
  reference mask. Full suite 407/408 — the one failure is the documented interactive-desktop flake
  RealMouseDrag_OnTheLayerList_PersistsTheReorder (passed in the run before; code path untouched).
- Docs: ai.md reversal + section + licensing guard, THIRD-PARTY-NOTICES 11-12, TASK 51, indexes,
  MyMistakes recipe, HANDOFF.

References: obs-backgroundremoval (approach only — ONNX Runtime on CPU, 2 threads; GPL-3.0, no code
used) https://github.com/royshil/obs-backgroundremoval ; MediaPipe image segmenter
https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
This commit is contained in:
2026-10-09 09:24:13 -07:00
parent f8dd1079f3
commit 9f3c154d84
20 changed files with 796 additions and 78 deletions
+34 -20
View File
@@ -4,7 +4,8 @@
> **Booked 2026-10-08 by creator ruling** — *"we're doing the bg removal and blur for v1 - book it, Danno"*.
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).
**Status:** ☐ Booked — **v1 requirement**. Queued behind the zoom build (current in-flight work unit).
**Status:** 🔶 In progress — **slice 1 (engine: bundled model + runner + Remove) shipped 2026-10-09**.
Next: slice 2 (Blur), then config + persistence, then the camera-path hookup + UI.
**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
selected per element in its properties panel, composited into stream + recording like any
@@ -34,7 +35,8 @@ delivery shape changed:
same posture as the ffmpeg bundling in TASK 48 item 1.)
2. **No hardware requirement.** CPU path must work; DirectML is an optional accelerator.
3. **Per-element (Meld-style), not global.** Each webcam element carries its own effect.
4. **No feature gating.** Works for free users; only the branding flash is paid.
4. **No feature gating.** Everything is free (free-forever ruling 2026-10-08, TASK 54 — the old
"only the branding flash is paid" line here was superseded).
5. **Scene change / webcam add/remove must not crash or stall the pump** — a failing model
run skips to passthrough, never drops the layer.
@@ -44,17 +46,21 @@ delivery shape changed:
- ❌ **Super resolution / denoise / prettify passes**
- ❌ **Virtual camera output** — permanently out of product (no vcam)
## 1. ☐ Engine
## 1. Engine
- ☐ Bundle an ONNX person-segmentation model (MediaPipe Selfie-Segmentation class, ~2–5 MB)
into the app (csproj copy like `ffmpeg/`, or as an `Asset`). Verify license + THIRD-PARTY
notices (TASK 36 item 6 gate).
- ☐ `onnxruntime` via NuGet. CPU EP by default; optionally DirectML EP later. Failing EP
init → passthrough, log once, never throw from the pump.
- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region. No
model needed for blur once the mask exists.
- ☐ Remove = **alpha 0 outside the person region.** The compositor already alpha-blends per
element (`SceneCompositor`, source-over) — no compositor change expected.
- ✅ **Model bundled (slice 1, 2026-10-09):** Google's official MediaPipe Selfie Segmenter
**landscape** (144×256, Apache-2.0 per the model card — license + changes recorded in
THIRD-PARTY-NOTICES item 12). Converted to ONNX by `tools/selfie-seg/convert_tflite_to_onnx.py`
(custom op `Convolution2DTransposeBias` → `ConvTranspose`); verified against MediaPipe's own
runtime by `tools/selfie-seg/verify_against_mediapipe.py` (max |diff| 5e-5). 430 KB,
`EmbeddedResource` in the exe. Rejected sources: PINTO #109 (ML Kit-extracted, unclear license),
obs-backgroundremoval code (GPL-3.0) — referenced for approach only (ONNX Runtime, CPU, 2 threads).
- ✅ `Microsoft.ML.OnnxRuntime` 1.31.0, CPU EP (`OnnxPersonSegmenter`). Failing load/run →
logs once, returns null, caller passes through. DirectML still optional/later.
- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region — slice 2.
- ✅ Remove = alpha = mask outside the person (`BackgroundEffectProcessor`), `IsOpaque = false`.
The compositor already alpha-blends per element — no compositor change expected (confirm in
the hookup slice).
- ☐ Feed the **VideoFrame seam** (`Services/VideoFrame.cs` doc explicitly anticipates
*"every future capture source (screen, background-removed webcam) feeds the same seam"*):
the frame handed to BOTH the preview bitmap path and the compositor's
@@ -71,19 +77,27 @@ delivery shape changed:
## 3. ☐ Test seam (Good Dog — one integration test per unit)
- ☐ Model-runner interface seam (like `CameraEnumeratorOverride`) so tests inject a fake
segmenter — no real ONNX inference in the test suite.
- ✅ Model-runner interface seam (`IPersonSegmenter`) so tests inject a fake segmenter.
- ☐ Assert the **processed frame reaches both consumers** (preview bitmap + compositor frame).
- ☐ `None` = byte-identical passthrough of the raw frame.
- ☐ `Remove` = alpha 0 outside person rect; `Blur` = background region blurred, person sharp.
- ✅ `None` = same-instance passthrough of the raw frame (`BackgroundEffectEngineTests`).
- ✅ `Remove` = alpha 0 outside the person, RGB untouched, non-opaque (`BackgroundEffectEngineTests`);
☐ `Blur` = background region blurred, person sharp.
- ✅ **Bundled-model contract:** the embedded ONNX model through the C# preprocessing reproduces
MediaPipe's reference mask on a fixture (≥99.5% agreement). This is a fixed, deterministic
contract check of the shipped bytes — not a camera/lighting quality test.
- ☐ Config → persistence roundtrip (per webcam element).
- ☐ Engine failure → passthrough + single log, layer never dropped.
- ☐ Engine failure → passthrough + single log, layer never dropped (code path exists; dedicated
test lands with the hookup slice).
- ☐ Real inference: **manual on the creator's box** (camera + lighting), not in CI.
- ☐ **Perf check in the hookup slice:** time `Segment` + `Apply` per 720p frame on the creator's
box before wiring it into the 60fps path (expect a few ms; the mask can run at a lower rate
than the camera if needed).
## 4. ☐ Sequencing
## 4. Sequencing
- ☐ **Separate work unit AFTER the zoom build.** This file is booked + scoped; do not
co-commit it with the zoom work.
- ✅ Separate work unit after the zoom build (zoom shipped 2026-10-08).
- Slice order: 1 engine ✅ → 2 Blur → 3 config + persistence → 4 camera-path hookup (one
processing point, both consumers) + perf check → 5 properties-panel UI.
## Related (not duplicated here)