TASK 51 slice 1: background-effects engine — bundled MediaPipe Selfie Segmenter (ONNX) + Remove (Unit: bg-effects engine)
- Model: Google's official MediaPipe Selfie Segmenter (landscape, Apache-2.0 per model card), converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op Convolution2DTransposeBias -> ConvTranspose) and verified against MediaPipe's own runtime (verify_against_mediapipe.py: max |diff| 5e-5, 100% binary agreement). Embedded resource, 430 KB. - Services/BackgroundEffects: IPersonSegmenter seam, OnnxPersonSegmenter (onnxruntime 1.31.0 CPU, never throws), BackgroundEffectProcessor (None passthrough, Remove = alpha mask). Not wired yet. - Tests: Remove integration test (fake segmenter) + bundled-model contract test vs MediaPipe reference mask. Full suite 407/408 — the one failure is the documented interactive-desktop flake RealMouseDrag_OnTheLayerList_PersistsTheReorder (passed in the run before; code path untouched). - Docs: ai.md reversal + section + licensing guard, THIRD-PARTY-NOTICES 11-12, TASK 51, indexes, MyMistakes recipe, HANDOFF. References: obs-backgroundremoval (approach only — ONNX Runtime on CPU, 2 threads; GPL-3.0, no code used) https://github.com/royshil/obs-backgroundremoval ; MediaPipe image segmenter https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
This commit is contained in:
@@ -602,12 +602,47 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||||
to the encoder task.
|
||||
- **Background removal = PERMANENTLY OUT** (creator ruling 2026-10-05, confirmed: "no - not doing
|
||||
it. the note is correct and accurate"). The line here used to read "milestone 2", which was stale —
|
||||
it was never a plan, only a leftover from TASK 3's original scoping. The rationale that stands:
|
||||
a vcam app (NVIDIA Broadcast, XSplit VCam) does it free, and an in-app model means a runtime, a
|
||||
GPU/driver matrix, and per-scene lighting tuning — support hell for a solo dev
|
||||
(`TASKS/task-36-gold-pass.md`, `TASKS/task-03-capture-pipeline.md` item 19). Don't reopen it.
|
||||
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
|
||||
2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
|
||||
and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
|
||||
run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
|
||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||
|
||||
### Webcam background effects (TASK 51 — slice 1 engine shipped 2026-10-09)
|
||||
|
||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||
float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
|
||||
`tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
|
||||
`Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
|
||||
translation was smaller than any workaround). **Verified against Google's own runtime:**
|
||||
`verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
|
||||
.tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
|
||||
⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance) or
|
||||
copy code from obs-backgroundremoval (GPL-3.0).
|
||||
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
|
||||
whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
|
||||
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
|
||||
inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
|
||||
`Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
|
||||
`onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
|
||||
publish (TASK 48) must pin the RID so only win-x64 ships.
|
||||
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
|
||||
SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
|
||||
resource; **never throws** — a load/run failure logs once and returns null forever after);
|
||||
`BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
|
||||
passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
|
||||
`IsOpaque = false` (the compositor's straight-alpha blend shows the layers below); `Blur` passes
|
||||
through until slice 2. No mask / segmenter exception → passthrough + one log line, never a dropped
|
||||
layer.
|
||||
- **Not wired yet:** nothing calls the processor — no config, no UI, no `CameraManager` hookup. The
|
||||
next slices add Blur, the per-webcam `BackgroundEffect` config + persistence, then the single
|
||||
processing point on the camera path (both preview and compositor consume the processed frame).
|
||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`).
|
||||
|
||||
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
|
||||
|
||||
@@ -1652,6 +1687,11 @@ guardrails below — written down so a future "quick fix" never reintroduces a G
|
||||
another public-domain source; a logo asset under a copyleft or attribution license would contaminate
|
||||
the paid product. The About hub's logo is the creator's own art ("llama fortnite superman logo",
|
||||
`Assets/llama-logo.png`) — owned by us, so it carries no third-party license either.
|
||||
- **Ship an ML model without clean provenance** — the background-effects model (TASK 51) is Google's
|
||||
official MediaPipe Selfie Segmenter download (Apache-2.0, model card), converted by our own script.
|
||||
Model-zoo re-hosts (e.g. PINTO #109 = ML Kit's model, extracted) and GPL plugins
|
||||
(obs-backgroundremoval, GPL-3.0) are off-limits. Apache-2.0 §4 obligations (license notice + "we
|
||||
converted it") live in `THIRD-PARTY-NOTICES.txt`.
|
||||
- **Forget the v1 license-texts gate** — `THIRD-PARTY-NOTICES.txt` links the canonical license texts; at
|
||||
**v1 (GA)** the full texts of every license it names MUST ship alongside it (TASK 4 requirement 9 is the
|
||||
release blocker). Queued early is wrong; the release pass owns it.
|
||||
|
||||
Reference in New Issue
Block a user