diff --git a/Assets/Models/selfie_segmenter_landscape.onnx b/Assets/Models/selfie_segmenter_landscape.onnx new file mode 100644 index 0000000..154384a Binary files /dev/null and b/Assets/Models/selfie_segmenter_landscape.onnx differ diff --git a/HANDOFF.md b/HANDOFF.md index 5bdeb6d..924fbc5 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,60 +1,42 @@ -# HANDOFF — 2026-10-08 (zoom box shipped, READY FOR PUSH) +# HANDOFF — 2026-10-09 (TASK 51 slice 1 shipped: background-effects engine) ## Branch -- main (no feature branch) +- main (no feature branch). Pushed through `f8dd107`; the TASK 51 slice-1 commit sits on top, + **not pushed** (push at the user's call). -## Push readiness -- **5 commits ahead of origin/main, clean to push** (fetch confirmed origin has not moved): - `966e718` zoom box visuals → `4b3d82f` zoom as layered + TASK 51 booking → `6bff21b` - anchored-capture rework → `948dd08` dims-shading fix → (docs unit: TASK 52 record + - numbering fixes + this HANDOFF). -- The pinned-chat work unit below is **uncommitted on purpose** — `git push` sends only - commits, so pushing is safe; the dirty files stay local. +## Dirty files (NOT mine — do NOT commit or co-edit) +- The in-flight "pinned chat" work unit: `Models/YouTube.cs`, `ViewModels/MainViewModel.Chat.cs`, + `Controls/LeftPanel.xaml`, untracked `Helpers/RelayCommandGeneric.cs`. +- The creator's untracked logo art in `Assets/` (`llamacasty text.*`, `llamacasty-text-nobg*.png`, + `thin logo wordmark.png`). -## Dirty files (NOT mine — the in-flight "pinned chat" work unit; do NOT commit or co-edit) -- `Models/YouTube.cs` -- `ViewModels/MainViewModel.Chat.cs` -- `Controls/LeftPanel.xaml` (pinned-message UI) -- Untracked: `Helpers/RelayCommandGeneric.cs` - -## What just happened (2026-10-08, third zoom pass — dims shading) -The zoom box shipped as a **loupe** (`4b3d82f`), then was corrected to an **anchored-capture -magnifier over the full composite drawn last** (commit `6bff21b`). The final correction: as the box -is dragged, the dim rectangles (55% black "div shading" around the box) were sliding under the -capture region — shown inside the box because its `VisualBrush` samples all of `CanvasGrid`, and the -`DimRects` ItemsControl previously lived INSIDE `CanvasGrid`. - -Fix: the `DimRects` overlay is now a **sibling of `CanvasGrid`** (renders above the composed -content, below the `ZoomBox`), never a descendant — so the zoom sample can never pick up the shading -and the captured area stays clear no matter where the box is dragged. Behavior consequence to note: -in non-zoom ("quality dims") mode the dims now also shade the global overlays (brand flash, alert -ticker, social bar) that sit above them in the old tree — consistent with "chrome outside the crop -is dimmed". Structural invariant added to `ZoomBoxInteractionTests`: `DimRectsLayer` is a direct -child of `PreviewCanvas`, not inside `CanvasGrid`, and renders below the `ZoomBox`. - -## Zoom feature state -- **TASK 52 (zoom box) is now a first-class record** — see [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md): - feature provenance + the three creator corrections, shapes, implementation map, tests. - TASK 51 (webcam background Remove + Blur) is unblocked for the next unit. -- Models: `ZoomState.cs`, `ZoomShape.cs` (Circle/VerticalRectangle/HorizontalRectangle), `ZoomSceneElement.cs`. -- ViewModel (`MainViewModel.Zoom.cs`): `ActivateZoom`/`ToggleZoom`/`DismissZoom`/`StartZoomMove`/`UpdateZoomMove`/`EndZoomMove` + `RebuildDimRectsForZoom`; transient element healed/mirrored by `HealZoomLayers`/`EnsureZoomLayer`. -- Output: box composited last via `BlitZoom`/`ZoomMagnifier`; output crop = tier (box never the crop). -- Preview: `ZoomBox` + `DimRectsLayer` are both SIBLINGS of `CanvasGrid` (box above dims above content); box samples the full composite at the fixed capture, 2x; circle→ellipse clip, rect/square outline per shape; `LeftPanel` hides the placeholder row via `IsZoomLayer`. -- Rectangle shape = 4:3; Circle square; Phone 9:16. Capture anchored on click (`CaptureX/Y`); dragging the box never changes it. +## What just happened — TASK 51 slice 1 (engine) +- **Model:** Google's official MediaPipe Selfie Segmenter (landscape, 144×256, Apache-2.0), + converted to ONNX by our own `tools/selfie-seg/convert_tflite_to_onnx.py` and **verified against + MediaPipe's own runtime** (`verify_against_mediapipe.py`, max |diff| 5e-5). Embedded in the exe + (`Assets/Models/selfie_segmenter_landscape.onnx`, 430 KB, in git). +- **Code:** `Services/BackgroundEffects/` — `IPersonSegmenter` seam, `OnnxPersonSegmenter` + (onnxruntime 1.31.0 CPU, never throws), `BackgroundEffectProcessor` (None passthrough, Remove = + alpha mask, Blur = passthrough until slice 2). `Models/BackgroundEffect.cs` enum. +- **Not wired:** nothing calls it yet — no config, UI, or camera hookup. Zero risk to the live path. +- **Docs:** ai.md (stale "background removal PERMANENTLY OUT" bullet replaced by the reversal + a + "Webcam background effects" section + a model-provenance licensing guard), THIRD-PARTY-NOTICES + items 11 (ONNX Runtime, MIT) + 12 (MediaPipe model, Apache-2.0 with §4b change note), + TASK 51 file + TASKS.md row, Services/Models indexes, MyMistakes conversion recipe. ## Tests -- Full suite: **406 passed / 0 failed** on a clean full run (the pre-existing audio-loopback timing - flake and `RealMouseDrag`'s interactive-desktop requirement recur occasionally — not zoom - regressions). Zoom tests 3/3 green, 0 build warnings. +- Full suite **408 passed / 0 failed**, 0 build warnings. New: `BackgroundEffectEngineTests` (Remove + integration test with a fake segmenter; bundled-model contract test vs a MediaPipe reference mask, + fixtures in `ytLive.Tests/Fixtures/`). ## Known issues / follow-ups -- **Background "Use default background" toggle still not working** (carried from 2026-10-07): - binding exists (`UseDefaultBackground`), VM handler invalidates bake; investigate when time permits. -- The 3px selection / crop-outline strokes could still show in the zoom box IF the capture overlaps - them in edit mode — not shading, thin outlines only; the dims (the user's complaint) are gone. +- `onnxruntime.dll` is ~17 MB; an RID-less build copies every RID's native folder — TASK 48's MSIX + publish must pin win-x64. +- `THIRD-PARTY-NOTICES.txt` header still says "paid, closed-source product" (pre free-forever) — noted, + not changed (out of scope). +- Carried: "Use default background" toggle still not working (from 2026-10-07). ## Next step -- User runs the app live to eyeball: drag the zoom box anywhere and the captured content stays clean - (no dim shading in the box; the scene outside the box stays dimmed). Then TASK 51 (webcam - background Remove + Blur) per `TASKS/task-51-webcam-bg-effects.md`, and the background-toggle - investigation if still open. \ No newline at end of file +- TASK 51 slice 2: **Blur** in `BackgroundEffectProcessor` (separable box/gaussian, strength), one + integration test. Then slice 3 config + persistence, slice 4 camera-path hookup + perf check on the + creator's box, slice 5 properties-panel UI. diff --git a/Models/BackgroundEffect.cs b/Models/BackgroundEffect.cs new file mode 100644 index 0000000..06a31c9 --- /dev/null +++ b/Models/BackgroundEffect.cs @@ -0,0 +1,11 @@ +namespace ytLive.Models; + +/// Per-webcam background effect (TASK 51). makes everything +/// outside the person transparent so the layers below show through; +/// blurs the background behind the person. +public enum BackgroundEffect +{ + None, + Remove, + Blur, +} diff --git a/Models/index.md b/Models/index.md index 4bacf08..a22d217 100644 --- a/Models/index.md +++ b/Models/index.md @@ -11,6 +11,7 @@ Plain data types. No logic beyond what a property can carry. See | `SceneElement.cs` | Base for anything placeable in a scene: shared layout/clip/mirror/border surface, `virtual IsWebcam`/`virtual IsImageSource`/`virtual IsZoomLayer` (TASK 52); `ToggleClipShape` + persisted `RectWidth`/`RectHeight` (pre-Round rect so round→rect restores after reload) | | `ZoomSceneElement.cs` | **Transient zoom box (TASK 52, creator-corrected 2026-10-08):** carries the box rect AND the anchored `CaptureX`/`CaptureY` (the fixed box/2 capture window set on `ActivateZoom`; moving the box never touches it). `IsZoomActive` flips `Kind` Static→Dynamic (leaves the baked base while active); never persisted (LayoutStore save skips non-Source/non-WebcamSceneConfig rows), never shown in the layers list or the element template (`IsZoomLayer` gates both). The compositor SKIPS the active element in every layer loop and draws its 2x content LAST (layer n+1) via `ZoomMagnifier`. | | `Webcam.cs` | Singleton webcam identity: `Id`, `DeviceId`, `Name` (one row app-wide) | +| `BackgroundEffect.cs` | **TASK 51:** per-webcam background effect enum `None` / `Remove` / `Blur` (not yet on `WebcamSceneConfig` — config + persistence is slice 3) | | `WebcamSceneConfig.cs` | Per-scene webcam placement (subclass of `SceneElement`): geometry + `IsVisible` + border (`BorderColor`/`BorderOpacity`/`BorderWidth`/`BorderAnimation`) + `VideoImageSource`; `WebcamId` links to `Webcam` | | `QualityOption.cs` | Resolution tier record `(Display, Fps, Bitrate, Width, Height)`; `Label` formats as `1080p60 (8Mbps)` — drives the bottom-bar dropdown and the output-rect math | | `StreamConfig.cs` | `StreamConfig` (note: `TargetBitrate`/`Resolution` defaults are stale — not yet wired to the dropdown), `StreamHealth` (bitrate/FPS/dropped/duration), `StreamStatus` enum | diff --git a/MyMistakes.md b/MyMistakes.md index a6073ca..054f24a 100644 --- a/MyMistakes.md +++ b/MyMistakes.md @@ -15,6 +15,22 @@ ## 🔬 Recipes registry +### CONVERTING A MEDIAPIPE .tflite (WITH CUSTOM OPS) TO ONNX — AND PROVING IT (RECIPE, TASK 51) + +**Problem:** MediaPipe segmentation models end in the custom op `Convolution2DTransposeBias`; +tf2onnx and tflite2onnx refuse it. **Recipe:** `tools/selfie-seg/convert_tflite_to_onnx.py` — +parse the flatbuffer with the `tflite` pip package, emit ONNX with `onnx.helper`, transpose +NHWC→NCHW once at input/output, fold every `DEQUANTIZE` (fp16 weights) into fp32 initializers. +Weight layouts: CONV OHWI→OIHW; DEPTHWISE [1,H,W,C]→[C,1,H,W] with `group=C`; transpose-conv +[O,H,W,I]→[I,O,H,W]. TF `SAME` padding = ONNX `auto_pad=SAME_UPPER`. +- ⚠️ **Enum trap:** the custom op's options are the C-API `TfLiteTransposeConvParams`, whose + `padding` uses `TfLitePadding` (Unknown=0, **SAME=1**, VALID=2) — NOT the flatbuffer `Padding` + enum (SAME=0). Comparing against `Padding.SAME` rejects the real model. +- **Proof, not hope:** `pip install mediapipe` ships the TFLite runtime WITH the custom op. Feed + an image already at the model's exact input size (no resampling on either side) to MediaPipe's + `ImageSegmenter` and to onnxruntime; compare masks (`verify_against_mediapipe.py`, exit 0 at + max |diff| < 1e-3 — we got 5e-5). Work in a throwaway venv under `/tmp/opencode/`. + ### ⛔ NEVER PRINT CREDENTIALS.md — NOT EVEN "FILTERED" (2026-10-09 incident) **What happened:** to look for an API key, a `sed -n '1,40p' CREDENTIALS.md` piped through a diff --git a/Services/BackgroundEffects/BackgroundEffectProcessor.cs b/Services/BackgroundEffects/BackgroundEffectProcessor.cs new file mode 100644 index 0000000..b026ac5 --- /dev/null +++ b/Services/BackgroundEffects/BackgroundEffectProcessor.cs @@ -0,0 +1,106 @@ +using ytLive.Helpers; +using ytLive.Models; + +namespace ytLive.Services.BackgroundEffects; + +/// +/// Applies a webcam's to one captured frame (TASK 51). +/// Pure pixel work over the seam — WPF-free and thread-agnostic, +/// so the camera worker thread can call it before the frame reaches the preview bitmap +/// and the compositor. +/// +/// +/// returns the SAME frame instance — a true +/// passthrough, no copy. +/// returns a copy whose alpha is the person +/// mask (bilinear-upscaled), RGB untouched, IsOpaque = false — the compositor's +/// straight-alpha blend then shows the layers below the webcam. +/// is the next slice; until it lands it passes +/// through. +/// +/// +/// A segmenter that returns no mask (or throws) never drops the layer: the frame passes +/// through and the failure is logged once. +/// +public sealed class BackgroundEffectProcessor +{ + private readonly IPersonSegmenter _segmenter; + private bool _loggedFailure; + + public BackgroundEffectProcessor(IPersonSegmenter segmenter) + { + _segmenter = segmenter; + } + + public VideoFrame Apply(VideoFrame frame, BackgroundEffect effect) + { + if (effect != BackgroundEffect.Remove) return frame; + + SegmentationMask? mask; + try + { + mask = _segmenter.Segment(frame); + } + catch (Exception ex) + { + LogFailureOnce($"{ex.GetType().Name}: {ex.Message}"); + return frame; + } + + if (mask == null || mask.Values.Length < mask.Width * mask.Height || mask.Width < 1 || mask.Height < 1) + { + LogFailureOnce("segmenter returned no usable mask"); + return frame; + } + + return ApplyRemove(frame, mask); + } + + internal static VideoFrame ApplyRemove(VideoFrame frame, SegmentationMask mask) + { + var src = frame.BgraPixels; + var dst = new byte[src.Length]; + Buffer.BlockCopy(src, 0, dst, 0, src.Length); + + var w = frame.Width; + var h = frame.Height; + var stride = frame.Stride; + for (var y = 0; y < h; y++) + { + var my = (y + 0.5f) * mask.Height / h - 0.5f; + var row = y * stride; + for (var x = 0; x < w; x++) + { + var mx = (x + 0.5f) * mask.Width / w - 0.5f; + var a = SampleBilinear(mask, mx, my); + var alpha = (int)(a * src[row + x * 4 + 3] + 0.5f); + dst[row + x * 4 + 3] = (byte)Math.Clamp(alpha, 0, 255); + } + } + + return new VideoFrame(w, h, dst) { Epoch = frame.Epoch, IsOpaque = false }; + } + + private static float SampleBilinear(SegmentationMask mask, float x, float y) + { + var x0 = (int)MathF.Floor(x); + var y0 = (int)MathF.Floor(y); + var fx = x - x0; + var fy = y - y0; + var x1 = Math.Clamp(x0 + 1, 0, mask.Width - 1); + var y1 = Math.Clamp(y0 + 1, 0, mask.Height - 1); + x0 = Math.Clamp(x0, 0, mask.Width - 1); + y0 = Math.Clamp(y0, 0, mask.Height - 1); + var v = mask.Values; + var top = v[y0 * mask.Width + x0] * (1 - fx) + v[y0 * mask.Width + x1] * fx; + var bottom = v[y1 * mask.Width + x0] * (1 - fx) + v[y1 * mask.Width + x1] * fx; + return Math.Clamp(top * (1 - fy) + bottom * fy, 0f, 1f); + } + + private void LogFailureOnce(string reason) + { + if (_loggedFailure) return; + _loggedFailure = true; + AppLog.Write($"BackgroundEffectProcessor: passing webcam through — {reason}"); + } +} diff --git a/Services/BackgroundEffects/IPersonSegmenter.cs b/Services/BackgroundEffects/IPersonSegmenter.cs new file mode 100644 index 0000000..c65cddb --- /dev/null +++ b/Services/BackgroundEffects/IPersonSegmenter.cs @@ -0,0 +1,13 @@ +namespace ytLive.Services.BackgroundEffects; + +/// A person-probability mask at the segmenter's own resolution: row-major, +/// one value per pixel, 0 = background, 1 = person. +public sealed record SegmentationMask(int Width, int Height, float[] Values); + +/// Seam over the person-segmentation model (TASK 51) so tests inject a fake and +/// never run real inference. Returns null when no mask can be produced; callers then pass +/// the frame through untouched. +public interface IPersonSegmenter +{ + SegmentationMask? Segment(VideoFrame frame); +} diff --git a/Services/BackgroundEffects/OnnxPersonSegmenter.cs b/Services/BackgroundEffects/OnnxPersonSegmenter.cs new file mode 100644 index 0000000..890f81f --- /dev/null +++ b/Services/BackgroundEffects/OnnxPersonSegmenter.cs @@ -0,0 +1,107 @@ +using System.IO; +using Microsoft.ML.OnnxRuntime; +using ytLive.Helpers; + +namespace ytLive.Services.BackgroundEffects; + +/// +/// The default : Google's MediaPipe Selfie Segmenter +/// (landscape, Apache-2.0) converted to ONNX by tools/selfie-seg/ and shipped as an +/// embedded resource — nothing is downloaded at runtime and no GPU is required. +/// +/// Contract: input [1,144,256,3] RGB in [0,1] (NHWC), output [1,144,256,1] person +/// probability. The whole frame is resampled to 256×144 (stretched, not cropped), so the +/// mask maps back to the frame by normalized coordinates. +/// +/// Never throws from : a model that fails to load or run logs once +/// and returns null, and the caller passes the webcam frame through. +/// +public sealed class OnnxPersonSegmenter : IPersonSegmenter, IDisposable +{ + public const int ModelWidth = 256; + public const int ModelHeight = 144; + internal const string ModelResourceName = "ytLive.Assets.Models.selfie_segmenter_landscape.onnx"; + + private static readonly long[] InputShape = { 1, ModelHeight, ModelWidth, 3 }; + + private readonly object _gate = new(); + private readonly float[] _input = new float[ModelWidth * ModelHeight * 3]; + private readonly RunOptions _runOptions = new(); + private InferenceSession? _session; + private bool _failed; + private bool _disposed; + + public SegmentationMask? Segment(VideoFrame frame) + { + lock (_gate) + { + if (_failed || _disposed) return null; + try + { + _session ??= CreateSession(); + FillInput(frame, _input); + using var input = OrtValue.CreateTensorValueFromMemory(_input, InputShape); + using var outputs = _session.Run(_runOptions, new[] { "input" }, new[] { input }, new[] { "mask" }); + var mask = outputs[0].GetTensorDataAsSpan().ToArray(); + return new SegmentationMask(ModelWidth, ModelHeight, mask); + } + catch (Exception ex) + { + _failed = true; + AppLog.Write($"OnnxPersonSegmenter: background effects disabled — {ex.GetType().Name}: {ex.Message}"); + return null; + } + } + } + + public void Dispose() + { + lock (_gate) + { + _disposed = true; + _session?.Dispose(); + _session = null; + _runOptions.Dispose(); + } + } + + private static InferenceSession CreateSession() + { + using var stream = typeof(OnnxPersonSegmenter).Assembly.GetManifestResourceStream(ModelResourceName) + ?? throw new InvalidOperationException($"embedded model '{ModelResourceName}' is missing"); + using var buffer = new MemoryStream(); + stream.CopyTo(buffer); + // Two intra-op threads: the model is tiny, and the webcam must not starve the + // compositor/encoder threads (obs-backgroundremoval reports 2 as the sweet spot). + using var options = new SessionOptions + { + IntraOpNumThreads = 2, + InterOpNumThreads = 1, + GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL, + }; + return new InferenceSession(buffer.ToArray(), options); + } + + /// BGRA frame → model input: nearest-sample stretch to 256×144, RGB order, + /// values /255. Nearest is enough — the model input is ~5× smaller than a 720p frame + /// and the mask is the only thing that comes back. + internal static void FillInput(VideoFrame frame, float[] input) + { + var src = frame.BgraPixels; + var stride = frame.Stride; + var o = 0; + for (var y = 0; y < ModelHeight; y++) + { + var sy = Math.Min(frame.Height - 1, (int)((y + 0.5) * frame.Height / ModelHeight)); + var row = sy * stride; + for (var x = 0; x < ModelWidth; x++) + { + var sx = Math.Min(frame.Width - 1, (int)((x + 0.5) * frame.Width / ModelWidth)); + var p = row + sx * 4; + input[o++] = src[p + 2] / 255f; + input[o++] = src[p + 1] / 255f; + input[o++] = src[p] / 255f; + } + } + } +} diff --git a/Services/index.md b/Services/index.md index a4882a5..d9d92dc 100644 --- a/Services/index.md +++ b/Services/index.md @@ -16,6 +16,9 @@ External-facing logic: YouTube API, persistence. See | `AlertClipDecoder.cs` | **Per-play alert video decoder (TASK 47):** `IAlertClipDecoder` (Start/Stop/Dispose + `FrameAvailable`/`AudioReady`/`Completed`) + ffmpeg child emitting `rawvideo bgra` (`-vf scale=W:H`, read via `RawVideoFrameReader`) and `f32le -ar 48000 -ac 2` on a **carry-buffer** audio loop (float boundary straddles never drop samples), both real-time paced; bgra is opaque (layer owns alpha). One-shot lifetime — created per alert, disposed at drain; tests inject a fake | | `LayoutStore.cs` | SQLite persistence (`Microsoft.Data.Sqlite`) at `%APPDATA%\ytLlive\ytLlive.db`; assets stored as BLOBs keyed by SHA-256 content hash; save/open layout files; schema `user_version` 10 (`Source.ClipShape`/`IsMirrored` via `ALTER TABLE` for pre-v2 DBs; v3 = singleton `Webcam` + per-scene `WebcamSceneConfig`, migrated idempotently **without backfill** — the stale `Source.DeviceId` column remains but is no longer read/written; v4 = `WebcamSceneConfig.RectWidth`/`RectHeight`, the pre-Round rect for the round-to-rect restore; v5 = `Source.IsBackdrop` + `Source.CaptureKey`, the live-capture backdrop; v6 = `Scene.HasBackdrop` — Live-only policy, one-time backfill turns Starting/BRB/Chat/Ending off + drops their backdrop sources; v9→v10 (TASK 39) = `YppSnapshot` history table; all later columns stay additive guarded ALTERs — TASK 47 adds the five alert `Source` columns (`AlertVideoAssetId`/`AlertVideoPath`/`AlertUseDefaultVideo`/`AlertShowTicker`/`AlertVideoVolume`) without a version bump; `MainViewModel.NormalizeBackgrounds` re-normalizes on every load — TASK 25: one locked "Background" per canonical screen (static art everywhere, DisplayCapture on Live), duplicates dropped, index 0, orphaned `BackgroundUseDefault_*`/`BackgroundPath_*` Settings keys purged at save). Settings key/value table holds app state: `MicSourceName`, `ReusableStream` (TASK 9 — the cached reusable stream's id/address/name so the pump has its RTMP URL at startup), `AlertDefaultVideoAssetId` (TASK 47 — the stamped built-in alert clip's asset row; save's prune exempts it via UNION) | | `VideoFrame.cs` | Normalized CPU frame seam (`Width`/`Height`/tightly-packed BGRA `byte[]`) — the only pixel type the rest of the app knows about; future capture sources (screen, background-removed webcam) feed the same seam | +| `BackgroundEffects/IPersonSegmenter.cs` | **TASK 51 seam:** `Segment(VideoFrame) → SegmentationMask?` (`SegmentationMask(Width, Height, float[] Values)`, 0 = background, 1 = person); null = no mask → caller passes through. Tests inject fakes | +| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) | +| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = bilinear mask × source alpha (`IsOpaque=false`), Blur = passthrough until slice 2; no mask/exception → passthrough + one log. Not wired into the camera path yet | | `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device | | `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) | | `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) | diff --git a/TASKS.md b/TASKS.md index bdb96e5..0071efc 100644 --- a/TASKS.md +++ b/TASKS.md @@ -66,7 +66,7 @@ | 48 | Distribution & packaging: **MSIX + Store IAP** | 🔶 In progress — **✅ route decided 2026-09-27 (Store MSIX + Store IAP)**; **item 1 (bundle ffmpeg) SHIPPED 2026-09-28** | [`TASKS/task-48-distribution-msix.md`](TASKS/task-48-distribution-msix.md) | | 49 | Chat profanity filter (local, opt-in, non-persistent, user word list) | ☐ Queued (2026-09-27) | [`TASKS/task-49-chat-profanity-filter.md`](TASKS/task-49-chat-profanity-filter.md) | | 50 | One-button "Publish this video" (resumable upload of the local recording) | ☐ Queued (2026-09-28) | [`TASKS/task-50-publish-recording.md`](TASKS/task-50-publish-recording.md) | -| 51 | **Webcam background effects: Remove + Blur** | ☐ **v1 requirement** — booked 2026-10-08; next up now that the zoom box shipped (TASK 52) | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) | +| 51 | **Webcam background effects: Remove + Blur** | 🔶 **v1 requirement — in progress**: slice 1 engine shipped 2026-10-09 (bundled MediaPipe Selfie Segmenter → ONNX, verified vs Google's runtime; Remove); next: Blur → config → camera hookup → UI | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) | | 52 | **Zoom box (anchored 2× magnifier)** | ✅ Done — shipped 2026-10-08 (3 creator corrections during the build: loupe → anchored capture over the full composite, drawn last → never shows the dim shading) | [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md) | | 53 | **Output-format verification: Shorts/TikTok geometry (9:16)** | ☐ Queued (2026-10-08) — output formats never tested end-to-end; pin + prove the vertical geometry | [`TASKS/task-53-output-formats.md`](TASKS/task-53-output-formats.md) | | 54 | **Marcom round: free-forever positioning + merch storefront** | 🔶 In progress — llamacasty.com copy live 2026-10-08 (pricing stripped); merch POD + app About links queued | [`TASKS/task-54-marcom-free-merch.md`](TASKS/task-54-marcom-free-merch.md) | diff --git a/TASKS/task-51-webcam-bg-effects.md b/TASKS/task-51-webcam-bg-effects.md index a0d9efa..199ebf0 100644 --- a/TASKS/task-51-webcam-bg-effects.md +++ b/TASKS/task-51-webcam-bg-effects.md @@ -4,7 +4,8 @@ > **Booked 2026-10-08 by creator ruling** — *"we're doing the bg removal and blur for v1 - book it, Danno"*. > This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19). -**Status:** ☐ Booked — **v1 requirement**. Queued behind the zoom build (current in-flight work unit). +**Status:** 🔶 In progress — **slice 1 (engine: bundled model + runner + Remove) shipped 2026-10-09**. +Next: slice 2 (Blur), then config + persistence, then the camera-path hookup + UI. **Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer, selected per element in its properties panel, composited into stream + recording like any @@ -34,7 +35,8 @@ delivery shape changed: same posture as the ffmpeg bundling in TASK 48 item 1.) 2. **No hardware requirement.** CPU path must work; DirectML is an optional accelerator. 3. **Per-element (Meld-style), not global.** Each webcam element carries its own effect. -4. **No feature gating.** Works for free users; only the branding flash is paid. +4. **No feature gating.** Everything is free (free-forever ruling 2026-10-08, TASK 54 — the old + "only the branding flash is paid" line here was superseded). 5. **Scene change / webcam add/remove must not crash or stall the pump** — a failing model run skips to passthrough, never drops the layer. @@ -44,17 +46,21 @@ delivery shape changed: - ❌ **Super resolution / denoise / prettify passes** - ❌ **Virtual camera output** — permanently out of product (no vcam) -## 1. ☐ Engine +## 1. Engine -- ☐ Bundle an ONNX person-segmentation model (MediaPipe Selfie-Segmentation class, ~2–5 MB) - into the app (csproj copy like `ffmpeg/`, or as an `Asset`). Verify license + THIRD-PARTY - notices (TASK 36 item 6 gate). -- ☐ `onnxruntime` via NuGet. CPU EP by default; optionally DirectML EP later. Failing EP - init → passthrough, log once, never throw from the pump. -- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region. No - model needed for blur once the mask exists. -- ☐ Remove = **alpha 0 outside the person region.** The compositor already alpha-blends per - element (`SceneCompositor`, source-over) — no compositor change expected. +- ✅ **Model bundled (slice 1, 2026-10-09):** Google's official MediaPipe Selfie Segmenter + **landscape** (144×256, Apache-2.0 per the model card — license + changes recorded in + THIRD-PARTY-NOTICES item 12). Converted to ONNX by `tools/selfie-seg/convert_tflite_to_onnx.py` + (custom op `Convolution2DTransposeBias` → `ConvTranspose`); verified against MediaPipe's own + runtime by `tools/selfie-seg/verify_against_mediapipe.py` (max |diff| 5e-5). 430 KB, + `EmbeddedResource` in the exe. Rejected sources: PINTO #109 (ML Kit-extracted, unclear license), + obs-backgroundremoval code (GPL-3.0) — referenced for approach only (ONNX Runtime, CPU, 2 threads). +- ✅ `Microsoft.ML.OnnxRuntime` 1.31.0, CPU EP (`OnnxPersonSegmenter`). Failing load/run → + logs once, returns null, caller passes through. DirectML still optional/later. +- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region — slice 2. +- ✅ Remove = alpha = mask outside the person (`BackgroundEffectProcessor`), `IsOpaque = false`. + The compositor already alpha-blends per element — no compositor change expected (confirm in + the hookup slice). - ☐ Feed the **VideoFrame seam** (`Services/VideoFrame.cs` doc explicitly anticipates *"every future capture source (screen, background-removed webcam) feeds the same seam"*): the frame handed to BOTH the preview bitmap path and the compositor's @@ -71,19 +77,27 @@ delivery shape changed: ## 3. ☐ Test seam (Good Dog — one integration test per unit) -- ☐ Model-runner interface seam (like `CameraEnumeratorOverride`) so tests inject a fake - segmenter — no real ONNX inference in the test suite. +- ✅ Model-runner interface seam (`IPersonSegmenter`) so tests inject a fake segmenter. - ☐ Assert the **processed frame reaches both consumers** (preview bitmap + compositor frame). -- ☐ `None` = byte-identical passthrough of the raw frame. -- ☐ `Remove` = alpha 0 outside person rect; `Blur` = background region blurred, person sharp. +- ✅ `None` = same-instance passthrough of the raw frame (`BackgroundEffectEngineTests`). +- ✅ `Remove` = alpha 0 outside the person, RGB untouched, non-opaque (`BackgroundEffectEngineTests`); + ☐ `Blur` = background region blurred, person sharp. +- ✅ **Bundled-model contract:** the embedded ONNX model through the C# preprocessing reproduces + MediaPipe's reference mask on a fixture (≥99.5% agreement). This is a fixed, deterministic + contract check of the shipped bytes — not a camera/lighting quality test. - ☐ Config → persistence roundtrip (per webcam element). -- ☐ Engine failure → passthrough + single log, layer never dropped. +- ☐ Engine failure → passthrough + single log, layer never dropped (code path exists; dedicated + test lands with the hookup slice). - ☐ Real inference: **manual on the creator's box** (camera + lighting), not in CI. +- ☐ **Perf check in the hookup slice:** time `Segment` + `Apply` per 720p frame on the creator's + box before wiring it into the 60fps path (expect a few ms; the mask can run at a lower rate + than the camera if needed). -## 4. ☐ Sequencing +## 4. Sequencing -- ☐ **Separate work unit AFTER the zoom build.** This file is booked + scoped; do not - co-commit it with the zoom work. +- ✅ Separate work unit after the zoom build (zoom shipped 2026-10-08). +- Slice order: 1 engine ✅ → 2 Blur → 3 config + persistence → 4 camera-path hookup (one + processing point, both consumers) + perf check → 5 properties-panel UI. ## Related (not duplicated here) diff --git a/THIRD-PARTY-NOTICES.txt b/THIRD-PARTY-NOTICES.txt index 7b2cc64..ec96224 100644 --- a/THIRD-PARTY-NOTICES.txt +++ b/THIRD-PARTY-NOTICES.txt @@ -115,7 +115,27 @@ there on purpose. Transitive dependencies (Microsoft.Xaml.Behaviors.Wpf, Microsoft.Extensions.DependencyInjection.Abstractions) are MIT, covered by item 7's .NET notice policy; this notice is kept per - this file's "list everything" policy. + this file's "list everything" policy. + +11. ONNX Runtime (Microsoft.ML.OnnxRuntime — runs the webcam background model, TASK 51) + Copyright (c) Microsoft Corporation + License: MIT + Home: https://github.com/microsoft/onnxruntime + Used as: the CPU inference runtime (managed package + native onnxruntime.dll) + behind the `IPersonSegmenter` seam. MIT imposes no source offer. + +12. MediaPipe Selfie Segmenter (landscape) (webcam background Remove/Blur, TASK 51) + Copyright (c) Google LLC (authors: Tingbo Hou, Siargey Pisarchyk, Karthik + Raveendran — model card "MediaPipe Selfie Segmentation", 2021-05-06) + License: Apache License 2.0 + Home: https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter + Source: https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/float16/latest/selfie_segmenter_landscape.tflite + Changes (Apache-2.0 §4b): converted from TensorFlow Lite to ONNX format by + tools/selfie-seg/convert_tflite_to_onnx.py — float16 weights widened + to float32, NHWC graph re-expressed in NCHW; the weights and the + network are otherwise unmodified. Embedded in the application as + selfie_segmenter_landscape.onnx. Runs entirely on the user's PC. + -------------------------------------------------------------------------------- diff --git a/ai.md b/ai.md index 1abdeb7..c4b0b5a 100644 --- a/ai.md +++ b/ai.md @@ -602,12 +602,47 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr - **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred to the encoder task. -- **Background removal = PERMANENTLY OUT** (creator ruling 2026-10-05, confirmed: "no - not doing - it. the note is correct and accurate"). The line here used to read "milestone 2", which was stale — - it was never a plan, only a leftover from TASK 3's original scoping. The rationale that stands: - a vcam app (NVIDIA Broadcast, XSplit VCam) does it free, and an in-app model means a runtime, a - GPU/driver matrix, and per-scene lighting tuning — support hell for a solo dev - (`TASKS/task-36-gold-pass.md`, `TASKS/task-03-capture-pipeline.md` item 19). Don't reopen it. +- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The + 2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix, + and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe, + run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video + Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background + effects" below and `TASKS/task-51-webcam-bg-effects.md`. + +### Webcam background effects (TASK 51 — slice 1 engine shipped 2026-10-09) + +- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06), + official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/ + float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own + `tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op + `Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct + translation was smaller than any workaround). **Verified against Google's own runtime:** + `verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original + .tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images. + ⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance) or + copy code from obs-backgroundremoval (GPL-3.0). +- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The + whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates. +- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git, + inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime = + `Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native + `onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX + publish (TASK 48) must pin the RID so only win-x64 ships. +- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) → + SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded + resource; **never throws** — a load/run failure logs once and returns null forever after); + `BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true + passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched, + `IsOpaque = false` (the compositor's straight-alpha blend shows the layers below); `Blur` passes + through until slice 2. No mask / segmenter exception → passthrough + one log line, never a dropped + layer. +- **Not wired yet:** nothing calls the processor — no config, no UI, no `CameraManager` hookup. The + next slices add Blur, the per-webcam `BackgroundEffect` config + persistence, then the single + processing point on the camera path (both preview and compositor consume the processed frame). +- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a + model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and + matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test + image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). ### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05) @@ -1652,6 +1687,11 @@ guardrails below — written down so a future "quick fix" never reintroduces a G another public-domain source; a logo asset under a copyleft or attribution license would contaminate the paid product. The About hub's logo is the creator's own art ("llama fortnite superman logo", `Assets/llama-logo.png`) — owned by us, so it carries no third-party license either. +- **Ship an ML model without clean provenance** — the background-effects model (TASK 51) is Google's + official MediaPipe Selfie Segmenter download (Apache-2.0, model card), converted by our own script. + Model-zoo re-hosts (e.g. PINTO #109 = ML Kit's model, extracted) and GPL plugins + (obs-backgroundremoval, GPL-3.0) are off-limits. Apache-2.0 §4 obligations (license notice + "we + converted it") live in `THIRD-PARTY-NOTICES.txt`. - **Forget the v1 license-texts gate** — `THIRD-PARTY-NOTICES.txt` links the canonical license texts; at **v1 (GA)** the full texts of every license it names MUST ship alongside it (TASK 4 requirement 9 is the release blocker). Queued early is wrong; the release pass owns it. diff --git a/tools/selfie-seg/convert_tflite_to_onnx.py b/tools/selfie-seg/convert_tflite_to_onnx.py new file mode 100644 index 0000000..d1951f5 --- /dev/null +++ b/tools/selfie-seg/convert_tflite_to_onnx.py @@ -0,0 +1,218 @@ +"""Convert Google's MediaPipe Selfie Segmenter (.tflite) to ONNX for LlamaCasty (TASK 51). + +Why a hand-rolled converter: the model ends in MediaPipe's custom op +`Convolution2DTransposeBias`, which tf2onnx / tflite2onnx don't support. The graph only +uses 11 op types, so a direct, auditable translation is smaller than any workaround. + +Source model (Apache-2.0, model card "MediaPipe Selfie Segmentation", Google, 2021-05-06): + https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/float16/latest/selfie_segmenter_landscape.tflite + +Layout: TFLite is NHWC. The ONNX graph keeps the SAME external contract +(input [1,144,256,3] float RGB in [0,1], output [1,144,256,1] person probability in [0,1]) +and transposes to NCHW internally. + +Usage: python convert_tflite_to_onnx.py in.tflite out.onnx +Requires: tflite, onnx, numpy. Verify with verify_against_mediapipe.py. +""" +import sys + +import numpy as np +import onnx +import tflite +from onnx import TensorProto, helper, numpy_helper +from tflite.ActivationFunctionType import ActivationFunctionType as Act +from tflite.BuiltinOperator import BuiltinOperator +from tflite.Padding import Padding +from tflite.TensorType import TensorType + +OPSET = 17 +OP_NAMES = {v: k for k, v in BuiltinOperator.__dict__.items() if not k.startswith("_")} +NP_TYPES = {TensorType.FLOAT32: np.float32, TensorType.FLOAT16: np.float16, TensorType.INT32: np.int32} + + +def main(src, dst): + model = tflite.Model.GetRootAsModel(open(src, "rb").read(), 0) + g = model.Subgraphs(0) + + def shape(ix): + return g.Tensors(ix).ShapeAsNumpy().tolist() + + def const(ix): + """Constant tensor value, or None for runtime tensors.""" + t = g.Tensors(ix) + buf = model.Buffers(t.Buffer()) + if buf is None or buf.DataLength() == 0: + return None + raw = buf.DataAsNumpy().tobytes() + return np.frombuffer(raw, dtype=NP_TYPES[t.Type()]).reshape(shape(ix)) + + nodes, inits = [], [] + consts = {} # tensor index -> float32 numpy (after DEQUANTIZE folding) + uid = [0] + + def name(ix): + return f"t{ix}" + + def fresh(prefix): + uid[0] += 1 + return f"{prefix}_{uid[0]}" + + def add_init(arr, nm=None): + nm = nm or fresh("c") + inits.append(numpy_helper.from_array(np.ascontiguousarray(arr), nm)) + return nm + + def value(ix): + if ix in consts: + return consts[ix] + c = const(ix) + if c is not None: + consts[ix] = c.astype(np.float32) if c.dtype == np.float16 else c + return consts[ix] + return None + + def activation(out_name, act, final_name): + if act == Act.NONE: + return out_name + if act == Act.RELU: + nodes.append(helper.make_node("Relu", [out_name], [final_name])) + elif act == Act.RELU6: + nodes.append(helper.make_node("Clip", [out_name, add_init(np.float32(0)), add_init(np.float32(6))], [final_name])) + else: + raise NotImplementedError(f"fused activation {act}") + return final_name + + def operand(ix): + """Runtime tensor name, or a constant reshaped from NHWC broadcast form to NCHW.""" + v = value(ix) + if v is None: + return name(ix) + if v.ndim == 1: + v = v.reshape(1, -1, 1, 1) + elif v.ndim == 4: + v = v.transpose(0, 3, 1, 2) + return add_init(v.astype(np.float32)) + + # NHWC -> NCHW at the input. + inp = g.Inputs(0) + nodes.append(helper.make_node("Transpose", ["input"], [name(inp)], perm=[0, 3, 1, 2])) + + for i in range(g.OperatorsLength()): + op = g.Operators(i) + oc = model.OperatorCodes(op.OpcodeIndex()) + code = max(oc.BuiltinCode(), oc.DeprecatedBuiltinCode()) + kind = "CUSTOM:" + oc.CustomCode().decode() if code == BuiltinOperator.CUSTOM else OP_NAMES[code] + ins = [op.Inputs(j) for j in range(op.InputsLength())] + out = op.Outputs(0) + o = name(out) + + if kind == "DEQUANTIZE": + consts[out] = const(ins[0]).astype(np.float32) + continue + + if kind in ("CONV_2D", "DEPTHWISE_CONV_2D"): + opt = (tflite.Conv2DOptions() if kind == "CONV_2D" else tflite.DepthwiseConv2DOptions()) + bo = op.BuiltinOptions() + opt.Init(bo.Bytes, bo.Pos) + w = value(ins[1]) + b = value(ins[2]) if len(ins) > 2 and ins[2] >= 0 else None + if kind == "CONV_2D": + W = w.transpose(0, 3, 1, 2) # OHWI -> OIHW + group = 1 + else: + # [1,H,W,C*mult] -> [C*mult,1,H,W], group = input channels + W = w.transpose(3, 0, 1, 2) + group = shape(ins[0])[3] + attrs = dict( + strides=[opt.StrideH(), opt.StrideW()], + dilations=[opt.DilationHFactor(), opt.DilationWFactor()], + kernel_shape=list(W.shape[2:]), + group=group, + auto_pad="SAME_UPPER" if opt.Padding() == Padding.SAME else "VALID", + ) + conv_in = [name(ins[0]), add_init(W)] + ([add_init(b)] if b is not None else []) + tmp = fresh("conv") if opt.FusedActivationFunction() != Act.NONE else o + nodes.append(helper.make_node("Conv", conv_in, [tmp], **attrs)) + activation(tmp, opt.FusedActivationFunction(), o) + continue + + if kind in ("ADD", "MUL"): + opt = tflite.AddOptions() if kind == "ADD" else tflite.MulOptions() + bo = op.BuiltinOptions() + act = Act.NONE + if bo is not None: + opt.Init(bo.Bytes, bo.Pos) + act = opt.FusedActivationFunction() + tmp = fresh("ew") if act != Act.NONE else o + nodes.append(helper.make_node("Add" if kind == "ADD" else "Mul", [operand(ins[0]), operand(ins[1])], [tmp])) + activation(tmp, act, o) + continue + + if kind == "RELU": + nodes.append(helper.make_node("Relu", [name(ins[0])], [o])) + continue + if kind == "HARD_SWISH": + nodes.append(helper.make_node("HardSwish", [name(ins[0])], [o])) + continue + if kind == "LOGISTIC": + nodes.append(helper.make_node("Sigmoid", [name(ins[0])], [o])) + continue + + if kind == "MEAN": + opt = tflite.ReducerOptions() + bo = op.BuiltinOptions() + opt.Init(bo.Bytes, bo.Pos) + nhwc_to_nchw = {0: 0, 1: 2, 2: 3, 3: 1} + axes = [nhwc_to_nchw[int(a)] for a in value(ins[1]).reshape(-1)] + nodes.append(helper.make_node("ReduceMean", [name(ins[0])], [o], axes=axes, keepdims=1 if opt.KeepDims() else 0)) + continue + + if kind == "RESIZE_BILINEAR": + opt = tflite.ResizeBilinearOptions() + bo = op.BuiltinOptions() + opt.Init(bo.Bytes, bo.Pos) + h, w = (int(x) for x in value(ins[1]).reshape(-1)) + c = shape(ins[0])[3] + mode = "align_corners" if opt.AlignCorners() else ("half_pixel" if opt.HalfPixelCenters() else "asymmetric") + sizes = add_init(np.array([1, c, h, w], dtype=np.int64)) + nodes.append(helper.make_node("Resize", [name(ins[0]), "", "", sizes], [o], mode="linear", coordinate_transformation_mode=mode)) + continue + + if kind == "CUSTOM:Convolution2DTransposeBias": + # Custom options = TfLiteTransposeConvParams {padding, stride_w, stride_h} (int32 each). + # NOTE: `padding` is the C-API TfLitePadding enum (Unknown=0, SAME=1, VALID=2), + # NOT the flatbuffer Padding enum (SAME=0, VALID=1). + padding, stride_w, stride_h = np.frombuffer(bytes(op.CustomOptionsAsNumpy().tolist()), dtype=np.int32)[:3] + w = value(ins[1]) # [O, kH, kW, I] + b = value(ins[2]) + W = w.transpose(3, 0, 1, 2) # -> [I, O, kH, kW] (ONNX ConvTranspose) + kh, kw = w.shape[1], w.shape[2] + if padding != 1 or kh != stride_h or kw != stride_w: + raise NotImplementedError("only SAME, kernel == stride transpose conv is supported") + nodes.append(helper.make_node("ConvTranspose", [name(ins[0]), add_init(W), add_init(b)], [o], + strides=[int(stride_h), int(stride_w)], kernel_shape=[kh, kw])) + continue + + raise NotImplementedError(kind) + + # NCHW -> NHWC at the output. + outp = g.Outputs(0) + nodes.append(helper.make_node("Transpose", [name(outp)], ["mask"], perm=[0, 2, 3, 1])) + + graph = helper.make_graph( + nodes, "mediapipe_selfie_segmenter_landscape", + [helper.make_tensor_value_info("input", TensorProto.FLOAT, shape(inp))], + [helper.make_tensor_value_info("mask", TensorProto.FLOAT, shape(outp))], + initializer=inits) + m = helper.make_model(graph, opset_imports=[helper.make_opsetid("", OPSET)], + producer_name="llamacasty-tflite2onnx", + doc_string="MediaPipe Selfie Segmenter (landscape), Apache-2.0, Google. " + "Converted for LlamaCasty by tools/selfie-seg/convert_tflite_to_onnx.py") + m.ir_version = 8 + onnx.checker.check_model(m) + onnx.save(m, dst) + print(f"wrote {dst}: {len(nodes)} nodes, {len(inits)} initializers") + + +if __name__ == "__main__": + main(sys.argv[1], sys.argv[2]) diff --git a/tools/selfie-seg/verify_against_mediapipe.py b/tools/selfie-seg/verify_against_mediapipe.py new file mode 100644 index 0000000..80c7c8e --- /dev/null +++ b/tools/selfie-seg/verify_against_mediapipe.py @@ -0,0 +1,59 @@ +"""Verify the converted ONNX model against Google's own MediaPipe runtime (TASK 51). + +Feeds the SAME 256x144 RGB frame to (a) MediaPipe's ImageSegmenter running the original +.tflite (with its custom op) and (b) onnxruntime running our converted .onnx, then compares +the person-probability masks. Also writes side-by-side PNGs for an eyeball check. + +Usage: python verify_against_mediapipe.py model.tflite model.onnx photo.jpg [out_prefix] +Requires: mediapipe, onnxruntime, numpy, pillow. +""" +import sys + +import mediapipe as mp +import numpy as np +import onnxruntime as ort +from mediapipe.tasks import python as mp_python +from mediapipe.tasks.python import vision +from PIL import Image + +W, H = 256, 144 + + +def main(tfl, onx, photo, prefix="verify"): + img = Image.open(photo).convert("RGB") + # Cover-crop to 16:9, then resize to the model's exact input so neither runtime resamples. + sw, sh = img.size + target = W / H + if sw / sh > target: + nw = int(sh * target); x = (sw - nw) // 2; img = img.crop((x, 0, x + nw, sh)) + else: + nh = int(sw / target); y = (sh - nh) // 2; img = img.crop((0, y, sw, y + nh)) + img = img.resize((W, H), Image.BILINEAR) + rgb = np.asarray(img, dtype=np.uint8) + + seg = vision.ImageSegmenter.create_from_options(vision.ImageSegmenterOptions( + base_options=mp_python.BaseOptions(model_asset_path=tfl), output_confidence_masks=True)) + res = seg.segment(mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb)) + ref = res.confidence_masks[0].numpy_view().astype(np.float32).reshape(H, W) + + sess = ort.InferenceSession(onx, providers=["CPUExecutionProvider"]) + x = (rgb.astype(np.float32) / 255.0)[None] + got = sess.run(None, {"input": x})[0].reshape(H, W) + + diff = np.abs(ref - got) + agree = np.mean((ref > 0.5) == (got > 0.5)) + print(f"ref range {ref.min():.3f}..{ref.max():.3f} person%={np.mean(ref > 0.5) * 100:.1f}") + print(f"onnx range {got.min():.3f}..{got.max():.3f} person%={np.mean(got > 0.5) * 100:.1f}") + print(f"max|diff|={diff.max():.5f} mean|diff|={diff.mean():.6f} binary-agreement={agree * 100:.2f}%") + + def to_img(m): + return Image.fromarray((np.clip(m, 0, 1) * 255).astype(np.uint8)).convert("RGB") + sheet = Image.new("RGB", (W * 3, H)) + sheet.paste(img, (0, 0)); sheet.paste(to_img(ref), (W, 0)); sheet.paste(to_img(got), (W * 2, 0)) + sheet.resize((W * 6, H * 2), Image.NEAREST).save(f"{prefix}_sheet.png") + return diff.max() + + +if __name__ == "__main__": + md = main(*sys.argv[1:]) + sys.exit(0 if md < 1e-3 else 1) diff --git a/ytLive.Tests/BackgroundEffectEngineTests.cs b/ytLive.Tests/BackgroundEffectEngineTests.cs new file mode 100644 index 0000000..48264bd --- /dev/null +++ b/ytLive.Tests/BackgroundEffectEngineTests.cs @@ -0,0 +1,115 @@ +using System.IO; +using System.Windows.Media; +using System.Windows.Media.Imaging; +using Xunit; +using ytLive.Models; +using ytLive.Services; +using ytLive.Services.BackgroundEffects; + +namespace ytLive.Tests; + +/// TASK 51 slice 1 — the background-effects engine. +public class BackgroundEffectEngineTests +{ + private sealed class FakeSegmenter : IPersonSegmenter + { + private readonly SegmentationMask? _mask; + public int Calls; + public FakeSegmenter(SegmentationMask? mask) => _mask = mask; + public SegmentationMask? Segment(VideoFrame frame) + { + Calls++; + return _mask; + } + } + + /// The integration test for this unit: a webcam frame run through the + /// processor with Remove comes back with the person kept and the background + /// transparent, colors untouched, flagged non-opaque so the compositor blends it. + /// None is a true passthrough (same instance, segmenter never called). + [Fact] + public void Remove_keeps_the_person_and_makes_the_background_transparent() + { + const int w = 64, h = 32; + var pixels = new byte[w * h * 4]; + for (var i = 0; i < pixels.Length; i += 4) + { + pixels[i] = 10; pixels[i + 1] = 20; pixels[i + 2] = 30; pixels[i + 3] = 255; + } + var frame = new VideoFrame(w, h, pixels) { IsOpaque = true }; + + // 4×2 mask: left half person, right half background. + var mask = new SegmentationMask(4, 2, new[] { 1f, 1f, 0f, 0f, 1f, 1f, 0f, 0f }); + var segmenter = new FakeSegmenter(mask); + var processor = new BackgroundEffectProcessor(segmenter); + + Assert.Same(frame, processor.Apply(frame, BackgroundEffect.None)); + Assert.Equal(0, segmenter.Calls); + + var result = processor.Apply(frame, BackgroundEffect.Remove); + + Assert.NotSame(frame, result); + Assert.False(result.IsOpaque); + Assert.Equal(w, result.Width); + Assert.Equal(h, result.Height); + for (var y = 0; y < h; y++) + { + Assert.Equal(255, Alpha(result, 4, y)); // well inside the person half + Assert.Equal(0, Alpha(result, w - 5, y)); // well inside the background half + } + for (var i = 0; i < pixels.Length; i += 4) + { + Assert.Equal(pixels[i], result.BgraPixels[i]); + Assert.Equal(pixels[i + 1], result.BgraPixels[i + 1]); + Assert.Equal(pixels[i + 2], result.BgraPixels[i + 2]); + } + Assert.Equal(255, frame.BgraPixels[(w - 5) * 4 + 3]); // source frame never mutated + } + + /// Model contract: the BUNDLED (embedded) ONNX model, driven through the C# + /// preprocessing, reproduces Google's own MediaPipe runtime on a fixed fixture. + /// The expected mask was produced by tools/selfie-seg/ running the original .tflite in + /// MediaPipe (fixture: MediaPipe's public test image mozart_square.jpg, a public-domain + /// painting, cropped to 256×144). Guards the embedding, the input layout (RGB, NHWC, + /// /255) and the conversion itself. + [Fact] + public void Bundled_model_matches_the_MediaPipe_reference_mask() + { + var frame = LoadBgra(Fixture("selfie-fixture-256x144.png")); + var expected = LoadBgra(Fixture("selfie-fixture-expected-mask.png")); + + using var segmenter = new OnnxPersonSegmenter(); + var mask = segmenter.Segment(frame); + + Assert.NotNull(mask); + Assert.Equal(OnnxPersonSegmenter.ModelWidth, mask!.Width); + Assert.Equal(OnnxPersonSegmenter.ModelHeight, mask.Height); + + var agree = 0; + var person = 0; + for (var i = 0; i < mask.Width * mask.Height; i++) + { + var expectPerson = expected.BgraPixels[i * 4] > 127; + var gotPerson = mask.Values[i] > 0.5f; + if (expectPerson == gotPerson) agree++; + if (gotPerson) person++; + } + var total = mask.Width * mask.Height; + Assert.True(agree >= total * 0.995, $"binary agreement {100.0 * agree / total:0.00}% < 99.5%"); + Assert.InRange(100.0 * person / total, 45, 60); // reference: 52.5% person + } + + private static int Alpha(VideoFrame f, int x, int y) => f.BgraPixels[(y * f.Width + x) * 4 + 3]; + + private static string Fixture(string name) => Path.Combine(AppContext.BaseDirectory, "Fixtures", name); + + private static VideoFrame LoadBgra(string path) + { + using var stream = File.OpenRead(path); + var decoded = BitmapDecoder.Create(stream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.OnLoad).Frames[0]; + var bgra = new FormatConvertedBitmap(decoded, PixelFormats.Bgra32, null, 0); + var pixels = new byte[bgra.PixelWidth * bgra.PixelHeight * 4]; + bgra.CopyPixels(pixels, bgra.PixelWidth * 4, 0); + return new VideoFrame(bgra.PixelWidth, bgra.PixelHeight, pixels); + } +} diff --git a/ytLive.Tests/Fixtures/selfie-fixture-256x144.png b/ytLive.Tests/Fixtures/selfie-fixture-256x144.png new file mode 100644 index 0000000..6fa1759 Binary files /dev/null and b/ytLive.Tests/Fixtures/selfie-fixture-256x144.png differ diff --git a/ytLive.Tests/Fixtures/selfie-fixture-expected-mask.png b/ytLive.Tests/Fixtures/selfie-fixture-expected-mask.png new file mode 100644 index 0000000..27e96ac Binary files /dev/null and b/ytLive.Tests/Fixtures/selfie-fixture-expected-mask.png differ diff --git a/ytLive.Tests/ytLive.Tests.csproj b/ytLive.Tests/ytLive.Tests.csproj index dc7b95b..7dd1042 100644 --- a/ytLive.Tests/ytLive.Tests.csproj +++ b/ytLive.Tests/ytLive.Tests.csproj @@ -18,4 +18,8 @@ + + + + diff --git a/ytLive.csproj b/ytLive.csproj index 135b622..043b185 100644 --- a/ytLive.csproj +++ b/ytLive.csproj @@ -75,6 +75,14 @@ + + + + +