TASK 51 slice 1: background-effects engine — bundled MediaPipe Selfie Segmenter (ONNX) + Remove (Unit: bg-effects engine)
- Model: Google's official MediaPipe Selfie Segmenter (landscape, Apache-2.0 per model card), converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op Convolution2DTransposeBias -> ConvTranspose) and verified against MediaPipe's own runtime (verify_against_mediapipe.py: max |diff| 5e-5, 100% binary agreement). Embedded resource, 430 KB. - Services/BackgroundEffects: IPersonSegmenter seam, OnnxPersonSegmenter (onnxruntime 1.31.0 CPU, never throws), BackgroundEffectProcessor (None passthrough, Remove = alpha mask). Not wired yet. - Tests: Remove integration test (fake segmenter) + bundled-model contract test vs MediaPipe reference mask. Full suite 407/408 — the one failure is the documented interactive-desktop flake RealMouseDrag_OnTheLayerList_PersistsTheReorder (passed in the run before; code path untouched). - Docs: ai.md reversal + section + licensing guard, THIRD-PARTY-NOTICES 11-12, TASK 51, indexes, MyMistakes recipe, HANDOFF. References: obs-backgroundremoval (approach only — ONNX Runtime on CPU, 2 threads; GPL-3.0, no code used) https://github.com/royshil/obs-backgroundremoval ; MediaPipe image segmenter https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
This commit is contained in:
Binary file not shown.
+32
-50
@@ -1,60 +1,42 @@
|
||||
# HANDOFF — 2026-10-08 (zoom box shipped, READY FOR PUSH)
|
||||
# HANDOFF — 2026-10-09 (TASK 51 slice 1 shipped: background-effects engine)
|
||||
|
||||
## Branch
|
||||
- main (no feature branch)
|
||||
- main (no feature branch). Pushed through `f8dd107`; the TASK 51 slice-1 commit sits on top,
|
||||
**not pushed** (push at the user's call).
|
||||
|
||||
## Push readiness
|
||||
- **5 commits ahead of origin/main, clean to push** (fetch confirmed origin has not moved):
|
||||
`966e718` zoom box visuals → `4b3d82f` zoom as layered + TASK 51 booking → `6bff21b`
|
||||
anchored-capture rework → `948dd08` dims-shading fix → (docs unit: TASK 52 record +
|
||||
numbering fixes + this HANDOFF).
|
||||
- The pinned-chat work unit below is **uncommitted on purpose** — `git push` sends only
|
||||
commits, so pushing is safe; the dirty files stay local.
|
||||
## Dirty files (NOT mine — do NOT commit or co-edit)
|
||||
- The in-flight "pinned chat" work unit: `Models/YouTube.cs`, `ViewModels/MainViewModel.Chat.cs`,
|
||||
`Controls/LeftPanel.xaml`, untracked `Helpers/RelayCommandGeneric.cs`.
|
||||
- The creator's untracked logo art in `Assets/` (`llamacasty text.*`, `llamacasty-text-nobg*.png`,
|
||||
`thin logo wordmark.png`).
|
||||
|
||||
## Dirty files (NOT mine — the in-flight "pinned chat" work unit; do NOT commit or co-edit)
|
||||
- `Models/YouTube.cs`
|
||||
- `ViewModels/MainViewModel.Chat.cs`
|
||||
- `Controls/LeftPanel.xaml` (pinned-message UI)
|
||||
- Untracked: `Helpers/RelayCommandGeneric.cs`
|
||||
|
||||
## What just happened (2026-10-08, third zoom pass — dims shading)
|
||||
The zoom box shipped as a **loupe** (`4b3d82f`), then was corrected to an **anchored-capture
|
||||
magnifier over the full composite drawn last** (commit `6bff21b`). The final correction: as the box
|
||||
is dragged, the dim rectangles (55% black "div shading" around the box) were sliding under the
|
||||
capture region — shown inside the box because its `VisualBrush` samples all of `CanvasGrid`, and the
|
||||
`DimRects` ItemsControl previously lived INSIDE `CanvasGrid`.
|
||||
|
||||
Fix: the `DimRects` overlay is now a **sibling of `CanvasGrid`** (renders above the composed
|
||||
content, below the `ZoomBox`), never a descendant — so the zoom sample can never pick up the shading
|
||||
and the captured area stays clear no matter where the box is dragged. Behavior consequence to note:
|
||||
in non-zoom ("quality dims") mode the dims now also shade the global overlays (brand flash, alert
|
||||
ticker, social bar) that sit above them in the old tree — consistent with "chrome outside the crop
|
||||
is dimmed". Structural invariant added to `ZoomBoxInteractionTests`: `DimRectsLayer` is a direct
|
||||
child of `PreviewCanvas`, not inside `CanvasGrid`, and renders below the `ZoomBox`.
|
||||
|
||||
## Zoom feature state
|
||||
- **TASK 52 (zoom box) is now a first-class record** — see [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md):
|
||||
feature provenance + the three creator corrections, shapes, implementation map, tests.
|
||||
TASK 51 (webcam background Remove + Blur) is unblocked for the next unit.
|
||||
- Models: `ZoomState.cs`, `ZoomShape.cs` (Circle/VerticalRectangle/HorizontalRectangle), `ZoomSceneElement.cs`.
|
||||
- ViewModel (`MainViewModel.Zoom.cs`): `ActivateZoom`/`ToggleZoom`/`DismissZoom`/`StartZoomMove`/`UpdateZoomMove`/`EndZoomMove` + `RebuildDimRectsForZoom`; transient element healed/mirrored by `HealZoomLayers`/`EnsureZoomLayer`.
|
||||
- Output: box composited last via `BlitZoom`/`ZoomMagnifier`; output crop = tier (box never the crop).
|
||||
- Preview: `ZoomBox` + `DimRectsLayer` are both SIBLINGS of `CanvasGrid` (box above dims above content); box samples the full composite at the fixed capture, 2x; circle→ellipse clip, rect/square outline per shape; `LeftPanel` hides the placeholder row via `IsZoomLayer`.
|
||||
- Rectangle shape = 4:3; Circle square; Phone 9:16. Capture anchored on click (`CaptureX/Y`); dragging the box never changes it.
|
||||
## What just happened — TASK 51 slice 1 (engine)
|
||||
- **Model:** Google's official MediaPipe Selfie Segmenter (landscape, 144×256, Apache-2.0),
|
||||
converted to ONNX by our own `tools/selfie-seg/convert_tflite_to_onnx.py` and **verified against
|
||||
MediaPipe's own runtime** (`verify_against_mediapipe.py`, max |diff| 5e-5). Embedded in the exe
|
||||
(`Assets/Models/selfie_segmenter_landscape.onnx`, 430 KB, in git).
|
||||
- **Code:** `Services/BackgroundEffects/` — `IPersonSegmenter` seam, `OnnxPersonSegmenter`
|
||||
(onnxruntime 1.31.0 CPU, never throws), `BackgroundEffectProcessor` (None passthrough, Remove =
|
||||
alpha mask, Blur = passthrough until slice 2). `Models/BackgroundEffect.cs` enum.
|
||||
- **Not wired:** nothing calls it yet — no config, UI, or camera hookup. Zero risk to the live path.
|
||||
- **Docs:** ai.md (stale "background removal PERMANENTLY OUT" bullet replaced by the reversal + a
|
||||
"Webcam background effects" section + a model-provenance licensing guard), THIRD-PARTY-NOTICES
|
||||
items 11 (ONNX Runtime, MIT) + 12 (MediaPipe model, Apache-2.0 with §4b change note),
|
||||
TASK 51 file + TASKS.md row, Services/Models indexes, MyMistakes conversion recipe.
|
||||
|
||||
## Tests
|
||||
- Full suite: **406 passed / 0 failed** on a clean full run (the pre-existing audio-loopback timing
|
||||
flake and `RealMouseDrag`'s interactive-desktop requirement recur occasionally — not zoom
|
||||
regressions). Zoom tests 3/3 green, 0 build warnings.
|
||||
- Full suite **408 passed / 0 failed**, 0 build warnings. New: `BackgroundEffectEngineTests` (Remove
|
||||
integration test with a fake segmenter; bundled-model contract test vs a MediaPipe reference mask,
|
||||
fixtures in `ytLive.Tests/Fixtures/`).
|
||||
|
||||
## Known issues / follow-ups
|
||||
- **Background "Use default background" toggle still not working** (carried from 2026-10-07):
|
||||
binding exists (`UseDefaultBackground`), VM handler invalidates bake; investigate when time permits.
|
||||
- The 3px selection / crop-outline strokes could still show in the zoom box IF the capture overlaps
|
||||
them in edit mode — not shading, thin outlines only; the dims (the user's complaint) are gone.
|
||||
- `onnxruntime.dll` is ~17 MB; an RID-less build copies every RID's native folder — TASK 48's MSIX
|
||||
publish must pin win-x64.
|
||||
- `THIRD-PARTY-NOTICES.txt` header still says "paid, closed-source product" (pre free-forever) — noted,
|
||||
not changed (out of scope).
|
||||
- Carried: "Use default background" toggle still not working (from 2026-10-07).
|
||||
|
||||
## Next step
|
||||
- User runs the app live to eyeball: drag the zoom box anywhere and the captured content stays clean
|
||||
(no dim shading in the box; the scene outside the box stays dimmed). Then TASK 51 (webcam
|
||||
background Remove + Blur) per `TASKS/task-51-webcam-bg-effects.md`, and the background-toggle
|
||||
investigation if still open.
|
||||
- TASK 51 slice 2: **Blur** in `BackgroundEffectProcessor` (separable box/gaussian, strength), one
|
||||
integration test. Then slice 3 config + persistence, slice 4 camera-path hookup + perf check on the
|
||||
creator's box, slice 5 properties-panel UI.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
namespace ytLive.Models;
|
||||
|
||||
/// <summary>Per-webcam background effect (TASK 51). <see cref="Remove"/> makes everything
|
||||
/// outside the person transparent so the layers below show through; <see cref="Blur"/>
|
||||
/// blurs the background behind the person.</summary>
|
||||
public enum BackgroundEffect
|
||||
{
|
||||
None,
|
||||
Remove,
|
||||
Blur,
|
||||
}
|
||||
@@ -11,6 +11,7 @@ Plain data types. No logic beyond what a property can carry. See
|
||||
| `SceneElement.cs` | Base for anything placeable in a scene: shared layout/clip/mirror/border surface, `virtual IsWebcam`/`virtual IsImageSource`/`virtual IsZoomLayer` (TASK 52); `ToggleClipShape` + persisted `RectWidth`/`RectHeight` (pre-Round rect so round→rect restores after reload) |
|
||||
| `ZoomSceneElement.cs` | **Transient zoom box (TASK 52, creator-corrected 2026-10-08):** carries the box rect AND the anchored `CaptureX`/`CaptureY` (the fixed box/2 capture window set on `ActivateZoom`; moving the box never touches it). `IsZoomActive` flips `Kind` Static→Dynamic (leaves the baked base while active); never persisted (LayoutStore save skips non-Source/non-WebcamSceneConfig rows), never shown in the layers list or the element template (`IsZoomLayer` gates both). The compositor SKIPS the active element in every layer loop and draws its 2x content LAST (layer n+1) via `ZoomMagnifier`. |
|
||||
| `Webcam.cs` | Singleton webcam identity: `Id`, `DeviceId`, `Name` (one row app-wide) |
|
||||
| `BackgroundEffect.cs` | **TASK 51:** per-webcam background effect enum `None` / `Remove` / `Blur` (not yet on `WebcamSceneConfig` — config + persistence is slice 3) |
|
||||
| `WebcamSceneConfig.cs` | Per-scene webcam placement (subclass of `SceneElement`): geometry + `IsVisible` + border (`BorderColor`/`BorderOpacity`/`BorderWidth`/`BorderAnimation`) + `VideoImageSource`; `WebcamId` links to `Webcam` |
|
||||
| `QualityOption.cs` | Resolution tier record `(Display, Fps, Bitrate, Width, Height)`; `Label` formats as `1080p60 (8Mbps)` — drives the bottom-bar dropdown and the output-rect math |
|
||||
| `StreamConfig.cs` | `StreamConfig` (note: `TargetBitrate`/`Resolution` defaults are stale — not yet wired to the dropdown), `StreamHealth` (bitrate/FPS/dropped/duration), `StreamStatus` enum |
|
||||
|
||||
@@ -15,6 +15,22 @@
|
||||
|
||||
## 🔬 Recipes registry
|
||||
|
||||
### CONVERTING A MEDIAPIPE .tflite (WITH CUSTOM OPS) TO ONNX — AND PROVING IT (RECIPE, TASK 51)
|
||||
|
||||
**Problem:** MediaPipe segmentation models end in the custom op `Convolution2DTransposeBias`;
|
||||
tf2onnx and tflite2onnx refuse it. **Recipe:** `tools/selfie-seg/convert_tflite_to_onnx.py` —
|
||||
parse the flatbuffer with the `tflite` pip package, emit ONNX with `onnx.helper`, transpose
|
||||
NHWC→NCHW once at input/output, fold every `DEQUANTIZE` (fp16 weights) into fp32 initializers.
|
||||
Weight layouts: CONV OHWI→OIHW; DEPTHWISE [1,H,W,C]→[C,1,H,W] with `group=C`; transpose-conv
|
||||
[O,H,W,I]→[I,O,H,W]. TF `SAME` padding = ONNX `auto_pad=SAME_UPPER`.
|
||||
- ⚠️ **Enum trap:** the custom op's options are the C-API `TfLiteTransposeConvParams`, whose
|
||||
`padding` uses `TfLitePadding` (Unknown=0, **SAME=1**, VALID=2) — NOT the flatbuffer `Padding`
|
||||
enum (SAME=0). Comparing against `Padding.SAME` rejects the real model.
|
||||
- **Proof, not hope:** `pip install mediapipe` ships the TFLite runtime WITH the custom op. Feed
|
||||
an image already at the model's exact input size (no resampling on either side) to MediaPipe's
|
||||
`ImageSegmenter` and to onnxruntime; compare masks (`verify_against_mediapipe.py`, exit 0 at
|
||||
max |diff| < 1e-3 — we got 5e-5). Work in a throwaway venv under `/tmp/opencode/`.
|
||||
|
||||
### ⛔ NEVER PRINT CREDENTIALS.md — NOT EVEN "FILTERED" (2026-10-09 incident)
|
||||
|
||||
**What happened:** to look for an API key, a `sed -n '1,40p' CREDENTIALS.md` piped through a
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
using ytLive.Helpers;
|
||||
using ytLive.Models;
|
||||
|
||||
namespace ytLive.Services.BackgroundEffects;
|
||||
|
||||
/// <summary>
|
||||
/// Applies a webcam's <see cref="BackgroundEffect"/> to one captured frame (TASK 51).
|
||||
/// Pure pixel work over the <see cref="VideoFrame"/> seam — WPF-free and thread-agnostic,
|
||||
/// so the camera worker thread can call it before the frame reaches the preview bitmap
|
||||
/// and the compositor.
|
||||
///
|
||||
/// <list type="bullet">
|
||||
/// <item><see cref="BackgroundEffect.None"/> returns the SAME frame instance — a true
|
||||
/// passthrough, no copy.</item>
|
||||
/// <item><see cref="BackgroundEffect.Remove"/> returns a copy whose alpha is the person
|
||||
/// mask (bilinear-upscaled), RGB untouched, <c>IsOpaque = false</c> — the compositor's
|
||||
/// straight-alpha blend then shows the layers below the webcam.</item>
|
||||
/// <item><see cref="BackgroundEffect.Blur"/> is the next slice; until it lands it passes
|
||||
/// through.</item>
|
||||
/// </list>
|
||||
///
|
||||
/// A segmenter that returns no mask (or throws) never drops the layer: the frame passes
|
||||
/// through and the failure is logged once.
|
||||
/// </summary>
|
||||
public sealed class BackgroundEffectProcessor
|
||||
{
|
||||
private readonly IPersonSegmenter _segmenter;
|
||||
private bool _loggedFailure;
|
||||
|
||||
public BackgroundEffectProcessor(IPersonSegmenter segmenter)
|
||||
{
|
||||
_segmenter = segmenter;
|
||||
}
|
||||
|
||||
public VideoFrame Apply(VideoFrame frame, BackgroundEffect effect)
|
||||
{
|
||||
if (effect != BackgroundEffect.Remove) return frame;
|
||||
|
||||
SegmentationMask? mask;
|
||||
try
|
||||
{
|
||||
mask = _segmenter.Segment(frame);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
LogFailureOnce($"{ex.GetType().Name}: {ex.Message}");
|
||||
return frame;
|
||||
}
|
||||
|
||||
if (mask == null || mask.Values.Length < mask.Width * mask.Height || mask.Width < 1 || mask.Height < 1)
|
||||
{
|
||||
LogFailureOnce("segmenter returned no usable mask");
|
||||
return frame;
|
||||
}
|
||||
|
||||
return ApplyRemove(frame, mask);
|
||||
}
|
||||
|
||||
internal static VideoFrame ApplyRemove(VideoFrame frame, SegmentationMask mask)
|
||||
{
|
||||
var src = frame.BgraPixels;
|
||||
var dst = new byte[src.Length];
|
||||
Buffer.BlockCopy(src, 0, dst, 0, src.Length);
|
||||
|
||||
var w = frame.Width;
|
||||
var h = frame.Height;
|
||||
var stride = frame.Stride;
|
||||
for (var y = 0; y < h; y++)
|
||||
{
|
||||
var my = (y + 0.5f) * mask.Height / h - 0.5f;
|
||||
var row = y * stride;
|
||||
for (var x = 0; x < w; x++)
|
||||
{
|
||||
var mx = (x + 0.5f) * mask.Width / w - 0.5f;
|
||||
var a = SampleBilinear(mask, mx, my);
|
||||
var alpha = (int)(a * src[row + x * 4 + 3] + 0.5f);
|
||||
dst[row + x * 4 + 3] = (byte)Math.Clamp(alpha, 0, 255);
|
||||
}
|
||||
}
|
||||
|
||||
return new VideoFrame(w, h, dst) { Epoch = frame.Epoch, IsOpaque = false };
|
||||
}
|
||||
|
||||
private static float SampleBilinear(SegmentationMask mask, float x, float y)
|
||||
{
|
||||
var x0 = (int)MathF.Floor(x);
|
||||
var y0 = (int)MathF.Floor(y);
|
||||
var fx = x - x0;
|
||||
var fy = y - y0;
|
||||
var x1 = Math.Clamp(x0 + 1, 0, mask.Width - 1);
|
||||
var y1 = Math.Clamp(y0 + 1, 0, mask.Height - 1);
|
||||
x0 = Math.Clamp(x0, 0, mask.Width - 1);
|
||||
y0 = Math.Clamp(y0, 0, mask.Height - 1);
|
||||
var v = mask.Values;
|
||||
var top = v[y0 * mask.Width + x0] * (1 - fx) + v[y0 * mask.Width + x1] * fx;
|
||||
var bottom = v[y1 * mask.Width + x0] * (1 - fx) + v[y1 * mask.Width + x1] * fx;
|
||||
return Math.Clamp(top * (1 - fy) + bottom * fy, 0f, 1f);
|
||||
}
|
||||
|
||||
private void LogFailureOnce(string reason)
|
||||
{
|
||||
if (_loggedFailure) return;
|
||||
_loggedFailure = true;
|
||||
AppLog.Write($"BackgroundEffectProcessor: passing webcam through — {reason}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
namespace ytLive.Services.BackgroundEffects;
|
||||
|
||||
/// <summary>A person-probability mask at the segmenter's own resolution: row-major,
|
||||
/// one value per pixel, 0 = background, 1 = person.</summary>
|
||||
public sealed record SegmentationMask(int Width, int Height, float[] Values);
|
||||
|
||||
/// <summary>Seam over the person-segmentation model (TASK 51) so tests inject a fake and
|
||||
/// never run real inference. Returns null when no mask can be produced; callers then pass
|
||||
/// the frame through untouched.</summary>
|
||||
public interface IPersonSegmenter
|
||||
{
|
||||
SegmentationMask? Segment(VideoFrame frame);
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
using System.IO;
|
||||
using Microsoft.ML.OnnxRuntime;
|
||||
using ytLive.Helpers;
|
||||
|
||||
namespace ytLive.Services.BackgroundEffects;
|
||||
|
||||
/// <summary>
|
||||
/// The default <see cref="IPersonSegmenter"/>: Google's MediaPipe Selfie Segmenter
|
||||
/// (landscape, Apache-2.0) converted to ONNX by <c>tools/selfie-seg/</c> and shipped as an
|
||||
/// embedded resource — nothing is downloaded at runtime and no GPU is required.
|
||||
///
|
||||
/// Contract: input [1,144,256,3] RGB in [0,1] (NHWC), output [1,144,256,1] person
|
||||
/// probability. The whole frame is resampled to 256×144 (stretched, not cropped), so the
|
||||
/// mask maps back to the frame by normalized coordinates.
|
||||
///
|
||||
/// Never throws from <see cref="Segment"/>: a model that fails to load or run logs once
|
||||
/// and returns null, and the caller passes the webcam frame through.
|
||||
/// </summary>
|
||||
public sealed class OnnxPersonSegmenter : IPersonSegmenter, IDisposable
|
||||
{
|
||||
public const int ModelWidth = 256;
|
||||
public const int ModelHeight = 144;
|
||||
internal const string ModelResourceName = "ytLive.Assets.Models.selfie_segmenter_landscape.onnx";
|
||||
|
||||
private static readonly long[] InputShape = { 1, ModelHeight, ModelWidth, 3 };
|
||||
|
||||
private readonly object _gate = new();
|
||||
private readonly float[] _input = new float[ModelWidth * ModelHeight * 3];
|
||||
private readonly RunOptions _runOptions = new();
|
||||
private InferenceSession? _session;
|
||||
private bool _failed;
|
||||
private bool _disposed;
|
||||
|
||||
public SegmentationMask? Segment(VideoFrame frame)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
if (_failed || _disposed) return null;
|
||||
try
|
||||
{
|
||||
_session ??= CreateSession();
|
||||
FillInput(frame, _input);
|
||||
using var input = OrtValue.CreateTensorValueFromMemory(_input, InputShape);
|
||||
using var outputs = _session.Run(_runOptions, new[] { "input" }, new[] { input }, new[] { "mask" });
|
||||
var mask = outputs[0].GetTensorDataAsSpan<float>().ToArray();
|
||||
return new SegmentationMask(ModelWidth, ModelHeight, mask);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_failed = true;
|
||||
AppLog.Write($"OnnxPersonSegmenter: background effects disabled — {ex.GetType().Name}: {ex.Message}");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
_disposed = true;
|
||||
_session?.Dispose();
|
||||
_session = null;
|
||||
_runOptions.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
private static InferenceSession CreateSession()
|
||||
{
|
||||
using var stream = typeof(OnnxPersonSegmenter).Assembly.GetManifestResourceStream(ModelResourceName)
|
||||
?? throw new InvalidOperationException($"embedded model '{ModelResourceName}' is missing");
|
||||
using var buffer = new MemoryStream();
|
||||
stream.CopyTo(buffer);
|
||||
// Two intra-op threads: the model is tiny, and the webcam must not starve the
|
||||
// compositor/encoder threads (obs-backgroundremoval reports 2 as the sweet spot).
|
||||
using var options = new SessionOptions
|
||||
{
|
||||
IntraOpNumThreads = 2,
|
||||
InterOpNumThreads = 1,
|
||||
GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL,
|
||||
};
|
||||
return new InferenceSession(buffer.ToArray(), options);
|
||||
}
|
||||
|
||||
/// <summary>BGRA frame → model input: nearest-sample stretch to 256×144, RGB order,
|
||||
/// values /255. Nearest is enough — the model input is ~5× smaller than a 720p frame
|
||||
/// and the mask is the only thing that comes back.</summary>
|
||||
internal static void FillInput(VideoFrame frame, float[] input)
|
||||
{
|
||||
var src = frame.BgraPixels;
|
||||
var stride = frame.Stride;
|
||||
var o = 0;
|
||||
for (var y = 0; y < ModelHeight; y++)
|
||||
{
|
||||
var sy = Math.Min(frame.Height - 1, (int)((y + 0.5) * frame.Height / ModelHeight));
|
||||
var row = sy * stride;
|
||||
for (var x = 0; x < ModelWidth; x++)
|
||||
{
|
||||
var sx = Math.Min(frame.Width - 1, (int)((x + 0.5) * frame.Width / ModelWidth));
|
||||
var p = row + sx * 4;
|
||||
input[o++] = src[p + 2] / 255f;
|
||||
input[o++] = src[p + 1] / 255f;
|
||||
input[o++] = src[p] / 255f;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,9 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `AlertClipDecoder.cs` | **Per-play alert video decoder (TASK 47):** `IAlertClipDecoder` (Start/Stop/Dispose + `FrameAvailable`/`AudioReady`/`Completed`) + ffmpeg child emitting `rawvideo bgra` (`-vf scale=W:H`, read via `RawVideoFrameReader`) and `f32le -ar 48000 -ac 2` on a **carry-buffer** audio loop (float boundary straddles never drop samples), both real-time paced; bgra is opaque (layer owns alpha). One-shot lifetime — created per alert, disposed at drain; tests inject a fake |
|
||||
| `LayoutStore.cs` | SQLite persistence (`Microsoft.Data.Sqlite`) at `%APPDATA%\ytLlive\ytLlive.db`; assets stored as BLOBs keyed by SHA-256 content hash; save/open layout files; schema `user_version` 10 (`Source.ClipShape`/`IsMirrored` via `ALTER TABLE` for pre-v2 DBs; v3 = singleton `Webcam` + per-scene `WebcamSceneConfig`, migrated idempotently **without backfill** — the stale `Source.DeviceId` column remains but is no longer read/written; v4 = `WebcamSceneConfig.RectWidth`/`RectHeight`, the pre-Round rect for the round-to-rect restore; v5 = `Source.IsBackdrop` + `Source.CaptureKey`, the live-capture backdrop; v6 = `Scene.HasBackdrop` — Live-only policy, one-time backfill turns Starting/BRB/Chat/Ending off + drops their backdrop sources; v9→v10 (TASK 39) = `YppSnapshot` history table; all later columns stay additive guarded ALTERs — TASK 47 adds the five alert `Source` columns (`AlertVideoAssetId`/`AlertVideoPath`/`AlertUseDefaultVideo`/`AlertShowTicker`/`AlertVideoVolume`) without a version bump; `MainViewModel.NormalizeBackgrounds` re-normalizes on every load — TASK 25: one locked "Background" per canonical screen (static art everywhere, DisplayCapture on Live), duplicates dropped, index 0, orphaned `BackgroundUseDefault_*`/`BackgroundPath_*` Settings keys purged at save). Settings key/value table holds app state: `MicSourceName`, `ReusableStream` (TASK 9 — the cached reusable stream's id/address/name so the pump has its RTMP URL at startup), `AlertDefaultVideoAssetId` (TASK 47 — the stamped built-in alert clip's asset row; save's prune exempts it via UNION) |
|
||||
| `VideoFrame.cs` | Normalized CPU frame seam (`Width`/`Height`/tightly-packed BGRA `byte[]`) — the only pixel type the rest of the app knows about; future capture sources (screen, background-removed webcam) feed the same seam |
|
||||
| `BackgroundEffects/IPersonSegmenter.cs` | **TASK 51 seam:** `Segment(VideoFrame) → SegmentationMask?` (`SegmentationMask(Width, Height, float[] Values)`, 0 = background, 1 = person); null = no mask → caller passes through. Tests inject fakes |
|
||||
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
||||
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = bilinear mask × source alpha (`IsOpaque=false`), Blur = passthrough until slice 2; no mask/exception → passthrough + one log. Not wired into the camera path yet |
|
||||
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
||||
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
||||
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
||||
|
||||
@@ -66,7 +66,7 @@
|
||||
| 48 | Distribution & packaging: **MSIX + Store IAP** | 🔶 In progress — **✅ route decided 2026-09-27 (Store MSIX + Store IAP)**; **item 1 (bundle ffmpeg) SHIPPED 2026-09-28** | [`TASKS/task-48-distribution-msix.md`](TASKS/task-48-distribution-msix.md) |
|
||||
| 49 | Chat profanity filter (local, opt-in, non-persistent, user word list) | ☐ Queued (2026-09-27) | [`TASKS/task-49-chat-profanity-filter.md`](TASKS/task-49-chat-profanity-filter.md) |
|
||||
| 50 | One-button "Publish this video" (resumable upload of the local recording) | ☐ Queued (2026-09-28) | [`TASKS/task-50-publish-recording.md`](TASKS/task-50-publish-recording.md) |
|
||||
| 51 | **Webcam background effects: Remove + Blur** | ☐ **v1 requirement** — booked 2026-10-08; next up now that the zoom box shipped (TASK 52) | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) |
|
||||
| 51 | **Webcam background effects: Remove + Blur** | 🔶 **v1 requirement — in progress**: slice 1 engine shipped 2026-10-09 (bundled MediaPipe Selfie Segmenter → ONNX, verified vs Google's runtime; Remove); next: Blur → config → camera hookup → UI | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) |
|
||||
| 52 | **Zoom box (anchored 2× magnifier)** | ✅ Done — shipped 2026-10-08 (3 creator corrections during the build: loupe → anchored capture over the full composite, drawn last → never shows the dim shading) | [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md) |
|
||||
| 53 | **Output-format verification: Shorts/TikTok geometry (9:16)** | ☐ Queued (2026-10-08) — output formats never tested end-to-end; pin + prove the vertical geometry | [`TASKS/task-53-output-formats.md`](TASKS/task-53-output-formats.md) |
|
||||
| 54 | **Marcom round: free-forever positioning + merch storefront** | 🔶 In progress — llamacasty.com copy live 2026-10-08 (pricing stripped); merch POD + app About links queued | [`TASKS/task-54-marcom-free-merch.md`](TASKS/task-54-marcom-free-merch.md) |
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
> **Booked 2026-10-08 by creator ruling** — *"we're doing the bg removal and blur for v1 - book it, Danno"*.
|
||||
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).
|
||||
|
||||
**Status:** ☐ Booked — **v1 requirement**. Queued behind the zoom build (current in-flight work unit).
|
||||
**Status:** 🔶 In progress — **slice 1 (engine: bundled model + runner + Remove) shipped 2026-10-09**.
|
||||
Next: slice 2 (Blur), then config + persistence, then the camera-path hookup + UI.
|
||||
|
||||
**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
|
||||
selected per element in its properties panel, composited into stream + recording like any
|
||||
@@ -34,7 +35,8 @@ delivery shape changed:
|
||||
same posture as the ffmpeg bundling in TASK 48 item 1.)
|
||||
2. **No hardware requirement.** CPU path must work; DirectML is an optional accelerator.
|
||||
3. **Per-element (Meld-style), not global.** Each webcam element carries its own effect.
|
||||
4. **No feature gating.** Works for free users; only the branding flash is paid.
|
||||
4. **No feature gating.** Everything is free (free-forever ruling 2026-10-08, TASK 54 — the old
|
||||
"only the branding flash is paid" line here was superseded).
|
||||
5. **Scene change / webcam add/remove must not crash or stall the pump** — a failing model
|
||||
run skips to passthrough, never drops the layer.
|
||||
|
||||
@@ -44,17 +46,21 @@ delivery shape changed:
|
||||
- ❌ **Super resolution / denoise / prettify passes**
|
||||
- ❌ **Virtual camera output** — permanently out of product (no vcam)
|
||||
|
||||
## 1. ☐ Engine
|
||||
## 1. Engine
|
||||
|
||||
- ☐ Bundle an ONNX person-segmentation model (MediaPipe Selfie-Segmentation class, ~2–5 MB)
|
||||
into the app (csproj copy like `ffmpeg/`, or as an `Asset`). Verify license + THIRD-PARTY
|
||||
notices (TASK 36 item 6 gate).
|
||||
- ☐ `onnxruntime` via NuGet. CPU EP by default; optionally DirectML EP later. Failing EP
|
||||
init → passthrough, log once, never throw from the pump.
|
||||
- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region. No
|
||||
model needed for blur once the mask exists.
|
||||
- ☐ Remove = **alpha 0 outside the person region.** The compositor already alpha-blends per
|
||||
element (`SceneCompositor`, source-over) — no compositor change expected.
|
||||
- ✅ **Model bundled (slice 1, 2026-10-09):** Google's official MediaPipe Selfie Segmenter
|
||||
**landscape** (144×256, Apache-2.0 per the model card — license + changes recorded in
|
||||
THIRD-PARTY-NOTICES item 12). Converted to ONNX by `tools/selfie-seg/convert_tflite_to_onnx.py`
|
||||
(custom op `Convolution2DTransposeBias` → `ConvTranspose`); verified against MediaPipe's own
|
||||
runtime by `tools/selfie-seg/verify_against_mediapipe.py` (max |diff| 5e-5). 430 KB,
|
||||
`EmbeddedResource` in the exe. Rejected sources: PINTO #109 (ML Kit-extracted, unclear license),
|
||||
obs-backgroundremoval code (GPL-3.0) — referenced for approach only (ONNX Runtime, CPU, 2 threads).
|
||||
- ✅ `Microsoft.ML.OnnxRuntime` 1.31.0, CPU EP (`OnnxPersonSegmenter`). Failing load/run →
|
||||
logs once, returns null, caller passes through. DirectML still optional/later.
|
||||
- ☐ Blur = **separable in-app blur** (box/gaussian) over the masked background region — slice 2.
|
||||
- ✅ Remove = alpha = mask outside the person (`BackgroundEffectProcessor`), `IsOpaque = false`.
|
||||
The compositor already alpha-blends per element — no compositor change expected (confirm in
|
||||
the hookup slice).
|
||||
- ☐ Feed the **VideoFrame seam** (`Services/VideoFrame.cs` doc explicitly anticipates
|
||||
*"every future capture source (screen, background-removed webcam) feeds the same seam"*):
|
||||
the frame handed to BOTH the preview bitmap path and the compositor's
|
||||
@@ -71,19 +77,27 @@ delivery shape changed:
|
||||
|
||||
## 3. ☐ Test seam (Good Dog — one integration test per unit)
|
||||
|
||||
- ☐ Model-runner interface seam (like `CameraEnumeratorOverride`) so tests inject a fake
|
||||
segmenter — no real ONNX inference in the test suite.
|
||||
- ✅ Model-runner interface seam (`IPersonSegmenter`) so tests inject a fake segmenter.
|
||||
- ☐ Assert the **processed frame reaches both consumers** (preview bitmap + compositor frame).
|
||||
- ☐ `None` = byte-identical passthrough of the raw frame.
|
||||
- ☐ `Remove` = alpha 0 outside person rect; `Blur` = background region blurred, person sharp.
|
||||
- ✅ `None` = same-instance passthrough of the raw frame (`BackgroundEffectEngineTests`).
|
||||
- ✅ `Remove` = alpha 0 outside the person, RGB untouched, non-opaque (`BackgroundEffectEngineTests`);
|
||||
☐ `Blur` = background region blurred, person sharp.
|
||||
- ✅ **Bundled-model contract:** the embedded ONNX model through the C# preprocessing reproduces
|
||||
MediaPipe's reference mask on a fixture (≥99.5% agreement). This is a fixed, deterministic
|
||||
contract check of the shipped bytes — not a camera/lighting quality test.
|
||||
- ☐ Config → persistence roundtrip (per webcam element).
|
||||
- ☐ Engine failure → passthrough + single log, layer never dropped.
|
||||
- ☐ Engine failure → passthrough + single log, layer never dropped (code path exists; dedicated
|
||||
test lands with the hookup slice).
|
||||
- ☐ Real inference: **manual on the creator's box** (camera + lighting), not in CI.
|
||||
- ☐ **Perf check in the hookup slice:** time `Segment` + `Apply` per 720p frame on the creator's
|
||||
box before wiring it into the 60fps path (expect a few ms; the mask can run at a lower rate
|
||||
than the camera if needed).
|
||||
|
||||
## 4. ☐ Sequencing
|
||||
## 4. Sequencing
|
||||
|
||||
- ☐ **Separate work unit AFTER the zoom build.** This file is booked + scoped; do not
|
||||
co-commit it with the zoom work.
|
||||
- ✅ Separate work unit after the zoom build (zoom shipped 2026-10-08).
|
||||
- Slice order: 1 engine ✅ → 2 Blur → 3 config + persistence → 4 camera-path hookup (one
|
||||
processing point, both consumers) + perf check → 5 properties-panel UI.
|
||||
|
||||
## Related (not duplicated here)
|
||||
|
||||
|
||||
+21
-1
@@ -115,7 +115,27 @@ there on purpose.
|
||||
Transitive dependencies (Microsoft.Xaml.Behaviors.Wpf,
|
||||
Microsoft.Extensions.DependencyInjection.Abstractions) are MIT,
|
||||
covered by item 7's .NET notice policy; this notice is kept per
|
||||
this file's "list everything" policy.
|
||||
this file's "list everything" policy.
|
||||
|
||||
11. ONNX Runtime (Microsoft.ML.OnnxRuntime — runs the webcam background model, TASK 51)
|
||||
Copyright (c) Microsoft Corporation
|
||||
License: MIT
|
||||
Home: https://github.com/microsoft/onnxruntime
|
||||
Used as: the CPU inference runtime (managed package + native onnxruntime.dll)
|
||||
behind the `IPersonSegmenter` seam. MIT imposes no source offer.
|
||||
|
||||
12. MediaPipe Selfie Segmenter (landscape) (webcam background Remove/Blur, TASK 51)
|
||||
Copyright (c) Google LLC (authors: Tingbo Hou, Siargey Pisarchyk, Karthik
|
||||
Raveendran — model card "MediaPipe Selfie Segmentation", 2021-05-06)
|
||||
License: Apache License 2.0
|
||||
Home: https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
|
||||
Source: https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/float16/latest/selfie_segmenter_landscape.tflite
|
||||
Changes (Apache-2.0 §4b): converted from TensorFlow Lite to ONNX format by
|
||||
tools/selfie-seg/convert_tflite_to_onnx.py — float16 weights widened
|
||||
to float32, NHWC graph re-expressed in NCHW; the weights and the
|
||||
network are otherwise unmodified. Embedded in the application as
|
||||
selfie_segmenter_landscape.onnx. Runs entirely on the user's PC.
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -602,12 +602,47 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||||
to the encoder task.
|
||||
- **Background removal = PERMANENTLY OUT** (creator ruling 2026-10-05, confirmed: "no - not doing
|
||||
it. the note is correct and accurate"). The line here used to read "milestone 2", which was stale —
|
||||
it was never a plan, only a leftover from TASK 3's original scoping. The rationale that stands:
|
||||
a vcam app (NVIDIA Broadcast, XSplit VCam) does it free, and an in-app model means a runtime, a
|
||||
GPU/driver matrix, and per-scene lighting tuning — support hell for a solo dev
|
||||
(`TASKS/task-36-gold-pass.md`, `TASKS/task-03-capture-pipeline.md` item 19). Don't reopen it.
|
||||
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
|
||||
2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
|
||||
and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
|
||||
run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
|
||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||
|
||||
### Webcam background effects (TASK 51 — slice 1 engine shipped 2026-10-09)
|
||||
|
||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||
float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
|
||||
`tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
|
||||
`Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
|
||||
translation was smaller than any workaround). **Verified against Google's own runtime:**
|
||||
`verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
|
||||
.tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
|
||||
⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance) or
|
||||
copy code from obs-backgroundremoval (GPL-3.0).
|
||||
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
|
||||
whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
|
||||
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
|
||||
inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
|
||||
`Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
|
||||
`onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
|
||||
publish (TASK 48) must pin the RID so only win-x64 ships.
|
||||
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
|
||||
SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
|
||||
resource; **never throws** — a load/run failure logs once and returns null forever after);
|
||||
`BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
|
||||
passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
|
||||
`IsOpaque = false` (the compositor's straight-alpha blend shows the layers below); `Blur` passes
|
||||
through until slice 2. No mask / segmenter exception → passthrough + one log line, never a dropped
|
||||
layer.
|
||||
- **Not wired yet:** nothing calls the processor — no config, no UI, no `CameraManager` hookup. The
|
||||
next slices add Blur, the per-webcam `BackgroundEffect` config + persistence, then the single
|
||||
processing point on the camera path (both preview and compositor consume the processed frame).
|
||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`).
|
||||
|
||||
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
|
||||
|
||||
@@ -1652,6 +1687,11 @@ guardrails below — written down so a future "quick fix" never reintroduces a G
|
||||
another public-domain source; a logo asset under a copyleft or attribution license would contaminate
|
||||
the paid product. The About hub's logo is the creator's own art ("llama fortnite superman logo",
|
||||
`Assets/llama-logo.png`) — owned by us, so it carries no third-party license either.
|
||||
- **Ship an ML model without clean provenance** — the background-effects model (TASK 51) is Google's
|
||||
official MediaPipe Selfie Segmenter download (Apache-2.0, model card), converted by our own script.
|
||||
Model-zoo re-hosts (e.g. PINTO #109 = ML Kit's model, extracted) and GPL plugins
|
||||
(obs-backgroundremoval, GPL-3.0) are off-limits. Apache-2.0 §4 obligations (license notice + "we
|
||||
converted it") live in `THIRD-PARTY-NOTICES.txt`.
|
||||
- **Forget the v1 license-texts gate** — `THIRD-PARTY-NOTICES.txt` links the canonical license texts; at
|
||||
**v1 (GA)** the full texts of every license it names MUST ship alongside it (TASK 4 requirement 9 is the
|
||||
release blocker). Queued early is wrong; the release pass owns it.
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Convert Google's MediaPipe Selfie Segmenter (.tflite) to ONNX for LlamaCasty (TASK 51).
|
||||
|
||||
Why a hand-rolled converter: the model ends in MediaPipe's custom op
|
||||
`Convolution2DTransposeBias`, which tf2onnx / tflite2onnx don't support. The graph only
|
||||
uses 11 op types, so a direct, auditable translation is smaller than any workaround.
|
||||
|
||||
Source model (Apache-2.0, model card "MediaPipe Selfie Segmentation", Google, 2021-05-06):
|
||||
https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/float16/latest/selfie_segmenter_landscape.tflite
|
||||
|
||||
Layout: TFLite is NHWC. The ONNX graph keeps the SAME external contract
|
||||
(input [1,144,256,3] float RGB in [0,1], output [1,144,256,1] person probability in [0,1])
|
||||
and transposes to NCHW internally.
|
||||
|
||||
Usage: python convert_tflite_to_onnx.py in.tflite out.onnx
|
||||
Requires: tflite, onnx, numpy. Verify with verify_against_mediapipe.py.
|
||||
"""
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
import onnx
|
||||
import tflite
|
||||
from onnx import TensorProto, helper, numpy_helper
|
||||
from tflite.ActivationFunctionType import ActivationFunctionType as Act
|
||||
from tflite.BuiltinOperator import BuiltinOperator
|
||||
from tflite.Padding import Padding
|
||||
from tflite.TensorType import TensorType
|
||||
|
||||
OPSET = 17
|
||||
OP_NAMES = {v: k for k, v in BuiltinOperator.__dict__.items() if not k.startswith("_")}
|
||||
NP_TYPES = {TensorType.FLOAT32: np.float32, TensorType.FLOAT16: np.float16, TensorType.INT32: np.int32}
|
||||
|
||||
|
||||
def main(src, dst):
|
||||
model = tflite.Model.GetRootAsModel(open(src, "rb").read(), 0)
|
||||
g = model.Subgraphs(0)
|
||||
|
||||
def shape(ix):
|
||||
return g.Tensors(ix).ShapeAsNumpy().tolist()
|
||||
|
||||
def const(ix):
|
||||
"""Constant tensor value, or None for runtime tensors."""
|
||||
t = g.Tensors(ix)
|
||||
buf = model.Buffers(t.Buffer())
|
||||
if buf is None or buf.DataLength() == 0:
|
||||
return None
|
||||
raw = buf.DataAsNumpy().tobytes()
|
||||
return np.frombuffer(raw, dtype=NP_TYPES[t.Type()]).reshape(shape(ix))
|
||||
|
||||
nodes, inits = [], []
|
||||
consts = {} # tensor index -> float32 numpy (after DEQUANTIZE folding)
|
||||
uid = [0]
|
||||
|
||||
def name(ix):
|
||||
return f"t{ix}"
|
||||
|
||||
def fresh(prefix):
|
||||
uid[0] += 1
|
||||
return f"{prefix}_{uid[0]}"
|
||||
|
||||
def add_init(arr, nm=None):
|
||||
nm = nm or fresh("c")
|
||||
inits.append(numpy_helper.from_array(np.ascontiguousarray(arr), nm))
|
||||
return nm
|
||||
|
||||
def value(ix):
|
||||
if ix in consts:
|
||||
return consts[ix]
|
||||
c = const(ix)
|
||||
if c is not None:
|
||||
consts[ix] = c.astype(np.float32) if c.dtype == np.float16 else c
|
||||
return consts[ix]
|
||||
return None
|
||||
|
||||
def activation(out_name, act, final_name):
|
||||
if act == Act.NONE:
|
||||
return out_name
|
||||
if act == Act.RELU:
|
||||
nodes.append(helper.make_node("Relu", [out_name], [final_name]))
|
||||
elif act == Act.RELU6:
|
||||
nodes.append(helper.make_node("Clip", [out_name, add_init(np.float32(0)), add_init(np.float32(6))], [final_name]))
|
||||
else:
|
||||
raise NotImplementedError(f"fused activation {act}")
|
||||
return final_name
|
||||
|
||||
def operand(ix):
|
||||
"""Runtime tensor name, or a constant reshaped from NHWC broadcast form to NCHW."""
|
||||
v = value(ix)
|
||||
if v is None:
|
||||
return name(ix)
|
||||
if v.ndim == 1:
|
||||
v = v.reshape(1, -1, 1, 1)
|
||||
elif v.ndim == 4:
|
||||
v = v.transpose(0, 3, 1, 2)
|
||||
return add_init(v.astype(np.float32))
|
||||
|
||||
# NHWC -> NCHW at the input.
|
||||
inp = g.Inputs(0)
|
||||
nodes.append(helper.make_node("Transpose", ["input"], [name(inp)], perm=[0, 3, 1, 2]))
|
||||
|
||||
for i in range(g.OperatorsLength()):
|
||||
op = g.Operators(i)
|
||||
oc = model.OperatorCodes(op.OpcodeIndex())
|
||||
code = max(oc.BuiltinCode(), oc.DeprecatedBuiltinCode())
|
||||
kind = "CUSTOM:" + oc.CustomCode().decode() if code == BuiltinOperator.CUSTOM else OP_NAMES[code]
|
||||
ins = [op.Inputs(j) for j in range(op.InputsLength())]
|
||||
out = op.Outputs(0)
|
||||
o = name(out)
|
||||
|
||||
if kind == "DEQUANTIZE":
|
||||
consts[out] = const(ins[0]).astype(np.float32)
|
||||
continue
|
||||
|
||||
if kind in ("CONV_2D", "DEPTHWISE_CONV_2D"):
|
||||
opt = (tflite.Conv2DOptions() if kind == "CONV_2D" else tflite.DepthwiseConv2DOptions())
|
||||
bo = op.BuiltinOptions()
|
||||
opt.Init(bo.Bytes, bo.Pos)
|
||||
w = value(ins[1])
|
||||
b = value(ins[2]) if len(ins) > 2 and ins[2] >= 0 else None
|
||||
if kind == "CONV_2D":
|
||||
W = w.transpose(0, 3, 1, 2) # OHWI -> OIHW
|
||||
group = 1
|
||||
else:
|
||||
# [1,H,W,C*mult] -> [C*mult,1,H,W], group = input channels
|
||||
W = w.transpose(3, 0, 1, 2)
|
||||
group = shape(ins[0])[3]
|
||||
attrs = dict(
|
||||
strides=[opt.StrideH(), opt.StrideW()],
|
||||
dilations=[opt.DilationHFactor(), opt.DilationWFactor()],
|
||||
kernel_shape=list(W.shape[2:]),
|
||||
group=group,
|
||||
auto_pad="SAME_UPPER" if opt.Padding() == Padding.SAME else "VALID",
|
||||
)
|
||||
conv_in = [name(ins[0]), add_init(W)] + ([add_init(b)] if b is not None else [])
|
||||
tmp = fresh("conv") if opt.FusedActivationFunction() != Act.NONE else o
|
||||
nodes.append(helper.make_node("Conv", conv_in, [tmp], **attrs))
|
||||
activation(tmp, opt.FusedActivationFunction(), o)
|
||||
continue
|
||||
|
||||
if kind in ("ADD", "MUL"):
|
||||
opt = tflite.AddOptions() if kind == "ADD" else tflite.MulOptions()
|
||||
bo = op.BuiltinOptions()
|
||||
act = Act.NONE
|
||||
if bo is not None:
|
||||
opt.Init(bo.Bytes, bo.Pos)
|
||||
act = opt.FusedActivationFunction()
|
||||
tmp = fresh("ew") if act != Act.NONE else o
|
||||
nodes.append(helper.make_node("Add" if kind == "ADD" else "Mul", [operand(ins[0]), operand(ins[1])], [tmp]))
|
||||
activation(tmp, act, o)
|
||||
continue
|
||||
|
||||
if kind == "RELU":
|
||||
nodes.append(helper.make_node("Relu", [name(ins[0])], [o]))
|
||||
continue
|
||||
if kind == "HARD_SWISH":
|
||||
nodes.append(helper.make_node("HardSwish", [name(ins[0])], [o]))
|
||||
continue
|
||||
if kind == "LOGISTIC":
|
||||
nodes.append(helper.make_node("Sigmoid", [name(ins[0])], [o]))
|
||||
continue
|
||||
|
||||
if kind == "MEAN":
|
||||
opt = tflite.ReducerOptions()
|
||||
bo = op.BuiltinOptions()
|
||||
opt.Init(bo.Bytes, bo.Pos)
|
||||
nhwc_to_nchw = {0: 0, 1: 2, 2: 3, 3: 1}
|
||||
axes = [nhwc_to_nchw[int(a)] for a in value(ins[1]).reshape(-1)]
|
||||
nodes.append(helper.make_node("ReduceMean", [name(ins[0])], [o], axes=axes, keepdims=1 if opt.KeepDims() else 0))
|
||||
continue
|
||||
|
||||
if kind == "RESIZE_BILINEAR":
|
||||
opt = tflite.ResizeBilinearOptions()
|
||||
bo = op.BuiltinOptions()
|
||||
opt.Init(bo.Bytes, bo.Pos)
|
||||
h, w = (int(x) for x in value(ins[1]).reshape(-1))
|
||||
c = shape(ins[0])[3]
|
||||
mode = "align_corners" if opt.AlignCorners() else ("half_pixel" if opt.HalfPixelCenters() else "asymmetric")
|
||||
sizes = add_init(np.array([1, c, h, w], dtype=np.int64))
|
||||
nodes.append(helper.make_node("Resize", [name(ins[0]), "", "", sizes], [o], mode="linear", coordinate_transformation_mode=mode))
|
||||
continue
|
||||
|
||||
if kind == "CUSTOM:Convolution2DTransposeBias":
|
||||
# Custom options = TfLiteTransposeConvParams {padding, stride_w, stride_h} (int32 each).
|
||||
# NOTE: `padding` is the C-API TfLitePadding enum (Unknown=0, SAME=1, VALID=2),
|
||||
# NOT the flatbuffer Padding enum (SAME=0, VALID=1).
|
||||
padding, stride_w, stride_h = np.frombuffer(bytes(op.CustomOptionsAsNumpy().tolist()), dtype=np.int32)[:3]
|
||||
w = value(ins[1]) # [O, kH, kW, I]
|
||||
b = value(ins[2])
|
||||
W = w.transpose(3, 0, 1, 2) # -> [I, O, kH, kW] (ONNX ConvTranspose)
|
||||
kh, kw = w.shape[1], w.shape[2]
|
||||
if padding != 1 or kh != stride_h or kw != stride_w:
|
||||
raise NotImplementedError("only SAME, kernel == stride transpose conv is supported")
|
||||
nodes.append(helper.make_node("ConvTranspose", [name(ins[0]), add_init(W), add_init(b)], [o],
|
||||
strides=[int(stride_h), int(stride_w)], kernel_shape=[kh, kw]))
|
||||
continue
|
||||
|
||||
raise NotImplementedError(kind)
|
||||
|
||||
# NCHW -> NHWC at the output.
|
||||
outp = g.Outputs(0)
|
||||
nodes.append(helper.make_node("Transpose", [name(outp)], ["mask"], perm=[0, 2, 3, 1]))
|
||||
|
||||
graph = helper.make_graph(
|
||||
nodes, "mediapipe_selfie_segmenter_landscape",
|
||||
[helper.make_tensor_value_info("input", TensorProto.FLOAT, shape(inp))],
|
||||
[helper.make_tensor_value_info("mask", TensorProto.FLOAT, shape(outp))],
|
||||
initializer=inits)
|
||||
m = helper.make_model(graph, opset_imports=[helper.make_opsetid("", OPSET)],
|
||||
producer_name="llamacasty-tflite2onnx",
|
||||
doc_string="MediaPipe Selfie Segmenter (landscape), Apache-2.0, Google. "
|
||||
"Converted for LlamaCasty by tools/selfie-seg/convert_tflite_to_onnx.py")
|
||||
m.ir_version = 8
|
||||
onnx.checker.check_model(m)
|
||||
onnx.save(m, dst)
|
||||
print(f"wrote {dst}: {len(nodes)} nodes, {len(inits)} initializers")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1], sys.argv[2])
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Verify the converted ONNX model against Google's own MediaPipe runtime (TASK 51).
|
||||
|
||||
Feeds the SAME 256x144 RGB frame to (a) MediaPipe's ImageSegmenter running the original
|
||||
.tflite (with its custom op) and (b) onnxruntime running our converted .onnx, then compares
|
||||
the person-probability masks. Also writes side-by-side PNGs for an eyeball check.
|
||||
|
||||
Usage: python verify_against_mediapipe.py model.tflite model.onnx photo.jpg [out_prefix]
|
||||
Requires: mediapipe, onnxruntime, numpy, pillow.
|
||||
"""
|
||||
import sys
|
||||
|
||||
import mediapipe as mp
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
from mediapipe.tasks import python as mp_python
|
||||
from mediapipe.tasks.python import vision
|
||||
from PIL import Image
|
||||
|
||||
W, H = 256, 144
|
||||
|
||||
|
||||
def main(tfl, onx, photo, prefix="verify"):
|
||||
img = Image.open(photo).convert("RGB")
|
||||
# Cover-crop to 16:9, then resize to the model's exact input so neither runtime resamples.
|
||||
sw, sh = img.size
|
||||
target = W / H
|
||||
if sw / sh > target:
|
||||
nw = int(sh * target); x = (sw - nw) // 2; img = img.crop((x, 0, x + nw, sh))
|
||||
else:
|
||||
nh = int(sw / target); y = (sh - nh) // 2; img = img.crop((0, y, sw, y + nh))
|
||||
img = img.resize((W, H), Image.BILINEAR)
|
||||
rgb = np.asarray(img, dtype=np.uint8)
|
||||
|
||||
seg = vision.ImageSegmenter.create_from_options(vision.ImageSegmenterOptions(
|
||||
base_options=mp_python.BaseOptions(model_asset_path=tfl), output_confidence_masks=True))
|
||||
res = seg.segment(mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb))
|
||||
ref = res.confidence_masks[0].numpy_view().astype(np.float32).reshape(H, W)
|
||||
|
||||
sess = ort.InferenceSession(onx, providers=["CPUExecutionProvider"])
|
||||
x = (rgb.astype(np.float32) / 255.0)[None]
|
||||
got = sess.run(None, {"input": x})[0].reshape(H, W)
|
||||
|
||||
diff = np.abs(ref - got)
|
||||
agree = np.mean((ref > 0.5) == (got > 0.5))
|
||||
print(f"ref range {ref.min():.3f}..{ref.max():.3f} person%={np.mean(ref > 0.5) * 100:.1f}")
|
||||
print(f"onnx range {got.min():.3f}..{got.max():.3f} person%={np.mean(got > 0.5) * 100:.1f}")
|
||||
print(f"max|diff|={diff.max():.5f} mean|diff|={diff.mean():.6f} binary-agreement={agree * 100:.2f}%")
|
||||
|
||||
def to_img(m):
|
||||
return Image.fromarray((np.clip(m, 0, 1) * 255).astype(np.uint8)).convert("RGB")
|
||||
sheet = Image.new("RGB", (W * 3, H))
|
||||
sheet.paste(img, (0, 0)); sheet.paste(to_img(ref), (W, 0)); sheet.paste(to_img(got), (W * 2, 0))
|
||||
sheet.resize((W * 6, H * 2), Image.NEAREST).save(f"{prefix}_sheet.png")
|
||||
return diff.max()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
md = main(*sys.argv[1:])
|
||||
sys.exit(0 if md < 1e-3 else 1)
|
||||
@@ -0,0 +1,115 @@
|
||||
using System.IO;
|
||||
using System.Windows.Media;
|
||||
using System.Windows.Media.Imaging;
|
||||
using Xunit;
|
||||
using ytLive.Models;
|
||||
using ytLive.Services;
|
||||
using ytLive.Services.BackgroundEffects;
|
||||
|
||||
namespace ytLive.Tests;
|
||||
|
||||
/// <summary>TASK 51 slice 1 — the background-effects engine.</summary>
|
||||
public class BackgroundEffectEngineTests
|
||||
{
|
||||
private sealed class FakeSegmenter : IPersonSegmenter
|
||||
{
|
||||
private readonly SegmentationMask? _mask;
|
||||
public int Calls;
|
||||
public FakeSegmenter(SegmentationMask? mask) => _mask = mask;
|
||||
public SegmentationMask? Segment(VideoFrame frame)
|
||||
{
|
||||
Calls++;
|
||||
return _mask;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The integration test for this unit: a webcam frame run through the
|
||||
/// processor with Remove comes back with the person kept and the background
|
||||
/// transparent, colors untouched, flagged non-opaque so the compositor blends it.
|
||||
/// None is a true passthrough (same instance, segmenter never called).</summary>
|
||||
[Fact]
|
||||
public void Remove_keeps_the_person_and_makes_the_background_transparent()
|
||||
{
|
||||
const int w = 64, h = 32;
|
||||
var pixels = new byte[w * h * 4];
|
||||
for (var i = 0; i < pixels.Length; i += 4)
|
||||
{
|
||||
pixels[i] = 10; pixels[i + 1] = 20; pixels[i + 2] = 30; pixels[i + 3] = 255;
|
||||
}
|
||||
var frame = new VideoFrame(w, h, pixels) { IsOpaque = true };
|
||||
|
||||
// 4×2 mask: left half person, right half background.
|
||||
var mask = new SegmentationMask(4, 2, new[] { 1f, 1f, 0f, 0f, 1f, 1f, 0f, 0f });
|
||||
var segmenter = new FakeSegmenter(mask);
|
||||
var processor = new BackgroundEffectProcessor(segmenter);
|
||||
|
||||
Assert.Same(frame, processor.Apply(frame, BackgroundEffect.None));
|
||||
Assert.Equal(0, segmenter.Calls);
|
||||
|
||||
var result = processor.Apply(frame, BackgroundEffect.Remove);
|
||||
|
||||
Assert.NotSame(frame, result);
|
||||
Assert.False(result.IsOpaque);
|
||||
Assert.Equal(w, result.Width);
|
||||
Assert.Equal(h, result.Height);
|
||||
for (var y = 0; y < h; y++)
|
||||
{
|
||||
Assert.Equal(255, Alpha(result, 4, y)); // well inside the person half
|
||||
Assert.Equal(0, Alpha(result, w - 5, y)); // well inside the background half
|
||||
}
|
||||
for (var i = 0; i < pixels.Length; i += 4)
|
||||
{
|
||||
Assert.Equal(pixels[i], result.BgraPixels[i]);
|
||||
Assert.Equal(pixels[i + 1], result.BgraPixels[i + 1]);
|
||||
Assert.Equal(pixels[i + 2], result.BgraPixels[i + 2]);
|
||||
}
|
||||
Assert.Equal(255, frame.BgraPixels[(w - 5) * 4 + 3]); // source frame never mutated
|
||||
}
|
||||
|
||||
/// <summary>Model contract: the BUNDLED (embedded) ONNX model, driven through the C#
|
||||
/// preprocessing, reproduces Google's own MediaPipe runtime on a fixed fixture.
|
||||
/// The expected mask was produced by tools/selfie-seg/ running the original .tflite in
|
||||
/// MediaPipe (fixture: MediaPipe's public test image mozart_square.jpg, a public-domain
|
||||
/// painting, cropped to 256×144). Guards the embedding, the input layout (RGB, NHWC,
|
||||
/// /255) and the conversion itself.</summary>
|
||||
[Fact]
|
||||
public void Bundled_model_matches_the_MediaPipe_reference_mask()
|
||||
{
|
||||
var frame = LoadBgra(Fixture("selfie-fixture-256x144.png"));
|
||||
var expected = LoadBgra(Fixture("selfie-fixture-expected-mask.png"));
|
||||
|
||||
using var segmenter = new OnnxPersonSegmenter();
|
||||
var mask = segmenter.Segment(frame);
|
||||
|
||||
Assert.NotNull(mask);
|
||||
Assert.Equal(OnnxPersonSegmenter.ModelWidth, mask!.Width);
|
||||
Assert.Equal(OnnxPersonSegmenter.ModelHeight, mask.Height);
|
||||
|
||||
var agree = 0;
|
||||
var person = 0;
|
||||
for (var i = 0; i < mask.Width * mask.Height; i++)
|
||||
{
|
||||
var expectPerson = expected.BgraPixels[i * 4] > 127;
|
||||
var gotPerson = mask.Values[i] > 0.5f;
|
||||
if (expectPerson == gotPerson) agree++;
|
||||
if (gotPerson) person++;
|
||||
}
|
||||
var total = mask.Width * mask.Height;
|
||||
Assert.True(agree >= total * 0.995, $"binary agreement {100.0 * agree / total:0.00}% < 99.5%");
|
||||
Assert.InRange(100.0 * person / total, 45, 60); // reference: 52.5% person
|
||||
}
|
||||
|
||||
private static int Alpha(VideoFrame f, int x, int y) => f.BgraPixels[(y * f.Width + x) * 4 + 3];
|
||||
|
||||
private static string Fixture(string name) => Path.Combine(AppContext.BaseDirectory, "Fixtures", name);
|
||||
|
||||
private static VideoFrame LoadBgra(string path)
|
||||
{
|
||||
using var stream = File.OpenRead(path);
|
||||
var decoded = BitmapDecoder.Create(stream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.OnLoad).Frames[0];
|
||||
var bgra = new FormatConvertedBitmap(decoded, PixelFormats.Bgra32, null, 0);
|
||||
var pixels = new byte[bgra.PixelWidth * bgra.PixelHeight * 4];
|
||||
bgra.CopyPixels(pixels, bgra.PixelWidth * 4, 0);
|
||||
return new VideoFrame(bgra.PixelWidth, bgra.PixelHeight, pixels);
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1019 B |
@@ -18,4 +18,8 @@
|
||||
<ProjectReference Include="..\ytLive.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<None Include="Fixtures\**" CopyToOutputDirectory="PreserveNewest"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -75,6 +75,14 @@
|
||||
<None Include="THIRD-PARTY-NOTICES.txt" CopyToOutputDirectory="PreserveNewest" />
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Webcam background effects (TASK 51): Google's MediaPipe Selfie Segmenter (landscape,
|
||||
Apache-2.0), converted to ONNX by tools/selfie-seg/ and embedded in the assembly —
|
||||
~430 KB, so it lives in git, ships inside the exe, and is never downloaded at runtime
|
||||
(Store policy 10.2.2, same posture as the bundled ffmpeg). -->
|
||||
<ItemGroup>
|
||||
<EmbeddedResource Include="Assets\Models\selfie_segmenter_landscape.onnx" LogicalName="ytLive.Assets.Models.selfie_segmenter_landscape.onnx" />
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Bundled FFmpeg (TASK 48 item 1, Store policy 10.2.2 dynamic code inclusion).
|
||||
The app used to DOWNLOAD an unsigned ffmpeg from GitHub at first run and execute
|
||||
it; that 404'd on the creator's first real recording (2026-09-01) and is the one
|
||||
@@ -102,6 +110,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.Sqlite" Version="8.0.10" />
|
||||
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.31.0" />
|
||||
<PackageReference Include="NAudio.Wasapi" Version="2.2.1" />
|
||||
<PackageReference Include="NAudio.WinMM" Version="2.2.1" />
|
||||
<PackageReference Include="Notification.Wpf" Version="11.0.0" />
|
||||
|
||||
Reference in New Issue
Block a user