From 305706956e8dc9120893b39580dc6cf26e9cb1aa Mon Sep 17 00:00:00 2001 From: gramps Date: Fri, 9 Oct 2026 22:15:59 -0700 Subject: [PATCH] TASK 51 jitter fix: temporal mask smoothing. MaskSmoother blends each raw segmentation 0.75xnew + 0.25xprevious at the one CameraManager worker (state per CameraSession, not in the shared segmenter); null resets history, first frame passes through, input array never aliased, ping-pong buffers (no LOH alloc for a 147KB 256x144 mask). Creator's real-camera check: Remove was 'jittery' (per-frame independent segmentation). Weights from the OBS/Jitsi consensus. One integration test asserts the second frame keeps partial alpha (x=6 -> 58) where the raw mask cuts to 0. 416 pass / 2 known env flakes. Refs: https://deepwiki.com/royshil/obs-backgroundremoval/2-core-features https://github.com/jitsi/jitsi-meet/issues/17080 (Unit: bg-removal jitter fix) --- HANDOFF.md | 105 +++++++++++---------- Services/BackgroundEffects/MaskSmoother.cs | 68 +++++++++++++ Services/CameraManager.cs | 8 +- Services/index.md | 3 +- TASKS.md | 2 +- TASKS/task-51-webcam-bg-effects.md | 26 ++++- ai.md | 31 ++++-- ytLive.Tests/CameraManagerTests.cs | 72 ++++++++++++++ 8 files changed, 252 insertions(+), 63 deletions(-) create mode 100644 Services/BackgroundEffects/MaskSmoother.cs diff --git a/HANDOFF.md b/HANDOFF.md index 928a355..644d594 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,54 +1,63 @@ -# HANDOFF — 2026-10-09 (TASK 51 webcam background Remove + Blur — BUILT) +# HANDOFF — 2026-10-09 (TASK 51 quality pass: jitter FIXED; blur-heavy is the next unit) ## Branch -- main (no feature branch). Pushed through `f8dd107`; everything after is local, **not pushed** - (push at the creator's call). +- main (no feature branch). Pushed through `5685c5d` (TRAX toggle); everything after is local, + **not pushed** (push at the creator's call). -## Dirty files (NOT mine — do NOT commit) -- The in-flight "pinned chat" work unit: `Models/YouTube.cs`, `ViewModels/MainViewModel.Chat.cs`, - untracked `Helpers/RelayCommandGeneric.cs`, and **its hunks in `Controls/LeftPanel.xaml`** - (lines ~599–610, the chat area). TASK 51 slice 5 edited the SAME file in a different region (the - webcam properties, ~line 287); only the TASK 51 hunk was staged (`git apply --cached` of that hunk - alone) — the pinned-chat hunks are still unstaged and untouched. -- The creator's untracked logo art in `Assets/` (`llamacasty text.*`, `llamacasty-text-nobg*.png`, - `thin logo wordmark.png`). +## Dirty files — split by ownership +- **This work unit (COMMIT these):** `Services/BackgroundEffects/MaskSmoother.cs` (new), the smoother + wiring in `Services/CameraManager.cs`, `ytLive.Tests/CameraManagerTests.cs` (new integration test), + and the same-commit memory updates (`Services/index.md`, `ai.md`, `TASKS/task-51-webcam-bg-effects.md`, + `TASKS.md` row 51, and this file) — all ship in the SAME commit. +- **NOT mine — do NOT commit:** the in-flight "pinned chat" work unit (`Models/YouTube.cs`, + `ViewModels/MainViewModel.Chat.cs`, untracked `Helpers/RelayCommandGeneric.cs`, and **its hunks in + `Controls/LeftPanel.xaml`** — only the webcam-properties region is TASK 51; the pinned-chat hunks + stayed unstaged/untouched), plus the creator's untracked `Assets/llamacasty*` art. +- ⚠️ Pre-commit: `scripts/verify.sh` runs `scripts/scope-check.sh` against ALL working-tree changes — + the not-mine files WILL flag. Pass them to the script as declared-known (checksums unchanged since + `5685c5d`) or let the script's output be reconciled before committing. -## What just happened — TASK 51, all five slices -1. **Engine** — Google's MediaPipe Selfie Segmenter (landscape, Apache-2.0) converted to ONNX by - `tools/selfie-seg/` and verified vs MediaPipe's own runtime; embedded resource; onnxruntime CPU. -2. **Blur** — background at ¼ res, 3 box passes, mask-blended; Remove gets edge feather. -3. **Config** — `WebcamSceneConfig.BackgroundEffect` + `BackgroundStrength`, persisted (additive cols). -4. **Hookup** — `CameraManager` renders per-scene effect **variants** (one segmentation per raw frame - on a latest-frame-drop worker; raw path untouched); VM `SyncWebcamEffects()` declares/adopts them. - Synthetic 720p Debug: segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms. -5. **UI** — Background section in the webcam's Layers properties: Remove / Blur `PillToggle`s, - radio-exclusive, + Strength slider (visible while an effect is on). -Also this session: 64-bit requirement (`PlatformTarget=x64`), GPL-3.0-or-later licensing, README -rewrite, TASK 55 (disk hygiene) + TASK 56 (secret scan before going public) queued. - -## Tests -- Full suite **412 passed / 0 failed**, 0 build warnings. New: `BackgroundEffectEngineTests` (Remove, - Blur, bundled-model contract), `CameraManagerTests` (variant reaches compositor + preview, raw - stays raw), `LayoutStorePersistenceTests` (per-scene roundtrip + legacy migration), - `WebcamBackgroundPillTests` (radio semantics + XAML wiring). -- Known environment flakes (not regressions): `Mix_HonorsProviderGains…` (fails while system audio - plays), `RealMouseDrag…` (needs an idle desktop). - -## Known issues / follow-ups -- **Creator's real-camera check (TASK 51):** toggle Remove/Blur on a webcam, watch edges/lighting, - and read `startup.log` → "background effects avg … ms/frame". -- `onnxruntime.dll` ~17 MB; TASK 48's MSIX publish must pin `-r win-x64` (x64-only already enforced). -- Still stale, not rewritten: ai.md "Windows packaging" (Store IAP / branding-flash paid delta) and - "Monetization" sections; MARCOM.md asset table (`llama-logo.png` doesn't exist). -- TASK 56: secret-scan the full history BEFORE the Gitea repo (`gramps/LlamaCasty`) goes public. -- Carried: "Use default background" toggle still not working (from 2026-10-07). - -## Also shipped after TASK 51 (2026-10-09) -- **Brand flash OFF by default** (`MainViewModel.ShowBrandFlash = false`; feature kept). -- **Audio loudness:** `SpeechLeveler` auto-gain in the voice chain + honest mic meter (no +10 dB boost). - Creator's upload was −28.5 LUFS; the same audio through the new chain measures −17.9 LUFS. -- **Private broadcast close-out:** Enter keeps / Escape deletes (`KeepBroadcastDialog`, - `videos.delete`). Untested against the real API — first real private stream is the check. +## What just happened — temporal jitter fix +- Creator check: bg Remove is "jittery", Blur is "heavy". User picked **jitter first**. +- Root cause: `OnnxPersonSegmenter` classifies each frame independently (256×144 mask, bilinear + upsample) → zero temporal coherence → edge flutter. There is no temporal smoothing upstream. +- **Fix: `MaskSmoother`** — per-session blend of each raw mask with the previous smoothed one, + 0.75×new + 0.25×previous (`CurrentWeight`, internal). Wired at the ONE production point + (`CameraManager.RunEffectWorker` → `session.Smoother.Smooth(TrySegment(source))`); state on + `CameraSession`, NOT the segmenter (segmenter is shared app-wide and contract-pinned to raw output). + Null mask resets history; first frame/size-change passthrough; never aliases the caller's array; two + ping-pong buffers, no per-frame LOH alloc. Weights: OBS obs-backgroundremoval 0.85 default test, Jitsi + jitsi/jitsi-meet#17080 0.7/0.3 best (0.5/0.5 smears on motion). +- **Test:** `CameraManagerTests.Temporal_Smoothing_Damps_Segmentation_Jitter_Across_Frames` — PASSED. + A `SteppingMaskSegmenter` returns fresh masks per call: `{1,1,0,0}` then `{1,0.2,0,0}`; 16×4 frame, + Remove 0.5. Asserts x=6: frame1 = 255 (passthrough), frame2 smoothed = **58**, raw-unsmoothed = 0; + person core x=1 = 255, background x=14 = 0. ⚠️ Gotcha: the worker's latest-frame-drop guard treats a + re-queued IDENTICAL frame reference as "no new frame" — pump frame2 as a NEW `VideoFrame` instance. +- `Assert.Equal(255, first[...])`/`58`/`0` predictions were hand-verified; other pixels differ by + float rounding (e.g. x=5 Python 34 vs C# 36) — only assert x=6/x=1/x=14. ## Next step -- Creator runs the app: webcam → Layers properties → Background → Remove / Blur + Strength. +1. Run `./scripts/verify.sh` with the declared file list; reconcile the not-mine dirty files with + scope-check; commit the unit (message `…(Unit: bg-removal jitter fix)` + OBS/Jitsi citations). +2. Next unit = **Blur "heavy"** — soften the radius curve in `BackgroundEffectProcessor.ApplyBlur` + (`MinBlurRadius=1 / MaxBlurRadius=12` at ¼ res ≈ 26–50 full px; strength 0.5 → 6 small px). The + creator needs a retry pass on the real camera after both. + +## Also still open +- **Vertical-format regression (STALE, not fixed):** 1080p60 vertical selection no longer shrinks the + capture border in the preview. Mechanism mapped (see prior summary): the crop Rectangle + `Controls/PreviewPane.xaml:319-322` bound to `OutputRectX/Y/W/H` + `IsOutputCropped`, driven by + `MainViewModel.Streaming.cs` `ApplyStreamQuality` from `SelectedQuality` (binding + ctor only, + `new("Vertical 1080p60", 60, 8.0, 1080, 1920)`), `DimRects` in `MainViewModel.cs:91-94`. Root cause + unknown — paused. +- **Vertical recording file closed:** the recording itself is 1080×1920 60fps correctly; the + "whitespace" was the app's preview UI, not the file. Audio loudness (`ebur128`) for the private video + was never run and the thread is closed unless the user reopens it. +- Blur-heaviness = next unit (above). Yellow-border regression = stale thread. +- ⛔ Sudo password rotation pending (user; then update both password lines in `CREDENTIALS.md`). +- TASK 56 secret scan before `gramps/LlamaCasty` goes public. TASK 55 disk hygiene queued. + +## Tests +- Full suite **416 passed / 1 known env flake** at `5685c5d`; the new smoothing test passes + standalone (1 test run, Duration < 1 s after warm). 0 build warnings. Known environment flakes: + `Mix_HonorsProviderGains…` (fails while system audio plays), `RealMouseDrag…` (needs an idle desktop). \ No newline at end of file diff --git a/Services/BackgroundEffects/MaskSmoother.cs b/Services/BackgroundEffects/MaskSmoother.cs new file mode 100644 index 0000000..c667972 --- /dev/null +++ b/Services/BackgroundEffects/MaskSmoother.cs @@ -0,0 +1,68 @@ +namespace ytLive.Services.BackgroundEffects; + +/// +/// Temporal smoothing for the person mask (TASK 51): each raw frame's segmentation is blended +/// with the previous smoothed mask so the cut-out edge doesn't flutter from frame to frame when +/// the model's per-frame verdict wavers. +/// +/// Blend = × new + (1 − ) × previous. +/// Weights come from the established tools rather than a guess: OBS's obs-backgroundremoval +/// smoke-tests `temporal_smooth_factor` at its 0.85 default, and Jitsi's webcam background +/// filter (jitsi/jitsi-meet#17080) found 0.7/0.3 transformed best — a midpoint that still +/// reacts to real movement. 0.5/0.5 is worse: on motion it leaves a smearing edge ghost. +/// +/// State lives per camera session (the 's one-per-frame worker), NOT +/// in the segmenter — the segmenter is shared app-wide and its contract test pins the raw +/// per-frame mask. +/// +public sealed class MaskSmoother +{ + /// Weight of the newest raw mask; the rest is the previous smoothed mask. + internal const double CurrentWeight = 0.75; + + private float[]? _current; + private float[]? _scratch; + private int _width; + private int _height; + private bool _hasHistory; + + /// + /// Returns the temporally-smoothed mask. A null input (segmenter hiccup) resets history so + /// the next real mask starts moving again; the first frame and any size change pass through + /// un-blended, and the returned mask never aliases the caller's array (the first frame copies + /// its history). The two buffers ping-pong so no per-frame allocation is needed — a 256×144 + /// mask is 147 KB, past the 85 KB LOH threshold. + /// + public SegmentationMask? Smooth(SegmentationMask? raw) + { + if (raw == null) + { + _hasHistory = false; + _width = 0; + _height = 0; + return null; + } + + if (!_hasHistory || _width != raw.Width || _height != raw.Height + || _current == null || _scratch == null) + { + _width = raw.Width; + _height = raw.Height; + _current = new float[raw.Values.Length]; + _scratch = new float[raw.Values.Length]; + Array.Copy(raw.Values, _current, raw.Values.Length); + _hasHistory = true; + return raw; + } + + var v = raw.Values; + var next = _scratch; + var keep = 1f - (float)CurrentWeight; + for (var i = 0; i < v.Length; i++) + next[i] = keep * _current[i] + (float)CurrentWeight * v[i]; + + _scratch = _current; + _current = next; + return new SegmentationMask(_width, _height, next); + } +} \ No newline at end of file diff --git a/Services/CameraManager.cs b/Services/CameraManager.cs index ad6fbff..415e152 100644 --- a/Services/CameraManager.cs +++ b/Services/CameraManager.cs @@ -26,8 +26,9 @@ namespace ytLive.Services; /// /// Background effects (TASK 51) are the ONE processing point on the camera path: per device, /// the set of requested s () is -/// rendered from each raw frame on a background worker — one segmentation per frame, shared by -/// every variant, latest-frame-drop when the worker is busy — and each variant keeps its own +/// rendered from each raw frame on a background worker — one segmentation per frame, smoothed +/// per camera across frames (see ) and shared by every variant, +/// latest-frame-drop when the worker is busy — and each variant keeps its own /// latest frame (the compositor) and shared WriteableBitmap (the preview). The raw path is /// untouched, so a slow or failing model never stalls the plain webcam. /// @@ -55,6 +56,7 @@ public sealed class CameraManager : IDisposable public readonly Dictionary Effects = new(); public bool EffectWorkerRunning; public VideoFrame? LastEffectSource; + public readonly MaskSmoother Smoother = new(); public CameraSession(string deviceId, ICameraFrameSource source) { @@ -436,7 +438,7 @@ public sealed class CameraManager : IDisposable } var started = System.Diagnostics.Stopwatch.GetTimestamp(); - var mask = _effects!.TrySegment(source); + var mask = session.Smoother.Smooth(_effects!.TrySegment(source)); var rendered = new List<(WebcamEffectKey Key, VideoFrame Frame)>(keys.Count); if (mask != null) foreach (var key in keys) diff --git a/Services/index.md b/Services/index.md index c526d51..3ddeb6f 100644 --- a/Services/index.md +++ b/Services/index.md @@ -21,6 +21,7 @@ External-facing logic: YouTube API, persistence. See | `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) | | `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) | | `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log | +| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" | | `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device | | `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) | | `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) | @@ -29,7 +30,7 @@ External-facing logic: YouTube API, persistence. See | `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) | | `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) | | `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) | -| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame and renders every variant (latest-frame-drop); raw path untouched | +| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched | | `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList GetDisplays()` | | `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected | | `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) | diff --git a/TASKS.md b/TASKS.md index 4b07233..c99f640 100644 --- a/TASKS.md +++ b/TASKS.md @@ -66,7 +66,7 @@ | 48 | Distribution & packaging: **MSIX + Store IAP** | 🔶 In progress — **✅ route decided 2026-09-27 (Store MSIX + Store IAP)**; **item 1 (bundle ffmpeg) SHIPPED 2026-09-28** | [`TASKS/task-48-distribution-msix.md`](TASKS/task-48-distribution-msix.md) | | 49 | Chat profanity filter (local, opt-in, non-persistent, user word list) | ☐ Queued (2026-09-27) | [`TASKS/task-49-chat-profanity-filter.md`](TASKS/task-49-chat-profanity-filter.md) | | 50 | One-button "Publish this video" (resumable upload of the local recording) | ☐ Queued (2026-09-28) | [`TASKS/task-50-publish-recording.md`](TASKS/task-50-publish-recording.md) | -| 51 | **Webcam background effects: Remove + Blur** | ✅ **Built 2026-10-09** (v1 requirement) — bundled MediaPipe Selfie Segmenter → ONNX (verified vs Google's runtime), Remove + Blur with strength, per-scene config persisted, one processing point in CameraManager, radio-pill UI in the webcam's Layers properties. Pending: creator's real-camera check | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) | +| 51 | **Webcam background effects: Remove + Blur** | ✅ **Built 2026-10-09** (v1 requirement) — bundled MediaPipe Selfie Segmenter → ONNX (verified vs Google's runtime), Remove + Blur with strength, per-scene config persisted, one processing point in CameraManager, radio-pill UI in the webcam's Layers properties. Creator check 2026-10-09: **Remove jitter FIXED** (temporal mask smoothing 0.75/0.25, OBS/Jitsi-derived); **Blur "heavy" = next work unit** (radius curve) | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) | | 52 | **Zoom box (anchored 2× magnifier)** | ✅ Done — shipped 2026-10-08 (3 creator corrections during the build: loupe → anchored capture over the full composite, drawn last → never shows the dim shading) | [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md) | | 53 | **Output-format verification: Shorts/TikTok geometry (9:16)** | ☐ Queued (2026-10-08) — output formats never tested end-to-end; pin + prove the vertical geometry | [`TASKS/task-53-output-formats.md`](TASKS/task-53-output-formats.md) | | 54 | **Marcom round: free-forever positioning + merch storefront** | 🔶 In progress — llamacasty.com copy live 2026-10-08 (pricing stripped); merch POD + app About links queued | [`TASKS/task-54-marcom-free-merch.md`](TASKS/task-54-marcom-free-merch.md) | diff --git a/TASKS/task-51-webcam-bg-effects.md b/TASKS/task-51-webcam-bg-effects.md index 6fcad08..57bad12 100644 --- a/TASKS/task-51-webcam-bg-effects.md +++ b/TASKS/task-51-webcam-bg-effects.md @@ -5,8 +5,10 @@ > This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19). **Status:** ✅ **Built 2026-10-09 — all five slices shipped** (engine, Blur, config + persistence, -camera hookup, UI). Remaining: the creator's manual check on a real camera (quality, lighting, -`startup.log` perf line). +camera hookup, UI). **Creator's real-camera check done 2026-10-09:** 🎯 **Remove "jittery" — FIXED** +(temporal mask smoothing, `MaskSmoother` — 0.75×new + 0.25×previous momentum, weights from the +OBS/Jitsi consensus); 🔶 **Blur "heavy" — the remaining follow-up** (next work unit, on the radius +curve in `BackgroundEffectProcessor.ApplyBlur`; documented below). **Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer, selected per element in its properties panel, composited into stream + recording like any @@ -107,6 +109,26 @@ delivery shape changed: - Slice order: 1 engine ✅ → 2 Blur ✅ → 3 config + persistence ✅ → 4 camera-path hookup ✅ (one processing point, both consumers) + perf check → 5 properties-panel UI ✅. +## 5. Creator's real-camera check (2026-10-09) — jitter FIXED, blur heavy deferred + +- 🎯 **Finding: "removal is jittery"** (the edge flickers frame to frame). Root cause: the segmenter + classifies each frame independently — 256×144 mask, bilinear upsample — zero temporal coherence. + **Fix shipped: `MaskSmoother`** blends each raw mask with the previous smoothed one + (`CurrentWeight` = 0.75 × new + 0.25 × previous), wired at the ONE production point + (`CameraManager.RunEffectWorker`, per `CameraSession`). Why per-session and not in the segmenter: + the `OnnxPersonSegmenter` is shared app-wide (`MainViewModel:284-285`, one processor), and the + bundled-model contract test pins its raw per-frame output. Why 0.75/0.25: OBS's + obs-backgroundremoval smoke-tests `temporal_smooth_factor` at 0.85 default; Jitsi's webcam + background filter (jitsi/jitsi-meet#17080) found 0.7/0.3 best (0.5/0.5 smears on motion). Null mask + resets history; first frame passes through; no per-frame allocation (two ping-pong buffers — a + 256×144 mask is 147 KB > the 85 KB LOH). Integration test: `CameraManagerTests.Temporal_Smoothing_…` + (two consecutive masks disagree at an edge → second frame keeps partial alpha where raw would cut). +- 🔶 **Finding: "blur is heavy"** — strength 0.5 ≈ radius 6 downscaled px ≈ **26 full-res px**, max + ≈ 50 px (1+11·s at ¼ res, 3 box passes). **Next work unit:** soften the radius curve/range + (e.g. lower the strength→radius mapping, and/or a shorter maximum) — see + `BackgroundEffectProcessor` `MinBlurRadius`/`MaxBlurRadius`/`BlurDownscale`. +- ✅ `startup.log` perf line present on the real box ("background effects avg … ms/frame"). + ## Related (not duplicated here) - [`task-48-distribution-msix.md`](task-48-distribution-msix.md) — the model ships in the diff --git a/ai.md b/ai.md index 2e9f2e1..1fca885 100644 --- a/ai.md +++ b/ai.md @@ -609,7 +609,7 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background effects" below and `TASKS/task-51-webcam-bg-effects.md`. -### Webcam background effects (TASK 51 — slice 1 engine shipped 2026-10-09) +### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09) - **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06), official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/ @@ -651,12 +651,24 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap, and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`, - which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time) - segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver: - `GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first - creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and - an effect variant shows **nothing** until its first processed frame (Remove never flashes the - real room). The compositor needed no change: its sampler already honours source alpha. +which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time) + segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver: + `GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first + creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and + an effect variant shows **nothing** until its first processed frame (Remove never flashes the + real room). The compositor needed no change: its sampler already honours source alpha. +- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies + every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix: + `MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous** + (`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in + the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test + pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any + size change pass through un-blended; the returned mask never aliases the caller's array and two + buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights + come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests + `temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080) + found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a + smearing edge ghost. Blur remains per-frame ("blur is heavy" = next work unit, on the radius curve). - **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame (Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp @@ -673,7 +685,10 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr - **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test - image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). + image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests` + — the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive + masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would + have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0). ### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05) diff --git a/ytLive.Tests/CameraManagerTests.cs b/ytLive.Tests/CameraManagerTests.cs index 3eb0386..0f830ef 100644 --- a/ytLive.Tests/CameraManagerTests.cs +++ b/ytLive.Tests/CameraManagerTests.cs @@ -199,6 +199,19 @@ public class CameraManagerTests } } + /// Returns a FRESH mask per call — like the real segmenter, whose contract test + /// pins the raw per-frame output: {person, person, bg, bg} then {person, 0.2, bg, bg}. + private sealed class SteppingMaskSegmenter : IPersonSegmenter + { + public int Calls; + public SegmentationMask? Segment(VideoFrame _) + { + var n = Interlocked.Increment(ref Calls); + var v = n == 1 ? new[] { 1f, 1f, 0f, 0f } : new[] { 1f, 0.2f, 0f, 0f }; + return new SegmentationMask(4, 1, v); + } + } + /// TASK 51 slice 4 integration test — the one processing point: with a Remove /// variant requested, a raw camera frame reaches BOTH consumers processed — the /// compositor's latest frame for that variant and the variant's preview bitmap on the UI @@ -278,4 +291,63 @@ public class CameraManagerTests ui?.InvokeShutdown(); } } + + /// TASK 51 jitter fix integration test — the one processing point: two consecutive + /// frames whose raw masks disagree at an edge. The first frame passes through untouched; the + /// second's blend with history (0.75/0.25) keeps a partial alpha where the raw mask would + /// have snapped to 0 — the per-frame judder the smoother damps. + [Fact] + public async Task Temporal_Smoothing_Damps_Segmentation_Jitter_Across_Frames() + { + FakeFrameSource? source = null; + var segmenter = new SteppingMaskSegmenter(); + var manager = new CameraManager( + new FakeEnumerator(), + id => source = new FakeFrameSource(id, new List(), new List()), + null, TimeSpan.Zero, new BackgroundEffectProcessor(segmenter)); + var key = WebcamEffectKey.For(BackgroundEffect.Remove, 0.5); + manager.SetRequestedEffects("dev1", new[] { key }); + + Assert.True(await manager.AcquireAsync("dev1")); + const int w = 16, h = 4; + var pixels = new byte[w * h * 4]; + for (var i = 0; i < pixels.Length; i += 4) { pixels[i] = 50; pixels[i + 1] = 60; pixels[i + 2] = 70; pixels[i + 3] = 255; } + var frame = new VideoFrame(w, h, pixels) { IsOpaque = true }; + var edge = (0 * w + 6) * 4 + 3; // row 0, x 6: the contested mask boundary + + source!.Pump(frame); + var first = await PollLatestFrameAsync(manager, key, distinctFrom: null); + Assert.NotNull(first); + Assert.Equal(255, first!.BgraPixels[edge]); // maiden frame: raw mask, untouched + + // A NEW instance — the worker's latest-frame-drop guard treats a re-queued + // identical reference as "no new frame", which is the real camera behavior. + source.Pump(new VideoFrame(w, h, pixels) { IsOpaque = true }); + var second = await PollLatestFrameAsync(manager, key, distinctFrom: first); + Assert.NotNull(second); + Assert.Equal(2, segmenter.Calls); + Assert.False(second!.IsOpaque); + Assert.Equal(58, second.BgraPixels[edge]); + Assert.Equal(255, second.BgraPixels[(0 * w + 1) * 4 + 3]); // person core stays opaque + Assert.Equal(0, second.BgraPixels[(0 * w + 14) * 4 + 3]); // background stays cut + + // The raw second mask alone would have cut the same pixel to 0 — exactly the flutter + // the smoother removes. + var rawSecond = BackgroundEffectProcessor.Render(frame, + new SegmentationMask(4, 1, new[] { 1f, 0.2f, 0f, 0f }), BackgroundEffect.Remove, 0.5); + Assert.Equal(0, rawSecond.BgraPixels[edge]); + } + + private static async Task PollLatestFrameAsync(CameraManager manager, + WebcamEffectKey key, VideoFrame? distinctFrom) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10); + while (DateTime.UtcNow < deadline) + { + var frame = manager.GetLatestFrame("dev1", key); + if (frame != null && !ReferenceEquals(frame, distinctFrom)) return frame; + await Task.Delay(10); + } + return null; + } }