TASK 51 jitter fix: temporal mask smoothing. MaskSmoother blends each raw segmentation 0.75xnew + 0.25xprevious at the one CameraManager worker (state per CameraSession, not in the shared segmenter); null resets history, first frame passes through, input array never aliased, ping-pong buffers (no LOH alloc for a 147KB 256x144 mask). Creator's real-camera check: Remove was 'jittery' (per-frame independent segmentation). Weights from the OBS/Jitsi consensus. One integration test asserts the second frame keeps partial alpha (x=6 -> 58) where the raw mask cuts to 0. 416 pass / 2 known env flakes. Refs: https://deepwiki.com/royshil/obs-backgroundremoval/2-core-features https://github.com/jitsi/jitsi-meet/issues/17080 (Unit: bg-removal jitter fix)
This commit is contained in:
+57
-48
@@ -1,54 +1,63 @@
|
||||
# HANDOFF — 2026-10-09 (TASK 51 webcam background Remove + Blur — BUILT)
|
||||
# HANDOFF — 2026-10-09 (TASK 51 quality pass: jitter FIXED; blur-heavy is the next unit)
|
||||
|
||||
## Branch
|
||||
- main (no feature branch). Pushed through `f8dd107`; everything after is local, **not pushed**
|
||||
(push at the creator's call).
|
||||
- main (no feature branch). Pushed through `5685c5d` (TRAX toggle); everything after is local,
|
||||
**not pushed** (push at the creator's call).
|
||||
|
||||
## Dirty files (NOT mine — do NOT commit)
|
||||
- The in-flight "pinned chat" work unit: `Models/YouTube.cs`, `ViewModels/MainViewModel.Chat.cs`,
|
||||
untracked `Helpers/RelayCommandGeneric.cs`, and **its hunks in `Controls/LeftPanel.xaml`**
|
||||
(lines ~599–610, the chat area). TASK 51 slice 5 edited the SAME file in a different region (the
|
||||
webcam properties, ~line 287); only the TASK 51 hunk was staged (`git apply --cached` of that hunk
|
||||
alone) — the pinned-chat hunks are still unstaged and untouched.
|
||||
- The creator's untracked logo art in `Assets/` (`llamacasty text.*`, `llamacasty-text-nobg*.png`,
|
||||
`thin logo wordmark.png`).
|
||||
## Dirty files — split by ownership
|
||||
- **This work unit (COMMIT these):** `Services/BackgroundEffects/MaskSmoother.cs` (new), the smoother
|
||||
wiring in `Services/CameraManager.cs`, `ytLive.Tests/CameraManagerTests.cs` (new integration test),
|
||||
and the same-commit memory updates (`Services/index.md`, `ai.md`, `TASKS/task-51-webcam-bg-effects.md`,
|
||||
`TASKS.md` row 51, and this file) — all ship in the SAME commit.
|
||||
- **NOT mine — do NOT commit:** the in-flight "pinned chat" work unit (`Models/YouTube.cs`,
|
||||
`ViewModels/MainViewModel.Chat.cs`, untracked `Helpers/RelayCommandGeneric.cs`, and **its hunks in
|
||||
`Controls/LeftPanel.xaml`** — only the webcam-properties region is TASK 51; the pinned-chat hunks
|
||||
stayed unstaged/untouched), plus the creator's untracked `Assets/llamacasty*` art.
|
||||
- ⚠️ Pre-commit: `scripts/verify.sh` runs `scripts/scope-check.sh` against ALL working-tree changes —
|
||||
the not-mine files WILL flag. Pass them to the script as declared-known (checksums unchanged since
|
||||
`5685c5d`) or let the script's output be reconciled before committing.
|
||||
|
||||
## What just happened — TASK 51, all five slices
|
||||
1. **Engine** — Google's MediaPipe Selfie Segmenter (landscape, Apache-2.0) converted to ONNX by
|
||||
`tools/selfie-seg/` and verified vs MediaPipe's own runtime; embedded resource; onnxruntime CPU.
|
||||
2. **Blur** — background at ¼ res, 3 box passes, mask-blended; Remove gets edge feather.
|
||||
3. **Config** — `WebcamSceneConfig.BackgroundEffect` + `BackgroundStrength`, persisted (additive cols).
|
||||
4. **Hookup** — `CameraManager` renders per-scene effect **variants** (one segmentation per raw frame
|
||||
on a latest-frame-drop worker; raw path untouched); VM `SyncWebcamEffects()` declares/adopts them.
|
||||
Synthetic 720p Debug: segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms.
|
||||
5. **UI** — Background section in the webcam's Layers properties: Remove / Blur `PillToggle`s,
|
||||
radio-exclusive, + Strength slider (visible while an effect is on).
|
||||
Also this session: 64-bit requirement (`PlatformTarget=x64`), GPL-3.0-or-later licensing, README
|
||||
rewrite, TASK 55 (disk hygiene) + TASK 56 (secret scan before going public) queued.
|
||||
|
||||
## Tests
|
||||
- Full suite **412 passed / 0 failed**, 0 build warnings. New: `BackgroundEffectEngineTests` (Remove,
|
||||
Blur, bundled-model contract), `CameraManagerTests` (variant reaches compositor + preview, raw
|
||||
stays raw), `LayoutStorePersistenceTests` (per-scene roundtrip + legacy migration),
|
||||
`WebcamBackgroundPillTests` (radio semantics + XAML wiring).
|
||||
- Known environment flakes (not regressions): `Mix_HonorsProviderGains…` (fails while system audio
|
||||
plays), `RealMouseDrag…` (needs an idle desktop).
|
||||
|
||||
## Known issues / follow-ups
|
||||
- **Creator's real-camera check (TASK 51):** toggle Remove/Blur on a webcam, watch edges/lighting,
|
||||
and read `startup.log` → "background effects avg … ms/frame".
|
||||
- `onnxruntime.dll` ~17 MB; TASK 48's MSIX publish must pin `-r win-x64` (x64-only already enforced).
|
||||
- Still stale, not rewritten: ai.md "Windows packaging" (Store IAP / branding-flash paid delta) and
|
||||
"Monetization" sections; MARCOM.md asset table (`llama-logo.png` doesn't exist).
|
||||
- TASK 56: secret-scan the full history BEFORE the Gitea repo (`gramps/LlamaCasty`) goes public.
|
||||
- Carried: "Use default background" toggle still not working (from 2026-10-07).
|
||||
|
||||
## Also shipped after TASK 51 (2026-10-09)
|
||||
- **Brand flash OFF by default** (`MainViewModel.ShowBrandFlash = false`; feature kept).
|
||||
- **Audio loudness:** `SpeechLeveler` auto-gain in the voice chain + honest mic meter (no +10 dB boost).
|
||||
Creator's upload was −28.5 LUFS; the same audio through the new chain measures −17.9 LUFS.
|
||||
- **Private broadcast close-out:** Enter keeps / Escape deletes (`KeepBroadcastDialog`,
|
||||
`videos.delete`). Untested against the real API — first real private stream is the check.
|
||||
## What just happened — temporal jitter fix
|
||||
- Creator check: bg Remove is "jittery", Blur is "heavy". User picked **jitter first**.
|
||||
- Root cause: `OnnxPersonSegmenter` classifies each frame independently (256×144 mask, bilinear
|
||||
upsample) → zero temporal coherence → edge flutter. There is no temporal smoothing upstream.
|
||||
- **Fix: `MaskSmoother`** — per-session blend of each raw mask with the previous smoothed one,
|
||||
0.75×new + 0.25×previous (`CurrentWeight`, internal). Wired at the ONE production point
|
||||
(`CameraManager.RunEffectWorker` → `session.Smoother.Smooth(TrySegment(source))`); state on
|
||||
`CameraSession`, NOT the segmenter (segmenter is shared app-wide and contract-pinned to raw output).
|
||||
Null mask resets history; first frame/size-change passthrough; never aliases the caller's array; two
|
||||
ping-pong buffers, no per-frame LOH alloc. Weights: OBS obs-backgroundremoval 0.85 default test, Jitsi
|
||||
jitsi/jitsi-meet#17080 0.7/0.3 best (0.5/0.5 smears on motion).
|
||||
- **Test:** `CameraManagerTests.Temporal_Smoothing_Damps_Segmentation_Jitter_Across_Frames` — PASSED.
|
||||
A `SteppingMaskSegmenter` returns fresh masks per call: `{1,1,0,0}` then `{1,0.2,0,0}`; 16×4 frame,
|
||||
Remove 0.5. Asserts x=6: frame1 = 255 (passthrough), frame2 smoothed = **58**, raw-unsmoothed = 0;
|
||||
person core x=1 = 255, background x=14 = 0. ⚠️ Gotcha: the worker's latest-frame-drop guard treats a
|
||||
re-queued IDENTICAL frame reference as "no new frame" — pump frame2 as a NEW `VideoFrame` instance.
|
||||
- `Assert.Equal(255, first[...])`/`58`/`0` predictions were hand-verified; other pixels differ by
|
||||
float rounding (e.g. x=5 Python 34 vs C# 36) — only assert x=6/x=1/x=14.
|
||||
|
||||
## Next step
|
||||
- Creator runs the app: webcam → Layers properties → Background → Remove / Blur + Strength.
|
||||
1. Run `./scripts/verify.sh` with the declared file list; reconcile the not-mine dirty files with
|
||||
scope-check; commit the unit (message `…(Unit: bg-removal jitter fix)` + OBS/Jitsi citations).
|
||||
2. Next unit = **Blur "heavy"** — soften the radius curve in `BackgroundEffectProcessor.ApplyBlur`
|
||||
(`MinBlurRadius=1 / MaxBlurRadius=12` at ¼ res ≈ 26–50 full px; strength 0.5 → 6 small px). The
|
||||
creator needs a retry pass on the real camera after both.
|
||||
|
||||
## Also still open
|
||||
- **Vertical-format regression (STALE, not fixed):** 1080p60 vertical selection no longer shrinks the
|
||||
capture border in the preview. Mechanism mapped (see prior summary): the crop Rectangle
|
||||
`Controls/PreviewPane.xaml:319-322` bound to `OutputRectX/Y/W/H` + `IsOutputCropped`, driven by
|
||||
`MainViewModel.Streaming.cs` `ApplyStreamQuality` from `SelectedQuality` (binding + ctor only,
|
||||
`new("Vertical 1080p60", 60, 8.0, 1080, 1920)`), `DimRects` in `MainViewModel.cs:91-94`. Root cause
|
||||
unknown — paused.
|
||||
- **Vertical recording file closed:** the recording itself is 1080×1920 60fps correctly; the
|
||||
"whitespace" was the app's preview UI, not the file. Audio loudness (`ebur128`) for the private video
|
||||
was never run and the thread is closed unless the user reopens it.
|
||||
- Blur-heaviness = next unit (above). Yellow-border regression = stale thread.
|
||||
- ⛔ Sudo password rotation pending (user; then update both password lines in `CREDENTIALS.md`).
|
||||
- TASK 56 secret scan before `gramps/LlamaCasty` goes public. TASK 55 disk hygiene queued.
|
||||
|
||||
## Tests
|
||||
- Full suite **416 passed / 1 known env flake** at `5685c5d`; the new smoothing test passes
|
||||
standalone (1 test run, Duration < 1 s after warm). 0 build warnings. Known environment flakes:
|
||||
`Mix_HonorsProviderGains…` (fails while system audio plays), `RealMouseDrag…` (needs an idle desktop).
|
||||
@@ -0,0 +1,68 @@
|
||||
namespace ytLive.Services.BackgroundEffects;
|
||||
|
||||
/// <summary>
|
||||
/// Temporal smoothing for the person mask (TASK 51): each raw frame's segmentation is blended
|
||||
/// with the previous smoothed mask so the cut-out edge doesn't flutter from frame to frame when
|
||||
/// the model's per-frame verdict wavers.
|
||||
///
|
||||
/// Blend = <see cref="CurrentWeight"/> × new + (1 − <see cref="CurrentWeight"/>) × previous.
|
||||
/// Weights come from the established tools rather than a guess: OBS's obs-backgroundremoval
|
||||
/// smoke-tests `temporal_smooth_factor` at its 0.85 default, and Jitsi's webcam background
|
||||
/// filter (jitsi/jitsi-meet#17080) found 0.7/0.3 transformed best — a midpoint that still
|
||||
/// reacts to real movement. 0.5/0.5 is worse: on motion it leaves a smearing edge ghost.
|
||||
///
|
||||
/// State lives per camera session (the <see cref="CameraManager"/>'s one-per-frame worker), NOT
|
||||
/// in the segmenter — the segmenter is shared app-wide and its contract test pins the raw
|
||||
/// per-frame mask.
|
||||
/// </summary>
|
||||
public sealed class MaskSmoother
|
||||
{
|
||||
/// <summary>Weight of the newest raw mask; the rest is the previous smoothed mask.</summary>
|
||||
internal const double CurrentWeight = 0.75;
|
||||
|
||||
private float[]? _current;
|
||||
private float[]? _scratch;
|
||||
private int _width;
|
||||
private int _height;
|
||||
private bool _hasHistory;
|
||||
|
||||
/// <summary>
|
||||
/// Returns the temporally-smoothed mask. A null input (segmenter hiccup) resets history so
|
||||
/// the next real mask starts moving again; the first frame and any size change pass through
|
||||
/// un-blended, and the returned mask never aliases the caller's array (the first frame copies
|
||||
/// its history). The two buffers ping-pong so no per-frame allocation is needed — a 256×144
|
||||
/// mask is 147 KB, past the 85 KB LOH threshold.
|
||||
/// </summary>
|
||||
public SegmentationMask? Smooth(SegmentationMask? raw)
|
||||
{
|
||||
if (raw == null)
|
||||
{
|
||||
_hasHistory = false;
|
||||
_width = 0;
|
||||
_height = 0;
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_hasHistory || _width != raw.Width || _height != raw.Height
|
||||
|| _current == null || _scratch == null)
|
||||
{
|
||||
_width = raw.Width;
|
||||
_height = raw.Height;
|
||||
_current = new float[raw.Values.Length];
|
||||
_scratch = new float[raw.Values.Length];
|
||||
Array.Copy(raw.Values, _current, raw.Values.Length);
|
||||
_hasHistory = true;
|
||||
return raw;
|
||||
}
|
||||
|
||||
var v = raw.Values;
|
||||
var next = _scratch;
|
||||
var keep = 1f - (float)CurrentWeight;
|
||||
for (var i = 0; i < v.Length; i++)
|
||||
next[i] = keep * _current[i] + (float)CurrentWeight * v[i];
|
||||
|
||||
_scratch = _current;
|
||||
_current = next;
|
||||
return new SegmentationMask(_width, _height, next);
|
||||
}
|
||||
}
|
||||
@@ -26,8 +26,9 @@ namespace ytLive.Services;
|
||||
///
|
||||
/// Background effects (TASK 51) are the ONE processing point on the camera path: per device,
|
||||
/// the set of requested <see cref="WebcamEffectKey"/>s (<see cref="SetRequestedEffects"/>) is
|
||||
/// rendered from each raw frame on a background worker — one segmentation per frame, shared by
|
||||
/// every variant, latest-frame-drop when the worker is busy — and each variant keeps its own
|
||||
/// rendered from each raw frame on a background worker — one segmentation per frame, smoothed
|
||||
/// per camera across frames (see <see cref="MaskSmoother"/>) and shared by every variant,
|
||||
/// latest-frame-drop when the worker is busy — and each variant keeps its own
|
||||
/// latest frame (the compositor) and shared WriteableBitmap (the preview). The raw path is
|
||||
/// untouched, so a slow or failing model never stalls the plain webcam.
|
||||
/// </summary>
|
||||
@@ -55,6 +56,7 @@ public sealed class CameraManager : IDisposable
|
||||
public readonly Dictionary<WebcamEffectKey, EffectVariant> Effects = new();
|
||||
public bool EffectWorkerRunning;
|
||||
public VideoFrame? LastEffectSource;
|
||||
public readonly MaskSmoother Smoother = new();
|
||||
|
||||
public CameraSession(string deviceId, ICameraFrameSource source)
|
||||
{
|
||||
@@ -436,7 +438,7 @@ public sealed class CameraManager : IDisposable
|
||||
}
|
||||
|
||||
var started = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var mask = _effects!.TrySegment(source);
|
||||
var mask = session.Smoother.Smooth(_effects!.TrySegment(source));
|
||||
var rendered = new List<(WebcamEffectKey Key, VideoFrame Frame)>(keys.Count);
|
||||
if (mask != null)
|
||||
foreach (var key in keys)
|
||||
|
||||
+2
-1
@@ -21,6 +21,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
||||
| `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) |
|
||||
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log |
|
||||
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" |
|
||||
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
||||
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
||||
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
||||
@@ -29,7 +30,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) |
|
||||
| `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) |
|
||||
| `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList<DisplayInfo> GetDisplays()` |
|
||||
| `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected |
|
||||
| `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) |
|
||||
|
||||
@@ -66,7 +66,7 @@
|
||||
| 48 | Distribution & packaging: **MSIX + Store IAP** | 🔶 In progress — **✅ route decided 2026-09-27 (Store MSIX + Store IAP)**; **item 1 (bundle ffmpeg) SHIPPED 2026-09-28** | [`TASKS/task-48-distribution-msix.md`](TASKS/task-48-distribution-msix.md) |
|
||||
| 49 | Chat profanity filter (local, opt-in, non-persistent, user word list) | ☐ Queued (2026-09-27) | [`TASKS/task-49-chat-profanity-filter.md`](TASKS/task-49-chat-profanity-filter.md) |
|
||||
| 50 | One-button "Publish this video" (resumable upload of the local recording) | ☐ Queued (2026-09-28) | [`TASKS/task-50-publish-recording.md`](TASKS/task-50-publish-recording.md) |
|
||||
| 51 | **Webcam background effects: Remove + Blur** | ✅ **Built 2026-10-09** (v1 requirement) — bundled MediaPipe Selfie Segmenter → ONNX (verified vs Google's runtime), Remove + Blur with strength, per-scene config persisted, one processing point in CameraManager, radio-pill UI in the webcam's Layers properties. Pending: creator's real-camera check | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) |
|
||||
| 51 | **Webcam background effects: Remove + Blur** | ✅ **Built 2026-10-09** (v1 requirement) — bundled MediaPipe Selfie Segmenter → ONNX (verified vs Google's runtime), Remove + Blur with strength, per-scene config persisted, one processing point in CameraManager, radio-pill UI in the webcam's Layers properties. Creator check 2026-10-09: **Remove jitter FIXED** (temporal mask smoothing 0.75/0.25, OBS/Jitsi-derived); **Blur "heavy" = next work unit** (radius curve) | [`TASKS/task-51-webcam-bg-effects.md`](TASKS/task-51-webcam-bg-effects.md) |
|
||||
| 52 | **Zoom box (anchored 2× magnifier)** | ✅ Done — shipped 2026-10-08 (3 creator corrections during the build: loupe → anchored capture over the full composite, drawn last → never shows the dim shading) | [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md) |
|
||||
| 53 | **Output-format verification: Shorts/TikTok geometry (9:16)** | ☐ Queued (2026-10-08) — output formats never tested end-to-end; pin + prove the vertical geometry | [`TASKS/task-53-output-formats.md`](TASKS/task-53-output-formats.md) |
|
||||
| 54 | **Marcom round: free-forever positioning + merch storefront** | 🔶 In progress — llamacasty.com copy live 2026-10-08 (pricing stripped); merch POD + app About links queued | [`TASKS/task-54-marcom-free-merch.md`](TASKS/task-54-marcom-free-merch.md) |
|
||||
|
||||
@@ -5,8 +5,10 @@
|
||||
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).
|
||||
|
||||
**Status:** ✅ **Built 2026-10-09 — all five slices shipped** (engine, Blur, config + persistence,
|
||||
camera hookup, UI). Remaining: the creator's manual check on a real camera (quality, lighting,
|
||||
`startup.log` perf line).
|
||||
camera hookup, UI). **Creator's real-camera check done 2026-10-09:** 🎯 **Remove "jittery" — FIXED**
|
||||
(temporal mask smoothing, `MaskSmoother` — 0.75×new + 0.25×previous momentum, weights from the
|
||||
OBS/Jitsi consensus); 🔶 **Blur "heavy" — the remaining follow-up** (next work unit, on the radius
|
||||
curve in `BackgroundEffectProcessor.ApplyBlur`; documented below).
|
||||
|
||||
**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
|
||||
selected per element in its properties panel, composited into stream + recording like any
|
||||
@@ -107,6 +109,26 @@ delivery shape changed:
|
||||
- Slice order: 1 engine ✅ → 2 Blur ✅ → 3 config + persistence ✅ → 4 camera-path hookup ✅ (one
|
||||
processing point, both consumers) + perf check → 5 properties-panel UI ✅.
|
||||
|
||||
## 5. Creator's real-camera check (2026-10-09) — jitter FIXED, blur heavy deferred
|
||||
|
||||
- 🎯 **Finding: "removal is jittery"** (the edge flickers frame to frame). Root cause: the segmenter
|
||||
classifies each frame independently — 256×144 mask, bilinear upsample — zero temporal coherence.
|
||||
**Fix shipped: `MaskSmoother`** blends each raw mask with the previous smoothed one
|
||||
(`CurrentWeight` = 0.75 × new + 0.25 × previous), wired at the ONE production point
|
||||
(`CameraManager.RunEffectWorker`, per `CameraSession`). Why per-session and not in the segmenter:
|
||||
the `OnnxPersonSegmenter` is shared app-wide (`MainViewModel:284-285`, one processor), and the
|
||||
bundled-model contract test pins its raw per-frame output. Why 0.75/0.25: OBS's
|
||||
obs-backgroundremoval smoke-tests `temporal_smooth_factor` at 0.85 default; Jitsi's webcam
|
||||
background filter (jitsi/jitsi-meet#17080) found 0.7/0.3 best (0.5/0.5 smears on motion). Null mask
|
||||
resets history; first frame passes through; no per-frame allocation (two ping-pong buffers — a
|
||||
256×144 mask is 147 KB > the 85 KB LOH). Integration test: `CameraManagerTests.Temporal_Smoothing_…`
|
||||
(two consecutive masks disagree at an edge → second frame keeps partial alpha where raw would cut).
|
||||
- 🔶 **Finding: "blur is heavy"** — strength 0.5 ≈ radius 6 downscaled px ≈ **26 full-res px**, max
|
||||
≈ 50 px (1+11·s at ¼ res, 3 box passes). **Next work unit:** soften the radius curve/range
|
||||
(e.g. lower the strength→radius mapping, and/or a shorter maximum) — see
|
||||
`BackgroundEffectProcessor` `MinBlurRadius`/`MaxBlurRadius`/`BlurDownscale`.
|
||||
- ✅ `startup.log` perf line present on the real box ("background effects avg … ms/frame").
|
||||
|
||||
## Related (not duplicated here)
|
||||
|
||||
- [`task-48-distribution-msix.md`](task-48-distribution-msix.md) — the model ships in the
|
||||
|
||||
@@ -609,7 +609,7 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||
|
||||
### Webcam background effects (TASK 51 — slice 1 engine shipped 2026-10-09)
|
||||
### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)
|
||||
|
||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||
@@ -651,12 +651,24 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
|
||||
config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
|
||||
and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
|
||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
|
||||
every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
|
||||
`MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
|
||||
(`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
|
||||
the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
|
||||
pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
|
||||
size change pass through un-blended; the returned mask never aliases the caller's array and two
|
||||
buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
|
||||
come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
|
||||
`temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
|
||||
found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
|
||||
smearing edge ghost. Blur remains per-frame ("blur is heavy" = next work unit, on the radius curve).
|
||||
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
|
||||
(Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
|
||||
bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
|
||||
@@ -673,7 +685,10 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`).
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
|
||||
— the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
|
||||
masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
|
||||
have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
|
||||
|
||||
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
|
||||
|
||||
|
||||
@@ -199,6 +199,19 @@ public class CameraManagerTests
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Returns a FRESH mask per call — like the real segmenter, whose contract test
|
||||
/// pins the raw per-frame output: {person, person, bg, bg} then {person, 0.2, bg, bg}.</summary>
|
||||
private sealed class SteppingMaskSegmenter : IPersonSegmenter
|
||||
{
|
||||
public int Calls;
|
||||
public SegmentationMask? Segment(VideoFrame _)
|
||||
{
|
||||
var n = Interlocked.Increment(ref Calls);
|
||||
var v = n == 1 ? new[] { 1f, 1f, 0f, 0f } : new[] { 1f, 0.2f, 0f, 0f };
|
||||
return new SegmentationMask(4, 1, v);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>TASK 51 slice 4 integration test — the one processing point: with a Remove
|
||||
/// variant requested, a raw camera frame reaches BOTH consumers processed — the
|
||||
/// compositor's latest frame for that variant and the variant's preview bitmap on the UI
|
||||
@@ -278,4 +291,63 @@ public class CameraManagerTests
|
||||
ui?.InvokeShutdown();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>TASK 51 jitter fix integration test — the one processing point: two consecutive
|
||||
/// frames whose raw masks disagree at an edge. The first frame passes through untouched; the
|
||||
/// second's blend with history (0.75/0.25) keeps a partial alpha where the raw mask would
|
||||
/// have snapped to 0 — the per-frame judder the smoother damps.</summary>
|
||||
[Fact]
|
||||
public async Task Temporal_Smoothing_Damps_Segmentation_Jitter_Across_Frames()
|
||||
{
|
||||
FakeFrameSource? source = null;
|
||||
var segmenter = new SteppingMaskSegmenter();
|
||||
var manager = new CameraManager(
|
||||
new FakeEnumerator(),
|
||||
id => source = new FakeFrameSource(id, new List<string>(), new List<string>()),
|
||||
null, TimeSpan.Zero, new BackgroundEffectProcessor(segmenter));
|
||||
var key = WebcamEffectKey.For(BackgroundEffect.Remove, 0.5);
|
||||
manager.SetRequestedEffects("dev1", new[] { key });
|
||||
|
||||
Assert.True(await manager.AcquireAsync("dev1"));
|
||||
const int w = 16, h = 4;
|
||||
var pixels = new byte[w * h * 4];
|
||||
for (var i = 0; i < pixels.Length; i += 4) { pixels[i] = 50; pixels[i + 1] = 60; pixels[i + 2] = 70; pixels[i + 3] = 255; }
|
||||
var frame = new VideoFrame(w, h, pixels) { IsOpaque = true };
|
||||
var edge = (0 * w + 6) * 4 + 3; // row 0, x 6: the contested mask boundary
|
||||
|
||||
source!.Pump(frame);
|
||||
var first = await PollLatestFrameAsync(manager, key, distinctFrom: null);
|
||||
Assert.NotNull(first);
|
||||
Assert.Equal(255, first!.BgraPixels[edge]); // maiden frame: raw mask, untouched
|
||||
|
||||
// A NEW instance — the worker's latest-frame-drop guard treats a re-queued
|
||||
// identical reference as "no new frame", which is the real camera behavior.
|
||||
source.Pump(new VideoFrame(w, h, pixels) { IsOpaque = true });
|
||||
var second = await PollLatestFrameAsync(manager, key, distinctFrom: first);
|
||||
Assert.NotNull(second);
|
||||
Assert.Equal(2, segmenter.Calls);
|
||||
Assert.False(second!.IsOpaque);
|
||||
Assert.Equal(58, second.BgraPixels[edge]);
|
||||
Assert.Equal(255, second.BgraPixels[(0 * w + 1) * 4 + 3]); // person core stays opaque
|
||||
Assert.Equal(0, second.BgraPixels[(0 * w + 14) * 4 + 3]); // background stays cut
|
||||
|
||||
// The raw second mask alone would have cut the same pixel to 0 — exactly the flutter
|
||||
// the smoother removes.
|
||||
var rawSecond = BackgroundEffectProcessor.Render(frame,
|
||||
new SegmentationMask(4, 1, new[] { 1f, 0.2f, 0f, 0f }), BackgroundEffect.Remove, 0.5);
|
||||
Assert.Equal(0, rawSecond.BgraPixels[edge]);
|
||||
}
|
||||
|
||||
private static async Task<VideoFrame?> PollLatestFrameAsync(CameraManager manager,
|
||||
WebcamEffectKey key, VideoFrame? distinctFrom)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
var frame = manager.GetLatestFrame("dev1", key);
|
||||
if (frame != null && !ReferenceEquals(frame, distinctFrom)) return frame;
|
||||
await Task.Delay(10);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user