TASK 51 jitter fix: temporal mask smoothing. MaskSmoother blends each raw segmentation 0.75xnew + 0.25xprevious at the one CameraManager worker (state per CameraSession, not in the shared segmenter); null resets history, first frame passes through, input array never aliased, ping-pong buffers (no LOH alloc for a 147KB 256x144 mask). Creator's real-camera check: Remove was 'jittery' (per-frame independent segmentation). Weights from the OBS/Jitsi consensus. One integration test asserts the second frame keeps partial alpha (x=6 -> 58) where the raw mask cuts to 0. 416 pass / 2 known env flakes. Refs: https://deepwiki.com/royshil/obs-backgroundremoval/2-core-features https://github.com/jitsi/jitsi-meet/issues/17080 (Unit: bg-removal jitter fix)
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
namespace ytLive.Services.BackgroundEffects;
|
||||
|
||||
/// <summary>
|
||||
/// Temporal smoothing for the person mask (TASK 51): each raw frame's segmentation is blended
|
||||
/// with the previous smoothed mask so the cut-out edge doesn't flutter from frame to frame when
|
||||
/// the model's per-frame verdict wavers.
|
||||
///
|
||||
/// Blend = <see cref="CurrentWeight"/> × new + (1 − <see cref="CurrentWeight"/>) × previous.
|
||||
/// Weights come from the established tools rather than a guess: OBS's obs-backgroundremoval
|
||||
/// smoke-tests `temporal_smooth_factor` at its 0.85 default, and Jitsi's webcam background
|
||||
/// filter (jitsi/jitsi-meet#17080) found 0.7/0.3 transformed best — a midpoint that still
|
||||
/// reacts to real movement. 0.5/0.5 is worse: on motion it leaves a smearing edge ghost.
|
||||
///
|
||||
/// State lives per camera session (the <see cref="CameraManager"/>'s one-per-frame worker), NOT
|
||||
/// in the segmenter — the segmenter is shared app-wide and its contract test pins the raw
|
||||
/// per-frame mask.
|
||||
/// </summary>
|
||||
public sealed class MaskSmoother
|
||||
{
|
||||
/// <summary>Weight of the newest raw mask; the rest is the previous smoothed mask.</summary>
|
||||
internal const double CurrentWeight = 0.75;
|
||||
|
||||
private float[]? _current;
|
||||
private float[]? _scratch;
|
||||
private int _width;
|
||||
private int _height;
|
||||
private bool _hasHistory;
|
||||
|
||||
/// <summary>
|
||||
/// Returns the temporally-smoothed mask. A null input (segmenter hiccup) resets history so
|
||||
/// the next real mask starts moving again; the first frame and any size change pass through
|
||||
/// un-blended, and the returned mask never aliases the caller's array (the first frame copies
|
||||
/// its history). The two buffers ping-pong so no per-frame allocation is needed — a 256×144
|
||||
/// mask is 147 KB, past the 85 KB LOH threshold.
|
||||
/// </summary>
|
||||
public SegmentationMask? Smooth(SegmentationMask? raw)
|
||||
{
|
||||
if (raw == null)
|
||||
{
|
||||
_hasHistory = false;
|
||||
_width = 0;
|
||||
_height = 0;
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_hasHistory || _width != raw.Width || _height != raw.Height
|
||||
|| _current == null || _scratch == null)
|
||||
{
|
||||
_width = raw.Width;
|
||||
_height = raw.Height;
|
||||
_current = new float[raw.Values.Length];
|
||||
_scratch = new float[raw.Values.Length];
|
||||
Array.Copy(raw.Values, _current, raw.Values.Length);
|
||||
_hasHistory = true;
|
||||
return raw;
|
||||
}
|
||||
|
||||
var v = raw.Values;
|
||||
var next = _scratch;
|
||||
var keep = 1f - (float)CurrentWeight;
|
||||
for (var i = 0; i < v.Length; i++)
|
||||
next[i] = keep * _current[i] + (float)CurrentWeight * v[i];
|
||||
|
||||
_scratch = _current;
|
||||
_current = next;
|
||||
return new SegmentationMask(_width, _height, next);
|
||||
}
|
||||
}
|
||||
@@ -26,8 +26,9 @@ namespace ytLive.Services;
|
||||
///
|
||||
/// Background effects (TASK 51) are the ONE processing point on the camera path: per device,
|
||||
/// the set of requested <see cref="WebcamEffectKey"/>s (<see cref="SetRequestedEffects"/>) is
|
||||
/// rendered from each raw frame on a background worker — one segmentation per frame, shared by
|
||||
/// every variant, latest-frame-drop when the worker is busy — and each variant keeps its own
|
||||
/// rendered from each raw frame on a background worker — one segmentation per frame, smoothed
|
||||
/// per camera across frames (see <see cref="MaskSmoother"/>) and shared by every variant,
|
||||
/// latest-frame-drop when the worker is busy — and each variant keeps its own
|
||||
/// latest frame (the compositor) and shared WriteableBitmap (the preview). The raw path is
|
||||
/// untouched, so a slow or failing model never stalls the plain webcam.
|
||||
/// </summary>
|
||||
@@ -55,6 +56,7 @@ public sealed class CameraManager : IDisposable
|
||||
public readonly Dictionary<WebcamEffectKey, EffectVariant> Effects = new();
|
||||
public bool EffectWorkerRunning;
|
||||
public VideoFrame? LastEffectSource;
|
||||
public readonly MaskSmoother Smoother = new();
|
||||
|
||||
public CameraSession(string deviceId, ICameraFrameSource source)
|
||||
{
|
||||
@@ -436,7 +438,7 @@ public sealed class CameraManager : IDisposable
|
||||
}
|
||||
|
||||
var started = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var mask = _effects!.TrySegment(source);
|
||||
var mask = session.Smoother.Smooth(_effects!.TrySegment(source));
|
||||
var rendered = new List<(WebcamEffectKey Key, VideoFrame Frame)>(keys.Count);
|
||||
if (mask != null)
|
||||
foreach (var key in keys)
|
||||
|
||||
+2
-1
@@ -21,6 +21,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
||||
| `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) |
|
||||
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log |
|
||||
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" |
|
||||
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
||||
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
||||
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
||||
@@ -29,7 +30,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) |
|
||||
| `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) |
|
||||
| `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList<DisplayInfo> GetDisplays()` |
|
||||
| `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected |
|
||||
| `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) |
|
||||
|
||||
Reference in New Issue
Block a user