TASK 51 jitter fix: temporal mask smoothing. MaskSmoother blends each raw segmentation 0.75xnew + 0.25xprevious at the one CameraManager worker (state per CameraSession, not in the shared segmenter); null resets history, first frame passes through, input array never aliased, ping-pong buffers (no LOH alloc for a 147KB 256x144 mask). Creator's real-camera check: Remove was 'jittery' (per-frame independent segmentation). Weights from the OBS/Jitsi consensus. One integration test asserts the second frame keeps partial alpha (x=6 -> 58) where the raw mask cuts to 0. 416 pass / 2 known env flakes. Refs: https://deepwiki.com/royshil/obs-backgroundremoval/2-core-features https://github.com/jitsi/jitsi-meet/issues/17080 (Unit: bg-removal jitter fix)
This commit is contained in:
+2
-1
@@ -21,6 +21,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
||||
| `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) |
|
||||
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log |
|
||||
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" |
|
||||
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
||||
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
||||
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
||||
@@ -29,7 +30,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) |
|
||||
| `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) |
|
||||
| `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList<DisplayInfo> GetDisplays()` |
|
||||
| `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected |
|
||||
| `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) |
|
||||
|
||||
Reference in New Issue
Block a user