TASK 51 jitter fix: temporal mask smoothing. MaskSmoother blends each raw segmentation 0.75xnew + 0.25xprevious at the one CameraManager worker (state per CameraSession, not in the shared segmenter); null resets history, first frame passes through, input array never aliased, ping-pong buffers (no LOH alloc for a 147KB 256x144 mask). Creator's real-camera check: Remove was 'jittery' (per-frame independent segmentation). Weights from the OBS/Jitsi consensus. One integration test asserts the second frame keeps partial alpha (x=6 -> 58) where the raw mask cuts to 0. 416 pass / 2 known env flakes. Refs: https://deepwiki.com/royshil/obs-backgroundremoval/2-core-features https://github.com/jitsi/jitsi-meet/issues/17080 (Unit: bg-removal jitter fix)
This commit is contained in:
@@ -5,8 +5,10 @@
|
||||
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).
|
||||
|
||||
**Status:** ✅ **Built 2026-10-09 — all five slices shipped** (engine, Blur, config + persistence,
|
||||
camera hookup, UI). Remaining: the creator's manual check on a real camera (quality, lighting,
|
||||
`startup.log` perf line).
|
||||
camera hookup, UI). **Creator's real-camera check done 2026-10-09:** 🎯 **Remove "jittery" — FIXED**
|
||||
(temporal mask smoothing, `MaskSmoother` — 0.75×new + 0.25×previous momentum, weights from the
|
||||
OBS/Jitsi consensus); 🔶 **Blur "heavy" — the remaining follow-up** (next work unit, on the radius
|
||||
curve in `BackgroundEffectProcessor.ApplyBlur`; documented below).
|
||||
|
||||
**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
|
||||
selected per element in its properties panel, composited into stream + recording like any
|
||||
@@ -107,6 +109,26 @@ delivery shape changed:
|
||||
- Slice order: 1 engine ✅ → 2 Blur ✅ → 3 config + persistence ✅ → 4 camera-path hookup ✅ (one
|
||||
processing point, both consumers) + perf check → 5 properties-panel UI ✅.
|
||||
|
||||
## 5. Creator's real-camera check (2026-10-09) — jitter FIXED, blur heavy deferred
|
||||
|
||||
- 🎯 **Finding: "removal is jittery"** (the edge flickers frame to frame). Root cause: the segmenter
|
||||
classifies each frame independently — 256×144 mask, bilinear upsample — zero temporal coherence.
|
||||
**Fix shipped: `MaskSmoother`** blends each raw mask with the previous smoothed one
|
||||
(`CurrentWeight` = 0.75 × new + 0.25 × previous), wired at the ONE production point
|
||||
(`CameraManager.RunEffectWorker`, per `CameraSession`). Why per-session and not in the segmenter:
|
||||
the `OnnxPersonSegmenter` is shared app-wide (`MainViewModel:284-285`, one processor), and the
|
||||
bundled-model contract test pins its raw per-frame output. Why 0.75/0.25: OBS's
|
||||
obs-backgroundremoval smoke-tests `temporal_smooth_factor` at 0.85 default; Jitsi's webcam
|
||||
background filter (jitsi/jitsi-meet#17080) found 0.7/0.3 best (0.5/0.5 smears on motion). Null mask
|
||||
resets history; first frame passes through; no per-frame allocation (two ping-pong buffers — a
|
||||
256×144 mask is 147 KB > the 85 KB LOH). Integration test: `CameraManagerTests.Temporal_Smoothing_…`
|
||||
(two consecutive masks disagree at an edge → second frame keeps partial alpha where raw would cut).
|
||||
- 🔶 **Finding: "blur is heavy"** — strength 0.5 ≈ radius 6 downscaled px ≈ **26 full-res px**, max
|
||||
≈ 50 px (1+11·s at ¼ res, 3 box passes). **Next work unit:** soften the radius curve/range
|
||||
(e.g. lower the strength→radius mapping, and/or a shorter maximum) — see
|
||||
`BackgroundEffectProcessor` `MinBlurRadius`/`MaxBlurRadius`/`BlurDownscale`.
|
||||
- ✅ `startup.log` perf line present on the real box ("background effects avg … ms/frame").
|
||||
|
||||
## Related (not duplicated here)
|
||||
|
||||
- [`task-48-distribution-msix.md`](task-48-distribution-msix.md) — the model ships in the
|
||||
|
||||
Reference in New Issue
Block a user