TASK 51 jitter fix: temporal mask smoothing. MaskSmoother blends each raw segmentation 0.75xnew + 0.25xprevious at the one CameraManager worker (state per CameraSession, not in the shared segmenter); null resets history, first frame passes through, input array never aliased, ping-pong buffers (no LOH alloc for a 147KB 256x144 mask). Creator's real-camera check: Remove was 'jittery' (per-frame independent segmentation). Weights from the OBS/Jitsi consensus. One integration test asserts the second frame keeps partial alpha (x=6 -> 58) where the raw mask cuts to 0. 416 pass / 2 known env flakes. Refs: https://deepwiki.com/royshil/obs-backgroundremoval/2-core-features https://github.com/jitsi/jitsi-meet/issues/17080 (Unit: bg-removal jitter fix)
This commit is contained in:
@@ -609,7 +609,7 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||
|
||||
### Webcam background effects (TASK 51 — slice 1 engine shipped 2026-10-09)
|
||||
### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)
|
||||
|
||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||
@@ -651,12 +651,24 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
|
||||
config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
|
||||
and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
|
||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
|
||||
every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
|
||||
`MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
|
||||
(`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
|
||||
the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
|
||||
pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
|
||||
size change pass through un-blended; the returned mask never aliases the caller's array and two
|
||||
buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
|
||||
come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
|
||||
`temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
|
||||
found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
|
||||
smearing edge ghost. Blur remains per-frame ("blur is heavy" = next work unit, on the radius curve).
|
||||
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
|
||||
(Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
|
||||
bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
|
||||
@@ -673,7 +685,10 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`).
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
|
||||
— the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
|
||||
masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
|
||||
have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
|
||||
|
||||
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user