docs(memory): split capture (backdrop, webcam, effects, vertical tier) out of ai.md
Unit 2 of slimming the mandatory read. Verbatim move again - the four capture blocks leave ai.md unchanged in content, and one stub carries the law. - docs/ai/capture.md: 'Screen backdrop capture', 'Webcam capture', 'Webcam background effects' and 'Vertical (9:16) tier camera placement', byte- identical, grouped under backdrop / webcam / vertical headings, with a 'Related in ai.md' footer. - ai.md stub keeps: the locked-Background model and NormalizeBackgrounds; ActiveBackgroundImage is fed by static art only (2026-08-23 regression); window:/picker: keys are session-scoped and a live pin beats auto-detect; the settled WinRT interop paths (TryGetDataUnsafe, from-DXGIDevice); the VideoFrame seam, own-catch subtype ladder and no fallback camera; one app- level webcam identity with live-lock attainability; CameraManager as the one effects processing point with per-session mask smoothing; geometry-only vertical tier and the model-level rect lock. - docs/ai/index.md gains the capture.md row; Services/index.md (2) and TASKS/task-38-window-backdrop.md repointed. ai.md 1,975 -> 1,558 lines. No source touched; no push.
This commit is contained in:
+2
-2
@@ -21,7 +21,7 @@ External-facing logic: YouTube API, persistence. See
|
|||||||
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
||||||
| `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) |
|
| `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) |
|
||||||
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; blur radius is **quadratic** via `BlurRadius(strength)` — 1→6 small px (default 0.5 = 2, was a linear 1→12 ≈ 48 full px; the "blur heavy" fix 2026-10-09); `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log |
|
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; blur radius is **quadratic** via `BlurRadius(strength)` — 1→6 small px (default 0.5 = 2, was a linear 1→12 ≈ 48 full px; the "blur heavy" fix 2026-10-09); `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log |
|
||||||
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" |
|
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see [`docs/ai/capture.md`](../docs/ai/capture.md) → "Webcam background effects" |
|
||||||
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
||||||
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
||||||
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
||||||
@@ -30,7 +30,7 @@ External-facing logic: YouTube API, persistence. See
|
|||||||
| `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) |
|
| `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) |
|
||||||
| `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) |
|
| `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) |
|
||||||
| `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) |
|
| `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) |
|
||||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see [`docs/ai/capture.md`](../docs/ai/capture.md) → "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
||||||
| `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList<DisplayInfo> GetDisplays()` |
|
| `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList<DisplayInfo> GetDisplays()` |
|
||||||
| `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected |
|
| `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected |
|
||||||
| `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) |
|
| `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) |
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ change to the file). This is the fix for the "committed syntax error" landmine.
|
|||||||
## Model / naming facts
|
## Model / naming facts
|
||||||
|
|
||||||
- `Source.CaptureKey` is persisted (`window:<hwnd>`, `monitor:<n>`, `picker:<displayname>`);
|
- `Source.CaptureKey` is persisted (`window:<hwnd>`, `monitor:<n>`, `picker:<displayname>`);
|
||||||
`Source.IsBackground` marks the one Background per scene. See ai.md §Screen backdrop capture.
|
`Source.IsBackground` marks the one Background per scene. See [`docs/ai/capture.md`](../docs/ai/capture.md) → "Screen backdrop capture".
|
||||||
- `WindowInfo` (Models) — `Title`, `ProcessName`, `Hwnd` (long), `ProcessId`;
|
- `WindowInfo` (Models) — `Title`, `ProcessName`, `Hwnd` (long), `ProcessId`;
|
||||||
`Label => "{Title} — {ProcessName}"`; `CaptureKey => $"window:0x{Hwnd:X}"`.
|
`Label => "{Title} — {ProcessName}"`; `CaptureKey => $"window:0x{Hwnd:X}"`.
|
||||||
- `IWindowEnumerator` (Services) — `IReadOnlyList<WindowInfo> Enumerate()`; seam like
|
- `IWindowEnumerator` (Services) — `IReadOnlyList<WindowInfo> Enumerate()`; seam like
|
||||||
|
|||||||
@@ -396,459 +396,42 @@ The app is a **director's control surface**, not an editor. This replaces the OB
|
|||||||
|
|
||||||
Test seam note: `YouTubeAuthService.SetSession()` is public — tests inject a channel with future `TokenExpiry` so `EnsureToken()` passes offline; pair with an `HttpClient(new StubHandler())`.
|
Test seam note: `YouTubeAuthService.SetSession()` is public — tests inject a channel with future `TokenExpiry` so `EnsureToken()` passes offline; pair with an `HttpClient(new StubHandler())`.
|
||||||
|
|
||||||
### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25
|
### Capture — backdrop, webcam, effects & the vertical tier — moved
|
||||||
|
|
||||||
**The Background is a locked per-screen layer (TASK 25): exactly one per canonical
|
Screen/game/window backdrop capture, the webcam seam and sessions, the MediaPipe background effects
|
||||||
scene, always named "Background", pinned at index 0, never addable/removable/reorderable,
|
and the vertical tier's per-scene camera placement live in
|
||||||
absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time
|
**[`docs/ai/capture.md`](docs/ai/capture.md)** — read it before touching capture, cameras, the
|
||||||
desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back
|
Background row, or anything the 9:16 tier re-places.
|
||||||
to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.**
|
|
||||||
`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this
|
|
||||||
model on every load: keeps the correctly-flavored row (or converts a survivor in place — note
|
|
||||||
`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops
|
|
||||||
duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0;
|
|
||||||
non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()`
|
|
||||||
(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads
|
|
||||||
`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins).
|
|
||||||
This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`,
|
|
||||||
`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`.
|
|
||||||
|
|
||||||
- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey`
|
Invariants that hold even if you never open that file:
|
||||||
(persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The Live row is
|
|
||||||
a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`,
|
|
||||||
internal static — also used by the `StagedScene` setter so a scene switch can never expose a
|
|
||||||
missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
|
|
||||||
the element template sets `IsHitTestVisible=false` for
|
|
||||||
backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground`
|
|
||||||
(persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls —
|
|
||||||
"Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is
|
|
||||||
staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row
|
|
||||||
menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live
|
|
||||||
screens keep the properties panel pill ("Use default background") + Browse for custom art.
|
|
||||||
- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` /
|
|
||||||
`BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
|
|
||||||
- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above*
|
|
||||||
`BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored
|
|
||||||
Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows
|
|
||||||
`DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for
|
|
||||||
persistence/heal/minis — not this overlay.
|
|
||||||
- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)`
|
|
||||||
— fires on the UI thread whenever the foreground window changes. The callback runs the same
|
|
||||||
`GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
|
|
||||||
`GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own
|
|
||||||
process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
|
|
||||||
`ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
|
|
||||||
`IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
|
|
||||||
friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
|
|
||||||
submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch
|
|
||||||
`ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the
|
|
||||||
full-screen game's
|
|
||||||
monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey`
|
|
||||||
values are cleared, no capture session is started, and `Source.DisplaySource` falls back to
|
|
||||||
`_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of
|
|
||||||
`ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears
|
|
||||||
on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps
|
|
||||||
running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to
|
|
||||||
"Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual
|
|
||||||
fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground
|
|
||||||
changes.
|
|
||||||
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
|
|
||||||
`Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
|
|
||||||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
|
|
||||||
`WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
|
|
||||||
`IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
|
|
||||||
flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
|
|
||||||
1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear —
|
|
||||||
slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel
|
|
||||||
version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take).
|
|
||||||
Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks
|
|
||||||
(slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a
|
|
||||||
monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never
|
|
||||||
overwrite a newer frame), and are spaced by a 10ms floor
|
|
||||||
(`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for
|
|
||||||
the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry).
|
|
||||||
Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a
|
|
||||||
buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a
|
|
||||||
consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags)
|
|
||||||
can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms,
|
|
||||||
drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion
|
|
||||||
failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black
|
|
||||||
frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps
|
|
||||||
running, the pool just stops delivering.
|
|
||||||
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
|
|
||||||
`WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
|
|
||||||
events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
|
|
||||||
a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
|
|
||||||
`IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
|
|
||||||
auto-detection.
|
|
||||||
- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus
|
|
||||||
(PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes
|
|
||||||
(`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`:
|
|
||||||
visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs
|
|
||||||
`RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the
|
|
||||||
game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there).
|
|
||||||
**Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` —
|
|
||||||
`ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be
|
|
||||||
recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong
|
|
||||||
window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while
|
|
||||||
`IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an
|
|
||||||
explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session,
|
|
||||||
`OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback
|
|
||||||
(fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo`
|
|
||||||
(`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors
|
|
||||||
`CameraEnumeratorOverride`.
|
|
||||||
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
|
|
||||||
so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
|
|
||||||
feature array) → QI `IDXGIDevice` → the WinRT interop export
|
|
||||||
`CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
|
|
||||||
process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
|
|
||||||
no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
|
|
||||||
WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
|
|
||||||
`IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
|
|
||||||
reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
|
|
||||||
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
|
|
||||||
task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and
|
|
||||||
window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet).
|
|
||||||
- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses
|
|
||||||
focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture —
|
|
||||||
the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never
|
|
||||||
pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the
|
|
||||||
app is in the background (worse under a full-screen game on 24H2/26100), GPU readback
|
|
||||||
(`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending`
|
|
||||||
gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as
|
|
||||||
WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor)
|
|
||||||
but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted
|
|
||||||
yet (deferred by user decision).
|
|
||||||
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
|
|
||||||
to the encoder task.
|
|
||||||
- **Preview watermark:** the "Preview" placeholder hides while a background capture renders —
|
|
||||||
`ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with
|
|
||||||
live capture shows the feed instead of the "nothing here" label.
|
|
||||||
|
|
||||||
### Webcam capture (TASK 3 milestone 1)
|
- **The Background is a locked per-screen layer:** exactly one per canonical scene, named
|
||||||
|
"Background", pinned at index 0, never addable/removable/reorderable — Live is `DisplayCapture`,
|
||||||
- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
|
every other screen is static art. `NormalizeBackgrounds` heals any layout to that model on load.
|
||||||
BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes;
|
- **Never feed a capture row to the `ActiveBackgroundImage` overlay.** That layer is fed by
|
||||||
screen capture and background removal later feed the same seam.
|
static-art rows only; a flavor-blind `IsBackground` lookup there permanently covers the live
|
||||||
- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video,
|
desktop/game feed (the 2026-08-23 regression).
|
||||||
SharingMode = SharedReadOnly }`, frames pulled via
|
- **`window:`/`picker:` capture keys are session-scoped and stripped on load** (a recycled HWND
|
||||||
`CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format
|
would capture the wrong window), and an explicit window pin beats auto game/desktop detection
|
||||||
conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via
|
while that window is alive.
|
||||||
`WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies).
|
- **The WinRT interop paths are settled:** read frame bytes with `WindowsRuntimeMarshal.TryGetDataUnsafe`
|
||||||
- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate
|
(the `IMemoryBufferByteAccess` ComImport throws `Invalid cast` on every frame under CsWinRT) and
|
||||||
by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then
|
build the D3D device via `CreateDirect3D11DeviceFromDXGIDevice` (the
|
||||||
source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as
|
QI-for-`IDirect3DDxgiInterfaceAccess` trick returns E_NOINTERFACE on current Windows).
|
||||||
Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes
|
- **`VideoFrame` + the capture interfaces are the seam** — nothing above the WinRT layer leaks WinRT
|
||||||
NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate
|
types; capture is CPU-first (`MemoryPreference.Cpu`, `SharedReadOnly`), the reader-subtype ladder
|
||||||
(our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under
|
tries each candidate under its OWN catch (one rejected format must never abort acquisition), and
|
||||||
msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition.
|
there is deliberately **no fallback camera** — the user's pick is never overridden.
|
||||||
Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype
|
- **One webcam identity app-wide, placements per scene.** `IsWebcamAttainable` needs a live lock, not
|
||||||
with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures
|
just a saved id; the app keeps its own base ref, so removing every placement leaves the session up.
|
||||||
degrade to the next candidate; rejections are logged + collected into the final error; only the outer
|
- **`CameraManager` is the ONE background-effects processing point** — segment once per raw frame,
|
||||||
catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not
|
render per-scene variants keyed by effect+strength, mask smoothed per camera session (0.75 new /
|
||||||
Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps.
|
0.25 previous). The raw path stays untouched so a slow model never stalls the webcam, and the
|
||||||
- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview`
|
bundled model's provenance is fixed (do not swap in PINTO #109).
|
||||||
(preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the
|
- **The vertical tier is geometry only:** `BeginVerticalLayout`/`EndVerticalLayout` park and swap the
|
||||||
failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture
|
live placement, so the compositor and `PreviewPane.xaml` never learn about tiers, and the stored
|
||||||
coexist with other apps that share the camera.
|
X/Y/W/H/ClipShape columns stay the **16:9** layout. The Chat camera is rect-only through the MODEL
|
||||||
- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync`
|
(`VerticalShapeLocked`), not merely through a hidden button.
|
||||||
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
|
|
||||||
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
|
|
||||||
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
|
|
||||||
Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
|
|
||||||
inside the physical camera's own driver, so it never appears as a selectable device. Only tools
|
|
||||||
that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
|
|
||||||
the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
|
|
||||||
bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
|
|
||||||
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
|
|
||||||
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
|
|
||||||
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
|
|
||||||
- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam`
|
|
||||||
row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig`
|
|
||||||
(position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at
|
|
||||||
most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager`
|
|
||||||
refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps
|
|
||||||
it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the
|
|
||||||
app default), usable in every scene that can host it.** "Add Webcam" places the existing default
|
|
||||||
DIRECTLY (no picker); the Windows camera picker runs only for the initial selection
|
|
||||||
(`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via
|
|
||||||
`SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide**
|
|
||||||
(TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable`
|
|
||||||
— a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene
|
|
||||||
holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a
|
|
||||||
saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)`
|
|
||||||
— an identity whose device was unplugged, or whose session can't start, leaves the row greyed with
|
|
||||||
`WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The
|
|
||||||
max-1 reason gets the gentle "a second camera means you've graduated to OBS" line.
|
|
||||||
Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never
|
|
||||||
nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock
|
|
||||||
(below).
|
|
||||||
- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and
|
|
||||||
locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app
|
|
||||||
instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget
|
|
||||||
right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the
|
|
||||||
OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt
|
|
||||||
`CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert**
|
|
||||||
(`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s
|
|
||||||
(`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** →
|
|
||||||
deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40
|
|
||||||
Unit A (App Settings round, plan saved 2026-09-22)**.
|
|
||||||
The single-camera branch **always acquires** (it no longer skips when configs already hold the
|
|
||||||
session): that acquire is the app's own BASE ref, so removing every scene's placement leaves
|
|
||||||
`RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered,
|
|
||||||
i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps
|
|
||||||
the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting —
|
|
||||||
a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves
|
|
||||||
the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the
|
|
||||||
camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then
|
|
||||||
offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never
|
|
||||||
overwritten. Test seams mirror `LayoutPathOverride`:
|
|
||||||
`CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests`
|
|
||||||
drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail
|
|
||||||
→ alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure
|
|
||||||
below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device
|
|
||||||
handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`).
|
|
||||||
The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27):
|
|
||||||
`CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on
|
|
||||||
staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created
|
|
||||||
before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) —
|
|
||||||
LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
|
|
||||||
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
|
|
||||||
snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
|
|
||||||
restores them when switching back — otherwise the Round resize lock (square) would leave a square
|
|
||||||
behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a
|
|
||||||
reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time
|
|
||||||
`HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9
|
|
||||||
(keeps height; Round and explicit rect dims are untouched).
|
|
||||||
- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load)
|
|
||||||
release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and
|
|
||||||
stops the source regardless of how many scenes held it (the per-config count isn't known once the
|
|
||||||
scenes are replaced) — then `AcquireAsync` the new device once per config.
|
|
||||||
- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread
|
|
||||||
at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via
|
|
||||||
`PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
|
|
||||||
(at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
|
|
||||||
device never drowns the render thread.
|
|
||||||
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
|
|
||||||
frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
|
|
||||||
exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
|
|
||||||
webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
|
|
||||||
current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
|
|
||||||
`AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
|
|
||||||
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
|
|
||||||
(`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
|
|
||||||
0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
|
|
||||||
Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click
|
|
||||||
context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist,
|
|
||||||
rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
|
|
||||||
this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
|
|
||||||
preview canvas has its own Show Webcam entry.
|
|
||||||
- The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
|
|
||||||
(`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
|
|
||||||
renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
|
|
||||||
the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
|
|
||||||
geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
|
|
||||||
frame on the render thread, which is what made the live webcam crawl while round. The Round border is
|
|
||||||
a centered `Ellipse` at `Width/Height = RoundBorderSize`.
|
|
||||||
- Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
|
|
||||||
- **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
|
|
||||||
max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
|
|
||||||
**by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
|
|
||||||
may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
|
|
||||||
(`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
|
|
||||||
`RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
|
|
||||||
- **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
|
|
||||||
the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected
|
|
||||||
the element — making the corner handle ungrabbable. The element Grid carries
|
|
||||||
`Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for
|
|
||||||
right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is
|
|
||||||
`IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work:
|
|
||||||
(1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu`
|
|
||||||
isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the
|
|
||||||
IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem`
|
|
||||||
template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness
|
|
||||||
sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow
|
|
||||||
driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never
|
|
||||||
opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the
|
|
||||||
Click handlers were fine.
|
|
||||||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
|
||||||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
|
||||||
to the encoder task.
|
|
||||||
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
|
|
||||||
2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
|
|
||||||
and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
|
|
||||||
run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
|
|
||||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
|
||||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
|
||||||
|
|
||||||
### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)
|
|
||||||
|
|
||||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
|
||||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
|
||||||
float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
|
|
||||||
`tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
|
|
||||||
`Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
|
|
||||||
translation was smaller than any workaround). **Verified against Google's own runtime:**
|
|
||||||
`verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
|
|
||||||
.tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
|
|
||||||
⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance).
|
|
||||||
obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09),
|
|
||||||
but none of its code is used; if any is ever borrowed, it needs its copyright notice kept.
|
|
||||||
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
|
|
||||||
whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
|
|
||||||
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
|
|
||||||
inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
|
|
||||||
`Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
|
|
||||||
`onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
|
|
||||||
publish (TASK 48) must pin the RID so only win-x64 ships.
|
|
||||||
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
|
|
||||||
SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
|
|
||||||
resource; **never throws** — a load/run failure logs once and returns null forever after);
|
|
||||||
`BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
|
|
||||||
passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
|
|
||||||
`IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength =
|
|
||||||
edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4×
|
|
||||||
by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person
|
|
||||||
pixels (mask = 1) are copied
|
|
||||||
untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px,
|
|
||||||
default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was
|
|
||||||
far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author
|
|
||||||
stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and
|
|
||||||
rendered per effect (`Render`), so several variants share one inference. No mask / segmenter
|
|
||||||
exception → passthrough + one log line, never a dropped layer.
|
|
||||||
- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength`
|
|
||||||
(0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig`
|
|
||||||
(`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump —
|
|
||||||
the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds.
|
|
||||||
- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the
|
|
||||||
camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect,
|
|
||||||
strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every
|
|
||||||
config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
|
|
||||||
config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
|
|
||||||
and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
|
|
||||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
|
||||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
|
||||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
|
||||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
|
||||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
|
||||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
|
||||||
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
|
|
||||||
every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
|
|
||||||
`MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
|
|
||||||
(`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
|
|
||||||
the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
|
|
||||||
pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
|
|
||||||
size change pass through un-blended; the returned mask never aliases the caller's array and two
|
|
||||||
buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
|
|
||||||
come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
|
|
||||||
`temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
|
|
||||||
found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
|
|
||||||
smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6
|
|
||||||
small px (see the `Blur` bullet above).
|
|
||||||
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
|
|
||||||
(Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
|
|
||||||
bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
|
|
||||||
per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real
|
|
||||||
average once per session after 120 processed frames (`startup.log`: "background effects avg …").
|
|
||||||
`MainViewModel.PersonSegmenterOverride` is the hermetic-test seam.
|
|
||||||
- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel
|
|
||||||
(`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and
|
|
||||||
Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off,
|
|
||||||
turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an
|
|
||||||
effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam:
|
|
||||||
a new strength variant inherits the dropped same-effect variant's last frame + bitmap
|
|
||||||
(`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders.
|
|
||||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
|
||||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
|
||||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
|
||||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
|
|
||||||
— the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
|
|
||||||
masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
|
|
||||||
have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
|
|
||||||
`Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping
|
|
||||||
(`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears
|
|
||||||
light ~9px into the dark half vs ~26px at full strength.
|
|
||||||
|
|
||||||
### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21)
|
|
||||||
|
|
||||||
**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080
|
|
||||||
master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default
|
|
||||||
webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the
|
|
||||||
creator is simply not in the picture.
|
|
||||||
|
|
||||||
**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical
|
|
||||||
placement — position, size and shape — and switching back to 16:9 restores the desktop layout
|
|
||||||
exactly:
|
|
||||||
|
|
||||||
| mode | shape | size | position |
|
|
||||||
|---|---|---|---|
|
|
||||||
| 16:9 | as today | as today | as today |
|
|
||||||
| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right |
|
|
||||||
| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area |
|
|
||||||
| Starting / BRB / Ending | unchanged | unchanged | unchanged |
|
|
||||||
|
|
||||||
**Design — geometry only, no render-path change.** The preview and the compositor both read the
|
|
||||||
element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live
|
|
||||||
geometry** instead of teaching the render path about tiers:
|
|
||||||
|
|
||||||
- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape`
|
|
||||||
+ `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical
|
|
||||||
placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient).
|
|
||||||
- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in;
|
|
||||||
`EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the
|
|
||||||
parked 16:9 layout. The live geometry IS the render input → **the compositor and
|
|
||||||
`PreviewPane.xaml` are untouched**.
|
|
||||||
- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is
|
|
||||||
live, the parked values are written there and the live ones into the `Vertical*` columns).
|
|
||||||
- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write
|
|
||||||
(drag, resize handle, properties fields) into the band — one choke point instead of three
|
|
||||||
writers re-implementing the rule. Outside the band = invisible in the vertical stream.
|
|
||||||
|
|
||||||
**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is
|
|
||||||
656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns
|
|
||||||
(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master
|
|
||||||
px from the band's outer edge (the app's own placement margin) and never outside the band. The
|
|
||||||
default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right**
|
|
||||||
spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the
|
|
||||||
first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the
|
|
||||||
other scenes keep their 16:9 placement until the creator drags them into the band.
|
|
||||||
|
|
||||||
**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**:
|
|
||||||
the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button
|
|
||||||
disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a
|
|
||||||
locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The
|
|
||||||
lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the
|
|
||||||
16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile
|
|
||||||
(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two
|
|
||||||
(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top
|
|
||||||
or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is
|
|
||||||
picked (like Live) — otherwise the vertical chat stream has no camera at all.
|
|
||||||
|
|
||||||
**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section
|
|
||||||
(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's
|
|
||||||
geometry setters clamp it into the band, so a typed value can never push the camera out of the
|
|
||||||
cropped window.
|
|
||||||
|
|
||||||
**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the
|
|
||||||
maintainers' answer is "create a new empty scene and add the scene you want to send as source"
|
|
||||||
(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no
|
|
||||||
built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and
|
|
||||||
plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored
|
|
||||||
placement, resolved on the tier change, is the step past that.
|
|
||||||
|
|
||||||
**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier
|
|
||||||
is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it
|
|
||||||
lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both
|
|
||||||
placements survive a save + reload. (The store writes the webcam rows only when the app has a
|
|
||||||
webcam identity — the config rows hang off it — so the test supplies one.)
|
|
||||||
`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked
|
|
||||||
→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and
|
|
||||||
cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back
|
|
||||||
to 16:9 the desktop layout is restored exactly and the circle option is offered again.
|
|
||||||
|
|
||||||
**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam`
|
|
||||||
broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a
|
|
||||||
new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice.
|
|
||||||
|
|
||||||
### Rendering budget, measurement & the compositor — moved
|
### Rendering budget, measurement & the compositor — moved
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,484 @@
|
|||||||
|
# Capture — backdrop, webcam, background effects, and the vertical tier
|
||||||
|
|
||||||
|
> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches;
|
||||||
|
> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md).
|
||||||
|
|
||||||
|
Every frame source in the app lands here: the locked Background layer (screen/game/window capture),
|
||||||
|
the webcam seam and its sessions, the MediaPipe background effects, and the per-scene camera placement
|
||||||
|
the vertical 9:16 tier uses. Content is the architecture plus the measured history that produced it;
|
||||||
|
superseded designs are marked inline.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The locked Background layer — screen, game, window capture
|
||||||
|
|
||||||
|
### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25
|
||||||
|
|
||||||
|
**The Background is a locked per-screen layer (TASK 25): exactly one per canonical
|
||||||
|
scene, always named "Background", pinned at index 0, never addable/removable/reorderable,
|
||||||
|
absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time
|
||||||
|
desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back
|
||||||
|
to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.**
|
||||||
|
`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this
|
||||||
|
model on every load: keeps the correctly-flavored row (or converts a survivor in place — note
|
||||||
|
`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops
|
||||||
|
duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0;
|
||||||
|
non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()`
|
||||||
|
(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads
|
||||||
|
`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins).
|
||||||
|
This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`,
|
||||||
|
`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`.
|
||||||
|
|
||||||
|
- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey`
|
||||||
|
(persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The Live row is
|
||||||
|
a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`,
|
||||||
|
internal static — also used by the `StagedScene` setter so a scene switch can never expose a
|
||||||
|
missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
|
||||||
|
the element template sets `IsHitTestVisible=false` for
|
||||||
|
backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground`
|
||||||
|
(persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls —
|
||||||
|
"Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is
|
||||||
|
staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row
|
||||||
|
menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live
|
||||||
|
screens keep the properties panel pill ("Use default background") + Browse for custom art.
|
||||||
|
- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` /
|
||||||
|
`BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
|
||||||
|
- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above*
|
||||||
|
`BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored
|
||||||
|
Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows
|
||||||
|
`DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for
|
||||||
|
persistence/heal/minis — not this overlay.
|
||||||
|
- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)`
|
||||||
|
— fires on the UI thread whenever the foreground window changes. The callback runs the same
|
||||||
|
`GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
|
||||||
|
`GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own
|
||||||
|
process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
|
||||||
|
`ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
|
||||||
|
`IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
|
||||||
|
friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
|
||||||
|
submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch
|
||||||
|
`ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the
|
||||||
|
full-screen game's
|
||||||
|
monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey`
|
||||||
|
values are cleared, no capture session is started, and `Source.DisplaySource` falls back to
|
||||||
|
`_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of
|
||||||
|
`ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears
|
||||||
|
on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps
|
||||||
|
running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to
|
||||||
|
"Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual
|
||||||
|
fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground
|
||||||
|
changes.
|
||||||
|
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
|
||||||
|
`Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
|
||||||
|
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
|
||||||
|
`WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
|
||||||
|
`IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
|
||||||
|
flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
|
||||||
|
1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear —
|
||||||
|
slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel
|
||||||
|
version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take).
|
||||||
|
Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks
|
||||||
|
(slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a
|
||||||
|
monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never
|
||||||
|
overwrite a newer frame), and are spaced by a 10ms floor
|
||||||
|
(`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for
|
||||||
|
the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry).
|
||||||
|
Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a
|
||||||
|
buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a
|
||||||
|
consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags)
|
||||||
|
can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms,
|
||||||
|
drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion
|
||||||
|
failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black
|
||||||
|
frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps
|
||||||
|
running, the pool just stops delivering.
|
||||||
|
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
|
||||||
|
`WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
|
||||||
|
events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
|
||||||
|
a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
|
||||||
|
`IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
|
||||||
|
auto-detection.
|
||||||
|
- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus
|
||||||
|
(PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes
|
||||||
|
(`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`:
|
||||||
|
visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs
|
||||||
|
`RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the
|
||||||
|
game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there).
|
||||||
|
**Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` —
|
||||||
|
`ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be
|
||||||
|
recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong
|
||||||
|
window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while
|
||||||
|
`IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an
|
||||||
|
explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session,
|
||||||
|
`OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback
|
||||||
|
(fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo`
|
||||||
|
(`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors
|
||||||
|
`CameraEnumeratorOverride`.
|
||||||
|
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
|
||||||
|
so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
|
||||||
|
feature array) → QI `IDXGIDevice` → the WinRT interop export
|
||||||
|
`CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
|
||||||
|
process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
|
||||||
|
no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
|
||||||
|
WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
|
||||||
|
`IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
|
||||||
|
reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
|
||||||
|
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
|
||||||
|
task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and
|
||||||
|
window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet).
|
||||||
|
- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses
|
||||||
|
focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture —
|
||||||
|
the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never
|
||||||
|
pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the
|
||||||
|
app is in the background (worse under a full-screen game on 24H2/26100), GPU readback
|
||||||
|
(`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending`
|
||||||
|
gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as
|
||||||
|
WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor)
|
||||||
|
but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted
|
||||||
|
yet (deferred by user decision).
|
||||||
|
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
|
||||||
|
to the encoder task.
|
||||||
|
- **Preview watermark:** the "Preview" placeholder hides while a background capture renders —
|
||||||
|
`ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with
|
||||||
|
live capture shows the feed instead of the "nothing here" label.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Webcam — capture seam, sessions, and background effects
|
||||||
|
|
||||||
|
### Webcam capture (TASK 3 milestone 1)
|
||||||
|
|
||||||
|
- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
|
||||||
|
BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes;
|
||||||
|
screen capture and background removal later feed the same seam.
|
||||||
|
- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video,
|
||||||
|
SharingMode = SharedReadOnly }`, frames pulled via
|
||||||
|
`CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format
|
||||||
|
conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via
|
||||||
|
`WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies).
|
||||||
|
- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate
|
||||||
|
by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then
|
||||||
|
source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as
|
||||||
|
Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes
|
||||||
|
NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate
|
||||||
|
(our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under
|
||||||
|
msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition.
|
||||||
|
Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype
|
||||||
|
with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures
|
||||||
|
degrade to the next candidate; rejections are logged + collected into the final error; only the outer
|
||||||
|
catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not
|
||||||
|
Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps.
|
||||||
|
- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview`
|
||||||
|
(preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the
|
||||||
|
failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture
|
||||||
|
coexist with other apps that share the camera.
|
||||||
|
- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync`
|
||||||
|
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
|
||||||
|
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
|
||||||
|
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
|
||||||
|
Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
|
||||||
|
inside the physical camera's own driver, so it never appears as a selectable device. Only tools
|
||||||
|
that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
|
||||||
|
the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
|
||||||
|
bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
|
||||||
|
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
|
||||||
|
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
|
||||||
|
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
|
||||||
|
- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam`
|
||||||
|
row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig`
|
||||||
|
(position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at
|
||||||
|
most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager`
|
||||||
|
refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps
|
||||||
|
it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the
|
||||||
|
app default), usable in every scene that can host it.** "Add Webcam" places the existing default
|
||||||
|
DIRECTLY (no picker); the Windows camera picker runs only for the initial selection
|
||||||
|
(`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via
|
||||||
|
`SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide**
|
||||||
|
(TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable`
|
||||||
|
— a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene
|
||||||
|
holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a
|
||||||
|
saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)`
|
||||||
|
— an identity whose device was unplugged, or whose session can't start, leaves the row greyed with
|
||||||
|
`WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The
|
||||||
|
max-1 reason gets the gentle "a second camera means you've graduated to OBS" line.
|
||||||
|
Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never
|
||||||
|
nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock
|
||||||
|
(below).
|
||||||
|
- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and
|
||||||
|
locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app
|
||||||
|
instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget
|
||||||
|
right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the
|
||||||
|
OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt
|
||||||
|
`CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert**
|
||||||
|
(`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s
|
||||||
|
(`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** →
|
||||||
|
deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40
|
||||||
|
Unit A (App Settings round, plan saved 2026-09-22)**.
|
||||||
|
The single-camera branch **always acquires** (it no longer skips when configs already hold the
|
||||||
|
session): that acquire is the app's own BASE ref, so removing every scene's placement leaves
|
||||||
|
`RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered,
|
||||||
|
i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps
|
||||||
|
the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting —
|
||||||
|
a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves
|
||||||
|
the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the
|
||||||
|
camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then
|
||||||
|
offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never
|
||||||
|
overwritten. Test seams mirror `LayoutPathOverride`:
|
||||||
|
`CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests`
|
||||||
|
drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail
|
||||||
|
→ alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure
|
||||||
|
below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device
|
||||||
|
handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`).
|
||||||
|
The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27):
|
||||||
|
`CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on
|
||||||
|
staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created
|
||||||
|
before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) —
|
||||||
|
LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
|
||||||
|
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
|
||||||
|
snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
|
||||||
|
restores them when switching back — otherwise the Round resize lock (square) would leave a square
|
||||||
|
behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a
|
||||||
|
reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time
|
||||||
|
`HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9
|
||||||
|
(keeps height; Round and explicit rect dims are untouched).
|
||||||
|
- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load)
|
||||||
|
release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and
|
||||||
|
stops the source regardless of how many scenes held it (the per-config count isn't known once the
|
||||||
|
scenes are replaced) — then `AcquireAsync` the new device once per config.
|
||||||
|
- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread
|
||||||
|
at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via
|
||||||
|
`PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
|
||||||
|
(at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
|
||||||
|
device never drowns the render thread.
|
||||||
|
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
|
||||||
|
frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
|
||||||
|
exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
|
||||||
|
webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
|
||||||
|
current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
|
||||||
|
`AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
|
||||||
|
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
|
||||||
|
(`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
|
||||||
|
0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
|
||||||
|
Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click
|
||||||
|
context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist,
|
||||||
|
rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
|
||||||
|
this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
|
||||||
|
preview canvas has its own Show Webcam entry.
|
||||||
|
- The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
|
||||||
|
(`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
|
||||||
|
renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
|
||||||
|
the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
|
||||||
|
geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
|
||||||
|
frame on the render thread, which is what made the live webcam crawl while round. The Round border is
|
||||||
|
a centered `Ellipse` at `Width/Height = RoundBorderSize`.
|
||||||
|
- Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
|
||||||
|
- **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
|
||||||
|
max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
|
||||||
|
**by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
|
||||||
|
may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
|
||||||
|
(`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
|
||||||
|
`RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
|
||||||
|
- **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
|
||||||
|
the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected
|
||||||
|
the element — making the corner handle ungrabbable. The element Grid carries
|
||||||
|
`Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for
|
||||||
|
right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is
|
||||||
|
`IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work:
|
||||||
|
(1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu`
|
||||||
|
isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the
|
||||||
|
IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem`
|
||||||
|
template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness
|
||||||
|
sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow
|
||||||
|
driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never
|
||||||
|
opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the
|
||||||
|
Click handlers were fine.
|
||||||
|
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||||||
|
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||||||
|
to the encoder task.
|
||||||
|
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
|
||||||
|
2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
|
||||||
|
and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
|
||||||
|
run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
|
||||||
|
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||||
|
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||||
|
|
||||||
|
### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)
|
||||||
|
|
||||||
|
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||||
|
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||||
|
float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
|
||||||
|
`tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
|
||||||
|
`Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
|
||||||
|
translation was smaller than any workaround). **Verified against Google's own runtime:**
|
||||||
|
`verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
|
||||||
|
.tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
|
||||||
|
⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance).
|
||||||
|
obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09),
|
||||||
|
but none of its code is used; if any is ever borrowed, it needs its copyright notice kept.
|
||||||
|
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
|
||||||
|
whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
|
||||||
|
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
|
||||||
|
inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
|
||||||
|
`Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
|
||||||
|
`onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
|
||||||
|
publish (TASK 48) must pin the RID so only win-x64 ships.
|
||||||
|
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
|
||||||
|
SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
|
||||||
|
resource; **never throws** — a load/run failure logs once and returns null forever after);
|
||||||
|
`BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
|
||||||
|
passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
|
||||||
|
`IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength =
|
||||||
|
edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4×
|
||||||
|
by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person
|
||||||
|
pixels (mask = 1) are copied
|
||||||
|
untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px,
|
||||||
|
default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was
|
||||||
|
far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author
|
||||||
|
stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and
|
||||||
|
rendered per effect (`Render`), so several variants share one inference. No mask / segmenter
|
||||||
|
exception → passthrough + one log line, never a dropped layer.
|
||||||
|
- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength`
|
||||||
|
(0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig`
|
||||||
|
(`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump —
|
||||||
|
the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds.
|
||||||
|
- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the
|
||||||
|
camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect,
|
||||||
|
strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every
|
||||||
|
config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
|
||||||
|
config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
|
||||||
|
and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
|
||||||
|
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||||
|
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||||
|
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||||
|
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||||
|
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||||
|
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||||
|
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
|
||||||
|
every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
|
||||||
|
`MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
|
||||||
|
(`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
|
||||||
|
the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
|
||||||
|
pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
|
||||||
|
size change pass through un-blended; the returned mask never aliases the caller's array and two
|
||||||
|
buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
|
||||||
|
come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
|
||||||
|
`temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
|
||||||
|
found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
|
||||||
|
smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6
|
||||||
|
small px (see the `Blur` bullet above).
|
||||||
|
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
|
||||||
|
(Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
|
||||||
|
bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
|
||||||
|
per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real
|
||||||
|
average once per session after 120 processed frames (`startup.log`: "background effects avg …").
|
||||||
|
`MainViewModel.PersonSegmenterOverride` is the hermetic-test seam.
|
||||||
|
- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel
|
||||||
|
(`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and
|
||||||
|
Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off,
|
||||||
|
turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an
|
||||||
|
effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam:
|
||||||
|
a new strength variant inherits the dropped same-effect variant's last frame + bitmap
|
||||||
|
(`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders.
|
||||||
|
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||||
|
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||||
|
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||||
|
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
|
||||||
|
— the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
|
||||||
|
masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
|
||||||
|
have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
|
||||||
|
`Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping
|
||||||
|
(`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears
|
||||||
|
light ~9px into the dark half vs ~26px at full strength.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The vertical (9:16) tier — per-scene camera placement
|
||||||
|
|
||||||
|
### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21)
|
||||||
|
|
||||||
|
**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080
|
||||||
|
master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default
|
||||||
|
webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the
|
||||||
|
creator is simply not in the picture.
|
||||||
|
|
||||||
|
**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical
|
||||||
|
placement — position, size and shape — and switching back to 16:9 restores the desktop layout
|
||||||
|
exactly:
|
||||||
|
|
||||||
|
| mode | shape | size | position |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 16:9 | as today | as today | as today |
|
||||||
|
| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right |
|
||||||
|
| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area |
|
||||||
|
| Starting / BRB / Ending | unchanged | unchanged | unchanged |
|
||||||
|
|
||||||
|
**Design — geometry only, no render-path change.** The preview and the compositor both read the
|
||||||
|
element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live
|
||||||
|
geometry** instead of teaching the render path about tiers:
|
||||||
|
|
||||||
|
- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape`
|
||||||
|
+ `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical
|
||||||
|
placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient).
|
||||||
|
- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in;
|
||||||
|
`EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the
|
||||||
|
parked 16:9 layout. The live geometry IS the render input → **the compositor and
|
||||||
|
`PreviewPane.xaml` are untouched**.
|
||||||
|
- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is
|
||||||
|
live, the parked values are written there and the live ones into the `Vertical*` columns).
|
||||||
|
- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write
|
||||||
|
(drag, resize handle, properties fields) into the band — one choke point instead of three
|
||||||
|
writers re-implementing the rule. Outside the band = invisible in the vertical stream.
|
||||||
|
|
||||||
|
**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is
|
||||||
|
656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns
|
||||||
|
(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master
|
||||||
|
px from the band's outer edge (the app's own placement margin) and never outside the band. The
|
||||||
|
default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right**
|
||||||
|
spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the
|
||||||
|
first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the
|
||||||
|
other scenes keep their 16:9 placement until the creator drags them into the band.
|
||||||
|
|
||||||
|
**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**:
|
||||||
|
the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button
|
||||||
|
disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a
|
||||||
|
locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The
|
||||||
|
lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the
|
||||||
|
16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile
|
||||||
|
(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two
|
||||||
|
(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top
|
||||||
|
or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is
|
||||||
|
picked (like Live) — otherwise the vertical chat stream has no camera at all.
|
||||||
|
|
||||||
|
**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section
|
||||||
|
(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's
|
||||||
|
geometry setters clamp it into the band, so a typed value can never push the camera out of the
|
||||||
|
cropped window.
|
||||||
|
|
||||||
|
**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the
|
||||||
|
maintainers' answer is "create a new empty scene and add the scene you want to send as source"
|
||||||
|
(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no
|
||||||
|
built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and
|
||||||
|
plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored
|
||||||
|
placement, resolved on the tier change, is the step past that.
|
||||||
|
|
||||||
|
**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier
|
||||||
|
is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it
|
||||||
|
lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both
|
||||||
|
placements survive a save + reload. (The store writes the webcam rows only when the app has a
|
||||||
|
webcam identity — the config rows hang off it — so the test supplies one.)
|
||||||
|
`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked
|
||||||
|
→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and
|
||||||
|
cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back
|
||||||
|
to 16:9 the desktop layout is restored exactly and the circle option is offered again.
|
||||||
|
|
||||||
|
**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam`
|
||||||
|
broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a
|
||||||
|
new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related in [`ai.md`](../../ai.md)
|
||||||
|
|
||||||
|
- **Resolution tiers** — `OutputRect*` and the tier list (the vertical tier's 607×1080 band).
|
||||||
|
- **Key patterns** — MVVM/binding conventions the capture UI follows.
|
||||||
|
- **Tests** — hermetic-real-window rules that camera tests must obey.
|
||||||
|
- **Run** — launch/take procedure and the Release-only rule for measurements.
|
||||||
@@ -10,6 +10,7 @@ Read the file whose row matches the task. There is no required reading order.
|
|||||||
| File | Covers | Read when |
|
| File | Covers | Read when |
|
||||||
|------|--------|-----------|
|
|------|--------|-----------|
|
||||||
| [`frame-pipeline.md`](frame-pipeline.md) | Render budget & measurement (`FrameCadence`, per-element composite timing, scene dump), `SceneCompositor` design + `StripPool`, and `FramePump` from scene snapshot to encoder (pacing, pool/paste caches, backpressure, teardown, health stats) | Touching the compositor, the render loop, per-frame cost, `Services/Encoder/*`, or debugging dropped/juddering frames |
|
| [`frame-pipeline.md`](frame-pipeline.md) | Render budget & measurement (`FrameCadence`, per-element composite timing, scene dump), `SceneCompositor` design + `StripPool`, and `FramePump` from scene snapshot to encoder (pacing, pool/paste caches, backpressure, teardown, health stats) | Touching the compositor, the render loop, per-frame cost, `Services/Encoder/*`, or debugging dropped/juddering frames |
|
||||||
|
| [`capture.md`](capture.md) | The locked Background layer (screen/game/window capture, WinRT GraphicsCapture interop), the webcam seam + `CameraManager` sessions, MediaPipe background effects, and the vertical 9:16 tier's per-scene camera placement | Touching any frame source, the Background row, camera acquisition/conflicts, background effects, or vertical-tier camera geometry |
|
||||||
|
|
||||||
## Split-out yet to come
|
## Split-out yet to come
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user