From 096bf6debeaf8c5e981d9cf41684d6d4fa614c20 Mon Sep 17 00:00:00 2001 From: gramps Date: Mon, 26 Oct 2026 10:26:53 -0700 Subject: [PATCH] docs(memory): split capture (backdrop, webcam, effects, vertical tier) out of ai.md Unit 2 of slimming the mandatory read. Verbatim move again - the four capture blocks leave ai.md unchanged in content, and one stub carries the law. - docs/ai/capture.md: 'Screen backdrop capture', 'Webcam capture', 'Webcam background effects' and 'Vertical (9:16) tier camera placement', byte- identical, grouped under backdrop / webcam / vertical headings, with a 'Related in ai.md' footer. - ai.md stub keeps: the locked-Background model and NormalizeBackgrounds; ActiveBackgroundImage is fed by static art only (2026-08-23 regression); window:/picker: keys are session-scoped and a live pin beats auto-detect; the settled WinRT interop paths (TryGetDataUnsafe, from-DXGIDevice); the VideoFrame seam, own-catch subtype ladder and no fallback camera; one app- level webcam identity with live-lock attainability; CameraManager as the one effects processing point with per-session mask smoothing; geometry-only vertical tier and the model-level rect lock. - docs/ai/index.md gains the capture.md row; Services/index.md (2) and TASKS/task-38-window-backdrop.md repointed. ai.md 1,975 -> 1,558 lines. No source touched; no push. --- Services/index.md | 4 +- TASKS/task-38-window-backdrop.md | 2 +- ai.md | 483 +++--------------------------- docs/ai/capture.md | 484 +++++++++++++++++++++++++++++++ docs/ai/index.md | 1 + 5 files changed, 521 insertions(+), 453 deletions(-) create mode 100644 docs/ai/capture.md diff --git a/Services/index.md b/Services/index.md index bd887bb..7d4f558 100644 --- a/Services/index.md +++ b/Services/index.md @@ -21,7 +21,7 @@ External-facing logic: YouTube API, persistence. See | `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) | | `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) | | `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; blur radius is **quadratic** via `BlurRadius(strength)` — 1→6 small px (default 0.5 = 2, was a linear 1→12 ≈ 48 full px; the "blur heavy" fix 2026-10-09); `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log | -| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" | +| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see [`docs/ai/capture.md`](../docs/ai/capture.md) → "Webcam background effects" | | `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device | | `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) | | `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) | @@ -30,7 +30,7 @@ External-facing logic: YouTube API, persistence. See | `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) | | `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) | | `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) | -| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched | +| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see [`docs/ai/capture.md`](../docs/ai/capture.md) → "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched | | `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList GetDisplays()` | | `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected | | `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) | diff --git a/TASKS/task-38-window-backdrop.md b/TASKS/task-38-window-backdrop.md index 3921525..bc94fad 100644 --- a/TASKS/task-38-window-backdrop.md +++ b/TASKS/task-38-window-backdrop.md @@ -63,7 +63,7 @@ change to the file). This is the fix for the "committed syntax error" landmine. ## Model / naming facts - `Source.CaptureKey` is persisted (`window:`, `monitor:`, `picker:`); - `Source.IsBackground` marks the one Background per scene. See ai.md §Screen backdrop capture. + `Source.IsBackground` marks the one Background per scene. See [`docs/ai/capture.md`](../docs/ai/capture.md) → "Screen backdrop capture". - `WindowInfo` (Models) — `Title`, `ProcessName`, `Hwnd` (long), `ProcessId`; `Label => "{Title} — {ProcessName}"`; `CaptureKey => $"window:0x{Hwnd:X}"`. - `IWindowEnumerator` (Services) — `IReadOnlyList Enumerate()`; seam like diff --git a/ai.md b/ai.md index 3d4d31c..3c0700d 100644 --- a/ai.md +++ b/ai.md @@ -396,459 +396,42 @@ The app is a **director's control surface**, not an editor. This replaces the OB Test seam note: `YouTubeAuthService.SetSession()` is public — tests inject a channel with future `TokenExpiry` so `EnsureToken()` passes offline; pair with an `HttpClient(new StubHandler())`. -### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25 +### Capture — backdrop, webcam, effects & the vertical tier — moved -**The Background is a locked per-screen layer (TASK 25): exactly one per canonical -scene, always named "Background", pinned at index 0, never addable/removable/reorderable, -absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time -desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back -to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.** -`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this -model on every load: keeps the correctly-flavored row (or converts a survivor in place — note -`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops -duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0; -non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()` -(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads -`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins). -This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`, -`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`. +Screen/game/window backdrop capture, the webcam seam and sessions, the MediaPipe background effects +and the vertical tier's per-scene camera placement live in +**[`docs/ai/capture.md`](docs/ai/capture.md)** — read it before touching capture, cameras, the +Background row, or anything the 9:16 tier re-places. -- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey` - (persisted) names the target — `monitor:`, `window:`, or `picker:`. The Live row is - a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`, - internal static — also used by the `StagedScene` setter so a scene switch can never expose a - missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`; - the element template sets `IsHitTestVisible=false` for - backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground` - (persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls — - "Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is - staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row - menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live - screens keep the properties panel pill ("Use default background") + Browse for custom art. -- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` / - `BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate). -- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above* - `BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored - Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows - `DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for - persistence/heal/minis — not this overlay. -- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)` - — fires on the UI thread whenever the foreground window changes. The callback runs the same - `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + - `GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own - process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order - `ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`). - `IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`, - friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display" - submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch - `ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the - full-screen game's - monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey` - values are cleared, no capture session is started, and `Source.DisplaySource` falls back to - `_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of - `ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears - on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps - running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to - "Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual - fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground - changes. -- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded - `Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames → - `SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via - `WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the - `IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which - flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the - 1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear — - slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel - version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take). - Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks - (slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a - monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never - overwrite a newer frame), and are spaced by a 10ms floor - (`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for - the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry). - Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a - buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a - consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags) - can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms, - drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion - failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black - frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps - running, the pool just stops delivering. -- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared - `WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed` - events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into - a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the - `IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to - auto-detection. -- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus - (PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes - (`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`: - visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs - `RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the - game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there). - **Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` — - `ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be - recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong - window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while - `IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an - explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session, - `OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback - (fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo` - (`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors - `CameraEnumeratorOverride`. -- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected, - so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first - feature array) → QI `IDXGIDevice` → the WinRT interop export - `CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface.FromAbi` (one shared device per - process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device - no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and - WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working. - `IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by - reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop. -- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder - task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and - window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet). -- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses - focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture — - the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never - pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the - app is in the background (worse under a full-screen game on 24H2/26100), GPU readback - (`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending` - gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as - WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor) - but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted - yet (deferred by user decision). -- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred - to the encoder task. -- **Preview watermark:** the "Preview" placeholder hides while a background capture renders — - `ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with - live capture shows the feed instead of the "nothing here" label. +Invariants that hold even if you never open that file: -### Webcam capture (TASK 3 milestone 1) - -- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed - BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes; - screen capture and background removal later feed the same seam. -- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video, - SharingMode = SharedReadOnly }`, frames pulled via - `CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format - conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via - `WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies). -- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate - by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then - source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as - Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes - NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate - (our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under - msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition. - Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype - with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures - degrade to the next candidate; rejections are logged + collected into the final error; only the outer - catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not - Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps. -- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview` - (preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the - failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture - coexist with other apps that share the camera. -- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync` - fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix: - quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the - app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`). - Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs - inside the physical camera's own driver, so it never appears as a selectable device. Only tools - that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in - the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not - bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**. -- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference - packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy - camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working. -- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam` - row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig` - (position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at - most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager` - refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps - it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the - app default), usable in every scene that can host it.** "Add Webcam" places the existing default - DIRECTLY (no picker); the Windows camera picker runs only for the initial selection - (`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via - `SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide** - (TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable` - — a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene - holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a - saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)` - — an identity whose device was unplugged, or whose session can't start, leaves the row greyed with - `WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The - max-1 reason gets the gentle "a second camera means you've graduated to OBS" line. - Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never - nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock - (below). -- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and - locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app - instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget - right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the - OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt - `CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert** - (`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s - (`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** → - deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40 - Unit A (App Settings round, plan saved 2026-09-22)**. - The single-camera branch **always acquires** (it no longer skips when configs already hold the - session): that acquire is the app's own BASE ref, so removing every scene's placement leaves - `RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered, - i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps - the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting — - a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves - the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the - camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then - offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never - overwritten. Test seams mirror `LayoutPathOverride`: - `CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests` - drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail - → alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure - below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device - handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`). - The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27): - `CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on - staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created - before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) — - LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched. -- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()` - snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and - restores them when switching back — otherwise the Round resize lock (square) would leave a square - behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a - reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time - `HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9 - (keeps height; Round and explicit rect dims are untouched). -- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load) - release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and - stops the source regardless of how many scenes held it (the per-config count isn't known once the - scenes are replaced) — then `AcquireAsync` the new device once per config. -- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread - at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via - `PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher - (at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps - device never drowns the render thread. -- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first - frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't - exist at first-frame time would never receive it — the empty/transparent container you'd see adding a - webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the - current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before - `AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload. -- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored` - (`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity` - 0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow| - Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click - context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist, - rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in - this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty - preview canvas has its own Show Webcam entry. - - The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip - (`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it - renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to - the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is - geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per - frame on the render thread, which is what made the live webcam crawl while round. The Round border is - a centered `Ellipse` at `Width/Height = RoundBorderSize`. - - Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`. - - **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the - max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked - **by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which - may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better - (`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap). - `RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp. - - **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in - the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected - the element — making the corner handle ungrabbable. The element Grid carries - `Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for - right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is - `IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work: - (1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu` - isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the - IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem` - template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness - sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow - driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never - opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the - Click handlers were fine. -- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware - encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred - to the encoder task. -- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The - 2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix, - and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe, - run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video - Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background - effects" below and `TASKS/task-51-webcam-bg-effects.md`. - -### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09) - -- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06), - official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/ - float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own - `tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op - `Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct - translation was smaller than any workaround). **Verified against Google's own runtime:** - `verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original - .tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images. - ⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance). - obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09), - but none of its code is used; if any is ever borrowed, it needs its copyright notice kept. -- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The - whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates. -- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git, - inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime = - `Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native - `onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX - publish (TASK 48) must pin the RID so only win-x64 ships. -- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) → - SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded - resource; **never throws** — a load/run failure logs once and returns null forever after); - `BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true - passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched, - `IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength = - edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4× - by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person - pixels (mask = 1) are copied - untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px, - default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was - far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author - stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and - rendered per effect (`Render`), so several variants share one inference. No mask / segmenter - exception → passthrough + one log line, never a dropped layer. -- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength` - (0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig` - (`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump — - the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds. -- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the - camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect, - strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every - config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each - config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap, - and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`, -which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time) - segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver: - `GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first - creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and - an effect variant shows **nothing** until its first processed frame (Remove never flashes the - real room). The compositor needed no change: its sampler already honours source alpha. -- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies - every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix: - `MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous** - (`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in - the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test - pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any - size change pass through un-blended; the returned mask never aliases the caller's array and two - buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights - come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests - `temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080) - found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a - smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6 - small px (see the `Blur` bullet above). -- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame - (Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the - bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp - per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real - average once per session after 120 processed frames (`startup.log`: "background effects avg …"). - `MainViewModel.PersonSegmenterOverride` is the hermetic-test seam. -- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel - (`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and - Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off, - turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an - effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam: - a new strength variant inherits the dropped same-effect variant's last frame + bitmap - (`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders. -- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a - model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and - matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test - image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests` - — the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive - masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would - have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0). - `Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping - (`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears - light ~9px into the dark half vs ~26px at full strength. - -### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21) - -**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080 -master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default -webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the -creator is simply not in the picture. - -**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical -placement — position, size and shape — and switching back to 16:9 restores the desktop layout -exactly: - -| mode | shape | size | position | -|---|---|---|---| -| 16:9 | as today | as today | as today | -| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right | -| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area | -| Starting / BRB / Ending | unchanged | unchanged | unchanged | - -**Design — geometry only, no render-path change.** The preview and the compositor both read the -element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live -geometry** instead of teaching the render path about tiers: - -- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape` - + `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical - placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient). -- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in; - `EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the - parked 16:9 layout. The live geometry IS the render input → **the compositor and - `PreviewPane.xaml` are untouched**. -- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is - live, the parked values are written there and the live ones into the `Vertical*` columns). -- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write - (drag, resize handle, properties fields) into the band — one choke point instead of three - writers re-implementing the rule. Outside the band = invisible in the vertical stream. - -**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is -656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns -(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master -px from the band's outer edge (the app's own placement margin) and never outside the band. The -default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right** -spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the -first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the -other scenes keep their 16:9 placement until the creator drags them into the band. - -**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**: -the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button -disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a -locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The -lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the -16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile -(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two -(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top -or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is -picked (like Live) — otherwise the vertical chat stream has no camera at all. - -**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section -(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's -geometry setters clamp it into the band, so a typed value can never push the camera out of the -cropped window. - -**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the -maintainers' answer is "create a new empty scene and add the scene you want to send as source" -(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no -built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and -plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored -placement, resolved on the tier change, is the step past that. - -**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier -is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it -lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both -placements survive a save + reload. (The store writes the webcam rows only when the app has a -webcam identity — the config rows hang off it — so the test supplies one.) -`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked -→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and -cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back -to 16:9 the desktop layout is restored exactly and the circle option is offered again. - -**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam` -broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a -new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice. +- **The Background is a locked per-screen layer:** exactly one per canonical scene, named + "Background", pinned at index 0, never addable/removable/reorderable — Live is `DisplayCapture`, + every other screen is static art. `NormalizeBackgrounds` heals any layout to that model on load. +- **Never feed a capture row to the `ActiveBackgroundImage` overlay.** That layer is fed by + static-art rows only; a flavor-blind `IsBackground` lookup there permanently covers the live + desktop/game feed (the 2026-08-23 regression). +- **`window:`/`picker:` capture keys are session-scoped and stripped on load** (a recycled HWND + would capture the wrong window), and an explicit window pin beats auto game/desktop detection + while that window is alive. +- **The WinRT interop paths are settled:** read frame bytes with `WindowsRuntimeMarshal.TryGetDataUnsafe` + (the `IMemoryBufferByteAccess` ComImport throws `Invalid cast` on every frame under CsWinRT) and + build the D3D device via `CreateDirect3D11DeviceFromDXGIDevice` (the + QI-for-`IDirect3DDxgiInterfaceAccess` trick returns E_NOINTERFACE on current Windows). +- **`VideoFrame` + the capture interfaces are the seam** — nothing above the WinRT layer leaks WinRT + types; capture is CPU-first (`MemoryPreference.Cpu`, `SharedReadOnly`), the reader-subtype ladder + tries each candidate under its OWN catch (one rejected format must never abort acquisition), and + there is deliberately **no fallback camera** — the user's pick is never overridden. +- **One webcam identity app-wide, placements per scene.** `IsWebcamAttainable` needs a live lock, not + just a saved id; the app keeps its own base ref, so removing every placement leaves the session up. +- **`CameraManager` is the ONE background-effects processing point** — segment once per raw frame, + render per-scene variants keyed by effect+strength, mask smoothed per camera session (0.75 new / + 0.25 previous). The raw path stays untouched so a slow model never stalls the webcam, and the + bundled model's provenance is fixed (do not swap in PINTO #109). +- **The vertical tier is geometry only:** `BeginVerticalLayout`/`EndVerticalLayout` park and swap the + live placement, so the compositor and `PreviewPane.xaml` never learn about tiers, and the stored + X/Y/W/H/ClipShape columns stay the **16:9** layout. The Chat camera is rect-only through the MODEL + (`VerticalShapeLocked`), not merely through a hidden button. ### Rendering budget, measurement & the compositor — moved diff --git a/docs/ai/capture.md b/docs/ai/capture.md new file mode 100644 index 0000000..9545cdd --- /dev/null +++ b/docs/ai/capture.md @@ -0,0 +1,484 @@ +# Capture — backdrop, webcam, background effects, and the vertical tier + +> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches; +> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md). + +Every frame source in the app lands here: the locked Background layer (screen/game/window capture), +the webcam seam and its sessions, the MediaPipe background effects, and the per-scene camera placement +the vertical 9:16 tier uses. Content is the architecture plus the measured history that produced it; +superseded designs are marked inline. + +--- + +## The locked Background layer — screen, game, window capture + +### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25 + +**The Background is a locked per-screen layer (TASK 25): exactly one per canonical +scene, always named "Background", pinned at index 0, never addable/removable/reorderable, +absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time +desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back +to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.** +`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this +model on every load: keeps the correctly-flavored row (or converts a survivor in place — note +`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops +duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0; +non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()` +(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads +`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins). +This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`, +`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`. + +- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey` + (persisted) names the target — `monitor:`, `window:`, or `picker:`. The Live row is + a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`, + internal static — also used by the `StagedScene` setter so a scene switch can never expose a + missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`; + the element template sets `IsHitTestVisible=false` for + backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground` + (persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls — + "Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is + staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row + menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live + screens keep the properties panel pill ("Use default background") + Browse for custom art. +- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` / + `BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate). +- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above* + `BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored + Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows + `DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for + persistence/heal/minis — not this overlay. +- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)` + — fires on the UI thread whenever the foreground window changes. The callback runs the same + `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + + `GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own + process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order + `ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`). + `IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`, + friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display" + submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch + `ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the + full-screen game's + monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey` + values are cleared, no capture session is started, and `Source.DisplaySource` falls back to + `_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of + `ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears + on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps + running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to + "Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual + fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground + changes. +- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded + `Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames → + `SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via + `WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the + `IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which + flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the + 1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear — + slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel + version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take). + Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks + (slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a + monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never + overwrite a newer frame), and are spaced by a 10ms floor + (`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for + the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry). + Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a + buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a + consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags) + can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms, + drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion + failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black + frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps + running, the pool just stops delivering. +- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared + `WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed` + events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into + a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the + `IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to + auto-detection. +- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus + (PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes + (`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`: + visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs + `RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the + game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there). + **Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` — + `ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be + recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong + window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while + `IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an + explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session, + `OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback + (fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo` + (`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors + `CameraEnumeratorOverride`. +- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected, + so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first + feature array) → QI `IDXGIDevice` → the WinRT interop export + `CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface.FromAbi` (one shared device per + process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device + no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and + WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working. + `IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by + reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop. +- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder + task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and + window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet). +- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses + focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture — + the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never + pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the + app is in the background (worse under a full-screen game on 24H2/26100), GPU readback + (`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending` + gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as + WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor) + but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted + yet (deferred by user decision). +- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred + to the encoder task. +- **Preview watermark:** the "Preview" placeholder hides while a background capture renders — + `ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with + live capture shows the feed instead of the "nothing here" label. + +--- + +## Webcam — capture seam, sessions, and background effects + +### Webcam capture (TASK 3 milestone 1) + +- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed + BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes; + screen capture and background removal later feed the same seam. +- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video, + SharingMode = SharedReadOnly }`, frames pulled via + `CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format + conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via + `WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies). +- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate + by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then + source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as + Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes + NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate + (our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under + msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition. + Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype + with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures + degrade to the next candidate; rejections are logged + collected into the final error; only the outer + catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not + Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps. +- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview` + (preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the + failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture + coexist with other apps that share the camera. +- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync` + fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix: + quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the + app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`). + Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs + inside the physical camera's own driver, so it never appears as a selectable device. Only tools + that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in + the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not + bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**. +- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference + packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy + camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working. +- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam` + row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig` + (position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at + most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager` + refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps + it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the + app default), usable in every scene that can host it.** "Add Webcam" places the existing default + DIRECTLY (no picker); the Windows camera picker runs only for the initial selection + (`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via + `SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide** + (TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable` + — a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene + holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a + saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)` + — an identity whose device was unplugged, or whose session can't start, leaves the row greyed with + `WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The + max-1 reason gets the gentle "a second camera means you've graduated to OBS" line. + Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never + nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock + (below). +- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and + locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app + instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget + right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the + OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt + `CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert** + (`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s + (`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** → + deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40 + Unit A (App Settings round, plan saved 2026-09-22)**. + The single-camera branch **always acquires** (it no longer skips when configs already hold the + session): that acquire is the app's own BASE ref, so removing every scene's placement leaves + `RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered, + i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps + the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting — + a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves + the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the + camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then + offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never + overwritten. Test seams mirror `LayoutPathOverride`: + `CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests` + drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail + → alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure + below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device + handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`). + The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27): + `CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on + staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created + before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) — + LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched. +- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()` + snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and + restores them when switching back — otherwise the Round resize lock (square) would leave a square + behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a + reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time + `HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9 + (keeps height; Round and explicit rect dims are untouched). +- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load) + release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and + stops the source regardless of how many scenes held it (the per-config count isn't known once the + scenes are replaced) — then `AcquireAsync` the new device once per config. +- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread + at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via + `PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher + (at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps + device never drowns the render thread. +- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first + frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't + exist at first-frame time would never receive it — the empty/transparent container you'd see adding a + webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the + current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before + `AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload. +- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored` + (`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity` + 0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow| + Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click + context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist, + rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in + this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty + preview canvas has its own Show Webcam entry. + - The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip + (`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it + renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to + the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is + geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per + frame on the render thread, which is what made the live webcam crawl while round. The Round border is + a centered `Ellipse` at `Width/Height = RoundBorderSize`. + - Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`. + - **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the + max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked + **by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which + may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better + (`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap). + `RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp. + - **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in + the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected + the element — making the corner handle ungrabbable. The element Grid carries + `Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for + right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is + `IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work: + (1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu` + isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the + IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem` + template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness + sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow + driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never + opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the + Click handlers were fine. +- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware + encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred + to the encoder task. +- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The + 2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix, + and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe, + run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video + Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background + effects" below and `TASKS/task-51-webcam-bg-effects.md`. + +### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09) + +- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06), + official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/ + float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own + `tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op + `Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct + translation was smaller than any workaround). **Verified against Google's own runtime:** + `verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original + .tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images. + ⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance). + obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09), + but none of its code is used; if any is ever borrowed, it needs its copyright notice kept. +- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The + whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates. +- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git, + inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime = + `Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native + `onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX + publish (TASK 48) must pin the RID so only win-x64 ships. +- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) → + SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded + resource; **never throws** — a load/run failure logs once and returns null forever after); + `BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true + passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched, + `IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength = + edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4× + by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person + pixels (mask = 1) are copied + untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px, + default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was + far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author + stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and + rendered per effect (`Render`), so several variants share one inference. No mask / segmenter + exception → passthrough + one log line, never a dropped layer. +- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength` + (0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig` + (`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump — + the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds. +- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the + camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect, + strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every + config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each + config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap, + and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`, +which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time) + segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver: + `GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first + creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and + an effect variant shows **nothing** until its first processed frame (Remove never flashes the + real room). The compositor needed no change: its sampler already honours source alpha. +- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies + every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix: + `MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous** + (`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in + the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test + pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any + size change pass through un-blended; the returned mask never aliases the caller's array and two + buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights + come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests + `temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080) + found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a + smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6 + small px (see the `Blur` bullet above). +- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame + (Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the + bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp + per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real + average once per session after 120 processed frames (`startup.log`: "background effects avg …"). + `MainViewModel.PersonSegmenterOverride` is the hermetic-test seam. +- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel + (`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and + Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off, + turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an + effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam: + a new strength variant inherits the dropped same-effect variant's last frame + bitmap + (`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders. +- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a + model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and + matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test + image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests` + — the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive + masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would + have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0). + `Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping + (`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears + light ~9px into the dark half vs ~26px at full strength. + +--- + +## The vertical (9:16) tier — per-scene camera placement + +### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21) + +**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080 +master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default +webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the +creator is simply not in the picture. + +**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical +placement — position, size and shape — and switching back to 16:9 restores the desktop layout +exactly: + +| mode | shape | size | position | +|---|---|---|---| +| 16:9 | as today | as today | as today | +| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right | +| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area | +| Starting / BRB / Ending | unchanged | unchanged | unchanged | + +**Design — geometry only, no render-path change.** The preview and the compositor both read the +element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live +geometry** instead of teaching the render path about tiers: + +- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape` + + `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical + placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient). +- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in; + `EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the + parked 16:9 layout. The live geometry IS the render input → **the compositor and + `PreviewPane.xaml` are untouched**. +- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is + live, the parked values are written there and the live ones into the `Vertical*` columns). +- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write + (drag, resize handle, properties fields) into the band — one choke point instead of three + writers re-implementing the rule. Outside the band = invisible in the vertical stream. + +**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is +656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns +(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master +px from the band's outer edge (the app's own placement margin) and never outside the band. The +default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right** +spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the +first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the +other scenes keep their 16:9 placement until the creator drags them into the band. + +**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**: +the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button +disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a +locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The +lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the +16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile +(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two +(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top +or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is +picked (like Live) — otherwise the vertical chat stream has no camera at all. + +**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section +(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's +geometry setters clamp it into the band, so a typed value can never push the camera out of the +cropped window. + +**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the +maintainers' answer is "create a new empty scene and add the scene you want to send as source" +(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no +built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and +plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored +placement, resolved on the tier change, is the step past that. + +**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier +is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it +lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both +placements survive a save + reload. (The store writes the webcam rows only when the app has a +webcam identity — the config rows hang off it — so the test supplies one.) +`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked +→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and +cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back +to 16:9 the desktop layout is restored exactly and the circle option is offered again. + +**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam` +broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a +new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice. + +--- + +## Related in [`ai.md`](../../ai.md) + +- **Resolution tiers** — `OutputRect*` and the tier list (the vertical tier's 607×1080 band). +- **Key patterns** — MVVM/binding conventions the capture UI follows. +- **Tests** — hermetic-real-window rules that camera tests must obey. +- **Run** — launch/take procedure and the Release-only rule for measurements. diff --git a/docs/ai/index.md b/docs/ai/index.md index a31c524..db00bf6 100644 --- a/docs/ai/index.md +++ b/docs/ai/index.md @@ -10,6 +10,7 @@ Read the file whose row matches the task. There is no required reading order. | File | Covers | Read when | |------|--------|-----------| | [`frame-pipeline.md`](frame-pipeline.md) | Render budget & measurement (`FrameCadence`, per-element composite timing, scene dump), `SceneCompositor` design + `StripPool`, and `FramePump` from scene snapshot to encoder (pacing, pool/paste caches, backpressure, teardown, health stats) | Touching the compositor, the render loop, per-frame cost, `Services/Encoder/*`, or debugging dropped/juddering frames | +| [`capture.md`](capture.md) | The locked Background layer (screen/game/window capture, WinRT GraphicsCapture interop), the webcam seam + `CameraManager` sessions, MediaPipe background effects, and the vertical 9:16 tier's per-scene camera placement | Touching any frame source, the Background row, camera acquisition/conflicts, background effects, or vertical-tier camera geometry | ## Split-out yet to come