LlamaCasty/docs/ai/capture.md

# Capture — backdrop, webcam, background effects, and the vertical tier

> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches;
> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md).

Every frame source in the app lands here: the locked Background layer (screen/game/window capture),
the webcam seam and its sessions, the MediaPipe background effects, and the per-scene camera placement
the vertical 9:16 tier uses. Content is the architecture plus the measured history that produced it;
superseded designs are marked inline.

---

## The locked Background layer — screen, game, window capture

### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25

**The Background is a locked per-screen layer (TASK 25): exactly one per canonical
scene, always named "Background", pinned at index 0, never addable/removable/reorderable,
absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time
desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back
to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.**
`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this
model on every load: keeps the correctly-flavored row (or converts a survivor in place — note
`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops
duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0;
non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()`
(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads
`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins).
This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`,
`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`.

- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey`
  (persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The Live row is
  a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`,
  internal static — also used by the `StagedScene` setter so a scene switch can never expose a
  missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
  the element template sets `IsHitTestVisible=false` for
  backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground`
  (persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls —
  "Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is
  staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row
  menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live
  screens keep the properties panel pill ("Use default background") + Browse for custom art.
- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` /
  `BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above*
  `BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored
  Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows
  `DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for
  persistence/heal/minis — not this overlay.
- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)`
  — fires on the UI thread whenever the foreground window changes. The callback runs the same
  `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
  `GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own
  process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
  `ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
  `IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
  friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
  submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch
  `ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the
  full-screen game's
  monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey`
  values are cleared, no capture session is started, and `Source.DisplaySource` falls back to
  `_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of
  `ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears
  on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps
  running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to
  "Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual
  fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground
  changes.
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
  `Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
  `SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
  `WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
  `IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
  flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
  1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear —
  slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel
  version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take).
  Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks
  (slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a
  monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never
  overwrite a newer frame), and are spaced by a 10ms floor
  (`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for
  the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry).
  Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a
  buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a
  consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags)
  can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms,
  drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion
  failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black
  frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps
  running, the pool just stops delivering.
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
  `WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
  events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
  a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
  `IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
  auto-detection.
- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus
  (PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes
  (`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`:
  visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs
  `RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the
  game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there).
  **Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` —
  `ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be
  recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong
  window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while
  `IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an
  explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session,
  `OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback
  (fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo`
  (`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors
  `CameraEnumeratorOverride`.
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
  so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
  feature array) → QI `IDXGIDevice` → the WinRT interop export
  `CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
  process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
  no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
  WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
  `IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
  reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
  task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and
  window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet).
- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses
  focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture —
  the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never
  pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the
  app is in the background (worse under a full-screen game on 24H2/26100), GPU readback
  (`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending`
  gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as
  WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor)
  but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted
  yet (deferred by user decision).
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
  to the encoder task.
- **Preview watermark:** the "Preview" placeholder hides while a background capture renders —
  `ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with
  live capture shows the feed instead of the "nothing here" label.

---

## Webcam — capture seam, sessions, and background effects

### Webcam capture (TASK 3 milestone 1)

- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
  BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes;
  screen capture and background removal later feed the same seam.
- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video,
  SharingMode = SharedReadOnly }`, frames pulled via
  `CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format
  conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via
  `WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies).
- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate
  by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then
  source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as
  Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes
  NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate
  (our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under
  msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition.
  Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype
  with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures
  degrade to the next candidate; rejections are logged + collected into the final error; only the outer
  catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not
  Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps.
- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview`
  (preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the
  failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture
  coexist with other apps that share the camera.
- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync`
  fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
  quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
  app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
  Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
  inside the physical camera's own driver, so it never appears as a selectable device. Only tools
  that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
  the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
  bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
  packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
  camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam`
  row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig`
  (position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at
  most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager`
  refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps
  it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the
  app default), usable in every scene that can host it.** "Add Webcam" places the existing default
  DIRECTLY (no picker); the Windows camera picker runs only for the initial selection
  (`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via
  `SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide**
  (TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable`
  — a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene
  holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a
  saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)`
  — an identity whose device was unplugged, or whose session can't start, leaves the row greyed with
  `WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The
  max-1 reason gets the gentle "a second camera means you've graduated to OBS" line.
  Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never
  nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock
  (below).
- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and
  locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app
  instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget
  right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the
  OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt
  `CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert**
  (`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s
  (`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** →
  deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40
  Unit A (App Settings round, plan saved 2026-09-22)**.
  The single-camera branch **always acquires** (it no longer skips when configs already hold the
  session): that acquire is the app's own BASE ref, so removing every scene's placement leaves
  `RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered,
  i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps
  the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting —
  a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves
  the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the
  camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then
  offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never
  overwritten. Test seams mirror `LayoutPathOverride`:
  `CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests`
  drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail
  → alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure
  below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device
  handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`).
  The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27):
  `CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on
  staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created
  before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) —
  LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
  snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
  restores them when switching back — otherwise the Round resize lock (square) would leave a square
  behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a
  reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time
  `HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9
  (keeps height; Round and explicit rect dims are untouched).
- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load)
  release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and
  stops the source regardless of how many scenes held it (the per-config count isn't known once the
  scenes are replaced) — then `AcquireAsync` the new device once per config.
- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread
  at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via
  `PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
  (at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
  device never drowns the render thread.
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
  frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
  exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
  webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
  current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
  `AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
  (`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
  0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
  Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click
  context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist,
  rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
  this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
  preview canvas has its own Show Webcam entry.
  - The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
    (`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
    renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
    the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
    geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
    frame on the render thread, which is what made the live webcam crawl while round. The Round border is
    a centered `Ellipse` at `Width/Height = RoundBorderSize`.
  - Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
  - **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
    max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
    **by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
    may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
    (`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
    `RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
  - **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
    the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected
    the element — making the corner handle ungrabbable. The element Grid carries
    `Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for
    right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is
  `IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work:
  (1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu`
  isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the
  IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem`
  template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness
  sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow
  driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never
  opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the
  Click handlers were fine.
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
  encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
  to the encoder task.
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
  2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
  and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
  run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
  Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
  effects" below and `TASKS/task-51-webcam-bg-effects.md`.

### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)

- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
  official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
  float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
  `tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
  `Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
  translation was smaller than any workaround). **Verified against Google's own runtime:**
  `verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
  .tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
  ⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance).
  obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09),
  but none of its code is used; if any is ever borrowed, it needs its copyright notice kept.
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
  whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
  inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
  `Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
  `onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
  publish (TASK 48) must pin the RID so only win-x64 ships.
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
  SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
  resource; **never throws** — a load/run failure logs once and returns null forever after);
  `BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
  passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
  `IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength =
  edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4×
  by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person
  pixels (mask = 1) are copied
  untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px,
  default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was
  far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author
  stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and
  rendered per effect (`Render`), so several variants share one inference. No mask / segmenter
  exception → passthrough + one log line, never a dropped layer.
- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength`
  (0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig`
  (`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump —
  the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds.
- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the
  camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect,
  strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every
  config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
  config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
  and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
   segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
   `GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
   creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
   an effect variant shows **nothing** until its first processed frame (Remove never flashes the
   real room). The compositor needed no change: its sampler already honours source alpha.
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
  every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
  `MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
  (`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
  the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
  pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
  size change pass through un-blended; the returned mask never aliases the caller's array and two
  buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
  come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
  `temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
  found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
  smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6
  small px (see the `Blur` bullet above).
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
  (Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
  bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
  per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real
  average once per session after 120 processed frames (`startup.log`: "background effects avg …").
  `MainViewModel.PersonSegmenterOverride` is the hermetic-test seam.
- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel
  (`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and
  Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off,
  turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an
  effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam:
  a new strength variant inherits the dropped same-effect variant's last frame + bitmap
  (`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders.
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
  model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
  matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
  image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
  — the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
  masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
  have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
  `Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping
  (`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears
  light ~9px into the dark half vs ~26px at full strength.

---

## The vertical (9:16) tier — per-scene camera placement

### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21)

**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080
master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default
webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the
creator is simply not in the picture.

**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical
placement — position, size and shape — and switching back to 16:9 restores the desktop layout
exactly:

| mode | shape | size | position |
|---|---|---|---|
| 16:9 | as today | as today | as today |
| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right |
| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area |
| Starting / BRB / Ending | unchanged | unchanged | unchanged |

**Design — geometry only, no render-path change.** The preview and the compositor both read the
element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live
geometry** instead of teaching the render path about tiers:

- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape`
  + `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical
  placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient).
- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in;
  `EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the
  parked 16:9 layout. The live geometry IS the render input → **the compositor and
  `PreviewPane.xaml` are untouched**.
- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is
  live, the parked values are written there and the live ones into the `Vertical*` columns).
- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write
  (drag, resize handle, properties fields) into the band — one choke point instead of three
  writers re-implementing the rule. Outside the band = invisible in the vertical stream.

**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is
656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns
(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master
px from the band's outer edge (the app's own placement margin) and never outside the band. The
default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right**
spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the
first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the
other scenes keep their 16:9 placement until the creator drags them into the band.

**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**:
the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button
disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a
locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The
lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the
16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile
(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two
(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top
or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is
picked (like Live) — otherwise the vertical chat stream has no camera at all.

**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section
(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's
geometry setters clamp it into the band, so a typed value can never push the camera out of the
cropped window.

**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the
maintainers' answer is "create a new empty scene and add the scene you want to send as source"
(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no
built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and
plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored
placement, resolved on the tier change, is the step past that.

**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier
is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it
lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both
placements survive a save + reload. (The store writes the webcam rows only when the app has a
webcam identity — the config rows hang off it — so the test supplies one.)
`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked
→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and
cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back
to 16:9 the desktop layout is restored exactly and the circle option is offered again.

**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam`
broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a
new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice.

---

## Related files

- [`ui-surface.md`](ui-surface.md) — **resolution tiers** (`OutputRect*`, the tier list, the vertical tier's 607×1080 band) and **key patterns** (MVVM/binding conventions the capture UI follows).
- [`environment.md`](environment.md) — **Tests** (hermetic real-window rules camera tests must obey) and **Run** (launch/take procedure, the Release-only rule for measurements).