docs(memory): split capture (backdrop, webcam, effects, vertical tier) out of ai.md
Unit 2 of slimming the mandatory read. Verbatim move again - the four capture blocks leave ai.md unchanged in content, and one stub carries the law. - docs/ai/capture.md: 'Screen backdrop capture', 'Webcam capture', 'Webcam background effects' and 'Vertical (9:16) tier camera placement', byte- identical, grouped under backdrop / webcam / vertical headings, with a 'Related in ai.md' footer. - ai.md stub keeps: the locked-Background model and NormalizeBackgrounds; ActiveBackgroundImage is fed by static art only (2026-08-23 regression); window:/picker: keys are session-scoped and a live pin beats auto-detect; the settled WinRT interop paths (TryGetDataUnsafe, from-DXGIDevice); the VideoFrame seam, own-catch subtype ladder and no fallback camera; one app- level webcam identity with live-lock attainability; CameraManager as the one effects processing point with per-session mask smoothing; geometry-only vertical tier and the model-level rect lock. - docs/ai/index.md gains the capture.md row; Services/index.md (2) and TASKS/task-38-window-backdrop.md repointed. ai.md 1,975 -> 1,558 lines. No source touched; no push.
This commit is contained in:
+2
-2
@@ -21,7 +21,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `BackgroundEffects/OnnxPersonSegmenter.cs` | **TASK 51 default segmenter:** MediaPipe Selfie Segmenter (landscape, Apache-2.0) as an embedded ONNX resource, run by onnxruntime on the CPU (2 intra-op threads); frame stretched to 256×144 RGB/255 NHWC; lazy session; never throws (fail → log once, null forever) |
|
||||
| `BackgroundEffects/WebcamEffectKey.cs` | **TASK 51:** `(Effect, Strength)` variant key, strength rounded to 0.01; `None` normalizes to strength 0 (the raw path) |
|
||||
| `BackgroundEffects/BackgroundEffectProcessor.cs` | **TASK 51 pure pixel stage:** `Apply(frame, BackgroundEffect)` — None = same instance, Remove = copy with alpha = feathered mask × source alpha (`IsOpaque=false`), Blur = background blurred at ¼ res (3 box passes) and mask-blended, person untouched; strength 0..1; blur radius is **quadratic** via `BlurRadius(strength)` — 1→6 small px (default 0.5 = 2, was a linear 1→12 ≈ 48 full px; the "blur heavy" fix 2026-10-09); `TrySegment` once + `Render` per effect; no mask/exception → passthrough + one log |
|
||||
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see `ai.md` "Webcam background effects" |
|
||||
| `BackgroundEffects/MaskSmoother.cs` | **TASK 51 jitter fix (2026-10-09):** temporally blends each raw mask with the previous smoothed one — `CurrentWeight` 0.75 × new + 0.25 × previous — so a per-frame segmentation verdict that wavers at an edge doesn't flutter the cut-out. Null input resets history; first frame/size change pass through; the returned mask never aliases the caller's array and the two buffers ping-pong (no per-frame LOH alloc: a 256×144 mask is 147 KB). State is **per camera session**, not in the segmenter (which is shared app-wide and contract-pinned to raw output). Weights from the OBS/Jitsi consensus — see [`docs/ai/capture.md`](../docs/ai/capture.md) → "Webcam background effects" |
|
||||
| `CameraDeviceInfo.cs` | `(Id, DisplayName)` for a physical capture device |
|
||||
| `ICameraEnumerator.cs` | `GetCamerasAsync()` — seam so the picker/`CameraManager` never touch WinRT (tests inject fakes) |
|
||||
| `ICameraFrameSource.cs` | `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` + **`SourceFailed(string)`** — seam for a running capture source (tests inject fakes) |
|
||||
@@ -30,7 +30,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `IMicrophoneEnumerator.cs` | `GetMicrophonesAsync()` — seam so the mic picker never touches WinRT (tests inject fakes) |
|
||||
| `WinRtMicrophoneEnumerator.cs` | WinRT mic enumeration via `DeviceInformation.FindAllAsync(DeviceClass.AudioCapture)` (no NAudio needed — capture libs stay deferred to the audio pipeline milestone) |
|
||||
| `MediaCaptureFrameSource.cs` | CPU-first MediaCapture source (`MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`). **Validated, not trusted:** post-init checks (`VideoDeviceId` match, `FrameSources` non-empty, `VideoDeviceController.GetAvailableMediaStreamProperties` ≥ 1); `reader.StartAsync()` status read (throws on non-`Success`); `capture.Failed` + `CameraStreamStateChanged` subscribed → `SourceFailed` event. Fallback ladder: VideoPreview → VideoRecord retry. `SharingMode.Exclusive` + `DeviceLost` not in 19041 SDK projection; `CameraStreamState.Failed` compared by `(int)2` (projection omits member names) |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see `ai.md` "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `CameraManager.cs` | Webcam capture ownership: refcounted by `DeviceId`, one shared `WriteableBitmap` per camera, dispatcher-coalesced ~render-rate UI updates (latest-frame drop); `PreviewBitmapChanged`/`CameraFailed` events. **First-frame proof** (4s timeout, configurable via ctor `TimeSpan?`): `AcquireAsync` returns true only after a real frame arrives — a reader that starts but never delivers (locked, suspended, dead) fails and surfaces `CameraFailed` with suspect-app names (`CameraConflictProbe`) instead of a silent empty box. `SourceFailed` forwarded from the frame source as async death signal. **TASK 51:** the one background-effects processing point — `SetRequestedEffects(device, keys)` + per-variant `GetLatestFrame`/`GetPreviewBitmap(device, key)` + `EffectPreviewBitmapChanged`; a background worker segments once per raw frame (mask temporally smoothed per camera — `MaskSmoother` — see [`docs/ai/capture.md`](../docs/ai/capture.md) → "Webcam background effects") and renders every variant (latest-frame-drop); raw path untouched |
|
||||
| `IFullScreenDetector.cs` | Seam for the win32 full-screen detector: `int? GetForegroundFullScreenMonitorIndex()`, `int PrimaryMonitorIndex()`, `IReadOnlyList<DisplayInfo> GetDisplays()` |
|
||||
| `Win32FullScreenDetector.cs` | `GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` + `GetMonitorInfo`; monitor covers all four edges → full-screen; own process excluded; monitor order = `EnumDisplayMonitors` order (static `GetMonitorHandle(int)` maps index→HMONITOR for the capture factory). `GetDisplays()` returns `DisplayInfo` (index/name/resolution/bounds/`IsPrimary`, friendly name via `EnumDisplayDevices`) for the in-app "Capture Display" picker; `PrimaryMonitorIndex()` is the auto-key fallback when no full-screen game is detected |
|
||||
| `IScreenCaptureSource.cs` | `Key` + `StartAsync`/`StopAsync`/`FrameAvailable(VideoFrame)` — seam so `ScreenCaptureManager` never touches WinRT (tests inject fakes) |
|
||||
|
||||
@@ -63,7 +63,7 @@ change to the file). This is the fix for the "committed syntax error" landmine.
|
||||
## Model / naming facts
|
||||
|
||||
- `Source.CaptureKey` is persisted (`window:<hwnd>`, `monitor:<n>`, `picker:<displayname>`);
|
||||
`Source.IsBackground` marks the one Background per scene. See ai.md §Screen backdrop capture.
|
||||
`Source.IsBackground` marks the one Background per scene. See [`docs/ai/capture.md`](../docs/ai/capture.md) → "Screen backdrop capture".
|
||||
- `WindowInfo` (Models) — `Title`, `ProcessName`, `Hwnd` (long), `ProcessId`;
|
||||
`Label => "{Title} — {ProcessName}"`; `CaptureKey => $"window:0x{Hwnd:X}"`.
|
||||
- `IWindowEnumerator` (Services) — `IReadOnlyList<WindowInfo> Enumerate()`; seam like
|
||||
|
||||
@@ -396,459 +396,42 @@ The app is a **director's control surface**, not an editor. This replaces the OB
|
||||
|
||||
Test seam note: `YouTubeAuthService.SetSession()` is public — tests inject a channel with future `TokenExpiry` so `EnsureToken()` passes offline; pair with an `HttpClient(new StubHandler())`.
|
||||
|
||||
### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25
|
||||
### Capture — backdrop, webcam, effects & the vertical tier — moved
|
||||
|
||||
**The Background is a locked per-screen layer (TASK 25): exactly one per canonical
|
||||
scene, always named "Background", pinned at index 0, never addable/removable/reorderable,
|
||||
absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time
|
||||
desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back
|
||||
to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.**
|
||||
`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this
|
||||
model on every load: keeps the correctly-flavored row (or converts a survivor in place — note
|
||||
`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops
|
||||
duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0;
|
||||
non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()`
|
||||
(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads
|
||||
`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins).
|
||||
This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`,
|
||||
`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`.
|
||||
Screen/game/window backdrop capture, the webcam seam and sessions, the MediaPipe background effects
|
||||
and the vertical tier's per-scene camera placement live in
|
||||
**[`docs/ai/capture.md`](docs/ai/capture.md)** — read it before touching capture, cameras, the
|
||||
Background row, or anything the 9:16 tier re-places.
|
||||
|
||||
- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey`
|
||||
(persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The Live row is
|
||||
a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`,
|
||||
internal static — also used by the `StagedScene` setter so a scene switch can never expose a
|
||||
missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
|
||||
the element template sets `IsHitTestVisible=false` for
|
||||
backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground`
|
||||
(persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls —
|
||||
"Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is
|
||||
staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row
|
||||
menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live
|
||||
screens keep the properties panel pill ("Use default background") + Browse for custom art.
|
||||
- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` /
|
||||
`BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
|
||||
- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above*
|
||||
`BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored
|
||||
Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows
|
||||
`DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for
|
||||
persistence/heal/minis — not this overlay.
|
||||
- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)`
|
||||
— fires on the UI thread whenever the foreground window changes. The callback runs the same
|
||||
`GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
|
||||
`GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own
|
||||
process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
|
||||
`ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
|
||||
`IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
|
||||
friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
|
||||
submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch
|
||||
`ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the
|
||||
full-screen game's
|
||||
monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey`
|
||||
values are cleared, no capture session is started, and `Source.DisplaySource` falls back to
|
||||
`_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of
|
||||
`ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears
|
||||
on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps
|
||||
running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to
|
||||
"Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual
|
||||
fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground
|
||||
changes.
|
||||
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
|
||||
`Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
|
||||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
|
||||
`WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
|
||||
`IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
|
||||
flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
|
||||
1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear —
|
||||
slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel
|
||||
version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take).
|
||||
Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks
|
||||
(slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a
|
||||
monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never
|
||||
overwrite a newer frame), and are spaced by a 10ms floor
|
||||
(`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for
|
||||
the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry).
|
||||
Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a
|
||||
buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a
|
||||
consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags)
|
||||
can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms,
|
||||
drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion
|
||||
failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black
|
||||
frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps
|
||||
running, the pool just stops delivering.
|
||||
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
|
||||
`WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
|
||||
events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
|
||||
a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
|
||||
`IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
|
||||
auto-detection.
|
||||
- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus
|
||||
(PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes
|
||||
(`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`:
|
||||
visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs
|
||||
`RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the
|
||||
game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there).
|
||||
**Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` —
|
||||
`ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be
|
||||
recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong
|
||||
window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while
|
||||
`IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an
|
||||
explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session,
|
||||
`OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback
|
||||
(fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo`
|
||||
(`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors
|
||||
`CameraEnumeratorOverride`.
|
||||
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
|
||||
so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
|
||||
feature array) → QI `IDXGIDevice` → the WinRT interop export
|
||||
`CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
|
||||
process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
|
||||
no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
|
||||
WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
|
||||
`IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
|
||||
reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
|
||||
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
|
||||
task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and
|
||||
window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet).
|
||||
- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses
|
||||
focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture —
|
||||
the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never
|
||||
pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the
|
||||
app is in the background (worse under a full-screen game on 24H2/26100), GPU readback
|
||||
(`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending`
|
||||
gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as
|
||||
WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor)
|
||||
but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted
|
||||
yet (deferred by user decision).
|
||||
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
|
||||
to the encoder task.
|
||||
- **Preview watermark:** the "Preview" placeholder hides while a background capture renders —
|
||||
`ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with
|
||||
live capture shows the feed instead of the "nothing here" label.
|
||||
Invariants that hold even if you never open that file:
|
||||
|
||||
### Webcam capture (TASK 3 milestone 1)
|
||||
|
||||
- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
|
||||
BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes;
|
||||
screen capture and background removal later feed the same seam.
|
||||
- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video,
|
||||
SharingMode = SharedReadOnly }`, frames pulled via
|
||||
`CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format
|
||||
conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via
|
||||
`WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies).
|
||||
- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate
|
||||
by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then
|
||||
source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as
|
||||
Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes
|
||||
NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate
|
||||
(our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under
|
||||
msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition.
|
||||
Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype
|
||||
with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures
|
||||
degrade to the next candidate; rejections are logged + collected into the final error; only the outer
|
||||
catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not
|
||||
Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps.
|
||||
- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview`
|
||||
(preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the
|
||||
failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture
|
||||
coexist with other apps that share the camera.
|
||||
- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync`
|
||||
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
|
||||
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
|
||||
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
|
||||
Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
|
||||
inside the physical camera's own driver, so it never appears as a selectable device. Only tools
|
||||
that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
|
||||
the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
|
||||
bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
|
||||
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
|
||||
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
|
||||
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
|
||||
- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam`
|
||||
row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig`
|
||||
(position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at
|
||||
most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager`
|
||||
refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps
|
||||
it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the
|
||||
app default), usable in every scene that can host it.** "Add Webcam" places the existing default
|
||||
DIRECTLY (no picker); the Windows camera picker runs only for the initial selection
|
||||
(`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via
|
||||
`SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide**
|
||||
(TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable`
|
||||
— a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene
|
||||
holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a
|
||||
saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)`
|
||||
— an identity whose device was unplugged, or whose session can't start, leaves the row greyed with
|
||||
`WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The
|
||||
max-1 reason gets the gentle "a second camera means you've graduated to OBS" line.
|
||||
Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never
|
||||
nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock
|
||||
(below).
|
||||
- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and
|
||||
locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app
|
||||
instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget
|
||||
right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the
|
||||
OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt
|
||||
`CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert**
|
||||
(`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s
|
||||
(`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** →
|
||||
deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40
|
||||
Unit A (App Settings round, plan saved 2026-09-22)**.
|
||||
The single-camera branch **always acquires** (it no longer skips when configs already hold the
|
||||
session): that acquire is the app's own BASE ref, so removing every scene's placement leaves
|
||||
`RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered,
|
||||
i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps
|
||||
the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting —
|
||||
a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves
|
||||
the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the
|
||||
camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then
|
||||
offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never
|
||||
overwritten. Test seams mirror `LayoutPathOverride`:
|
||||
`CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests`
|
||||
drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail
|
||||
→ alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure
|
||||
below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device
|
||||
handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`).
|
||||
The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27):
|
||||
`CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on
|
||||
staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created
|
||||
before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) —
|
||||
LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
|
||||
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
|
||||
snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
|
||||
restores them when switching back — otherwise the Round resize lock (square) would leave a square
|
||||
behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a
|
||||
reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time
|
||||
`HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9
|
||||
(keeps height; Round and explicit rect dims are untouched).
|
||||
- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load)
|
||||
release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and
|
||||
stops the source regardless of how many scenes held it (the per-config count isn't known once the
|
||||
scenes are replaced) — then `AcquireAsync` the new device once per config.
|
||||
- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread
|
||||
at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via
|
||||
`PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
|
||||
(at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
|
||||
device never drowns the render thread.
|
||||
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
|
||||
frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
|
||||
exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
|
||||
webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
|
||||
current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
|
||||
`AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
|
||||
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
|
||||
(`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
|
||||
0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
|
||||
Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click
|
||||
context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist,
|
||||
rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
|
||||
this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
|
||||
preview canvas has its own Show Webcam entry.
|
||||
- The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
|
||||
(`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
|
||||
renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
|
||||
the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
|
||||
geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
|
||||
frame on the render thread, which is what made the live webcam crawl while round. The Round border is
|
||||
a centered `Ellipse` at `Width/Height = RoundBorderSize`.
|
||||
- Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
|
||||
- **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
|
||||
max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
|
||||
**by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
|
||||
may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
|
||||
(`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
|
||||
`RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
|
||||
- **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
|
||||
the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected
|
||||
the element — making the corner handle ungrabbable. The element Grid carries
|
||||
`Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for
|
||||
right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is
|
||||
`IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work:
|
||||
(1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu`
|
||||
isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the
|
||||
IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem`
|
||||
template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness
|
||||
sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow
|
||||
driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never
|
||||
opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the
|
||||
Click handlers were fine.
|
||||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||||
to the encoder task.
|
||||
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
|
||||
2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
|
||||
and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
|
||||
run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
|
||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||
|
||||
### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)
|
||||
|
||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||
float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
|
||||
`tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
|
||||
`Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
|
||||
translation was smaller than any workaround). **Verified against Google's own runtime:**
|
||||
`verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
|
||||
.tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
|
||||
⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance).
|
||||
obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09),
|
||||
but none of its code is used; if any is ever borrowed, it needs its copyright notice kept.
|
||||
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
|
||||
whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
|
||||
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
|
||||
inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
|
||||
`Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
|
||||
`onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
|
||||
publish (TASK 48) must pin the RID so only win-x64 ships.
|
||||
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
|
||||
SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
|
||||
resource; **never throws** — a load/run failure logs once and returns null forever after);
|
||||
`BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
|
||||
passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
|
||||
`IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength =
|
||||
edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4×
|
||||
by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person
|
||||
pixels (mask = 1) are copied
|
||||
untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px,
|
||||
default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was
|
||||
far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author
|
||||
stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and
|
||||
rendered per effect (`Render`), so several variants share one inference. No mask / segmenter
|
||||
exception → passthrough + one log line, never a dropped layer.
|
||||
- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength`
|
||||
(0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig`
|
||||
(`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump —
|
||||
the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds.
|
||||
- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the
|
||||
camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect,
|
||||
strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every
|
||||
config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
|
||||
config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
|
||||
and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
|
||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
|
||||
every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
|
||||
`MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
|
||||
(`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
|
||||
the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
|
||||
pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
|
||||
size change pass through un-blended; the returned mask never aliases the caller's array and two
|
||||
buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
|
||||
come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
|
||||
`temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
|
||||
found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
|
||||
smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6
|
||||
small px (see the `Blur` bullet above).
|
||||
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
|
||||
(Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
|
||||
bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
|
||||
per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real
|
||||
average once per session after 120 processed frames (`startup.log`: "background effects avg …").
|
||||
`MainViewModel.PersonSegmenterOverride` is the hermetic-test seam.
|
||||
- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel
|
||||
(`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and
|
||||
Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off,
|
||||
turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an
|
||||
effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam:
|
||||
a new strength variant inherits the dropped same-effect variant's last frame + bitmap
|
||||
(`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders.
|
||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
|
||||
— the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
|
||||
masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
|
||||
have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
|
||||
`Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping
|
||||
(`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears
|
||||
light ~9px into the dark half vs ~26px at full strength.
|
||||
|
||||
### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21)
|
||||
|
||||
**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080
|
||||
master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default
|
||||
webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the
|
||||
creator is simply not in the picture.
|
||||
|
||||
**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical
|
||||
placement — position, size and shape — and switching back to 16:9 restores the desktop layout
|
||||
exactly:
|
||||
|
||||
| mode | shape | size | position |
|
||||
|---|---|---|---|
|
||||
| 16:9 | as today | as today | as today |
|
||||
| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right |
|
||||
| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area |
|
||||
| Starting / BRB / Ending | unchanged | unchanged | unchanged |
|
||||
|
||||
**Design — geometry only, no render-path change.** The preview and the compositor both read the
|
||||
element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live
|
||||
geometry** instead of teaching the render path about tiers:
|
||||
|
||||
- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape`
|
||||
+ `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical
|
||||
placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient).
|
||||
- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in;
|
||||
`EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the
|
||||
parked 16:9 layout. The live geometry IS the render input → **the compositor and
|
||||
`PreviewPane.xaml` are untouched**.
|
||||
- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is
|
||||
live, the parked values are written there and the live ones into the `Vertical*` columns).
|
||||
- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write
|
||||
(drag, resize handle, properties fields) into the band — one choke point instead of three
|
||||
writers re-implementing the rule. Outside the band = invisible in the vertical stream.
|
||||
|
||||
**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is
|
||||
656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns
|
||||
(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master
|
||||
px from the band's outer edge (the app's own placement margin) and never outside the band. The
|
||||
default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right**
|
||||
spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the
|
||||
first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the
|
||||
other scenes keep their 16:9 placement until the creator drags them into the band.
|
||||
|
||||
**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**:
|
||||
the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button
|
||||
disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a
|
||||
locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The
|
||||
lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the
|
||||
16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile
|
||||
(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two
|
||||
(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top
|
||||
or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is
|
||||
picked (like Live) — otherwise the vertical chat stream has no camera at all.
|
||||
|
||||
**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section
|
||||
(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's
|
||||
geometry setters clamp it into the band, so a typed value can never push the camera out of the
|
||||
cropped window.
|
||||
|
||||
**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the
|
||||
maintainers' answer is "create a new empty scene and add the scene you want to send as source"
|
||||
(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no
|
||||
built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and
|
||||
plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored
|
||||
placement, resolved on the tier change, is the step past that.
|
||||
|
||||
**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier
|
||||
is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it
|
||||
lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both
|
||||
placements survive a save + reload. (The store writes the webcam rows only when the app has a
|
||||
webcam identity — the config rows hang off it — so the test supplies one.)
|
||||
`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked
|
||||
→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and
|
||||
cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back
|
||||
to 16:9 the desktop layout is restored exactly and the circle option is offered again.
|
||||
|
||||
**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam`
|
||||
broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a
|
||||
new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice.
|
||||
- **The Background is a locked per-screen layer:** exactly one per canonical scene, named
|
||||
"Background", pinned at index 0, never addable/removable/reorderable — Live is `DisplayCapture`,
|
||||
every other screen is static art. `NormalizeBackgrounds` heals any layout to that model on load.
|
||||
- **Never feed a capture row to the `ActiveBackgroundImage` overlay.** That layer is fed by
|
||||
static-art rows only; a flavor-blind `IsBackground` lookup there permanently covers the live
|
||||
desktop/game feed (the 2026-08-23 regression).
|
||||
- **`window:`/`picker:` capture keys are session-scoped and stripped on load** (a recycled HWND
|
||||
would capture the wrong window), and an explicit window pin beats auto game/desktop detection
|
||||
while that window is alive.
|
||||
- **The WinRT interop paths are settled:** read frame bytes with `WindowsRuntimeMarshal.TryGetDataUnsafe`
|
||||
(the `IMemoryBufferByteAccess` ComImport throws `Invalid cast` on every frame under CsWinRT) and
|
||||
build the D3D device via `CreateDirect3D11DeviceFromDXGIDevice` (the
|
||||
QI-for-`IDirect3DDxgiInterfaceAccess` trick returns E_NOINTERFACE on current Windows).
|
||||
- **`VideoFrame` + the capture interfaces are the seam** — nothing above the WinRT layer leaks WinRT
|
||||
types; capture is CPU-first (`MemoryPreference.Cpu`, `SharedReadOnly`), the reader-subtype ladder
|
||||
tries each candidate under its OWN catch (one rejected format must never abort acquisition), and
|
||||
there is deliberately **no fallback camera** — the user's pick is never overridden.
|
||||
- **One webcam identity app-wide, placements per scene.** `IsWebcamAttainable` needs a live lock, not
|
||||
just a saved id; the app keeps its own base ref, so removing every placement leaves the session up.
|
||||
- **`CameraManager` is the ONE background-effects processing point** — segment once per raw frame,
|
||||
render per-scene variants keyed by effect+strength, mask smoothed per camera session (0.75 new /
|
||||
0.25 previous). The raw path stays untouched so a slow model never stalls the webcam, and the
|
||||
bundled model's provenance is fixed (do not swap in PINTO #109).
|
||||
- **The vertical tier is geometry only:** `BeginVerticalLayout`/`EndVerticalLayout` park and swap the
|
||||
live placement, so the compositor and `PreviewPane.xaml` never learn about tiers, and the stored
|
||||
X/Y/W/H/ClipShape columns stay the **16:9** layout. The Chat camera is rect-only through the MODEL
|
||||
(`VerticalShapeLocked`), not merely through a hidden button.
|
||||
|
||||
### Rendering budget, measurement & the compositor — moved
|
||||
|
||||
|
||||
@@ -0,0 +1,484 @@
|
||||
# Capture — backdrop, webcam, background effects, and the vertical tier
|
||||
|
||||
> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches;
|
||||
> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md).
|
||||
|
||||
Every frame source in the app lands here: the locked Background layer (screen/game/window capture),
|
||||
the webcam seam and its sessions, the MediaPipe background effects, and the per-scene camera placement
|
||||
the vertical 9:16 tier uses. Content is the architecture plus the measured history that produced it;
|
||||
superseded designs are marked inline.
|
||||
|
||||
---
|
||||
|
||||
## The locked Background layer — screen, game, window capture
|
||||
|
||||
### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25
|
||||
|
||||
**The Background is a locked per-screen layer (TASK 25): exactly one per canonical
|
||||
scene, always named "Background", pinned at index 0, never addable/removable/reorderable,
|
||||
absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time
|
||||
desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back
|
||||
to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.**
|
||||
`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this
|
||||
model on every load: keeps the correctly-flavored row (or converts a survivor in place — note
|
||||
`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops
|
||||
duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0;
|
||||
non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()`
|
||||
(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads
|
||||
`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins).
|
||||
This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`,
|
||||
`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`.
|
||||
|
||||
- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey`
|
||||
(persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The Live row is
|
||||
a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`,
|
||||
internal static — also used by the `StagedScene` setter so a scene switch can never expose a
|
||||
missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
|
||||
the element template sets `IsHitTestVisible=false` for
|
||||
backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground`
|
||||
(persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls —
|
||||
"Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is
|
||||
staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row
|
||||
menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live
|
||||
screens keep the properties panel pill ("Use default background") + Browse for custom art.
|
||||
- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` /
|
||||
`BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
|
||||
- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above*
|
||||
`BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored
|
||||
Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows
|
||||
`DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for
|
||||
persistence/heal/minis — not this overlay.
|
||||
- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)`
|
||||
— fires on the UI thread whenever the foreground window changes. The callback runs the same
|
||||
`GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
|
||||
`GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own
|
||||
process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
|
||||
`ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
|
||||
`IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
|
||||
friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
|
||||
submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch
|
||||
`ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the
|
||||
full-screen game's
|
||||
monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey`
|
||||
values are cleared, no capture session is started, and `Source.DisplaySource` falls back to
|
||||
`_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of
|
||||
`ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears
|
||||
on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps
|
||||
running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to
|
||||
"Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual
|
||||
fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground
|
||||
changes.
|
||||
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
|
||||
`Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
|
||||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
|
||||
`WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
|
||||
`IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
|
||||
flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
|
||||
1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear —
|
||||
slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel
|
||||
version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take).
|
||||
Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks
|
||||
(slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a
|
||||
monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never
|
||||
overwrite a newer frame), and are spaced by a 10ms floor
|
||||
(`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for
|
||||
the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry).
|
||||
Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a
|
||||
buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a
|
||||
consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags)
|
||||
can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms,
|
||||
drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion
|
||||
failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black
|
||||
frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps
|
||||
running, the pool just stops delivering.
|
||||
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
|
||||
`WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
|
||||
events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
|
||||
a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
|
||||
`IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
|
||||
auto-detection.
|
||||
- **"Capture Window…" — in-app window pin (TASK 38, 2026-09-21):** the Live backdrop context menus
|
||||
(PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes
|
||||
(`Win32WindowEnumerator`, an `EnumWindows` pass filtered like `Win32FullScreenDetector`:
|
||||
visible, non-cloaked via `DWMWA_CLOAKED`, non-empty rect, other-process, titled). Picking one runs
|
||||
`RedesignateBackgroundAsync("window:0x{hwnd:X}")` — the same full-bleed layer-0 capture path as the
|
||||
game/desktop, keyed straight into `ScreenCaptureSourceFactory` (`window:` already resolved there).
|
||||
**Session-scoped, never persisted:** `window:` (and `picker:`) keys are `IsTransientCaptureKey` —
|
||||
`ReacquireScreenCaptures` strips them before auto-detection on every load, because an HWND can be
|
||||
recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong
|
||||
window). **Pin wins while alive:** `OnFullscreenMonitorChanged` skips re-designation while
|
||||
`IsAliveWindowPin(current.CaptureKey)` is true — the auto game/desktop detector never steals an
|
||||
explicit pin. **No dead ends:** if the pinned window's capture session dies mid-session,
|
||||
`OnScreenCaptureFailed` (transient key only) re-runs `ReacquireScreenCaptures` → auto-fallback
|
||||
(fullscreen game → desktop → static art). Seam: `IWindowEnumerator` + `WindowInfo`
|
||||
(`Label = "Title — Process"`), `WindowEnumeratorOverride` static test seam mirrors
|
||||
`CameraEnumeratorOverride`.
|
||||
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
|
||||
so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
|
||||
feature array) → QI `IDXGIDevice` → the WinRT interop export
|
||||
`CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
|
||||
process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
|
||||
no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
|
||||
WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
|
||||
`IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
|
||||
reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
|
||||
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
|
||||
task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and
|
||||
window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet).
|
||||
- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses
|
||||
focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture —
|
||||
the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never
|
||||
pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the
|
||||
app is in the background (worse under a full-screen game on 24H2/26100), GPU readback
|
||||
(`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending`
|
||||
gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as
|
||||
WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor)
|
||||
but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted
|
||||
yet (deferred by user decision).
|
||||
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
|
||||
to the encoder task.
|
||||
- **Preview watermark:** the "Preview" placeholder hides while a background capture renders —
|
||||
`ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with
|
||||
live capture shows the feed instead of the "nothing here" label.
|
||||
|
||||
---
|
||||
|
||||
## Webcam — capture seam, sessions, and background effects
|
||||
|
||||
### Webcam capture (TASK 3 milestone 1)
|
||||
|
||||
- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
|
||||
BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes;
|
||||
screen capture and background removal later feed the same seam.
|
||||
- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video,
|
||||
SharingMode = SharedReadOnly }`, frames pulled via
|
||||
`CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format
|
||||
conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via
|
||||
`WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies).
|
||||
- **Reader-output subtype ladder (2026-09-15 webcam-take fix):** the reader is created per-candidate
|
||||
by `ReaderSubtypeCandidates(activeSubtype)` — Bgra8 for uncompressed cameras (fast path), NV12 then
|
||||
source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as
|
||||
Bgra8 by a `MediaFrameReader` (`StartAsync` → `OutputFormatNotSupported`; the MJPG pipeline exposes
|
||||
NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate
|
||||
(our `SetMediaStreamPropertiesAsync` refuses with "file in use / CaptureMode is SharedReadOnly" under
|
||||
msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition.
|
||||
Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype
|
||||
with a THROW (`E_INVALIDARG`), not a status; one rejected format must never abort the ladder. Failures
|
||||
degrade to the next candidate; rejections are logged + collected into the final error; only the outer
|
||||
catch nets device-level errors (see `MyMistakes.md` → WINRT RESOURCE-ALLOCATION RECIPE). Anything not
|
||||
Bgra8 is converted in `OnFrameArrived`, which already handled non-BGRA software bitmaps.
|
||||
- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview`
|
||||
(preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the
|
||||
failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture
|
||||
coexist with other apps that share the camera.
|
||||
- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync`
|
||||
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
|
||||
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
|
||||
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
|
||||
Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
|
||||
inside the physical camera's own driver, so it never appears as a selectable device. Only tools
|
||||
that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
|
||||
the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
|
||||
bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
|
||||
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
|
||||
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
|
||||
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
|
||||
- **One webcam, many scenes (schema v3) — app-level default (2026-09-17):** a singleton `Webcam`
|
||||
row holds the identity (`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig`
|
||||
(position/size/clip/mirror/border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at
|
||||
most once, the webcam (`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager`
|
||||
refcounts capture sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps
|
||||
it; the last release stops + disposes). **The webcam is an app-level resource — ONE selection (the
|
||||
app default), usable in every scene that can host it.** "Add Webcam" places the existing default
|
||||
DIRECTLY (no picker); the Windows camera picker runs only for the initial selection
|
||||
(`_webcam == null`). Choosing a different camera in the picker swaps it app-wide via
|
||||
`SwapWebcamIdentityAsync` (same path "Change Webcam…" uses). **Per-scene gate, not app-wide**
|
||||
(TASK 26 superseded by creator directive 2026-09-17): `CanAddWebcam` = `StagedScene is { WebcamConfig: null } && IsWebcamAttainable`
|
||||
— a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene
|
||||
holding one does not, so Chat's camera never greys Live. **Attainable means a live lock, not just a
|
||||
saved identity:** `IsWebcamAttainable` = `_webcam != null && CameraManager.IsRunning(_webcam.DeviceId)`
|
||||
— an identity whose device was unplugged, or whose session can't start, leaves the row greyed with
|
||||
`WebcamAddToolTip` = "No webcam is currently available…" (identity alone no longer suffices). The
|
||||
max-1 reason gets the gentle "a second camera means you've graduated to OBS" line.
|
||||
Removing a scene's placement **keeps the identity** (it's the app default; `_webcam` is never
|
||||
nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock
|
||||
(below).
|
||||
- **Startup resource lifecycle (2026-09-17):** the webcam is a resource the app *allocates and
|
||||
locks*, validated at startup — this answers "why is the WebCam row greyed" from inside the app
|
||||
instead of a DB spelunk. `MainViewModel.ValidateWebcamResourceStartupAsync` (fired fire-and-forget
|
||||
right after `LoadLayout`, stored as `WebcamStartupValidationTask` so tests await it) enumerates the
|
||||
OS once and tri-states on the count: **0** → app runs on, layer inactive, no alarm; **1** → attempt
|
||||
`CameraManager.AcquireAsync(device)` as an app-wide lock — on failure a **persistent red alert**
|
||||
(`WebcamLockAlert`, bottom of the Layers panel with a Retry button) re-polls every 5 s
|
||||
(`_webcamLockPollTimer`) until a lock succeeds, and any first real frame also clears it; **≥2** →
|
||||
deliberately no auto-lock — selection belongs to the App Settings dialog (gear) **queued — TASK 40
|
||||
Unit A (App Settings round, plan saved 2026-09-22)**.
|
||||
The single-camera branch **always acquires** (it no longer skips when configs already hold the
|
||||
session): that acquire is the app's own BASE ref, so removing every scene's placement leaves
|
||||
`RefCount = 1` and the session alive — the gate stays satisfied and the Web Cam row stays offered,
|
||||
i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps
|
||||
the refcount. `CameraManager.IsRunning(deviceId)` = a session exists (started OR still starting —
|
||||
a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves
|
||||
the dictionary so the pass can retry. After a successful single-cam lock, the pass **adopts the
|
||||
camera as the app default** if no identity exists yet (fresh layout) — the Web Cam layer is then
|
||||
offered in Live/Chat immediately (`CanAddWebcam` true); an already-loaded identity is never
|
||||
overwritten. Test seams mirror `LayoutPathOverride`:
|
||||
`CameraEnumeratorOverride`/`CameraFrameSourceFactoryOverride` let `WebcamStartupResourceTests`
|
||||
drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail
|
||||
→ alert → clears on retry). **Attribution correction:** the "NVIDIA Broadcast opens the webcam exclusively" failure
|
||||
below is a *suspect-list* claim — `CameraConflictProbe` reads only running process names, no device
|
||||
handles; contention symptoms fit shared-mode/bandwidth just as well (see `MyMistakes.md`).
|
||||
The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27):
|
||||
`CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on
|
||||
staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created
|
||||
before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) —
|
||||
LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
|
||||
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
|
||||
snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
|
||||
restores them when switching back — otherwise the Round resize lock (square) would leave a square
|
||||
behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a
|
||||
reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time
|
||||
`HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9
|
||||
(keeps height; Round and explicit rect dims are untouched).
|
||||
- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load)
|
||||
release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and
|
||||
stops the source regardless of how many scenes held it (the per-config count isn't known once the
|
||||
scenes are replaced) — then `AcquireAsync` the new device once per config.
|
||||
- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread
|
||||
at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via
|
||||
`PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
|
||||
(at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
|
||||
device never drowns the render thread.
|
||||
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
|
||||
frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
|
||||
exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
|
||||
webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
|
||||
current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
|
||||
`AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
|
||||
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
|
||||
(`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
|
||||
0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
|
||||
Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click
|
||||
context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist,
|
||||
rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
|
||||
this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
|
||||
preview canvas has its own Show Webcam entry.
|
||||
- The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
|
||||
(`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
|
||||
renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
|
||||
the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
|
||||
geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
|
||||
frame on the render thread, which is what made the live webcam crawl while round. The Round border is
|
||||
a centered `Ellipse` at `Width/Height = RoundBorderSize`.
|
||||
- Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
|
||||
- **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
|
||||
max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
|
||||
**by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
|
||||
may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
|
||||
(`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
|
||||
`RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
|
||||
- **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
|
||||
the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected
|
||||
the element — making the corner handle ungrabbable. The element Grid carries
|
||||
`Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for
|
||||
right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is
|
||||
`IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work:
|
||||
(1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu`
|
||||
isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the
|
||||
IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem`
|
||||
template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness
|
||||
sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow
|
||||
driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never
|
||||
opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the
|
||||
Click handlers were fine.
|
||||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||||
to the encoder task.
|
||||
- **Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51).** The
|
||||
2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix,
|
||||
and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe,
|
||||
run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video
|
||||
Effects SDK stays **ruled out** (RTX-only + ~1 GB user-side download). See "Webcam background
|
||||
effects" below and `TASKS/task-51-webcam-bg-effects.md`.
|
||||
|
||||
### Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)
|
||||
|
||||
- **Model:** Google's **MediaPipe Selfie Segmenter, landscape** (Apache-2.0, model card 2021-05-06),
|
||||
official file `storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/
|
||||
float16/latest/selfie_segmenter_landscape.tflite`. Converted to ONNX by our own
|
||||
`tools/selfie-seg/convert_tflite_to_onnx.py` (the graph ends in MediaPipe's custom op
|
||||
`Convolution2DTransposeBias`, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct
|
||||
translation was smaller than any workaround). **Verified against Google's own runtime:**
|
||||
`verify_against_mediapipe.py` feeds the same frame to MediaPipe's `ImageSegmenter` (original
|
||||
.tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images.
|
||||
⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance).
|
||||
obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09),
|
||||
but none of its code is used; if any is ever borrowed, it needs its copyright notice kept.
|
||||
- **Contract:** input `[1,144,256,3]` RGB /255 NHWC → output `[1,144,256,1]` person probability. The
|
||||
whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
|
||||
- **Shipping:** `Assets/Models/selfie_segmenter_landscape.onnx` is an **EmbeddedResource** (in git,
|
||||
inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime =
|
||||
`Microsoft.ML.OnnxRuntime` 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native
|
||||
`onnxruntime.dll` is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX
|
||||
publish (TASK 48) must pin the RID so only win-x64 ships.
|
||||
- **Code (`Services/BackgroundEffects/`):** `IPersonSegmenter` seam (`Segment(VideoFrame) →
|
||||
SegmentationMask?`, tests inject fakes); `OnnxPersonSegmenter` (lazy session from the embedded
|
||||
resource; **never throws** — a load/run failure logs once and returns null forever after);
|
||||
`BackgroundEffectProcessor.Apply(frame, BackgroundEffect)` — `None` = same instance (true
|
||||
passthrough), `Remove` = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched,
|
||||
`IsOpaque = false` (the compositor's straight-alpha blend shows the layers below), strength =
|
||||
edge feather (mask band 0.5±(0.05+0.45·s) → alpha); `Blur` (slice 2) = background downscaled 4×
|
||||
by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person
|
||||
pixels (mask = 1) are copied
|
||||
untouched, opacity preserved. **Blur radius is quadratic** (`BlurRadius(strength)`: 1→6 small px,
|
||||
default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was
|
||||
far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author
|
||||
stops downscale-based blurs around 8 px). The mask is computed once per frame (`TrySegment`) and
|
||||
rendered per effect (`Render`), so several variants share one inference. No mask / segmenter
|
||||
exception → passthrough + one log line, never a dropped layer.
|
||||
- **Config (slice 3):** per scene, on `WebcamSceneConfig` — `BackgroundEffect` + `BackgroundStrength`
|
||||
(0..1, default 0.5), persisted as two additive guarded columns on `WebcamSceneConfig`
|
||||
(`BackgroundEffect TEXT DEFAULT 'None'`, `BackgroundStrength REAL DEFAULT 0.5`, no version bump —
|
||||
the TASK 47 convention). `RemoveBackground`/`BlurBackground` are the radio-pill bools the UI binds.
|
||||
- **Hookup (slice 4) — the ONE processing point is `CameraManager`.** Effects are per scene but the
|
||||
camera session is shared, so the manager renders **variants** keyed by `WebcamEffectKey(effect,
|
||||
strength rounded to 0.01)`. `MainViewModel.SyncWebcamEffects()` declares the union of every
|
||||
config's key (`SetRequestedEffects`; unused variants are dropped so their CPU stops) and points each
|
||||
config's `VideoImageSource` at its variant's bitmap; it re-runs on load, add, remove, device swap,
|
||||
and any `BackgroundEffect`/`BackgroundStrength` change (configs are watched via `PropertyChanged`,
|
||||
which also `ScheduleSave`s). Per raw frame a background worker (latest-frame-drop, one at a time)
|
||||
segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver:
|
||||
`GetLatestFrame(device, key)`) and WriteableBitmap (`EffectPreviewBitmapChanged` on first
|
||||
creation). **The raw path is untouched** — a slow/failed model never stalls the plain webcam — and
|
||||
an effect variant shows **nothing** until its first processed frame (Remove never flashes the
|
||||
real room). The compositor needed no change: its sampler already honours source alpha.
|
||||
- **Temporal smoothing (2026-10-09, creator check: "removal is jittery"):** the segmenter classifies
|
||||
every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix:
|
||||
`MaskSmoother` blends each raw mask with the previous smoothed one — **0.75 × new + 0.25 × previous**
|
||||
(`CurrentWeight`, internal const). State is **per `CameraSession`** (one per-camera worker), NOT in
|
||||
the segmenter — the `OnnxPersonSegmenter` is shared app-wide and its bundled-model contract test
|
||||
pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any
|
||||
size change pass through un-blended; the returned mask never aliases the caller's array and two
|
||||
buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights
|
||||
come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests
|
||||
`temporal_smooth_factor` at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080)
|
||||
found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a
|
||||
smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6
|
||||
small px (see the `Blur` bullet above).
|
||||
- **Perf (synthetic 1280×720, Debug build):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame
|
||||
(Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the
|
||||
bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp
|
||||
per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. `CameraManager` logs the real
|
||||
average once per session after 120 processed frames (`startup.log`: "background effects avg …").
|
||||
`MainViewModel.PersonSegmenterOverride` is the hermetic-test seam.
|
||||
- **UI (slice 5, creator spec 2026-10-09):** the webcam's properties in the Layers panel
|
||||
(`Controls/LeftPanel.xaml`) gain a **Background** section — two `PillToggle` switches, Remove and
|
||||
Blur, bound to `RemoveBackground`/`BlurBackground` (radio-exclusive: one on turns the other off,
|
||||
turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an
|
||||
effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam:
|
||||
a new strength variant inherits the dropped same-effect variant's last frame + bitmap
|
||||
(`CameraManager.SyncVariants`), and the VM keeps the shown bitmap until the new one renders.
|
||||
- **Tests:** `BackgroundEffectEngineTests` — the Remove integration test (fake segmenter), and a
|
||||
model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and
|
||||
matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test
|
||||
image `mozart_square.jpg`, public-domain painting, in `ytLive.Tests/Fixtures/`). `CameraManagerTests`
|
||||
— the ONE integration test for the jitter fix (a `SteppingMaskSegmenter` whose two consecutive
|
||||
masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would
|
||||
have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0).
|
||||
`Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the quadratic radius mapping
|
||||
(`BlurRadius(0.5)` = 2, `BlurRadius(1)` = 6) and the behaviour at a step edge: default smears
|
||||
light ~9px into the dark half vs ~26px at full strength.
|
||||
|
||||
---
|
||||
|
||||
## The vertical (9:16) tier — per-scene camera placement
|
||||
|
||||
### Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21)
|
||||
|
||||
**The problem.** The vertical tier (1080×1920) is a centred **band crop** of the SAME 1920×1080
|
||||
master: `OutputRect*` = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default
|
||||
webcam sits at x≈1408) lands **outside the cropped window entirely** — in the vertical stream the
|
||||
creator is simply not in the picture.
|
||||
|
||||
**The rule (creator's locked table, 2026-10-17).** Each scene keeps its **own** vertical
|
||||
placement — position, size and shape — and switching back to 16:9 restores the desktop layout
|
||||
exactly:
|
||||
|
||||
| mode | shape | size | position |
|
||||
|---|---|---|---|
|
||||
| 16:9 | as today | as today | as today |
|
||||
| **vv · Live** | circle **or** rect (the creator's choice, per scene) | small tile | one of **six** spots: top/bottom × left/centre/right |
|
||||
| vv · Chat | **rect only** (the circle option is not offered) | the small tile (normal), or enlarged to fill one 607×540 area | the same six spots; enlarged fills the top or the bottom area |
|
||||
| Starting / BRB / Ending | unchanged | unchanged | unchanged |
|
||||
|
||||
**Design — geometry only, no render-path change.** The preview and the compositor both read the
|
||||
element's OWN `X`/`Y`/`Width`/`Height` + `ClipShape`, so the tier switch **swaps the live
|
||||
geometry** instead of teaching the render path about tiers:
|
||||
|
||||
- `WebcamSceneConfig` carries the vertical placement (`VerticalX/Y/Width/Height/VerticalClipShape`
|
||||
+ `VerticalRectWidth/Height` — seven additive DB columns, NULL = "this scene has no vertical
|
||||
placement yet") and the **parked** 16:9 layout (`LandscapeX/…/LandscapeRectHeight`, transient).
|
||||
- `BeginVerticalLayout(...)` parks the live 16:9 layout and brings the vertical one in;
|
||||
`EndVerticalLayout()` commits the vertical placement back into the stored slot and restores the
|
||||
parked 16:9 layout. The live geometry IS the render input → **the compositor and
|
||||
`PreviewPane.xaml` are untouched**.
|
||||
- The save keeps the X/Y/W/H/ClipShape columns as the **16:9** layout (while the vertical tier is
|
||||
live, the parked values are written there and the live ones into the `Vertical*` columns).
|
||||
- `SceneElement.X/Y/Width/Height` became **virtual** so the webcam can CONSTRAIN every write
|
||||
(drag, resize handle, properties fields) into the band — one choke point instead of three
|
||||
writers re-implementing the rule. Outside the band = invisible in the vertical stream.
|
||||
|
||||
**The band and the six spots** (`Models/VerticalCameraLayout.cs`, pure math): the band is
|
||||
656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns
|
||||
(left/centre/right) → six spots. A tile snaps to the spot its **centre** falls in, inset 32 master
|
||||
px from the band's outer edge (the app's own placement margin) and never outside the band. The
|
||||
default tile is **270×152 master px** (480×270 in the 1080×1920 output) in the **bottom-right**
|
||||
spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the
|
||||
first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the
|
||||
other scenes keep their 16:9 placement until the creator drags them into the band.
|
||||
|
||||
**The Chat scene's own rules (Unit 2).** In the vertical tier the Chat camera is **rect only**:
|
||||
the circle is not *offered* (`MainViewModel.CanToggleClipShape` is false → the Shape button
|
||||
disappears) and the model refuses it too — `WebcamSceneConfig.ClipShape` is **overridden**, so a
|
||||
locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The
|
||||
lock is `WebcamSceneConfig.VerticalShapeLocked`, set by the VM per scene and cleared **before** the
|
||||
16:9 layout comes back, so the desktop shape is restored. Its **size is discrete**: the small tile
|
||||
(normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two
|
||||
(`VerticalCameraLayout.SizeFor`), and an enlarged camera **fills the area its centre is in** (top
|
||||
or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is
|
||||
picked (like Live) — otherwise the vertical chat stream has no camera at all.
|
||||
|
||||
**Manual entry** (creator rule): the webcam's Layers properties gained a **Geometry** section
|
||||
(X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's
|
||||
geometry setters clamp it into the band, so a typed value can never push the camera out of the
|
||||
cropped window.
|
||||
|
||||
**Derivative work.** OBS has no built-in way to store a scene's layout per aspect ratio — the
|
||||
maintainers' answer is "create a new empty scene and add the scene you want to send as source"
|
||||
(obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no
|
||||
built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and
|
||||
plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored
|
||||
placement, resolved on the tier change, is the step past that.
|
||||
|
||||
**Tests.** `VerticalWebcamLayoutTests` — the ONE integration test for Unit 1: the vertical tier
|
||||
is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it
|
||||
lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both
|
||||
placements survive a save + reload. (The store writes the webcam rows only when the app has a
|
||||
webcam identity — the config rows hang off it — so the test supplies one.)
|
||||
`VerticalChatCameraLayoutTests` — the ONE integration test for Unit 2: the vertical tier is picked
|
||||
→ the Chat camera comes into the band as a small RECT tile → the circle option is not offered and
|
||||
cannot be set even on the model → dragged up and enlarged it fills the **top** area exactly → back
|
||||
to 16:9 the desktop layout is restored exactly and the circle option is offered again.
|
||||
|
||||
**Known trap (see `MyMistakes.md`):** a second properties panel bound to `SelectedElement.IsWebcam`
|
||||
broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a
|
||||
new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice.
|
||||
|
||||
---
|
||||
|
||||
## Related in [`ai.md`](../../ai.md)
|
||||
|
||||
- **Resolution tiers** — `OutputRect*` and the tier list (the vertical tier's 607×1080 band).
|
||||
- **Key patterns** — MVVM/binding conventions the capture UI follows.
|
||||
- **Tests** — hermetic-real-window rules that camera tests must obey.
|
||||
- **Run** — launch/take procedure and the Release-only rule for measurements.
|
||||
@@ -10,6 +10,7 @@ Read the file whose row matches the task. There is no required reading order.
|
||||
| File | Covers | Read when |
|
||||
|------|--------|-----------|
|
||||
| [`frame-pipeline.md`](frame-pipeline.md) | Render budget & measurement (`FrameCadence`, per-element composite timing, scene dump), `SceneCompositor` design + `StripPool`, and `FramePump` from scene snapshot to encoder (pacing, pool/paste caches, backpressure, teardown, health stats) | Touching the compositor, the render loop, per-frame cost, `Services/Encoder/*`, or debugging dropped/juddering frames |
|
||||
| [`capture.md`](capture.md) | The locked Background layer (screen/game/window capture, WinRT GraphicsCapture interop), the webcam seam + `CameraManager` sessions, MediaPipe background effects, and the vertical 9:16 tier's per-scene camera placement | Touching any frame source, the Background row, camera acquisition/conflicts, background effects, or vertical-tier camera geometry |
|
||||
|
||||
## Split-out yet to come
|
||||
|
||||
|
||||
Reference in New Issue
Block a user