LlamaCasty/docs/ai/capture.md

Capture — backdrop, webcam, background effects, and the vertical tier

Split out of ai.md (2026-10-26) so a session reads only what its task touches; ai.md keeps a short invariant stub for this file. Directory map: index.md.

Every frame source in the app lands here: the locked Background layer (screen/game/window capture), the webcam seam and its sessions, the MediaPipe background effects, and the per-scene camera placement the vertical 9:16 tier uses. Content is the architecture plus the measured history that produced it; superseded designs are marked inline.

---

The locked Background layer — screen, game, window capture

Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25

The Background is a locked per-screen layer (TASK 25): exactly one per canonical scene, always named "Background", pinned at index 0, never addable/removable/reorderable, absent from the (+) menu. Flavor is decided by scene: Live = DisplayCapture (the real-time desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back to live-background.jpg); Starting/BRB/Chat/Ending = static Background art layers. MainViewModel.NormalizeBackgrounds(scenes) (internal static) heals any loaded layout to this model on every load: keeps the correctly-flavored row (or converts a survivor in place — note Source.Type's setter derives IsBackground, so conversions must re-assert the flag), drops duplicates/stale rows, seeds missing ones via CreateBackground(name), renames, moves to index 0; non-canonical scenes get HasBackground=false and lose any backgrounds. HealBackgrounds() (instance) runs Normalize + stamps default art (StampDefaultBackgroundAsset reads Assets/{scene}-background.jpg pack resources through AddAsset; custom Browse art wins). This replaces the old five-seeder cluster (Seed{Starting,Brb,Ending,Chat}Background, SeedLiveBackgroundAsset, EnsureDefaultBackground) and EnforceBackgroundPolicy.

• Model (schema v6): Source.IsBackground (persisted) marks the one Background per scene; Source.CaptureKey (persisted) names the target — monitor:<n>, window:<hwnd>, or picker:<displayname>. The Live row is a real Source of Type DisplayCapture, inserted first (MainViewModel.EnsureBackground(scene), internal static — also used by the StagedScene setter so a scene switch can never expose a missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in RemoveElement; the element template sets IsHitTestVisible=false for backgrounds; the row's remove button and "Remove Source" menu item are hidden). Scene.HasBackground (persisted) is true for all five canonical scenes (see SceneCatalog.HasBackground). Capture controls — "Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the Live scene is staged (CanChangeBackground = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row menu MultiBindings it with the row's own IsBackground via Helpers/AllTrueToVisibilityConverter). Non-Live screens keep the properties panel pill ("Use default background") + Browse for custom art.
• Settings hygiene: LayoutStore.Save purges orphaned BackgroundUseDefault_{id} / BackgroundPath_{id} Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
• Layering invariant (regression lesson, 2026-08-23): ActiveBackgroundImage renders above BackgroundImage, so it is fed by static-art rows only (Type == Background). The capture-flavored Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows DisplaySource (capture, falling back to its AssetId art). Flavor-blind IsBackground lookups are for persistence/heal/minis — not this overlay.
• Detection (event-driven): Win32FullScreenDetector hooks SetWinEventHook(EVENT_SYSTEM_FOREGROUND) — fires on the UI thread whenever the foreground window changes. The callback runs the same GetForegroundWindow + DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) + MonitorFromWindow + GetMonitorInfo check; a window is full-screen when its frame covers all four monitor edges; own process is excluded; monitor index = EnumDisplayMonitors enumeration order (the same order ScreenCaptureSourceFactory uses to map index→HMONITOR via Win32FullScreenDetector.GetMonitorHandle). IFullScreenDetector also exposes GetDisplays() (DisplayInfo: index/name/resolution/bounds/IsPrimary, friendly name via EnumDisplayDevices) + PrimaryMonitorIndex() for the in-app "Capture Display" submenu + FullscreenMonitorChanged event + StartWatching()/StopWatching(). At launch ReacquireScreenCaptures heals backgrounds (HealBackgrounds()), then keys the Live background to the full-screen game's monitor; when no game is detected, ResolveAutoCaptureKey() returns null — stale CaptureKey values are cleared, no capture session is started, and Source.DisplaySource falls back to _imageSource (the static asset on the backdrop element). StartWatching() is called at the end of ReacquireScreenCaptures. OnFullscreenMonitorChanged auto-redesignates when a fullscreen game appears on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps running so loading screens / transitions don't break the stream. RefreshBackdropAutoCapture (bound to "Refresh Capture" context menu + Activated handler) reads the current foreground directly as a manual fallback. Deduplication: _lastReportedMonitor prevents redundant redesignates from rapid foreground changes.
• Capture (WinRT GraphicsCapture): ScreenCaptureFrameSource creates a free-threaded Direct3D11CaptureFramePool (2 buffers, B8G8R8A8UIntNormalized) + GraphicsCaptureSession; frames → SoftwareBitmap.CreateCopyFromSurfaceAsync (alpha ignored) → VideoFrame (BGRA8), bytes read via WindowsRuntimeMarshal.TryGetDataUnsafe (the same CsWinRT-safe read the webcam path uses) — the IMemoryBufferByteAccess ComImport cast threw Invalid cast on every frame under CsWinRT, which flooded startup.log (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the 1920×1080 master are downscaled to the master (DownscaleBgra, integer 8.8 fixed-point bilinear — slice 16: the same two-stage math as SceneCompositor.Bilinear; the previous double-per-pixel version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take). Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks (slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a monotonic LatestFrame publish gate (MonotonicGate: a slow OLDER completion can never overwrite a newer frame), and are spaced by a 10ms floor (MinConvertInterval): the monitor delivers at the 240Hz DWM cadence (~4.2ms), far too fast for the ~60/s the pump can use, so the extra arrivals are dropped (skip busy/skip cadence telemetry). Hand-out buffers come from a reuse-distance ring (FrameRingBuffer, depth 8, redLine 4): a buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a consumer still holds (session.LatestFrame survives conversions; the dispatcher preview copy lags) can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms, drops, ring allocs) is logged to startup.log every 2s while converting. Per-frame conversion failures are logged at most once per 5 s (ErrorLogThrottle). DRM-protected content delivers black frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps running, the pool just stops delivering.
• Ownership: ScreenCaptureManager mirrors CameraManager — refcounted by target key, one shared WriteableBitmap per key, dispatcher-coalesced latest-frame copies, PreviewBitmapChanged/CaptureFailed events, ReleaseAllAsync on re-designation. ScreenCaptureSourceFactory.Resolve(key) parses the key into a source; PickAsync() shows the OS GraphicsCapturePicker ("Change Capture…", owner window set via the IInitializeWithWindow ComImport) and returns a transient picker: key — a reload falls back to auto-detection.
• "Capture Window…" — in-app window pin (TASK 38, 2026-09-21): the Live backdrop context menus (PreviewPane canvas + layer-row) offer a submenu listing visible top-level windows of other processes (Win32WindowEnumerator, an EnumWindows pass filtered like Win32FullScreenDetector: visible, non-cloaked via DWMWA_CLOAKED, non-empty rect, other-process, titled). Picking one runs RedesignateBackgroundAsync("window:0x{hwnd:X}") — the same full-bleed layer-0 capture path as the game/desktop, keyed straight into ScreenCaptureSourceFactory (window: already resolved there). Session-scoped, never persisted: window: (and picker:) keys are IsTransientCaptureKey — ReacquireScreenCaptures strips them before auto-detection on every load, because an HWND can be recycled to an unrelated window after restart (resurrecting a dead pin could capture the wrong window). Pin wins while alive: OnFullscreenMonitorChanged skips re-designation while IsAliveWindowPin(current.CaptureKey) is true — the auto game/desktop detector never steals an explicit pin. No dead ends: if the pinned window's capture session dies mid-session, OnScreenCaptureFailed (transient key only) re-runs ReacquireScreenCaptures → auto-fallback (fullscreen game → desktop → static art). Seam: IWindowEnumerator + WindowInfo (Label = "Title — Process"), WindowEnumeratorOverride static test seam mirrors CameraEnumeratorOverride.
• CsWinRT projection gaps hand-rolled: Windows.Graphics.Direct3D11.Direct3D11Helper is not projected, so Direct3D11Helper P/Invokes d3d11.dll!D3D11CreateDevice (hardware, BGRA_SUPPORT, explicit 11.1-first feature array) → QI IDXGIDevice → the WinRT interop export CreateDirect3D11DeviceFromDXGIDevice → MarshalInterface<IDirect3DDevice>.FromAbi (one shared device per process). Do not switch back to the QI-for-IDirect3DDxgiInterfaceAccess trick: the raw D3D11 device no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and WARP alike) while CreateDirect3D11DeviceFromDXGIDevice keeps working. IInitializeWithWindow are ComImports in CaptureInterop.cs. All WinRT projections were verified by reflection against the built Microsoft.Windows.SDK.NET.dll before writing the interop.
• Known v1 limits: full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder task); multi-monitor live re-targeting beyond the auto-detected game is behind the picker; picker- and window-pin captures don't survive reload (session-scoped by design — see the "Capture Window…" bullet).
• Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred): when the app window loses focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture — the only focus hooks (FullscreenMonitorChanged event + RefreshBackdropAutoCapture) re-target the backdrop, never pace it. The lag is inherited: Windows.Graphics.Capture is content-driven and DWM/OS-throttled while the app is in the background (worse under a full-screen game on 24H2/26100), GPU readback (CreateCopyFromSurfaceAsync) contends with the foreground game, the source's one-in-flight _framePending gate drops frames during stalls, and the manager's DispatcherPriority.Render copies only run as fast as WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor) but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted yet (deferred by user decision).
• GPU posture: same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred to the encoder task.
• Preview watermark: the "Preview" placeholder hides while a background capture renders — ShowPreviewPlaceholder also checks BackgroundImage (raised on change), so a scene with live capture shows the feed instead of the "nothing here" label.

---

Webcam — capture seam, sessions, and background effects

Webcam capture (TASK 3 milestone 1)

• Seam-first: everything above the WinRT layer speaks only VideoFrame (normalized tightly-packed BGRA8) + CameraDeviceInfo/ICameraEnumerator/ICameraFrameSource interfaces. Tests inject fakes; screen capture and background removal later feed the same seam.
• CPU-first: MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video, SharingMode = SharedReadOnly }, frames pulled via CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8) — the pipeline does any format conversion, so every FrameArrived yields a ready BGRA8 SoftwareBitmap (bytes read via WindowsRuntimeMarshal.TryGetDataUnsafe, not marshalled copies).
• Reader-output subtype ladder (2026-09-15 webcam-take fix): the reader is created per-candidate by ReaderSubtypeCandidates(activeSubtype) — Bgra8 for uncompressed cameras (fast path), NV12 then source-default for cameras sitting in MJPG. A camera another app left in MJPG mode cannot be read as Bgra8 by a MediaFrameReader (StartAsync → OutputFormatNotSupported; the MJPG pipeline exposes NV12, not BGRA) — exactly what a browser grabbing the webcam produces. When the app can't re-negotiate (our SetMediaStreamPropertiesAsync refuses with "file in use / CaptureMode is SharedReadOnly" under msedge + NVIDIA Broadcast contention), an MJPG-active reader otherwise aborts the whole acquisition. Each candidate is allocated AND started under its OWN catch — WinRT answers an unsupported subtype with a THROW (E_INVALIDARG), not a status; one rejected format must never abort the ladder. Failures degrade to the next candidate; rejections are logged + collected into the final error; only the outer catch nets device-level errors (see MyMistakes.md → WINRT RESOURCE-ALLOCATION RECIPE). Anything not Bgra8 is converted in OnFrameArrived, which already handled non-BGRA software bitmaps.
• Source pick, not first hit: the frame reader is bound to the first source that is VideoPreview (preferred) or VideoRecord, not blindly the first preview source. If a camera exposes neither, the failure names the device and the stream types it does expose. SharedReadOnly lets the capture coexist with other apps that share the camera.
• Known failure: NVIDIA Broadcast — it opens the physical webcam exclusively, so InitializeAsync fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix: quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the app captures the processed feed. This is a real-device finding (Logitech VID_046D&PID_082D). Correction (2026-10-05): Windows Studio Effects is NOT a second option of this kind — it runs inside the physical camera's own driver, so it never appears as a selectable device. Only tools that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not bypass the user's explicit camera choice, so there is deliberately no fallback-camera setting.
• TFM: net8.0-windows10.0.19041.0 (app + tests) pulls the WinRT projection from the SDK reference packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy camera toggle still applies). EnableWindowsTargeting keeps WSL builds working.
• One webcam, many scenes (schema v3) — app-level default (2026-09-17): a singleton Webcam row holds the identity (Id/DeviceId/Name); each scene gets its own WebcamSceneConfig (position/size/clip/mirror/border/IsVisible). Scene.Elements holds images (Source) and, at most once, the webcam (WebcamSceneConfig); Scene.WebcamConfig is the accessor. CameraManager refcounts capture sessions by DeviceId (a session starts at RefCount = 1; repeat acquire bumps it; the last release stops + disposes). The webcam is an app-level resource — ONE selection (the app default), usable in every scene that can host it. "Add Webcam" places the existing default DIRECTLY (no picker); the Windows camera picker runs only for the initial selection (_webcam == null). Choosing a different camera in the picker swaps it app-wide via SwapWebcamIdentityAsync (same path "Change Webcam…" uses). Per-scene gate, not app-wide (TASK 26 superseded by creator directive 2026-09-17): CanAddWebcam = StagedScene is { WebcamConfig: null } && IsWebcamAttainable — a scene holding its own placement greys the row (one webcam per stream), but ANOTHER scene holding one does not, so Chat's camera never greys Live. Attainable means a live lock, not just a saved identity: IsWebcamAttainable = _webcam != null && CameraManager.IsRunning(_webcam.DeviceId) — an identity whose device was unplugged, or whose session can't start, leaves the row greyed with WebcamAddToolTip = "No webcam is currently available…" (identity alone no longer suffices). The max-1 reason gets the gentle "a second camera means you've graduated to OBS" line. Removing a scene's placement keeps the identity (it's the app default; _webcam is never nulled by removal) — Add stays offered for the same camera, because the app holds its own base lock (below).
• Startup resource lifecycle (2026-09-17): the webcam is a resource the app allocates and locks, validated at startup — this answers "why is the WebCam row greyed" from inside the app instead of a DB spelunk. MainViewModel.ValidateWebcamResourceStartupAsync (fired fire-and-forget right after LoadLayout, stored as WebcamStartupValidationTask so tests await it) enumerates the OS once and tri-states on the count: 0 → app runs on, layer inactive, no alarm; 1 → attempt CameraManager.AcquireAsync(device) as an app-wide lock — on failure a persistent red alert (WebcamLockAlert, bottom of the Layers panel with a Retry button) re-polls every 5 s (_webcamLockPollTimer) until a lock succeeds, and any first real frame also clears it; ≥2 → deliberately no auto-lock — selection belongs to the App Settings dialog (gear) queued — TASK 40 Unit A (App Settings round, plan saved 2026-09-22). The single-camera branch always acquires (it no longer skips when configs already hold the session): that acquire is the app's own BASE ref, so removing every scene's placement leaves RefCount = 1 and the session alive — the gate stays satisfied and the Web Cam row stays offered, i.e. the app default outlives the scenes that render it. Acquire on a running session just bumps the refcount. CameraManager.IsRunning(deviceId) = a session exists (started OR still starting — a mid-start session is not "locked yet", it's in flight), and a rolled-back (failed) session leaves the dictionary so the pass can retry. After a successful single-cam lock, the pass adopts the camera as the app default if no identity exists yet (fresh layout) — the Web Cam layer is then offered in Live/Chat immediately (CanAddWebcam true); an already-loaded identity is never overwritten. Test seams mirror LayoutPathOverride: CameraEnumeratorOverride/CameraFrameSourceFactoryOverride let WebcamStartupResourceTests drive the probe without real hardware (0-cam no-alarm, 1-cam locked → identity adopted, lock-fail → alert → clears on retry). Attribution correction: the "NVIDIA Broadcast opens the webcam exclusively" failure below is a suspect-list claim — CameraConflictProbe reads only running process names, no device handles; contention symptoms fit shared-mode/bandwidth just as well (see MyMistakes.md). The YouTube Chat layer follows the same one-per-layout rule (TASK 27): CanAddYouTubeChat greys out the (+) item while any scene carries a ChatBox source (raised on staging + elements change; AddSource refuses a second as defense in depth). Chat layers created before the "YouTube Chat" default still carried the legacy name "Chat" (commit 65641d8 era) — LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
• Round→rect restores the aspect (persisted, schema v4): SceneElement.ToggleClipShape() snapshots the rectangular Width/Height into public RectWidth/RectHeight before going Round and restores them when switching back — otherwise the Round resize lock (square) would leave a square behind. The rect dims are persisted (WebcamSceneConfig.RectWidth/RectHeight, nullable), so a reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time HealLegacySquareRect (load only) widens a pre-v4 Traditional config that ended up square to 16:9 (keeps height; Round and explicit rect dims are untouched).
• Device swap / layout reload: ChangeWebcamAsync (picker) and ReacquireWebcam (after load) release the old device with ReleaseAllAsync — a forced full drop that zeroes the refcount and stops the source regardless of how many scenes held it (the per-config count isn't known once the scenes are replaced) — then AcquireAsync the new device once per config.
• Shared bitmap, coalesced updates: one WriteableBitmap per active camera, created on the UI thread at the device's frame size (first frame), forwarded to every WebcamSceneConfig.VideoImageSource via PreviewBitmapChanged. Frames arrive on a worker thread; CameraManager coalesces onto the dispatcher (at most one pending copy per session, at Render priority, always copying the latest frame) so a 60fps device never drowns the render thread.
• Webcam added mid-session must get the live frames: PreviewBitmapChanged fires once (the first frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't exist at first-frame time would never receive it — the empty/transparent container you'd see adding a webcam to Chat while Live already had the camera. CameraManager.GetPreviewBitmap(deviceId) exposes the current shared bitmap; AddWebcamToActiveSceneAsync assigns it to the new config right before AcquireAsync, and ReacquireWebcam re-propagates it to every config after a reload.
• Clip/mirror/border: per-element ClipShape (Traditional rectangle / Round ellipse) + IsMirrored (ScaleX = -1) + the OSB-standard static border (BorderColor #RRGGBB or ""=none, BorderOpacity 0–1, BorderWidth 0–20, BorderAnimation None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow| Electricity|Sparkles). Rendered in the preview DataTemplate; toggled from the element's right-click context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist, rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty preview canvas has its own Show Webcam entry.
  • The Round webcam is an Image Stretch="UniformToFill" with an EllipseGeometry clip (Center=0.5,0.5 RadiusX/Y=0.5), inside a Viewbox Stretch="Uniform" holding a 1x1 Grid, so it renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to the element rect — the traditional Image keeps UniformToFill over the full rect. The clip is geometry, not an ImageBrush: a brush re-rasterizes the frequently-updated WriteableBitmap per frame on the render thread, which is what made the live webcam crawl while round. The Round border is a centered Ellipse at Width/Height = RoundBorderSize.
  • Resizing locks to a square (_resizeAspect = 1) while ClipShape == Round.
  • Webcam size clamp: ClampWebcamToBounds(config, sceneName) (internal — test seam) enforces the max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked by canonical scene name: 50% per dimension (960×540) everywhere except the Chat scene, which may reach half the screen area (~1358×764 @16:9) so the viewer sees the creator better (MaxWebcamWidthFor/MaxWebcamHeightFor; a renamed Chat loses the bigger cap). RoundBorderSize follows the clamped height. WebcamSafeguardTests guards the clamp.
  • Hit-testing: a Grid without Background only hit-tests where its children draw, so clicks in the empty corners of a round clip fell through to Window_PreviewMouseLeftButtonDown and deselected the element — making the corner handle ungrabbable. The element Grid carries Background="Transparent" (whole rect draggable; the empty canvas Grid uses the same trick for right-click Show Webcam) and the SelectionOverlay (dashed border + corner dot) is IsHitTestVisible="False" so it never intercepts the click. Two things make the webcam menu work: (1) the ContextMenu pins its own DataContext to PlacementTarget.DataContext — a ContextMenu isn't in the visual tree, so without it the Click-handler DataContext: patterns (and the IsChecked/slider bindings) silently fail; (2) Themes/Controls.xaml ships a full dark MenuItem template — PART_Popup (submenu popups), a popup ItemsPresenter (the Border Opacity/Thickness sliders live in Items, so they render in a hover flyout), a ✓ checkmark column, and a › arrow driven by HasItems. An earlier bare Border + Header template dropped all three: submenus never opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the Click handlers were fine.
• GPU posture: webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred to the encoder task.
• Background removal — REVERSED, now IN for v1 (creator ruling 2026-10-08, TASK 51). The 2026-10-05 "permanently out" ruling stood on "an in-app model means a runtime, a GPU/driver matrix, and per-scene tuning". The delivery shape changed: a ~430 KB Apache-2.0 model embedded in the exe, run on the CPU by onnxruntime — no download, no GPU requirement, no driver matrix. NVIDIA's Video Effects SDK stays ruled out (RTX-only + ~1 GB user-side download). See "Webcam background effects" below and TASKS/task-51-webcam-bg-effects.md.

Webcam background effects (TASK 51 — built 2026-10-09; jitter fix 2026-10-09)

• Model: Google's MediaPipe Selfie Segmenter, landscape (Apache-2.0, model card 2021-05-06), official file storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/ float16/latest/selfie_segmenter_landscape.tflite. Converted to ONNX by our own tools/selfie-seg/convert_tflite_to_onnx.py (the graph ends in MediaPipe's custom op Convolution2DTransposeBias, which tf2onnx/tflite2onnx can't convert — 11 op types, so a direct translation was smaller than any workaround). Verified against Google's own runtime: verify_against_mediapipe.py feeds the same frame to MediaPipe's ImageSegmenter (original .tflite) and onnxruntime (our .onnx): max |diff| 5e-5, 100% binary agreement on two images. ⛔ Do NOT swap in PINTO model-zoo #109 (it's ML Kit's model, extracted — not Apache provenance). obs-backgroundremoval is GPL-3.0 — license-compatible since LlamaCasty went GPL-3.0 (2026-10-09), but none of its code is used; if any is ever borrowed, it needs its copyright notice kept.
• Contract: input [1,144,256,3] RGB /255 NHWC → output [1,144,256,1] person probability. The whole frame is stretched (not cropped) to 256×144, so the mask maps back by normalized coordinates.
• Shipping: Assets/Models/selfie_segmenter_landscape.onnx is an EmbeddedResource (in git, inside the exe — nothing to lose next to it, nothing fetched at runtime). Runtime = Microsoft.ML.OnnxRuntime 1.31.0 (MIT), CPU execution provider, 2 intra-op threads. Note the native onnxruntime.dll is ~17 MB (win-x64); an RID-less build copies every RID's native folder — the MSIX publish (TASK 48) must pin the RID so only win-x64 ships.
• Code (Services/BackgroundEffects/): IPersonSegmenter seam (Segment(VideoFrame) → SegmentationMask?, tests inject fakes); OnnxPersonSegmenter (lazy session from the embedded resource; never throws — a load/run failure logs once and returns null forever after); BackgroundEffectProcessor.Apply(frame, BackgroundEffect) — None = same instance (true passthrough), Remove = copy with alpha = bilinear-upscaled mask × source alpha, RGB untouched, IsOpaque = false (the compositor's straight-alpha blend shows the layers below), strength = edge feather (mask band 0.5±(0.05+0.45·s) → alpha); Blur (slice 2) = background downscaled 4× by box average, 3 separable box-blur passes and bilinear-upsampled, blended by the mask — person pixels (mask = 1) are copied untouched, opacity preserved. Blur radius is quadratic (BlurRadius(strength): 1→6 small px, default 0.5 = 2 — the "blur heavy" fix 2026-10-09; the old LINEAR 1→12 small px ≈ 48 full px was far beyond what the established tools use: StreamFX defaults its Box blur Size to 5 and its author stops downscale-based blurs around 8 px). The mask is computed once per frame (TrySegment) and rendered per effect (Render), so several variants share one inference. No mask / segmenter exception → passthrough + one log line, never a dropped layer.
• Config (slice 3): per scene, on WebcamSceneConfig — BackgroundEffect + BackgroundStrength (0..1, default 0.5), persisted as two additive guarded columns on WebcamSceneConfig (BackgroundEffect TEXT DEFAULT 'None', BackgroundStrength REAL DEFAULT 0.5, no version bump — the TASK 47 convention). RemoveBackground/BlurBackground are the radio-pill bools the UI binds.
• Hookup (slice 4) — the ONE processing point is CameraManager. Effects are per scene but the camera session is shared, so the manager renders variants keyed by WebcamEffectKey(effect, strength rounded to 0.01). MainViewModel.SyncWebcamEffects() declares the union of every config's key (SetRequestedEffects; unused variants are dropped so their CPU stops) and points each config's VideoImageSource at its variant's bitmap; it re-runs on load, add, remove, device swap, and any BackgroundEffect/BackgroundStrength change (configs are watched via PropertyChanged, which also ScheduleSaves). Per raw frame a background worker (latest-frame-drop, one at a time) segments ONCE and renders every variant; each variant keeps its own latest frame (the resolver: GetLatestFrame(device, key)) and WriteableBitmap (EffectPreviewBitmapChanged on first creation). The raw path is untouched — a slow/failed model never stalls the plain webcam — and an effect variant shows nothing until its first processed frame (Remove never flashes the real room). The compositor needed no change: its sampler already honours source alpha.
• Temporal smoothing (2026-10-09, creator check: "removal is jittery"): the segmenter classifies every frame independently, so a verdict that wavers at an edge makes the cut-out flutter. Fix: MaskSmoother blends each raw mask with the previous smoothed one — 0.75 × new + 0.25 × previous (CurrentWeight, internal const). State is per CameraSession (one per-camera worker), NOT in the segmenter — the OnnxPersonSegmenter is shared app-wide and its bundled-model contract test pins the raw per-frame output. A null mask (model hiccup) resets history; the first frame and any size change pass through un-blended; the returned mask never aliases the caller's array and two buffers ping-pong (no per-frame allocation — a 256×144 mask is 147 KB, past the 85 KB LOH). Weights come from the established tools, not a guess: OBS's obs-backgroundremoval smoke-tests temporal_smooth_factor at its 0.85 default, Jitsi's webcam background filter (jitsi/jitsi-meet#17080) found 0.7/0.3 transformed best — a midpoint that still reacts to real movement; 0.5/0.5 leaves a smearing edge ghost. The blur "heavy" follow-up was fixed the same day: quadratic radius, 1→6 small px (see the Blur bullet above).
• Perf (synthetic 1280×720, Debug build): segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms per frame (Release is several × faster). Pixel loops are row-parallel, capped at min(4, cores/2); the bilinear upsampling is separable (horizontal rows precomputed per mask/small row, one vertical lerp per pixel) — the first naive per-pixel 2-D bilinear cost 46/80 ms. CameraManager logs the real average once per session after 120 processed frames (startup.log: "background effects avg …"). MainViewModel.PersonSegmenterOverride is the hermetic-test seam.
• UI (slice 5, creator spec 2026-10-09): the webcam's properties in the Layers panel (Controls/LeftPanel.xaml) gain a Background section — two PillToggle switches, Remove and Blur, bound to RemoveBackground/BlurBackground (radio-exclusive: one on turns the other off, turning the active one off = None) and a Strength slider (0..1, 5% snaps, visible only while an effect is on; Blur = amount, Remove = edge softness). Dragging the slider never blanks the webcam: a new strength variant inherits the dropped same-effect variant's last frame + bitmap (CameraManager.SyncVariants), and the VM keeps the shown bitmap until the new one renders.
• Tests: BackgroundEffectEngineTests — the Remove integration test (fake segmenter), and a model contract test that runs the BUNDLED model through the C# preprocessing on a fixture and matches the MediaPipe-produced reference mask (≥99.5% agreement; fixture = MediaPipe's public test image mozart_square.jpg, public-domain painting, in ytLive.Tests/Fixtures/). CameraManagerTests — the ONE integration test for the jitter fix (a SteppingMaskSegmenter whose two consecutive masks disagree at an edge; asserts the second frame keeps a partial alpha where the raw mask would have cut to 0: x=6 → 58 upstairs of the raw 0, person core 255, background 0). Blur_default_strength_is_gentle_and_full_strength_is_strong pins the quadratic radius mapping (BlurRadius(0.5) = 2, BlurRadius(1) = 6) and the behaviour at a step edge: default smears light ~9px into the dark half vs ~26px at full strength.

---

The vertical (9:16) tier — per-scene camera placement

Vertical (9:16) tier camera placement (TASK 57 — "vv" Units 1+2, built 2026-10-21)

The problem. The vertical tier (1080×1920) is a centred band crop of the SAME 1920×1080 master: OutputRect* = x 656.25, y 0, w 607.5, h 1080. A camera placed for 16:9 (the default webcam sits at x≈1408) lands outside the cropped window entirely — in the vertical stream the creator is simply not in the picture.

The rule (creator's locked table, 2026-10-17). Each scene keeps its own vertical placement — position, size and shape — and switching back to 16:9 restores the desktop layout exactly:

mode	shape	size	position
16:9	as today	as today	as today
vv · Live	circle or rect (the creator's choice, per scene)	small tile	one of six spots: top/bottom × left/centre/right
vv · Chat	rect only (the circle option is not offered)	the small tile (normal), or enlarged to fill one 607×540 area	the same six spots; enlarged fills the top or the bottom area
Starting / BRB / Ending	unchanged	unchanged	unchanged

Design — geometry only, no render-path change. The preview and the compositor both read the element's OWN X/Y/Width/Height + ClipShape, so the tier switch swaps the live geometry instead of teaching the render path about tiers:

• WebcamSceneConfig carries the vertical placement (VerticalX/Y/Width/Height/VerticalClipShape
  • VerticalRectWidth/Height — seven additive DB columns, NULL = "this scene has no vertical placement yet") and the parked 16:9 layout (LandscapeX/…/LandscapeRectHeight, transient).
• BeginVerticalLayout(...) parks the live 16:9 layout and brings the vertical one in; EndVerticalLayout() commits the vertical placement back into the stored slot and restores the parked 16:9 layout. The live geometry IS the render input → the compositor and PreviewPane.xaml are untouched.
• The save keeps the X/Y/W/H/ClipShape columns as the 16:9 layout (while the vertical tier is live, the parked values are written there and the live ones into the Vertical* columns).
• SceneElement.X/Y/Width/Height became virtual so the webcam can CONSTRAIN every write (drag, resize handle, properties fields) into the band — one choke point instead of three writers re-implementing the rule. Outside the band = invisible in the vertical stream.

The band and the six spots (Models/VerticalCameraLayout.cs, pure math): the band is 656.25…1263.75 × 0…1080; it splits into two 607×540 areas (top/bottom), each with three columns (left/centre/right) → six spots. A tile snaps to the spot its centre falls in, inset 32 master px from the band's outer edge (the app's own placement margin) and never outside the band. The default tile is 270×152 master px (480×270 in the 1080×1920 output) in the bottom-right spot — the corner the 16:9 default webcam uses. The Live scene gets that tile automatically the first time the vertical tier is picked (otherwise the vertical stream has no camera at all); the other scenes keep their 16:9 placement until the creator drags them into the band.

The Chat scene's own rules (Unit 2). In the vertical tier the Chat camera is rect only: the circle is not offered (MainViewModel.CanToggleClipShape is false → the Shape button disappears) and the model refuses it too — WebcamSceneConfig.ClipShape is overridden, so a locked camera cannot become Round by ANY writer (the toggle, the button, the placement swap). The lock is WebcamSceneConfig.VerticalShapeLocked, set by the VM per scene and cleared before the 16:9 layout comes back, so the desktop shape is restored. Its size is discrete: the small tile (normal) or one whole 607×540 area (enlarged) — the resize handle snaps to the nearer of the two (VerticalCameraLayout.SizeFor), and an enlarged camera fills the area its centre is in (top or bottom). The Chat scene also gets its tile automatically the first time the vertical tier is picked (like Live) — otherwise the vertical chat stream has no camera at all.

Manual entry (creator rule): the webcam's Layers properties gained a Geometry section (X/Y/W/H) — it edits the LIVE placement, i.e. the one the current tier shows. The webcam's geometry setters clamp it into the band, so a typed value can never push the camera out of the cropped window.

Derivative work. OBS has no built-in way to store a scene's layout per aspect ratio — the maintainers' answer is "create a new empty scene and add the scene you want to send as source" (obsproject.com/forum/threads/…161307), and Streamyard's dual-aspect guide says the same ("no built-in way to store separate layouts per aspect ratio, so you rely on manual workarounds and plugins" — streamyard.com/blog/dual-aspect-ratio-streaming-tool). Our per-scene, per-tier stored placement, resolved on the tier change, is the step past that.

Tests. VerticalWebcamLayoutTests — the ONE integration test for Unit 1: the vertical tier is picked → the Live camera comes INTO the band as a small tile → dragged to the top-right it lands top-right inside the band → back to 16:9 the desktop layout is restored exactly → and both placements survive a save + reload. (The store writes the webcam rows only when the app has a webcam identity — the config rows hang off it — so the test supplies one.) VerticalChatCameraLayoutTests — the ONE integration test for Unit 2: the vertical tier is picked → the Chat camera comes into the band as a small RECT tile → the circle option is not offered and cannot be set even on the model → dragged up and enlarged it fills the top area exactly → back to 16:9 the desktop layout is restored exactly and the circle option is offered again.

Known trap (see MyMistakes.md): a second properties panel bound to SelectedElement.IsWebcam broke WebcamBackgroundPillTests — that test finds the webcam's panel by binding text, so a new panel must use a different binding (IsWebcamSelected) or the selector matches twice.

---

Related files

• ui-surface.md — resolution tiers (OutputRect*, the tier list, the vertical tier's 607×1080 band) and key patterns (MVVM/binding conventions the capture UI follows).
• environment.md — Tests (hermetic real-window rules camera tests must obey) and Run (launch/take procedure, the Release-only rule for measurements).