Scene compositor (TASK 4 ship step 1): encoder master-frame scaffolding

Software compositor that renders a scene into the encoder's master VideoFrame,
mirroring the XAML preview minus editing chrome: backdrop -> background ->
elements (UniformToFill cover-crop, round clip, mirror, opacity, border) ->
branding flash.

NOTE FOR USERS: this change shows NO difference in the app's UI — it is pure
backend scaffolding laying the groundwork for live video capture/streaming.
The preview you see is unchanged.

- Services/Compositor/: SceneCompositor (Render(scene, frameFor resolver,
  flashFrame, CompositorOptions)), CompositorOptions (source rect + output
  size; 16:9 full master, vertical 607x1080 -> 1080x1920), StretchMath (pure
  UniformToFill + bilinear), StaticPixelCache (asset bytes -> BGRA8 frame)
- Frame sources injected via Func<SceneElement, VideoFrame?> resolver, so the
  compositor is pure, WPF-free, and hermetic to test (D3D11 upgrade behind the
  same seam later)
- SceneElement.TryGetBorderColor public (shared hex parse), stale
  MainViewModel comment fixed, pre-existing CS1998 in YouTubeAuthServiceTests
  cleaned up
- Tests: SceneCompositorTests integration (full scene + vertical tier + flash)
  + StretchMath units, docs updated (72 tests passing, 0 warnings)
This commit is contained in:
2026-08-10 10:15:58 -07:00
parent b00a4cbd5e
commit 18a21010bb
11 changed files with 681 additions and 11 deletions
+33 -3
View File
@@ -59,8 +59,10 @@ ScreenCaptureManager refcount + shared-bitmap + coalescing
(fake `IScreenCaptureSource` + a real background-STA `Dispatcher`), BackdropTests (EnsureBackdrop
insert/idempotent/heal + HasBackdrop gate, IsLiveCapture, DisplaySource, INPC), SceneCatalogTests
(the five canonical scenes, Live-only backdrop policy, EnforceBackdropPolicy), WebcamSafeguardTests
(the per-scene size clamp incl. the Chat half-screen-area cap)
65 passing.
(the per-scene size clamp incl. the Chat half-screen-area cap), SceneCompositorTests (the full-scene
composite integration test: backdrop + round webcam + mirrored/bordered images + flash; the vertical
tier 1080×1920 crop/scale), StretchMathTests (UniformToFill cover-crop + bilinear) —
72 passing.
### Real-MainWindow tests MUST be hermetic (DB pollution bug)
@@ -110,7 +112,7 @@ C# / WPF (.NET 8) following MVVM:
- `Helpers/OAuthCredentials.cs` contains the real ClientId/ClientSecret. Auth is complete and the session **persists via Windows DPAPI** (`Helpers/TokenStore.cs``%APPDATA%\ytLlive\ytLlive.auth`, CurrentUser scope), reloaded best-effort at startup with a proactive refresh of a near-expiry access token. Sign-in/Change Account lives **inside the Start Stream dialog** (two-state flow — no separate Connect button). A **graceful End Livestream signs out**: `StopStream()` clears the session + token, so the next go-live needs a fresh sign-in; a crash never runs End, so the token survives and the creator stays signed in. `YouTubeAuthService` takes an optional `HttpClient` + `sessionChanged` callback (test seam + save hook; services are still constructed in `MainViewModel`)
- Scene/source/asset layout persists (SQLite, schema v6); the OAuth session persists (DPAPI); the paid-unlock state does not (yet — itch.io key verification pending)
- `YouTubeStreamService` uses hardcoded `1080p`/`60fps` and per-broadcast streams — must switch to the v3 `variable` reusable stream
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **window capture (non-backdrop), scene compositing/encoding, RTMP are next**
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **the output compositor (TASK 4 ship step 1) is the next build** — full plan in `TASKS.md`; window capture (non-backdrop), the encoder + RTMP push, and audio capture follow it
- `StreamConfig` defaults (`TargetBitrate=6000`, `Resolution="1920x1080"`) are stale — the live dropdown drives `StreamHealth.CurrentBitrate`/`FPS` instead
### Screen backdrop capture (TASK 3 ship task #1)
@@ -282,6 +284,34 @@ instead of a normal draggable source.
- **Background removal = milestone 2** — ONNX Runtime + DirectML (CPU fallback), MediaPipe Selfie
Segmentation (Apache-2.0), wired into the same `VideoFrame` seam. Not part of milestone 1.
### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)
The encoder needs the master 1920×1080 frame **without** the preview's editing chrome (SelectionOverlay,
DimRects, output-rect outline, badge, placeholder). WPF's `RenderTargetBitmap` is software-rendered and
captures the visual tree *including* chrome, so the preview can't be captured — the output is a **second,
parallel software compositor** over the `VideoFrame` (BGRA8) seam, and the XAML preview
(`MainWindow.xaml` CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two
renderers must agree: geometry, `UniformToFill` cover-crop, round clip, mirror, border, z-order. Preview
stays XAML (editing view); the compositor is the output view.
- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H,
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the
vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920.
- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame
anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers
pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition;
if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one **behind
the same seam** — the CPU master buffer stays the contract.
- **Branding flash is composited by the output path too** (it's on the live output, per Monetization),
passed in as a pre-rendered `VideoFrame?` — the compositor core stays pure byte-math, no WPF. Likely a
bundled asset rather than runtime text rendering (deterministic, no font/layout risk).
- **Frame sources are injected** via a `Func<SceneElement, VideoFrame?>` resolver
(`SceneCompositor.Render(scene, frameFor, flashFrame, options)`) — the caller maps each element to
its frame (webcam → `DeviceId`, image → `AssetId` via `StaticPixelCache`, backdrop → `CaptureKey`),
so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that
resolver in the encoder step, not the compositor step. The master buffer (the compositor's return
value) is the seam a future D3D11 compositor would honor identically.
## Design Principle
> This software is so intuitive that even the most right-brained person can easily intuit and use it.