Scene compositor (TASK 4 ship step 1): encoder master-frame scaffolding
Software compositor that renders a scene into the encoder's master VideoFrame, mirroring the XAML preview minus editing chrome: backdrop -> background -> elements (UniformToFill cover-crop, round clip, mirror, opacity, border) -> branding flash. NOTE FOR USERS: this change shows NO difference in the app's UI — it is pure backend scaffolding laying the groundwork for live video capture/streaming. The preview you see is unchanged. - Services/Compositor/: SceneCompositor (Render(scene, frameFor resolver, flashFrame, CompositorOptions)), CompositorOptions (source rect + output size; 16:9 full master, vertical 607x1080 -> 1080x1920), StretchMath (pure UniformToFill + bilinear), StaticPixelCache (asset bytes -> BGRA8 frame) - Frame sources injected via Func<SceneElement, VideoFrame?> resolver, so the compositor is pure, WPF-free, and hermetic to test (D3D11 upgrade behind the same seam later) - SceneElement.TryGetBorderColor public (shared hex parse), stale MainViewModel comment fixed, pre-existing CS1998 in YouTubeAuthServiceTests cleaned up - Tests: SceneCompositorTests integration (full scene + vertical tier + flash) + StretchMath units, docs updated (72 tests passing, 0 warnings)
This commit is contained in:
@@ -59,8 +59,10 @@ ScreenCaptureManager refcount + shared-bitmap + coalescing
|
||||
(fake `IScreenCaptureSource` + a real background-STA `Dispatcher`), BackdropTests (EnsureBackdrop
|
||||
insert/idempotent/heal + HasBackdrop gate, IsLiveCapture, DisplaySource, INPC), SceneCatalogTests
|
||||
(the five canonical scenes, Live-only backdrop policy, EnforceBackdropPolicy), WebcamSafeguardTests
|
||||
(the per-scene size clamp incl. the Chat half-screen-area cap) —
|
||||
65 passing.
|
||||
(the per-scene size clamp incl. the Chat half-screen-area cap), SceneCompositorTests (the full-scene
|
||||
composite integration test: backdrop + round webcam + mirrored/bordered images + flash; the vertical
|
||||
tier 1080×1920 crop/scale), StretchMathTests (UniformToFill cover-crop + bilinear) —
|
||||
72 passing.
|
||||
|
||||
### Real-MainWindow tests MUST be hermetic (DB pollution bug)
|
||||
|
||||
@@ -110,7 +112,7 @@ C# / WPF (.NET 8) following MVVM:
|
||||
- `Helpers/OAuthCredentials.cs` contains the real ClientId/ClientSecret. Auth is complete and the session **persists via Windows DPAPI** (`Helpers/TokenStore.cs` → `%APPDATA%\ytLlive\ytLlive.auth`, CurrentUser scope), reloaded best-effort at startup with a proactive refresh of a near-expiry access token. Sign-in/Change Account lives **inside the Start Stream dialog** (two-state flow — no separate Connect button). A **graceful End Livestream signs out**: `StopStream()` clears the session + token, so the next go-live needs a fresh sign-in; a crash never runs End, so the token survives and the creator stays signed in. `YouTubeAuthService` takes an optional `HttpClient` + `sessionChanged` callback (test seam + save hook; services are still constructed in `MainViewModel`)
|
||||
- Scene/source/asset layout persists (SQLite, schema v6); the OAuth session persists (DPAPI); the paid-unlock state does not (yet — itch.io key verification pending)
|
||||
- `YouTubeStreamService` uses hardcoded `1080p`/`60fps` and per-broadcast streams — must switch to the v3 `variable` reusable stream
|
||||
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **window capture (non-backdrop), scene compositing/encoding, RTMP are next**
|
||||
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **the output compositor (TASK 4 ship step 1) is the next build** — full plan in `TASKS.md`; window capture (non-backdrop), the encoder + RTMP push, and audio capture follow it
|
||||
- `StreamConfig` defaults (`TargetBitrate=6000`, `Resolution="1920x1080"`) are stale — the live dropdown drives `StreamHealth.CurrentBitrate`/`FPS` instead
|
||||
|
||||
### Screen backdrop capture (TASK 3 ship task #1)
|
||||
@@ -282,6 +284,34 @@ instead of a normal draggable source.
|
||||
- **Background removal = milestone 2** — ONNX Runtime + DirectML (CPU fallback), MediaPipe Selfie
|
||||
Segmentation (Apache-2.0), wired into the same `VideoFrame` seam. Not part of milestone 1.
|
||||
|
||||
### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)
|
||||
|
||||
The encoder needs the master 1920×1080 frame **without** the preview's editing chrome (SelectionOverlay,
|
||||
DimRects, output-rect outline, badge, placeholder). WPF's `RenderTargetBitmap` is software-rendered and
|
||||
captures the visual tree *including* chrome, so the preview can't be captured — the output is a **second,
|
||||
parallel software compositor** over the `VideoFrame` (BGRA8) seam, and the XAML preview
|
||||
(`MainWindow.xaml` CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two
|
||||
renderers must agree: geometry, `UniformToFill` cover-crop, round clip, mirror, border, z-order. Preview
|
||||
stays XAML (editing view); the compositor is the output view.
|
||||
|
||||
- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H,
|
||||
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the
|
||||
vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920.
|
||||
- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame
|
||||
anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers
|
||||
pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition;
|
||||
if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one **behind
|
||||
the same seam** — the CPU master buffer stays the contract.
|
||||
- **Branding flash is composited by the output path too** (it's on the live output, per Monetization),
|
||||
passed in as a pre-rendered `VideoFrame?` — the compositor core stays pure byte-math, no WPF. Likely a
|
||||
bundled asset rather than runtime text rendering (deterministic, no font/layout risk).
|
||||
- **Frame sources are injected** via a `Func<SceneElement, VideoFrame?>` resolver
|
||||
(`SceneCompositor.Render(scene, frameFor, flashFrame, options)`) — the caller maps each element to
|
||||
its frame (webcam → `DeviceId`, image → `AssetId` via `StaticPixelCache`, backdrop → `CaptureKey`),
|
||||
so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that
|
||||
resolver in the encoder step, not the compositor step. The master buffer (the compositor's return
|
||||
value) is the seam a future D3D11 compositor would honor identically.
|
||||
|
||||
## Design Principle
|
||||
|
||||
> This software is so intuitive that even the most right-brained person can easily intuit and use it.
|
||||
|
||||
Reference in New Issue
Block a user