Scene compositor (TASK 4 ship step 1): encoder master-frame scaffolding
Software compositor that renders a scene into the encoder's master VideoFrame, mirroring the XAML preview minus editing chrome: backdrop -> background -> elements (UniformToFill cover-crop, round clip, mirror, opacity, border) -> branding flash. NOTE FOR USERS: this change shows NO difference in the app's UI — it is pure backend scaffolding laying the groundwork for live video capture/streaming. The preview you see is unchanged. - Services/Compositor/: SceneCompositor (Render(scene, frameFor resolver, flashFrame, CompositorOptions)), CompositorOptions (source rect + output size; 16:9 full master, vertical 607x1080 -> 1080x1920), StretchMath (pure UniformToFill + bilinear), StaticPixelCache (asset bytes -> BGRA8 frame) - Frame sources injected via Func<SceneElement, VideoFrame?> resolver, so the compositor is pure, WPF-free, and hermetic to test (D3D11 upgrade behind the same seam later) - SceneElement.TryGetBorderColor public (shared hex parse), stale MainViewModel comment fixed, pre-existing CS1998 in YouTubeAuthServiceTests cleaned up - Tests: SceneCompositorTests integration (full scene + vertical tier + flash) + StretchMath units, docs updated (72 tests passing, 0 warnings)
This commit is contained in:
@@ -169,8 +169,8 @@ Preview shows the transition too (WYSIWYG). No wipes/slides/LUTs beyond the four
|
||||
|
||||
### Requirements:
|
||||
|
||||
1. **Encoding** — H.264 (hardware via NVENC/AMD, fallback x264) + AAC audio; **must comply**: keyframes ≤ 4s (gopSizeLong), closed GOP, AAC/MP3 @ 44.1/48kHz, mono/stereo only
|
||||
2. **RTMP push** — FFmpeg subprocess or native RTMP library, to the cached reusable stream's ingestion URL
|
||||
1. **Encoding** — H.264 (hardware via NVENC/AMD, fallback x264) + AAC audio; **must comply**: keyframes ≤ 4s (gopSizeLong), closed GOP, AAC/MP3 @ 44.1/48kHz, mono/stereo only. **License posture (decided): GPL-free build** — NVENC (NVIDIA) / QSV (Intel) / AMF (AMD) + OpenH264 software fallback + built-in AAC; no libx264 (GPL contaminates a paid product). Output containers are identical either way (H.264+AAC in `.flv` for RTMP, `.mp4`/`.ts` for VOD) — the format is NOT the differentiator, the license and per-GPU quality are.
|
||||
2. **RTMP push** — **FFmpeg subprocess (decided)**: app feeds raw frames via stdin, parses stderr for health; one battle-tested binary does encode + FLV mux + push + reconnect. **Binary distribution (decided): check-then-pull** — probe `where ffmpeg`/PATH at first go-live; if absent, download a **pinned** build (~30 MB, standard gyan.dev/BtB N — no custom minimal build) to `%APPDATA%\ytLlive\tools\ffmpeg.exe` and cache it, offline-friendly. Behind an `IFfmpegLocator` seam so tests fake it. Push goes to the cached reusable stream's ingestion URL
|
||||
3. **Quality ladder** — the offered tiers, with **1080p60 @ 8 Mbps as the standard/default**:
|
||||
- 720p30 @ 6 Mbps
|
||||
- 720p60 @ 6 Mbps
|
||||
@@ -191,8 +191,80 @@ Preview shows the transition too (WYSIWYG). No wipes/slides/LUTs beyond the four
|
||||
4. **Stream key management** — reuse the cached reusable stream (one per channel) instead of creating a new one per go-live; prefill default YouTube ingest URL `rtmp://a.rtmp.youtube.com/live2`
|
||||
5. **Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (encoder-side)
|
||||
6. **One-click go live** — defaults that work out of the box
|
||||
7. **Audio capture (feeds the meter — this task ships the wiring)** — WASAPI loopback (desktop/game at unity, zero UI — "it just is") + the picked mic (`MicSourceName` from the `MicPickerDialog`). The mic capture feeds `AudioLevel` so the realtime meter comes alive (today it reads 0 — the mixer feed is pending, see `ai.md` audio notes). AAC mono/stereo @ 48 kHz per the compliance rules.
|
||||
|
||||
### Status: Not started
|
||||
### Status: 🔶 In progress — **ship step 1 (the output compositor) SHIPPED** (2026-08-10); encoder/RTMP/audio follow it
|
||||
|
||||
The pipeline chain the encoder needs doesn't exist yet: **scene compositing** (the master 1920×1080 frame
|
||||
without the preview's editing chrome) → **audio capture** (WASAPI, feeds the meter) → **H.264+AAC encode**
|
||||
→ **vertical-tier crop/scale** → **RTMP push** → **health stats** into the bottom bar. Nothing can encode
|
||||
until a frame source exists, so the compositor is ship step 1.
|
||||
|
||||
#### Ship step 1 — Scene compositor (the frame source)
|
||||
|
||||
**Goal:** a pure-CPU software compositor producing the encoder's master frame (BGRA8, the `VideoFrame`
|
||||
seam) from the scene model. The preview stays XAML (the editing view); the compositor is the **output
|
||||
view** — WPF's `RenderTargetBitmap` can't be used (software-rendered + captures chrome). Two renderers
|
||||
must agree, so the XAML (`MainWindow.xaml` CanvasGrid + element DataTemplate) is the contract.
|
||||
|
||||
**Decisions (locked 2026-08-10):** **Path A CPU blitter** — GPU effort belongs to NVENC (the encoder),
|
||||
not composition; with an FFmpeg subprocess the master crosses a CPU readback to the pipe every frame
|
||||
anyway, so GPU compositing buys ~nothing at this layer count (2-3 live layers; static layers
|
||||
pre-composite once). A D3D11 compositor can replace this one later **behind the same seam** (the CPU
|
||||
master buffer stays the contract). **Render the output rect directly**: compositor is constructed with
|
||||
`CompositorOptions {SourceRectX/Y/W/H, OutputWidth, OutputHeight}`; 16:9 tiers = full 1920×1080 1:1;
|
||||
vertical (9:16) = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920. Reuses
|
||||
`MainViewModel.OutputRectX/Y/W/H` (note `(1920−607)/2 = 656.5` → align to integer pixels for output).
|
||||
|
||||
**Render spec (back → front, mirror the XAML exactly):**
|
||||
1. Backdrop — the Live scene's `IsBackdrop` Source (`CaptureKey` → live frame), `UniformToFill`
|
||||
full-frame (XAML's separate `BackdropImage` layer; the backdrop *element* renders nothing — its
|
||||
DataTemplate Image is Collapsed for DisplayCapture).
|
||||
2. Background — the scene's `Background` Source, `UniformToFill` full-frame (the `ActiveBackgroundImage`
|
||||
layer, not per-element).
|
||||
3. Elements in `Scene.Elements` order (back→front), skip `IsVisible=false`. What actually renders:
|
||||
- `Source` Type `Image` → static asset, `UniformToFill` cover-crop into (X, Y, W, H)
|
||||
- `WebcamSceneConfig` → latest frame by `DeviceId`: Traditional = `UniformToFill` rect; Round = circle
|
||||
diameter `min(W,H)` (alpha 0 outside — true circle, not oval); mirror = horizontal flip around
|
||||
element center (`MirrorScale`); opacity = per-pixel multiply (content + border); border = stroked
|
||||
rect / centered circle at `RoundBorderSize`, width `BorderWidth`, alpha `BorderOpacity`
|
||||
- `Background` / `IsBackdrop` / `TextOverlay` are NOT per-element (layers above; Text not shipped)
|
||||
4. Branding flash — pre-rendered full-frame "made with ytLlive!" at 25% alpha when live +
|
||||
`BrandFlashEnabled` + timer active. Passed in as a `VideoFrame?` (compositor core stays pure byte-math,
|
||||
no WPF; likely a bundled asset rather than runtime text rendering).
|
||||
5. NOT in output (preview chrome only): SelectionOverlay, DimRects, output-rect outline, badge, placeholder.
|
||||
|
||||
**New files (all in `Services/Compositor/`):**
|
||||
- `SceneCompositor.cs` — `Render(Scene, frameFor: Func<SceneElement, VideoFrame?>, flashFrame:
|
||||
VideoFrame?, CompositorOptions) → VideoFrame` (output-sized). The caller's `frameFor` resolver maps
|
||||
each element to its frame (webcam → DeviceId, image → AssetId via `StaticPixelCache`, backdrop →
|
||||
CaptureKey) — the compositor stays pure/hermetic/no WPF.
|
||||
- `CompositorOptions.cs` — source-rect + output W×H.
|
||||
- `StretchMath.cs` — `UniformToFill` cover-crop, ellipse mask, bilinear scale (pure, unit-tested).
|
||||
- `StaticPixelCache.cs` — asset `byte[]` → cached BGRA `VideoFrame` (WPF `BitmapDecoder` + `CopyPixels`,
|
||||
decode once per content hash).
|
||||
|
||||
**Test plan (Good Dog Rule — ONE integration test):** `SceneCompositorTests` — a scene with backdrop
|
||||
(solid red fake frame) + round webcam (solid green) + image (solid blue) → render 16:9 master → assert
|
||||
per-layer probe pixels (corner = backdrop color, element center = webcam color, outside the round clip =
|
||||
backdrop color, mirrored element swaps left/right); a vertical-tier variant asserts 1080×1920 output +
|
||||
crop fidelity. Focused unit tests on `StretchMath`. Tests push frames directly — no capture managers
|
||||
involved (they wire in a later step).
|
||||
|
||||
**Same-PR housekeeping:** fix the stale comment `MainViewModel.cs:324` ("shown under the meter on line 2"
|
||||
→ "shown left-justified INSIDE the meter bar" — `ai.md` is the authority); this task's requirements now
|
||||
include the explicit audio-capture/meter wiring (#7 above).
|
||||
|
||||
**Out of scope (later ship steps):** FFmpeg locator + license posture (covered in requirements 1-2),
|
||||
encoder + RTMP push, WASAPI audio capture (loopback + mic) feeding `AudioLevel`, wiring
|
||||
`CameraManager`/`ScreenCaptureManager` into the frame pipeline, brand-flash timer wiring, health stats
|
||||
(bitrate/FPS/dropped).
|
||||
|
||||
**Built (2026-08-10):** all four files shipped in `Services/Compositor/`, `SceneElement.TryGetBorderColor`
|
||||
made public (shared hex parse with the compositor — no duplicated color parsing), the stale
|
||||
`MainViewModel.cs:324` comment corrected, and the pre-existing CS1998 in `YouTubeAuthServiceTests`
|
||||
cleaned up — build **0 warnings**. Tests: the `SceneCompositorTests` integration test (full-scene master
|
||||
pixels, vertical tier, flash) + 4 `StretchMath` units — **72 passing**.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user