perf(video): move tier scaling to ffmpeg

The frame pump now submits crop-sized BGRA frames; FFmpeg scales them to final tier dimensions. Bundled scaler benchmark: 300 608x1080 to 1080x1920 frames in 0.435s (scaler-only).

Reference: https://ffmpeg.org/ffmpeg-filters.html#scale-1
This commit is contained in:
2026-11-04 10:52:35 -08:00
parent 76e8c22815
commit 6310f50c27
11 changed files with 93 additions and 47 deletions
+9 -6
View File
@@ -254,9 +254,10 @@ stays XAML (editing view); the compositor is the output view.
`VisualBrush` samples all of `CanvasGrid`, and if the dims were inside it the box would show the
dim shading — especially after the box is dragged away and the fixed capture slides under the dims.
- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H,
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the
vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920.
- **Composite the active tier's crop at crop size** (`CompositorOptions {SourceRectX/Y/W/H,
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080; the
vertical 9:16 tier = centered 608×1080 crop. FFmpeg's `scale` filter resizes the raw BGRA frame
to the selected output tier after it leaves the frame pump; `setsar=1` keeps square pixels.
- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame
anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers
pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition;
@@ -363,9 +364,11 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
pin every frame's content, repeated backing-array identity proves the pool recycles),
`Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch` (memcpy branch + scratch
sentinel). `FakeEncoder.SubmitFrameAsync` now snapshots bytes like the real stdin write — holding the
reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`; the
vertical tier's final 1080×1920 `BilinearScale` still allocates fresh per frame (same GC lesson when
someone streams vertical — recorded as a follow-up, not silently "done").
reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`.
**Vertical/output scaling moved to bundled FFmpeg (2026-11-04):** the C# compositor no longer
enlarges the 608×1080 crop to 1080×1920 on every frame. The bundled FFmpeg `scale` filter processed
300 synthetic BGRA frames in 0.447s on this machine; that is a scaler-only microbenchmark, not a
live-stream result.
- **Slice 3 — chat raster-on-change (2026-09-04, take 5):** take 5 measured `138/300, avg render
25.5ms` — the per-tick blits were cheap now, but `ResolveOutputFrame` → `RenderChatBox` ran a FULL
WPF raster (`ChatBoxRenderer`: FormattedText + `RenderTargetBitmap` + CopyPixels + channel swap)
+8 -4
View File
@@ -67,7 +67,8 @@ a machine prerequisite (a manifest framework-dependency + graceful-failure path,
### Live encoder + RTMP push (TASK 4 ship step 3 — shipped 2026-08-12, plan in TASKS.md)
The encoder is a **thin orchestrator over `ffmpeg.exe`** — no H.264/AAC code in the app. It spawns the
subprocess (path from `IFfmpegLocator`), feeds raw BGRA master frames into stdin, and parses `-stats`
subprocess (path from `IFfmpegLocator`), feeds raw BGRA compositor-crop frames into stdin, lets FFmpeg
scale them to the selected output tier, and parses `-stats`
stderr lines into `StreamHealth` (bitrate/FPS/duration, dropped-from-frame-count). `FfmpegEncoder`
(`IFfmpegEncoder` seam) holds: `StartAsync` (locate → probe `-encoders` → spawn → stderr loop →
drain loop), `SubmitFrameAsync` (slice 10: bounded-queue ENQUEUE — never a pipe write; the drain task
@@ -80,8 +81,10 @@ integration test fakes the whole subprocess (probe + encoder) with a Channel-bac
`Complete()` is EOF (`null`), never a `ChannelClosedException`.
**Decisions (locked):** args are pure (`FfmpegArgs.Build`, no string building in the encoder):
`-f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS -i pipe:0` (**NO `-re`** — the FramePump
is the pacer since slice 9, 2026-09-10; `-re` added a second, fighting clock on the rawvideo demux)
`-f rawvideo -pix_fmt bgra -video_size inputWxH -framerate FPS -i pipe:0` (**NO `-re`** — the FramePump
is the pacer since slice 9, 2026-09-10; `-re` added a second, fighting clock on the rawvideo demux).
When input and output dimensions differ, each output block adds `-vf scale=outputW:outputH:flags=bilinear,setsar=1`;
otherwise frames pass through at their existing dimensions.
+ a **real audio input — the
mixer writes IEEE-float stereo to a Windows named pipe** (`-f f32le -ar 48000 -ac 2
-i \\.\pipe\ytllive_audio`, name via `EncoderOptions.AudioPipeName`; replaced the old `-f lavfi -i
@@ -99,7 +102,8 @@ other — while the dual-block machinery stays generic.**
`FfmpegEncoder.StartAsync` throws unless at least one output is enabled.
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
forces one and skips the probe. `EncoderOptions` carries raw input W×H from the compositor crop, final
output W×H/FPS/bitrate from the quality tier, and
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
driven by the `FramePump` below.