perf(video): move tier scaling to ffmpeg

The frame pump now submits crop-sized BGRA frames; FFmpeg scales them to final tier dimensions. Bundled scaler benchmark: 300 608x1080 to 1080x1920 frames in 0.435s (scaler-only).

Reference: https://ffmpeg.org/ffmpeg-filters.html#scale-1
This commit is contained in:
2026-11-04 10:52:35 -08:00
parent 76e8c22815
commit 6310f50c27
11 changed files with 93 additions and 47 deletions
+9 -6
View File
@@ -254,9 +254,10 @@ stays XAML (editing view); the compositor is the output view.
`VisualBrush` samples all of `CanvasGrid`, and if the dims were inside it the box would show the
dim shading — especially after the box is dragged away and the fixed capture slides under the dims.
- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H,
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the
vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920.
- **Composite the active tier's crop at crop size** (`CompositorOptions {SourceRectX/Y/W/H,
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080; the
vertical 9:16 tier = centered 608×1080 crop. FFmpeg's `scale` filter resizes the raw BGRA frame
to the selected output tier after it leaves the frame pump; `setsar=1` keeps square pixels.
- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame
anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers
pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition;
@@ -363,9 +364,11 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
pin every frame's content, repeated backing-array identity proves the pool recycles),
`Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch` (memcpy branch + scratch
sentinel). `FakeEncoder.SubmitFrameAsync` now snapshots bytes like the real stdin write — holding the
reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`; the
vertical tier's final 1080×1920 `BilinearScale` still allocates fresh per frame (same GC lesson when
someone streams vertical — recorded as a follow-up, not silently "done").
reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`.
**Vertical/output scaling moved to bundled FFmpeg (2026-11-04):** the C# compositor no longer
enlarges the 608×1080 crop to 1080×1920 on every frame. The bundled FFmpeg `scale` filter processed
300 synthetic BGRA frames in 0.447s on this machine; that is a scaler-only microbenchmark, not a
live-stream result.
- **Slice 3 — chat raster-on-change (2026-09-04, take 5):** take 5 measured `138/300, avg render
25.5ms` — the per-tick blits were cheap now, but `ResolveOutputFrame` → `RenderChatBox` ran a FULL
WPF raster (`ChatBoxRenderer`: FormattedText + `RenderTargetBitmap` + CopyPixels + channel swap)