TASK 4 ship step 5: live frame pipeline — FramePump paces the active scene's composite into the encoder, ScreenCaptureManager.GetLatestFrame, MainViewModel resolver/option-builder/pump wiring, FramePumpTests (7) + GetLatestFrame test — 147 tests passing, 0 warnings

This commit is contained in:
2026-08-13 08:57:56 -07:00
parent 58c0f8e8c4
commit e72ba71165
9 changed files with 700 additions and 54 deletions
+35 -6
View File
@@ -91,7 +91,7 @@ C# / WPF (.NET 8) following MVVM:
|------|------|
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")** |
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
@@ -352,7 +352,8 @@ integration test fakes the whole subprocess (probe + encoder) with a Channel-bac
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
`GopSize` = FPS×4. Not yet constructed by the app — the frame-pipeline wiring (ship step 5) owns it.
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
driven by the `FramePump` below.
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12, plan in TASKS.md)
@@ -371,16 +372,44 @@ timers).
- **`AudioMixer`** owns both sources; `Start`/`Stop` follow go-live (`MainViewModel.BeginGoLive` success
→ `_audioMixer.Start()`, `StopStream` → `Stop()`). Mic samples feed a pure **`AudioLevelMeter`** (RMS
with 0.2 exponential smoothing) → `MicLevelChanged` → marshalled to the UI thread → `AudioLevel`.
Desktop samples are currently **dropped** — the future encoder's AAC mix (ship step 5) consumes them,
replacing the `-f lavfi -i anullsrc` placeholder. Failures log via `AppLog`; a mic failure zeroes the
Desktop samples are currently **dropped** — a later step's AAC mix consumes them, replacing the
`-f lavfi -i anullsrc` placeholder (the encoder construction itself shipped in ship step 5). Failures log via `AppLog`; a mic failure zeroes the
meter, a loopback failure never kills the mic.
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
- `FfmpegEncoder.cs:139` pre-existing CS8602 fixed (`process!`) — build **0 warnings**; **139 passing**.
**Deferred:** loopback→encoder mix wiring and the encoder construction (ship step 5); capture while not
live is deliberately not shipped (privacy indicator otherwise).
**Deferred:** the loopback→encoder AAC mix wiring (a later step — the encoder construction + frame
pipeline shipped in ship step 5); capture while not live is deliberately not shipped (privacy indicator
otherwise).
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, plan in TASKS.md)
The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active
scene each tick, resolves every element to its latest frame, composites it into the tier's output frame,
and paces frames into the encoder at the tier's FPS. **Pattern — everything is a constructor-injected
seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, and an injectable pacing delay
(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers.
- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log
+ surface via the `Failed` event. `EncoderOptions == null` means "no RTMP URL": the pump logs and skips
the encoder entirely. `MainViewModel._rtmpUrlProvider` is that seam — a `Func<string?>` returning null
until TASK 5 supplies the reusable stream's ingest URL, so go-live runs the current visual flow.
- **Stop ordering matters:** `StopAsync` stops the encoder (closes stdin → EOF → ffmpeg finalizes+exits)
**before** awaiting the loop, because closing stdin unblocks a write stuck on pipe backpressure — the
reverse order would deadlock. `ProcessFailed` self-stops the pump. `HealthUpdated` is forwarded
(ship step 6 binds it to the bottom bar); `Failed` while live flips `StreamStatus.Error` (minimal).
- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` →
`CameraManager.GetLatestFrame(WebcamId)`, `Source { IsLiveCapture, CaptureKey }` →
`ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/
background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*`
doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`,
ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one.
- Known consideration: the pump reads the active scene on a background thread while the UI can still edit
it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than
crashing. The background thread + video pipeline is the new reality since this step.
### Social bar (TASK 14 — shipped 2026-08-12, plan in TASKS.md)