TASK 4 ship step 5: live frame pipeline — FramePump paces the active scene's composite into the encoder, ScreenCaptureManager.GetLatestFrame, MainViewModel resolver/option-builder/pump wiring, FramePumpTests (7) + GetLatestFrame test — 147 tests passing, 0 warnings
This commit is contained in:
@@ -91,7 +91,7 @@ C# / WPF (.NET 8) following MVVM:
|
||||
|------|------|
|
||||
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
|
||||
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
|
||||
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")** |
|
||||
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")** |
|
||||
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
|
||||
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
|
||||
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
|
||||
@@ -352,7 +352,8 @@ integration test fakes the whole subprocess (probe + encoder) with a Channel-bac
|
||||
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
|
||||
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
|
||||
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
|
||||
`GopSize` = FPS×4. Not yet constructed by the app — the frame-pipeline wiring (ship step 5) owns it.
|
||||
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
|
||||
driven by the `FramePump` below.
|
||||
|
||||
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12, plan in TASKS.md)
|
||||
|
||||
@@ -371,16 +372,44 @@ timers).
|
||||
- **`AudioMixer`** owns both sources; `Start`/`Stop` follow go-live (`MainViewModel.BeginGoLive` success
|
||||
→ `_audioMixer.Start()`, `StopStream` → `Stop()`). Mic samples feed a pure **`AudioLevelMeter`** (RMS
|
||||
with 0.2 exponential smoothing) → `MicLevelChanged` → marshalled to the UI thread → `AudioLevel`.
|
||||
Desktop samples are currently **dropped** — the future encoder's AAC mix (ship step 5) consumes them,
|
||||
replacing the `-f lavfi -i anullsrc` placeholder. Failures log via `AppLog`; a mic failure zeroes the
|
||||
Desktop samples are currently **dropped** — a later step's AAC mix consumes them, replacing the
|
||||
`-f lavfi -i anullsrc` placeholder (the encoder construction itself shipped in ship step 5). Failures log via `AppLog`; a mic failure zeroes the
|
||||
meter, a loopback failure never kills the mic.
|
||||
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
|
||||
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
|
||||
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
|
||||
- `FfmpegEncoder.cs:139` pre-existing CS8602 fixed (`process!`) — build **0 warnings**; **139 passing**.
|
||||
|
||||
**Deferred:** loopback→encoder mix wiring and the encoder construction (ship step 5); capture while not
|
||||
live is deliberately not shipped (privacy indicator otherwise).
|
||||
**Deferred:** the loopback→encoder AAC mix wiring (a later step — the encoder construction + frame
|
||||
pipeline shipped in ship step 5); capture while not live is deliberately not shipped (privacy indicator
|
||||
otherwise).
|
||||
|
||||
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, plan in TASKS.md)
|
||||
|
||||
The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active
|
||||
scene each tick, resolves every element to its latest frame, composites it into the tier's output frame,
|
||||
and paces frames into the encoder at the tier's FPS. **Pattern — everything is a constructor-injected
|
||||
seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
|
||||
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, and an injectable pacing delay
|
||||
(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers.
|
||||
|
||||
- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log
|
||||
+ surface via the `Failed` event. `EncoderOptions == null` means "no RTMP URL": the pump logs and skips
|
||||
the encoder entirely. `MainViewModel._rtmpUrlProvider` is that seam — a `Func<string?>` returning null
|
||||
until TASK 5 supplies the reusable stream's ingest URL, so go-live runs the current visual flow.
|
||||
- **Stop ordering matters:** `StopAsync` stops the encoder (closes stdin → EOF → ffmpeg finalizes+exits)
|
||||
**before** awaiting the loop, because closing stdin unblocks a write stuck on pipe backpressure — the
|
||||
reverse order would deadlock. `ProcessFailed` self-stops the pump. `HealthUpdated` is forwarded
|
||||
(ship step 6 binds it to the bottom bar); `Failed` while live flips `StreamStatus.Error` (minimal).
|
||||
- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` →
|
||||
`CameraManager.GetLatestFrame(WebcamId)`, `Source { IsLiveCapture, CaptureKey }` →
|
||||
`ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/
|
||||
background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*`
|
||||
doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`,
|
||||
ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one.
|
||||
- Known consideration: the pump reads the active scene on a background thread while the UI can still edit
|
||||
it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than
|
||||
crashing. The background thread + video pipeline is the new reality since this step.
|
||||
|
||||
### Social bar (TASK 14 — shipped 2026-08-12, plan in TASKS.md)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user