TASK 4 ship step 5: live frame pipeline — FramePump paces the active scene's composite into the encoder, ScreenCaptureManager.GetLatestFrame, MainViewModel resolver/option-builder/pump wiring, FramePumpTests (7) + GetLatestFrame test — 147 tests passing, 0 warnings

This commit is contained in:
2026-08-13 08:57:56 -07:00
parent 58c0f8e8c4
commit e72ba71165
9 changed files with 700 additions and 54 deletions
+28 -43
View File
@@ -5,52 +5,37 @@
> a problem. Conventions: [`schema.md`](schema.md). Rewrite this file at session
> end, compaction, or any interruption.
## Session state (last updated: 2026-08-12)
## Session state (last updated: 2026-08-13)
- **Branch:** `main`. TASK 4 ship step 4 (WASAPI audio capture) is **built and
tested** but **NOT committed** — the working tree is dirty with the audio
layer (6 new files in `Services/Audio/` + `ytLive.Tests/AudioMixerTests.cs`,
the `NAudio.Wasapi` 2.2.1 package reference, the THIRD-PARTY-NOTICES entry,
the `MainViewModel` wiring, the `FfmpegEncoder.cs:139` CS8602 fix) and its
memory updates (TASKS.md, ai.md, Services/index.md, HANDOFF.md). Ready to
commit when the user says so.
- **Finished this session:** TASK 4 ship step 4 — the live audio capture layer:
`IAudioSource`/`AudioSample` seam, `WasapiLoopbackAudioSource`
(`WasapiLoopbackCapture` on the default render device), `WasapiMicAudioSource`
(`WasapiCapture`, NAudio device resolved by `FriendlyName` matching
`MicSourceName` via a re-read `Func<string?>` provider, default-endpoint
fallback), `AudioMixer` (owns both sources, starts/stops with go-live, feeds
the pure `AudioLevelMeter``MicLevelChanged`), `WaveToFloat` (IEEE-float /
PCM16 / extensible). `MainViewModel` constructs the mixer, `BeginGoLive`
success → `Start()`, `StopStream``Stop()`, `MicLevelChanged` marshalled to
the UI thread → `AudioLevel`. Loopback samples are currently dropped (the
encoder's AAC mix, ship step 5, consumes them). Build: **0 warnings** (a
pre-existing CS8602 in `FfmpegEncoder.cs:139` surfaced during the rebuild and
was fixed with `process!`). Tests: **139 passing** (was 122; +17 new audio).
- **Branch:** `main`. TASK 4 ship step 5 (the live frame pipeline) is **shipped
and committed** — the `FramePump` frame producer
(`Services/Encoder/FramePump.cs`), `ScreenCaptureManager.GetLatestFrame(key)`,
the `MainViewModel` resolver/option-builders/pump lifecycle wiring, `FramePumpTests`
(7) + the `GetLatestFrame` test (1), and the memory updates (TASKS.md, ai.md,
Services/index.md — the previous session left `Services/index.md` stale and
HANDOFF unrewritten; both were fixed before the commit). Build **0 warnings**,
**147 tests passing**. A Windows patch reboot killed the prior session right
after the work was verified but before the commit.
- **Finished this session:** ship step 5 (re-verified: 0 warnings, 147/147 pass),
the `Services/index.md` FramePump row + de-stale'd encoder rows, and this
HANDOFF rewrite.
- **Landmines:**
- `WaveFormatExtensible.SubFormat` is a **`Guid`** in NAudio 2.x, not a
`WaveFormatEncoding` — compare it to
`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT` (the IEEE-float
subtype constant; it lives in the `NAudio.Dmo` namespace, not CoreAudioApi).
- Use the **`NAudio.Wasapi` 2.2.1 feature package**, not the `NAudio`
meta-package — the wasapi types come from that package (`WasapiCapture` in
`NAudio.CoreAudioApi`, `WasapiLoopbackCapture` in `NAudio.Wave`); `NAudio.Core`
comes in transitively.
- The audio meter is an exponential smoother (0.2 factor) — a single pushed
sample only moves 20% toward its RMS. Tests must push repeatedly before
asserting a converged level.
- The pump reads the active scene on a background thread while the UI can still
edit it — a concurrent-mutation exception is contained (logged + `Failed` +
the pump stops), not a crash. The background thread + video pipeline is the
new reality since ship step 5.
- `StopAsync` must stop the encoder (closes stdin) **before** awaiting the pump
loop — closing stdin unblocks a write stuck on pipe backpressure; the reverse
order deadlocks.
- `FramePump.IsRunning` must be set true before the loop starts (a completed-task
delay can run the first iteration synchronously on the caller's thread).
- `StartAsync` never throws; the VM fires-and-forgets it. `Failed` while live
flips `StreamStatus.Error` (minimal — real health surfacing is ship step 6).
- Tests never instantiate `MainViewModel` directly except the round-clip
integration test (a real `MainWindow`), which never goes live — so the mixer
is constructed but never started there; NAudio types are only constructed,
never touching devices. Keep it that way.
- Mic device resolution must re-read `MicSourceName` at each `Start` (the app
persists only the DisplayName, not a device ID).
- Windows-only: all capture runs only while live (privacy indicator otherwise).
- **Next step:** TASK 4 ship step 5 — the frame-pipeline wiring
(`CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder
construction, plus the loopback → encoder AAC mix). Nothing else queued — do
not expand the task queue on your own.
integration test (a real `MainWindow`), which never goes live — keep it that way.
- **Next step:** TASK 4 ship step 6 — health stats: bind `FramePump.HealthUpdated`
(bitrate/FPS/duration) into the bottom bar. Nothing else queued — do not expand
the task queue on your own.
- **Secret/DB/port facts live:** OAuth client id/secret in `Helpers/OAuthCredentials.cs`;
OAuth session token in `Helpers/TokenStore.cs` (DPAPI → `%APPDATA%\ytLlive\ytLlive.auth`);
+256
View File
@@ -0,0 +1,256 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using ytLive.Models;
using ytLive.Services.Compositor;
namespace ytLive.Services.Encoder;
/// <summary>
/// The live frame producer (TASK 4 ship step 5): the bridge between the capture
/// managers + compositor and the encoder. While live it snapshots the active
/// scene each tick, resolves every element to its latest frame, composites it
/// into the tier's output frame, and paces frames into the encoder at the tier's
/// FPS. All collaborators are constructor-injected seams (scene, resolver,
/// options, encoder factory, pacing delay) so the pump stays free of WPF and of
/// the capture managers and is fully hermetic in tests.
///
/// The RTMP URL comes from the options provider: until the live-stream create
/// flow lands (TASK 5) it yields null, so go-live runs the existing visual flow
/// without actually pushing.
/// </summary>
public sealed class FramePump : IDisposable
{
private readonly Func<Scene?> _sceneProvider;
private readonly Func<SceneElement, VideoFrame?> _frameResolver;
private readonly Func<CompositorOptions> _compositorOptions;
private readonly Func<EncoderOptions?> _encoderOptions;
private readonly Func<IFfmpegEncoder> _encoderFactory;
private readonly Action<string>? _log;
private readonly Func<TimeSpan, CancellationToken, Task> _pacingDelay;
private readonly SceneCompositor _compositor = new();
private readonly object _gate = new();
private IFfmpegEncoder? _encoder;
private CancellationTokenSource? _cts;
private Task? _pumpTask;
private bool _started;
/// <summary>Forwards the encoder's parsed health — ship step 6 binds this to the bottom bar.</summary>
public event EventHandler<StreamHealth>? HealthUpdated;
/// <summary>Raised when the encoder cannot start or dies mid-stream. The pump stops itself.</summary>
public event EventHandler<string>? Failed;
public FramePump(
Func<Scene?> sceneProvider,
Func<SceneElement, VideoFrame?> frameResolver,
Func<CompositorOptions> compositorOptions,
Func<EncoderOptions?> encoderOptions,
Func<IFfmpegEncoder> encoderFactory,
Action<string>? log = null,
Func<TimeSpan, CancellationToken, Task>? pacingDelay = null)
{
_sceneProvider = sceneProvider ?? throw new ArgumentNullException(nameof(sceneProvider));
_frameResolver = frameResolver ?? throw new ArgumentNullException(nameof(frameResolver));
_compositorOptions = compositorOptions ?? throw new ArgumentNullException(nameof(compositorOptions));
_encoderOptions = encoderOptions ?? throw new ArgumentNullException(nameof(encoderOptions));
_encoderFactory = encoderFactory ?? throw new ArgumentNullException(nameof(encoderFactory));
_log = log;
_pacingDelay = pacingDelay ?? ((delay, ct) => Task.Delay(delay, ct));
}
public bool IsRunning { get; private set; }
/// <summary>Never throws: failures are logged and surfaced via <see cref="Failed"/>,
/// so the VM can fire-and-forget it from a sync command handler.</summary>
public async Task StartAsync(CancellationToken cancellationToken = default)
{
lock (_gate)
{
if (_started) return;
_started = true;
}
IFfmpegEncoder? encoder = null;
try
{
var options = _encoderOptions();
if (options == null)
{
_log?.Invoke("FramePump: no RTMP URL available (live-stream create lands in TASK 5) — encoder skipped");
lock (_gate) _started = false;
return;
}
encoder = _encoderFactory();
encoder.HealthUpdated += OnHealthUpdated;
encoder.ProcessFailed += OnProcessFailed;
await encoder.StartAsync(options, cancellationToken);
lock (_gate)
{
_encoder = encoder;
}
// IsRunning must be true before the loop starts: the loop reads it on
// its first iteration, and with a completed-task delay it can run
// synchronously on this thread before PumpAsync even returns.
IsRunning = true;
_cts = new CancellationTokenSource();
_pumpTask = PumpAsync(options, _cts.Token);
_log?.Invoke($"FramePump started ({options.Width}×{options.Height} @ {options.Fps} fps)");
}
catch (Exception ex)
{
_log?.Invoke($"FramePump: start failed: {ex.Message}");
if (encoder != null)
{
encoder.HealthUpdated -= OnHealthUpdated;
encoder.ProcessFailed -= OnProcessFailed;
try
{
encoder.Dispose();
}
catch (Exception disposeEx)
{
_log?.Invoke($"FramePump: disposing failed encoder: {disposeEx.Message}");
}
}
lock (_gate)
{
_started = false;
IsRunning = false;
}
Failed?.Invoke(this, ex.Message);
}
}
public async Task StopAsync(CancellationToken cancellationToken = default)
{
IFfmpegEncoder? encoder;
Task? pump;
lock (_gate)
{
if (!_started && _encoder == null) return;
_started = false;
IsRunning = false;
encoder = _encoder;
pump = _pumpTask;
_cts?.Cancel();
}
// Stop the encoder BEFORE awaiting the pump: closing its stdin unblocks a
// write stuck on pipe backpressure, otherwise the pump could await forever.
if (encoder != null)
{
try
{
await encoder.StopAsync(cancellationToken);
}
catch (Exception ex)
{
_log?.Invoke($"FramePump: encoder stop failed: {ex.Message}");
}
}
if (pump != null)
{
try
{
await pump;
}
catch (Exception ex)
{
_log?.Invoke($"FramePump: pump loop faulted during stop: {ex.Message}");
}
}
if (encoder != null)
{
encoder.HealthUpdated -= OnHealthUpdated;
encoder.ProcessFailed -= OnProcessFailed;
try
{
encoder.Dispose();
}
catch (Exception ex)
{
_log?.Invoke($"FramePump: encoder dispose failed: {ex.Message}");
}
}
lock (_gate)
{
_encoder = null;
_cts = null;
_pumpTask = null;
}
_log?.Invoke("FramePump stopped");
}
public void Dispose()
{
try
{
StopAsync().GetAwaiter().GetResult();
}
catch (Exception ex)
{
_log?.Invoke($"FramePump: dispose failed: {ex.Message}");
}
}
private async Task PumpAsync(EncoderOptions options, CancellationToken ct)
{
var interval = TimeSpan.FromSeconds(1d / Math.Max(1, options.Fps));
try
{
while (!ct.IsCancellationRequested)
{
var scene = _sceneProvider();
if (scene != null)
{
var frame = _compositor.Render(scene, _frameResolver, null, _compositorOptions());
IFfmpegEncoder? encoder;
lock (_gate) encoder = _encoder;
if (encoder == null) break; // _encoder is only cleared after the loop ends; defensive
await encoder.SubmitFrameAsync(frame, ct);
}
await _pacingDelay(interval, ct);
}
}
catch (OperationCanceledException)
{
// normal stop
}
catch (Exception ex)
{
// A failure while the pump is supposed to run (encoder died under us,
// scene provider faulted, ...) stops the pump and surfaces once.
if (ct.IsCancellationRequested)
{
_log?.Invoke($"FramePump: pump exited during stop: {ex.Message}");
}
else
{
_log?.Invoke($"FramePump: pump loop faulted: {ex.Message}");
Failed?.Invoke(this, ex.Message);
}
}
finally
{
lock (_gate) IsRunning = false;
}
}
private void OnHealthUpdated(object? sender, StreamHealth health) => HealthUpdated?.Invoke(this, health);
private void OnProcessFailed(object? sender, string message)
{
_log?.Invoke($"FramePump: encoder process failed: {message}");
Failed?.Invoke(this, message);
_ = StopAsync();
}
}
+9
View File
@@ -156,6 +156,15 @@ public sealed class ScreenCaptureManager : IDisposable
toStop.PreviewBitmap = null;
}
/// <summary>The most recent frame for a capture key, or null before the first
/// frame arrives (or if the key has no session). The live compositor reads
/// the backdrop from here.</summary>
public VideoFrame? GetLatestFrame(string key)
{
lock (_gate)
return _sessions.TryGetValue(key, out var session) ? session.LatestFrame : null;
}
public void Dispose()
{
List<CaptureSession> sessions;
+3 -2
View File
@@ -35,13 +35,14 @@ External-facing logic: YouTube API, persistence. See
| `SocialValidator.cs` | `ISocialValidator` seam + `HttpSocialValidator` default: validates a social handle/URL by constructing the canonical profile URL and issuing an HTTP GET — the "act of validation" before adding to the bar. 200/redirect = exists; 404/connection failure = rejected. Fediverse `@user@domain` additionally does a best-effort nodeinfo lookup (`/.well-known/nodeinfo``software.name`) so the entry can show the instance's real logo; nodeinfo failure still validates (generic fediverse glyph). If the identity domain's nodeinfo is blocked (SSO) but the bare root 302s to the real instance (YunoHost default-app subdomains), the lookup follows the redirect and asks that host instead. Constructor takes optional `HttpClient` for tests |
| `Encoder/EncoderOptions.cs` | One go-live's encoder config: full `RtmpUrl` (ingest + stream key), W×H/FPS/bitrate from the quality tier, `VideoEncoder` (null = probe), audio sample rate/channels, `GopSize` = FPS×4 (the ≤4s keyframe bound) |
| `Encoder/IFfmpegEncoder.cs` | **TASK 4 ship step 3 seam**: start/feed/stop the live encoder; `HealthUpdated` (parsed `StreamHealth`), `ProcessFailed` on unexpected non-zero exit. Constructor-injected locator + process factory (tests fake both) |
| `Encoder/FfmpegEncoder.cs` | The default encoder: spawn `ffmpeg.exe` (probe `-encoders` first → hardware NVENC/QSV/AMF, OpenH264 fallback — never libx264), feed raw BGRA frames into stdin, parse `-stats` lines into `StreamHealth`, graceful stop via stdin EOF + 10s kill watchdog. Not yet constructed by the app (ship step 5 wires it) |
| `Encoder/FfmpegEncoder.cs` | The default encoder: spawn `ffmpeg.exe` (probe `-encoders` first → hardware NVENC/QSV/AMF, OpenH264 fallback — never libx264), feed raw BGRA frames into stdin, parse `-stats` lines into `StreamHealth`, graceful stop via stdin EOF + 10s kill watchdog. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`), driven by the `FramePump` |
| `Encoder/IEncoderProcess.cs` | Seam around the spawned subprocess (probe + encoder): redirected stdin/stdout/stderr + exit control — the encoder never touches `Process` directly |
| `Encoder/FfmpegEncoderProcess.cs` | The real process wrapper (`Process` with all three std streams redirected) |
| `Encoder/FfmpegArgs.cs` | Pure FFmpeg command-line builder: rawvideo `pipe:0` input, `anullsrc` silent audio (WASAPI replaces it), H.264+AAC, closed GOP (`-g fps×4 -keyint_min -sc_threshold 0 -bf 0`, `yuv420p`), FLV → RTMP |
| `Encoder/FfmpegProgressParser.cs` | Pure parser for `frame=/fps=/size=/time=/bitrate=` stats lines → `FfmpegProgress` |
| `Encoder/FfmpegEncoderPicker.cs` | Pure H.264 encoder picker from `-encoders` output: NVENC → QSV → AMF → OpenH264; **never returns libx264** (GPL) |
| `Encoder/FfmpegLocator.cs` | The default locator: PATH probe wins, then the cache, then pull+extract (`ffmpeg.exe` **+ the `libav*.dll` family** via a staging dir, failures via `AppLog`). Pinned to BtbN `autobuild-2026-08-09-13-03` **lgpl-shared** (dynamic linking = LGPL compliance without static-relink §6 material; no GPL libx264; NVENC/QSV/AMF + libopenh264 + native AAC — see `ai.md` → Licensing). Search dirs / tools dir / downloader constructor-injected for hermetic tests. Not yet constructed by the app (the encoder step wires it) |
| `Encoder/FramePump.cs` | **TASK 4 ship step 5**: the live frame producer — while live it snapshots the active scene each tick, resolves every element to its latest frame (`Func<SceneElement, VideoFrame?>` resolver), composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. All collaborators constructor-injected seams; free of WPF and the capture managers. `StartAsync` never throws (failures log + `Failed`); no RTMP URL = encoder skipped; `StopAsync` stops the encoder before awaiting the loop (backpressure deadlock); `ProcessFailed` self-stops. See `ai.md` "Live frame pipeline" |
| `Encoder/FfmpegLocator.cs` | The default locator: PATH probe wins, then the cache, then pull+extract (`ffmpeg.exe` **+ the `libav*.dll` family** via a staging dir, failures via `AppLog`). Pinned to BtbN `autobuild-2026-08-09-13-03` **lgpl-shared** (dynamic linking = LGPL compliance without static-relink §6 material; no GPL libx264; NVENC/QSV/AMF + libopenh264 + native AAC — see `ai.md` → Licensing). Search dirs / tools dir / downloader constructor-injected for hermetic tests. Wired into the encoder since ship step 5 |
| `Audio/IAudioSource.cs` | **TASK 4 ship step 4 seam**: live capture source — `Start`/`Stop`/`SampleReady(AudioSample)`/`Failed(Exception)`, `IDisposable`; runs only while live. The app consumes this seam; tests inject hermetic fakes |
| `Audio/AudioSample.cs` | One captured chunk: interleaved PCM float (-1..1) + sample rate + channels |
| `Audio/WasapiLoopbackAudioSource.cs` | Desktop/game capture: NAudio `WasapiLoopbackCapture` on the default render device — automatic at unity, zero UI |
+52 -3
View File
@@ -201,7 +201,7 @@ Preview shows the transition too (WYSIWYG). No wipes/slides/LUTs beyond the four
2.**Ship step 2 — the FFmpeg locator SHIPPED** (2026-08-10)
3.**Encoder + RTMP push SHIPPED** (2026-08-12) — the FFmpeg subprocess: raw BGRA frames via stdin, stderr health parsing, FLV mux + push to the ingestion URL (see the ship step 3 plan below)
4.**WASAPI audio capture SHIPPED** (2026-08-12) — NAudio loopback (desktop/game) + the picked mic feeding `AudioLevel`, so the realtime meter comes alive (see the ship step 4 plan below)
5. **Frame-pipeline wiring**`CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder
5. **Frame-pipeline wiring SHIPPED** (2026-08-12)`CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven by a paced `FramePump` (see the ship step 5 plan below)
6.**Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (req 5)
7.**One-click go live + private-only enforcement** — Go Live always creates/updates the broadcast with `privacyStatus = "private"` + PRIVATE badge (req 8, test-verifiable)
@@ -423,8 +423,57 @@ lifecycle/forwarding/failure against fakes, meter RMS/smoothing/reset, `WaveToFl
extensible/truncation) — **139 passing**.
**Deferred (later ship steps):** wiring the desktop-capture samples into the encoder's AAC mix (replaces
the `-f lavfi -i anullsrc` placeholder; ship step 5 owns the encoder construction), WASAPI capture while
not live, and any audio UI beyond the existing mic controls.
the `-f lavfi -i anullsrc` placeholder; the encoder construction itself shipped in ship step 5), WASAPI
capture while not live, and any audio UI beyond the existing mic controls.
#### Ship step 5 — Frame-pipeline wiring (the encoder gets a frame source)
**Goal:** the chain `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven while
live by a paced frame pump: snapshot the active scene → resolve each element to its latest frame →
composite into the tier's output frame → pace into the encoder's stdin at the tier's FPS.
**Decisions (locked via user Q&A, 2026-08-12):**
1. **Video pipeline first** — the `-f lavfi -i anullsrc` silent track stays; mixing the loopback/mic
WASAPI samples into the encoder's AAC track is its own later step.
2. **RTMP URL via a provider seam** — `MainViewModel._rtmpUrlProvider` is a `Func<string?>` returning
null today (the reusable stream's ingest URL lands with TASK 5); when it yields null the pump logs
and skips the encoder entirely, so go-live runs the existing visual flow without pushing.
**Design:**
1. `Services/Encoder/FramePump.cs` — the frame producer. All collaborators constructor-injected seams
(`Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, injectable pacing delay) so it
stays free of WPF and of the capture managers and is hermetic in tests. `StartAsync` never throws
(failures log + surface via `Failed` — the VM fires-and-forgets from the sync command handler);
loop = snapshot → render → `SubmitFrameAsync`, paced at `1/options.Fps` (default `Task.Delay`; tests
inject `Task.Yield`). `StopAsync` stops the encoder (closes stdin) BEFORE awaiting the loop — closing
stdin unblocks a write stuck on pipe backpressure, so stop can't deadlock on the pump. `ProcessFailed`
self-stops the pump. `HealthUpdated` forwards the encoder's stats (ship step 6 binds the bottom bar).
2. `ScreenCaptureManager.GetLatestFrame(key)` — mirrors `CameraManager.GetLatestFrame(deviceId)`; the
backdrop's live frame for the compositor.
3. `MainViewModel` — owns the resolver (`WebcamSceneConfig` → `GetLatestFrame(WebcamId)`;
`Source.IsLiveCapture` → `GetLatestFrame(CaptureKey)`; image/background → `StaticPixelCache.Get(AssetId)`),
builds `CompositorOptions` from the tier + `OutputRect*` (doubles rounded to ints — the vertical
607.5 half-pixel crop rounds to a perfectly-centered 608), builds `EncoderOptions` from the tier when
the URL provider returns one, constructs the real `FfmpegEncoder(new FfmpegLocator())`, starts the
pump on go-live, stops it on end-stream, disposes in `Shutdown`, and flips `StreamStatus.Error` when
the pump fails while live (minimal — detailed health surfacing is ship step 6).
**Test plan (Good Dog Rule — ONE integration test):** `FramePumpTests.Start_CompositesScene_FeedsEncoder_StopsCleanly`
drives the full lifecycle against fakes — real `SceneCompositor` + real `FramePump`, fake `IFfmpegEncoder`
— asserting the composited red backdrop frame actually reaches the encoder at the tier size and that stop
tears everything down. Units: no-URL start skips the encoder, re-entrant start/stop no-ops, encoder
start-failure raises `Failed` + disposes, `ProcessFailed` self-stops the pump, `HealthUpdated` forwards.
`ScreenCaptureManagerTests.GetLatestFrame_ReturnsLatestPump_UntilReleased` pins the new accessor.
**Out of scope (later ship steps):** the loopback/mic → AAC mix (replaces `anullsrc`), health stats in the
bottom bar (ship step 6), scene-switching transitions, and any flash-frame wiring.
**Built (2026-08-12):** `FramePump` shipped in `Services/Encoder/`, `ScreenCaptureManager.GetLatestFrame`
added, `MainViewModel` wired end-to-end (resolver + both option builders + pump lifecycle), `FramePumpTests`
(7) + `GetLatestFrame` test (1) added — build **0 warnings**, **147 tests passing**. Known consideration:
the pump reads the active scene on a background thread while the UI can still edit it; a concurrent-mutation
exception is contained (logged + `Failed` + pump stops) rather than crashing.
---
+74
View File
@@ -15,6 +15,8 @@ using ytLive.Helpers;
using ytLive.Models;
using ytLive.Services;
using ytLive.Services.Audio;
using ytLive.Services.Compositor;
using ytLive.Services.Encoder;
namespace ytLive.ViewModels;
@@ -86,6 +88,12 @@ public class MainViewModel : ViewModelBase
// encoder mix. Private by design — no UI beyond the existing mic controls.
private readonly AudioMixer _audioMixer;
// Live frame pipeline (TASK 4 ship step 5): composites the active scene at the
// tier's FPS and paces frames into the encoder. The RTMP URL seam stays null
// until the live-stream create flow (TASK 5) supplies the reusable stream URL.
private readonly Func<string?> _rtmpUrlProvider;
private readonly FramePump _framePump;
// Screen backdrop: a permanent live capture (desktop/game) that every scene
// shows at the bottom layer. One shared capture session per key — the
// ScreenCaptureManager refcounts by key, mirroring CameraManager.
@@ -857,6 +865,16 @@ public class MainViewModel : ViewModelBase
_screenCaptureManager.CaptureFailed += (key, message) =>
AppLog.Write($"ScreenCaptureManager: capture '{key}' failed: {message}");
_rtmpUrlProvider = () => null; // TASK 5: the reusable stream's ingest URL
_framePump = new FramePump(
sceneProvider: () => ActiveScene,
frameResolver: ResolveOutputFrame,
compositorOptions: BuildCompositorOptions,
encoderOptions: BuildEncoderOptions,
encoderFactory: () => new FfmpegEncoder(new FfmpegLocator()),
log: message => AppLog.Write(message));
_framePump.Failed += OnFramePumpFailed;
LoadLayout();
_ = LoadSavedSessionAsync();
AppLog.Write("MainViewModel ctor end");
@@ -1169,6 +1187,7 @@ public class MainViewModel : ViewModelBase
{
_saveDebounce?.Stop();
SaveLayoutNow();
_framePump.Dispose();
_cameraManager.Dispose();
_screenCaptureManager.Dispose();
_layoutStore.Dispose();
@@ -1739,6 +1758,7 @@ public class MainViewModel : ViewModelBase
: $"{dialog.StreamTitle} — ytLlive";
StreamStatus = StreamStatus.Streaming;
_audioMixer.Start();
_ = _framePump.StartAsync(); // never throws; failures log + surface via Failed
}
}
@@ -1747,6 +1767,7 @@ public class MainViewModel : ViewModelBase
StreamStatus = StreamStatus.Offline;
WindowTitle = "ytLlive";
_audioMixer.Stop();
_ = _framePump.StopAsync();
// Graceful end completes the session = signs out (the DPAPI token is
// cleared so the next Start Stream requires a fresh sign-in). A crash
// never runs this, so the token survives and the creator stays signed in.
@@ -1768,6 +1789,59 @@ public class MainViewModel : ViewModelBase
AudioLevel = level;
}
// Scene-element → latest frame, for the live compositor. The map mirrors the
// preview: webcam by DeviceId, live captures (the backdrop) by CaptureKey,
// images/background by AssetId. A null frame leaves the element transparent.
private VideoFrame? ResolveOutputFrame(SceneElement element)
{
return element switch
{
WebcamSceneConfig webcam => _cameraManager.GetLatestFrame(webcam.WebcamId),
Source { IsLiveCapture: true, CaptureKey: not null } live => _screenCaptureManager.GetLatestFrame(live.CaptureKey),
Source { AssetId: not null } image => StaticPixelCache.Get(image.AssetId),
_ => null,
};
}
// The tier's crop rect over the 1920x1080 master, integer-aligned (the VM's
// OutputRect* are doubles — the vertical 607.5 half-pixel crop rounds to 608).
private CompositorOptions BuildCompositorOptions()
{
var quality = SelectedQuality;
return new CompositorOptions
{
SourceRectX = (int)Math.Round(OutputRectX),
SourceRectY = (int)Math.Round(OutputRectY),
SourceRectWidth = (int)Math.Round(OutputRectWidth),
SourceRectHeight = (int)Math.Round(OutputRectHeight),
OutputWidth = quality.Width,
OutputHeight = quality.Height,
};
}
// Full encoder options for the current tier, or null when no RTMP URL is
// available — the pump then skips the encoder entirely (TASK 5 fills the seam).
private EncoderOptions? BuildEncoderOptions()
{
var url = _rtmpUrlProvider();
if (string.IsNullOrWhiteSpace(url)) return null;
var quality = SelectedQuality;
return new EncoderOptions
{
RtmpUrl = url,
Width = quality.Width,
Height = quality.Height,
Fps = quality.Fps,
BitrateKbps = (int)Math.Round(quality.Bitrate * 1000),
};
}
private void OnFramePumpFailed(object? sender, string message)
{
AppLog.Write($"Frame pump failed: {message}");
if (IsLive) StreamStatus = StreamStatus.Error;
}
private void UpdateLiveVisuals()
{
var live = IsLive;
+35 -6
View File
@@ -91,7 +91,7 @@ C# / WPF (.NET 8) following MVVM:
|------|------|
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")** |
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
@@ -352,7 +352,8 @@ integration test fakes the whole subprocess (probe + encoder) with a Channel-bac
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
`GopSize` = FPS×4. Not yet constructed by the app — the frame-pipeline wiring (ship step 5) owns it.
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
driven by the `FramePump` below.
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12, plan in TASKS.md)
@@ -371,16 +372,44 @@ timers).
- **`AudioMixer`** owns both sources; `Start`/`Stop` follow go-live (`MainViewModel.BeginGoLive` success
`_audioMixer.Start()`, `StopStream``Stop()`). Mic samples feed a pure **`AudioLevelMeter`** (RMS
with 0.2 exponential smoothing) → `MicLevelChanged` → marshalled to the UI thread → `AudioLevel`.
Desktop samples are currently **dropped**the future encoder's AAC mix (ship step 5) consumes them,
replacing the `-f lavfi -i anullsrc` placeholder. Failures log via `AppLog`; a mic failure zeroes the
Desktop samples are currently **dropped**a later step's AAC mix consumes them, replacing the
`-f lavfi -i anullsrc` placeholder (the encoder construction itself shipped in ship step 5). Failures log via `AppLog`; a mic failure zeroes the
meter, a loopback failure never kills the mic.
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
- `FfmpegEncoder.cs:139` pre-existing CS8602 fixed (`process!`) — build **0 warnings**; **139 passing**.
**Deferred:** loopback→encoder mix wiring and the encoder construction (ship step 5); capture while not
live is deliberately not shipped (privacy indicator otherwise).
**Deferred:** the loopback→encoder AAC mix wiring (a later step — the encoder construction + frame
pipeline shipped in ship step 5); capture while not live is deliberately not shipped (privacy indicator
otherwise).
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, plan in TASKS.md)
The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active
scene each tick, resolves every element to its latest frame, composites it into the tier's output frame,
and paces frames into the encoder at the tier's FPS. **Pattern — everything is a constructor-injected
seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, and an injectable pacing delay
(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers.
- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log
+ surface via the `Failed` event. `EncoderOptions == null` means "no RTMP URL": the pump logs and skips
the encoder entirely. `MainViewModel._rtmpUrlProvider` is that seam — a `Func<string?>` returning null
until TASK 5 supplies the reusable stream's ingest URL, so go-live runs the current visual flow.
- **Stop ordering matters:** `StopAsync` stops the encoder (closes stdin → EOF → ffmpeg finalizes+exits)
**before** awaiting the loop, because closing stdin unblocks a write stuck on pipe backpressure — the
reverse order would deadlock. `ProcessFailed` self-stops the pump. `HealthUpdated` is forwarded
(ship step 6 binds it to the bottom bar); `Failed` while live flips `StreamStatus.Error` (minimal).
- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig`
`CameraManager.GetLatestFrame(WebcamId)`, `Source { IsLiveCapture, CaptureKey }`
`ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/
background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*`
doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`,
ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one.
- Known consideration: the pump reads the active scene on a background thread while the UI can still edit
it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than
crashing. The background thread + video pipeline is the new reality since this step.
### Social bar (TASK 14 — shipped 2026-08-12, plan in TASKS.md)
+221
View File
@@ -0,0 +1,221 @@
using Xunit;
using ytLive.Models;
using ytLive.Services;
using ytLive.Services.Compositor;
using ytLive.Services.Encoder;
namespace ytLive.Tests;
/// <summary>
/// The live frame producer (TASK 4 ship step 5): composites the active scene at
/// the tier's FPS and paces frames into the encoder. The integration test drives
/// the full lifecycle against fakes — real SceneCompositor + real FramePump, fake
/// IFfmpegEncoder — proving the composite frame actually reaches the encoder and
/// that stop tears the pump down cleanly. The units pin the failure edges: the
/// no-URL skip (the TASK 5 seam), re-entrancy, and encoder death.
/// </summary>
public class FramePumpTests
{
private sealed class FakeEncoder : IFfmpegEncoder
{
public readonly List<VideoFrame> Frames = new();
public int StartCount;
public int StopCount;
public bool Disposed;
public EncoderOptions? LastOptions;
public Exception? StartError;
public TaskCompletionSource FrameArrived = new(TaskCreationOptions.RunContinuationsAsynchronously);
public event EventHandler<StreamHealth>? HealthUpdated;
public event EventHandler<string>? ProcessFailed;
public Task StartAsync(EncoderOptions options, CancellationToken cancellationToken = default)
{
StartCount++;
LastOptions = options;
if (StartError != null) throw StartError;
return Task.CompletedTask;
}
public Task SubmitFrameAsync(VideoFrame frame, CancellationToken cancellationToken = default)
{
Frames.Add(frame);
FrameArrived.TrySetResult();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken = default)
{
StopCount++;
return Task.CompletedTask;
}
public void Dispose() => Disposed = true;
public void RaiseProcessFailed(string message) => ProcessFailed?.Invoke(this, message);
public void RaiseHealth(StreamHealth health) => HealthUpdated?.Invoke(this, health);
}
private static Scene BackdropScene()
{
var scene = new Scene { Name = "Live" };
scene.Elements.Add(new Source { Type = SourceType.DisplayCapture, IsBackdrop = true, CaptureKey = "monitor:0" });
return scene;
}
private static FramePump NewPump(FakeEncoder encoder, Func<EncoderOptions?>? options = null,
Func<Scene?>? scene = null, Func<SceneElement, VideoFrame?>? resolve = null,
List<string>? log = null)
{
return new FramePump(
sceneProvider: scene ?? (() => BackdropScene()),
frameResolver: resolve ?? (_ => null),
compositorOptions: () => new CompositorOptions
{
SourceRectX = 0, SourceRectY = 0, SourceRectWidth = 64, SourceRectHeight = 48,
OutputWidth = 64, OutputHeight = 48,
},
encoderOptions: options ?? (() => new EncoderOptions
{
RtmpUrl = "rtmp://a.rtmp.youtube.com/live2/abc",
Width = 64, Height = 48, Fps = 60,
}),
encoderFactory: () => encoder,
log: log != null ? m => log.Add(m) : null,
pacingDelay: async (_, _) => await Task.Yield()); // deterministic: no real waits
}
private static void AssertColor(VideoFrame frame, int x, int y, byte r, byte g, byte b)
{
var i = (y * frame.Width + x) * 4;
var br = frame.BgraPixels[i + 2];
var bg = frame.BgraPixels[i + 1];
var bb = frame.BgraPixels[i];
Assert.True(
Math.Abs(br - r) <= 2 && Math.Abs(bg - g) <= 2 && Math.Abs(bb - b) <= 2,
$"pixel ({x},{y}): expected rgb({r},{g},{b}), got rgb({br},{bg},{bb})");
}
[Fact]
public async Task Start_CompositesScene_FeedsEncoder_StopsCleanly()
{
var red = SceneCompositorTests.Solid(64, 48, 255, 0, 0);
var encoder = new FakeEncoder();
using var pump = NewPump(encoder,
resolve: e => e is Source { IsBackdrop: true } ? red : null);
await pump.StartAsync();
Assert.True(pump.IsRunning);
Assert.Equal(1, encoder.StartCount);
await encoder.FrameArrived.Task.WaitAsync(TimeSpan.FromSeconds(5));
Assert.NotEmpty(encoder.Frames);
var frame = encoder.Frames[0];
Assert.Equal(64, frame.Width);
Assert.Equal(48, frame.Height);
AssertColor(frame, 0, 0, 255, 0, 0); // the backdrop really was composited in
await pump.StopAsync();
Assert.False(pump.IsRunning);
Assert.Equal(1, encoder.StopCount);
Assert.True(encoder.Disposed);
}
[Fact]
public async Task Start_WithoutRtmpUrl_SkipsEncoder()
{
var encoder = new FakeEncoder();
var log = new List<string>();
using var pump = NewPump(encoder, options: () => null, log: log);
await pump.StartAsync();
Assert.False(pump.IsRunning);
Assert.Equal(0, encoder.StartCount);
Assert.Contains(log, m => m.Contains("RTMP"));
await pump.StopAsync(); // no-op after a skipped start
Assert.Equal(0, encoder.StopCount);
}
[Fact]
public async Task Start_WhileRunning_IsNoop()
{
var encoder = new FakeEncoder();
using var pump = NewPump(encoder);
await pump.StartAsync();
await pump.StartAsync();
Assert.Equal(1, encoder.StartCount);
}
[Fact]
public async Task Stop_WithoutStart_IsNoop()
{
var encoder = new FakeEncoder();
using var pump = NewPump(encoder);
await pump.StopAsync();
Assert.Equal(0, encoder.StopCount);
Assert.False(pump.IsRunning);
}
[Fact]
public async Task Start_EncoderThrows_RaisesFailed_AndDisposes()
{
var encoder = new FakeEncoder { StartError = new InvalidOperationException("access denied") };
var failed = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
using var pump = NewPump(encoder);
pump.Failed += (_, m) => failed.TrySetResult(m);
await pump.StartAsync();
Assert.False(pump.IsRunning);
var message = await failed.Task.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Contains("access denied", message);
Assert.True(encoder.Disposed);
}
[Fact]
public async Task ProcessDeath_StopsPump_AndRaisesFailed()
{
var encoder = new FakeEncoder();
var failed = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
using var pump = NewPump(encoder);
pump.Failed += (_, m) => failed.TrySetResult(m);
await pump.StartAsync();
encoder.RaiseProcessFailed("FFmpeg exited with code 1");
var message = await failed.Task.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Contains("1", message);
// The pump self-stops (fire-and-forget); poll the definitive teardown
// marker — the encoder's disposal — rather than the IsRunning flag, which
// StopAsync clears before the loop has fully drained.
var deadline = DateTime.UtcNow.AddSeconds(5);
while (!encoder.Disposed && DateTime.UtcNow < deadline)
await Task.Delay(10);
Assert.True(encoder.Disposed);
Assert.False(pump.IsRunning);
}
[Fact]
public async Task HealthUpdated_ForwardsEncoderHealth()
{
var encoder = new FakeEncoder();
var health = new TaskCompletionSource<StreamHealth>(TaskCreationOptions.RunContinuationsAsynchronously);
using var pump = NewPump(encoder);
pump.HealthUpdated += (_, h) => health.TrySetResult(h);
await pump.StartAsync();
encoder.RaiseHealth(new StreamHealth { Status = StreamStatus.Streaming, FPS = 59.9 });
var h = await health.Task.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Equal(StreamStatus.Streaming, h.Status);
Assert.Equal(59.9, h.FPS, 1);
}
}
+22
View File
@@ -145,6 +145,28 @@ public class ScreenCaptureManagerTests
Assert.False(await manager.AcquireAsync(" "));
}
[Fact]
public async Task GetLatestFrame_ReturnsLatestPump_UntilReleased()
{
FakeScreenSource? captured = null;
var manager = new ScreenCaptureManager(key => captured = new FakeScreenSource(key));
Assert.Null(manager.GetLatestFrame("monitor:0"));
Assert.True(await manager.AcquireAsync("monitor:0"));
Assert.Null(manager.GetLatestFrame("monitor:0")); // nothing pumped yet
var first = new VideoFrame(2, 2, Pixels(1, 0, 0, 255, 2, 0, 0, 255, 3, 0, 0, 255, 4, 0, 0, 255));
var second = new VideoFrame(2, 2, Pixels(5, 0, 0, 255, 6, 0, 0, 255, 7, 0, 0, 255, 8, 0, 0, 255));
captured!.Pump(first);
captured.Pump(second);
Assert.Same(second, manager.GetLatestFrame("monitor:0"));
await manager.ReleaseAsync("monitor:0");
Assert.Null(manager.GetLatestFrame("monitor:0"));
}
// The one integration test for this branch: one target creates one shared
// WriteableBitmap, published once, and back-to-back frames coalesce to the
// latest (a single pending UI copy per session).