TASK 4 ship step 5: live frame pipeline — FramePump paces the active scene's composite into the encoder, ScreenCaptureManager.GetLatestFrame, MainViewModel resolver/option-builder/pump wiring, FramePumpTests (7) + GetLatestFrame test — 147 tests passing, 0 warnings
This commit is contained in:
+28
-43
@@ -5,52 +5,37 @@
|
|||||||
> a problem. Conventions: [`schema.md`](schema.md). Rewrite this file at session
|
> a problem. Conventions: [`schema.md`](schema.md). Rewrite this file at session
|
||||||
> end, compaction, or any interruption.
|
> end, compaction, or any interruption.
|
||||||
|
|
||||||
## Session state (last updated: 2026-08-12)
|
## Session state (last updated: 2026-08-13)
|
||||||
|
|
||||||
- **Branch:** `main`. TASK 4 ship step 4 (WASAPI audio capture) is **built and
|
- **Branch:** `main`. TASK 4 ship step 5 (the live frame pipeline) is **shipped
|
||||||
tested** but **NOT committed** — the working tree is dirty with the audio
|
and committed** — the `FramePump` frame producer
|
||||||
layer (6 new files in `Services/Audio/` + `ytLive.Tests/AudioMixerTests.cs`,
|
(`Services/Encoder/FramePump.cs`), `ScreenCaptureManager.GetLatestFrame(key)`,
|
||||||
the `NAudio.Wasapi` 2.2.1 package reference, the THIRD-PARTY-NOTICES entry,
|
the `MainViewModel` resolver/option-builders/pump lifecycle wiring, `FramePumpTests`
|
||||||
the `MainViewModel` wiring, the `FfmpegEncoder.cs:139` CS8602 fix) and its
|
(7) + the `GetLatestFrame` test (1), and the memory updates (TASKS.md, ai.md,
|
||||||
memory updates (TASKS.md, ai.md, Services/index.md, HANDOFF.md). Ready to
|
Services/index.md — the previous session left `Services/index.md` stale and
|
||||||
commit when the user says so.
|
HANDOFF unrewritten; both were fixed before the commit). Build **0 warnings**,
|
||||||
- **Finished this session:** TASK 4 ship step 4 — the live audio capture layer:
|
**147 tests passing**. A Windows patch reboot killed the prior session right
|
||||||
`IAudioSource`/`AudioSample` seam, `WasapiLoopbackAudioSource`
|
after the work was verified but before the commit.
|
||||||
(`WasapiLoopbackCapture` on the default render device), `WasapiMicAudioSource`
|
- **Finished this session:** ship step 5 (re-verified: 0 warnings, 147/147 pass),
|
||||||
(`WasapiCapture`, NAudio device resolved by `FriendlyName` matching
|
the `Services/index.md` FramePump row + de-stale'd encoder rows, and this
|
||||||
`MicSourceName` via a re-read `Func<string?>` provider, default-endpoint
|
HANDOFF rewrite.
|
||||||
fallback), `AudioMixer` (owns both sources, starts/stops with go-live, feeds
|
|
||||||
the pure `AudioLevelMeter` → `MicLevelChanged`), `WaveToFloat` (IEEE-float /
|
|
||||||
PCM16 / extensible). `MainViewModel` constructs the mixer, `BeginGoLive`
|
|
||||||
success → `Start()`, `StopStream` → `Stop()`, `MicLevelChanged` marshalled to
|
|
||||||
the UI thread → `AudioLevel`. Loopback samples are currently dropped (the
|
|
||||||
encoder's AAC mix, ship step 5, consumes them). Build: **0 warnings** (a
|
|
||||||
pre-existing CS8602 in `FfmpegEncoder.cs:139` surfaced during the rebuild and
|
|
||||||
was fixed with `process!`). Tests: **139 passing** (was 122; +17 new audio).
|
|
||||||
- **Landmines:**
|
- **Landmines:**
|
||||||
- `WaveFormatExtensible.SubFormat` is a **`Guid`** in NAudio 2.x, not a
|
- The pump reads the active scene on a background thread while the UI can still
|
||||||
`WaveFormatEncoding` — compare it to
|
edit it — a concurrent-mutation exception is contained (logged + `Failed` +
|
||||||
`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT` (the IEEE-float
|
the pump stops), not a crash. The background thread + video pipeline is the
|
||||||
subtype constant; it lives in the `NAudio.Dmo` namespace, not CoreAudioApi).
|
new reality since ship step 5.
|
||||||
- Use the **`NAudio.Wasapi` 2.2.1 feature package**, not the `NAudio`
|
- `StopAsync` must stop the encoder (closes stdin) **before** awaiting the pump
|
||||||
meta-package — the wasapi types come from that package (`WasapiCapture` in
|
loop — closing stdin unblocks a write stuck on pipe backpressure; the reverse
|
||||||
`NAudio.CoreAudioApi`, `WasapiLoopbackCapture` in `NAudio.Wave`); `NAudio.Core`
|
order deadlocks.
|
||||||
comes in transitively.
|
- `FramePump.IsRunning` must be set true before the loop starts (a completed-task
|
||||||
- The audio meter is an exponential smoother (0.2 factor) — a single pushed
|
delay can run the first iteration synchronously on the caller's thread).
|
||||||
sample only moves 20% toward its RMS. Tests must push repeatedly before
|
- `StartAsync` never throws; the VM fires-and-forgets it. `Failed` while live
|
||||||
asserting a converged level.
|
flips `StreamStatus.Error` (minimal — real health surfacing is ship step 6).
|
||||||
- Tests never instantiate `MainViewModel` directly except the round-clip
|
- Tests never instantiate `MainViewModel` directly except the round-clip
|
||||||
integration test (a real `MainWindow`), which never goes live — so the mixer
|
integration test (a real `MainWindow`), which never goes live — keep it that way.
|
||||||
is constructed but never started there; NAudio types are only constructed,
|
- **Next step:** TASK 4 ship step 6 — health stats: bind `FramePump.HealthUpdated`
|
||||||
never touching devices. Keep it that way.
|
(bitrate/FPS/duration) into the bottom bar. Nothing else queued — do not expand
|
||||||
- Mic device resolution must re-read `MicSourceName` at each `Start` (the app
|
the task queue on your own.
|
||||||
persists only the DisplayName, not a device ID).
|
|
||||||
- Windows-only: all capture runs only while live (privacy indicator otherwise).
|
|
||||||
|
|
||||||
- **Next step:** TASK 4 ship step 5 — the frame-pipeline wiring
|
|
||||||
(`CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder
|
|
||||||
construction, plus the loopback → encoder AAC mix). Nothing else queued — do
|
|
||||||
not expand the task queue on your own.
|
|
||||||
|
|
||||||
- **Secret/DB/port facts live:** OAuth client id/secret in `Helpers/OAuthCredentials.cs`;
|
- **Secret/DB/port facts live:** OAuth client id/secret in `Helpers/OAuthCredentials.cs`;
|
||||||
OAuth session token in `Helpers/TokenStore.cs` (DPAPI → `%APPDATA%\ytLlive\ytLlive.auth`);
|
OAuth session token in `Helpers/TokenStore.cs` (DPAPI → `%APPDATA%\ytLlive\ytLlive.auth`);
|
||||||
|
|||||||
@@ -0,0 +1,256 @@
|
|||||||
|
using System;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using ytLive.Models;
|
||||||
|
using ytLive.Services.Compositor;
|
||||||
|
|
||||||
|
namespace ytLive.Services.Encoder;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The live frame producer (TASK 4 ship step 5): the bridge between the capture
|
||||||
|
/// managers + compositor and the encoder. While live it snapshots the active
|
||||||
|
/// scene each tick, resolves every element to its latest frame, composites it
|
||||||
|
/// into the tier's output frame, and paces frames into the encoder at the tier's
|
||||||
|
/// FPS. All collaborators are constructor-injected seams (scene, resolver,
|
||||||
|
/// options, encoder factory, pacing delay) so the pump stays free of WPF and of
|
||||||
|
/// the capture managers and is fully hermetic in tests.
|
||||||
|
///
|
||||||
|
/// The RTMP URL comes from the options provider: until the live-stream create
|
||||||
|
/// flow lands (TASK 5) it yields null, so go-live runs the existing visual flow
|
||||||
|
/// without actually pushing.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class FramePump : IDisposable
|
||||||
|
{
|
||||||
|
private readonly Func<Scene?> _sceneProvider;
|
||||||
|
private readonly Func<SceneElement, VideoFrame?> _frameResolver;
|
||||||
|
private readonly Func<CompositorOptions> _compositorOptions;
|
||||||
|
private readonly Func<EncoderOptions?> _encoderOptions;
|
||||||
|
private readonly Func<IFfmpegEncoder> _encoderFactory;
|
||||||
|
private readonly Action<string>? _log;
|
||||||
|
private readonly Func<TimeSpan, CancellationToken, Task> _pacingDelay;
|
||||||
|
private readonly SceneCompositor _compositor = new();
|
||||||
|
|
||||||
|
private readonly object _gate = new();
|
||||||
|
private IFfmpegEncoder? _encoder;
|
||||||
|
private CancellationTokenSource? _cts;
|
||||||
|
private Task? _pumpTask;
|
||||||
|
private bool _started;
|
||||||
|
|
||||||
|
/// <summary>Forwards the encoder's parsed health — ship step 6 binds this to the bottom bar.</summary>
|
||||||
|
public event EventHandler<StreamHealth>? HealthUpdated;
|
||||||
|
|
||||||
|
/// <summary>Raised when the encoder cannot start or dies mid-stream. The pump stops itself.</summary>
|
||||||
|
public event EventHandler<string>? Failed;
|
||||||
|
|
||||||
|
public FramePump(
|
||||||
|
Func<Scene?> sceneProvider,
|
||||||
|
Func<SceneElement, VideoFrame?> frameResolver,
|
||||||
|
Func<CompositorOptions> compositorOptions,
|
||||||
|
Func<EncoderOptions?> encoderOptions,
|
||||||
|
Func<IFfmpegEncoder> encoderFactory,
|
||||||
|
Action<string>? log = null,
|
||||||
|
Func<TimeSpan, CancellationToken, Task>? pacingDelay = null)
|
||||||
|
{
|
||||||
|
_sceneProvider = sceneProvider ?? throw new ArgumentNullException(nameof(sceneProvider));
|
||||||
|
_frameResolver = frameResolver ?? throw new ArgumentNullException(nameof(frameResolver));
|
||||||
|
_compositorOptions = compositorOptions ?? throw new ArgumentNullException(nameof(compositorOptions));
|
||||||
|
_encoderOptions = encoderOptions ?? throw new ArgumentNullException(nameof(encoderOptions));
|
||||||
|
_encoderFactory = encoderFactory ?? throw new ArgumentNullException(nameof(encoderFactory));
|
||||||
|
_log = log;
|
||||||
|
_pacingDelay = pacingDelay ?? ((delay, ct) => Task.Delay(delay, ct));
|
||||||
|
}
|
||||||
|
|
||||||
|
public bool IsRunning { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>Never throws: failures are logged and surfaced via <see cref="Failed"/>,
|
||||||
|
/// so the VM can fire-and-forget it from a sync command handler.</summary>
|
||||||
|
public async Task StartAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
if (_started) return;
|
||||||
|
_started = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
IFfmpegEncoder? encoder = null;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var options = _encoderOptions();
|
||||||
|
if (options == null)
|
||||||
|
{
|
||||||
|
_log?.Invoke("FramePump: no RTMP URL available (live-stream create lands in TASK 5) — encoder skipped");
|
||||||
|
lock (_gate) _started = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
encoder = _encoderFactory();
|
||||||
|
encoder.HealthUpdated += OnHealthUpdated;
|
||||||
|
encoder.ProcessFailed += OnProcessFailed;
|
||||||
|
await encoder.StartAsync(options, cancellationToken);
|
||||||
|
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
_encoder = encoder;
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsRunning must be true before the loop starts: the loop reads it on
|
||||||
|
// its first iteration, and with a completed-task delay it can run
|
||||||
|
// synchronously on this thread before PumpAsync even returns.
|
||||||
|
IsRunning = true;
|
||||||
|
_cts = new CancellationTokenSource();
|
||||||
|
_pumpTask = PumpAsync(options, _cts.Token);
|
||||||
|
_log?.Invoke($"FramePump started ({options.Width}×{options.Height} @ {options.Fps} fps)");
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: start failed: {ex.Message}");
|
||||||
|
if (encoder != null)
|
||||||
|
{
|
||||||
|
encoder.HealthUpdated -= OnHealthUpdated;
|
||||||
|
encoder.ProcessFailed -= OnProcessFailed;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
encoder.Dispose();
|
||||||
|
}
|
||||||
|
catch (Exception disposeEx)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: disposing failed encoder: {disposeEx.Message}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
_started = false;
|
||||||
|
IsRunning = false;
|
||||||
|
}
|
||||||
|
Failed?.Invoke(this, ex.Message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task StopAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
IFfmpegEncoder? encoder;
|
||||||
|
Task? pump;
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
if (!_started && _encoder == null) return;
|
||||||
|
_started = false;
|
||||||
|
IsRunning = false;
|
||||||
|
encoder = _encoder;
|
||||||
|
pump = _pumpTask;
|
||||||
|
_cts?.Cancel();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop the encoder BEFORE awaiting the pump: closing its stdin unblocks a
|
||||||
|
// write stuck on pipe backpressure, otherwise the pump could await forever.
|
||||||
|
if (encoder != null)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await encoder.StopAsync(cancellationToken);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: encoder stop failed: {ex.Message}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (pump != null)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await pump;
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: pump loop faulted during stop: {ex.Message}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (encoder != null)
|
||||||
|
{
|
||||||
|
encoder.HealthUpdated -= OnHealthUpdated;
|
||||||
|
encoder.ProcessFailed -= OnProcessFailed;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
encoder.Dispose();
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: encoder dispose failed: {ex.Message}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
_encoder = null;
|
||||||
|
_cts = null;
|
||||||
|
_pumpTask = null;
|
||||||
|
}
|
||||||
|
_log?.Invoke("FramePump stopped");
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
StopAsync().GetAwaiter().GetResult();
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: dispose failed: {ex.Message}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task PumpAsync(EncoderOptions options, CancellationToken ct)
|
||||||
|
{
|
||||||
|
var interval = TimeSpan.FromSeconds(1d / Math.Max(1, options.Fps));
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
while (!ct.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
var scene = _sceneProvider();
|
||||||
|
if (scene != null)
|
||||||
|
{
|
||||||
|
var frame = _compositor.Render(scene, _frameResolver, null, _compositorOptions());
|
||||||
|
IFfmpegEncoder? encoder;
|
||||||
|
lock (_gate) encoder = _encoder;
|
||||||
|
if (encoder == null) break; // _encoder is only cleared after the loop ends; defensive
|
||||||
|
await encoder.SubmitFrameAsync(frame, ct);
|
||||||
|
}
|
||||||
|
await _pacingDelay(interval, ct);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
// normal stop
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
// A failure while the pump is supposed to run (encoder died under us,
|
||||||
|
// scene provider faulted, ...) stops the pump and surfaces once.
|
||||||
|
if (ct.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: pump exited during stop: {ex.Message}");
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: pump loop faulted: {ex.Message}");
|
||||||
|
Failed?.Invoke(this, ex.Message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
lock (_gate) IsRunning = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnHealthUpdated(object? sender, StreamHealth health) => HealthUpdated?.Invoke(this, health);
|
||||||
|
|
||||||
|
private void OnProcessFailed(object? sender, string message)
|
||||||
|
{
|
||||||
|
_log?.Invoke($"FramePump: encoder process failed: {message}");
|
||||||
|
Failed?.Invoke(this, message);
|
||||||
|
_ = StopAsync();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -156,6 +156,15 @@ public sealed class ScreenCaptureManager : IDisposable
|
|||||||
toStop.PreviewBitmap = null;
|
toStop.PreviewBitmap = null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>The most recent frame for a capture key, or null before the first
|
||||||
|
/// frame arrives (or if the key has no session). The live compositor reads
|
||||||
|
/// the backdrop from here.</summary>
|
||||||
|
public VideoFrame? GetLatestFrame(string key)
|
||||||
|
{
|
||||||
|
lock (_gate)
|
||||||
|
return _sessions.TryGetValue(key, out var session) ? session.LatestFrame : null;
|
||||||
|
}
|
||||||
|
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
List<CaptureSession> sessions;
|
List<CaptureSession> sessions;
|
||||||
|
|||||||
+3
-2
@@ -35,13 +35,14 @@ External-facing logic: YouTube API, persistence. See
|
|||||||
| `SocialValidator.cs` | `ISocialValidator` seam + `HttpSocialValidator` default: validates a social handle/URL by constructing the canonical profile URL and issuing an HTTP GET — the "act of validation" before adding to the bar. 200/redirect = exists; 404/connection failure = rejected. Fediverse `@user@domain` additionally does a best-effort nodeinfo lookup (`/.well-known/nodeinfo` → `software.name`) so the entry can show the instance's real logo; nodeinfo failure still validates (generic fediverse glyph). If the identity domain's nodeinfo is blocked (SSO) but the bare root 302s to the real instance (YunoHost default-app subdomains), the lookup follows the redirect and asks that host instead. Constructor takes optional `HttpClient` for tests |
|
| `SocialValidator.cs` | `ISocialValidator` seam + `HttpSocialValidator` default: validates a social handle/URL by constructing the canonical profile URL and issuing an HTTP GET — the "act of validation" before adding to the bar. 200/redirect = exists; 404/connection failure = rejected. Fediverse `@user@domain` additionally does a best-effort nodeinfo lookup (`/.well-known/nodeinfo` → `software.name`) so the entry can show the instance's real logo; nodeinfo failure still validates (generic fediverse glyph). If the identity domain's nodeinfo is blocked (SSO) but the bare root 302s to the real instance (YunoHost default-app subdomains), the lookup follows the redirect and asks that host instead. Constructor takes optional `HttpClient` for tests |
|
||||||
| `Encoder/EncoderOptions.cs` | One go-live's encoder config: full `RtmpUrl` (ingest + stream key), W×H/FPS/bitrate from the quality tier, `VideoEncoder` (null = probe), audio sample rate/channels, `GopSize` = FPS×4 (the ≤4s keyframe bound) |
|
| `Encoder/EncoderOptions.cs` | One go-live's encoder config: full `RtmpUrl` (ingest + stream key), W×H/FPS/bitrate from the quality tier, `VideoEncoder` (null = probe), audio sample rate/channels, `GopSize` = FPS×4 (the ≤4s keyframe bound) |
|
||||||
| `Encoder/IFfmpegEncoder.cs` | **TASK 4 ship step 3 seam**: start/feed/stop the live encoder; `HealthUpdated` (parsed `StreamHealth`), `ProcessFailed` on unexpected non-zero exit. Constructor-injected locator + process factory (tests fake both) |
|
| `Encoder/IFfmpegEncoder.cs` | **TASK 4 ship step 3 seam**: start/feed/stop the live encoder; `HealthUpdated` (parsed `StreamHealth`), `ProcessFailed` on unexpected non-zero exit. Constructor-injected locator + process factory (tests fake both) |
|
||||||
| `Encoder/FfmpegEncoder.cs` | The default encoder: spawn `ffmpeg.exe` (probe `-encoders` first → hardware NVENC/QSV/AMF, OpenH264 fallback — never libx264), feed raw BGRA frames into stdin, parse `-stats` lines into `StreamHealth`, graceful stop via stdin EOF + 10s kill watchdog. Not yet constructed by the app (ship step 5 wires it) |
|
| `Encoder/FfmpegEncoder.cs` | The default encoder: spawn `ffmpeg.exe` (probe `-encoders` first → hardware NVENC/QSV/AMF, OpenH264 fallback — never libx264), feed raw BGRA frames into stdin, parse `-stats` lines into `StreamHealth`, graceful stop via stdin EOF + 10s kill watchdog. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`), driven by the `FramePump` |
|
||||||
| `Encoder/IEncoderProcess.cs` | Seam around the spawned subprocess (probe + encoder): redirected stdin/stdout/stderr + exit control — the encoder never touches `Process` directly |
|
| `Encoder/IEncoderProcess.cs` | Seam around the spawned subprocess (probe + encoder): redirected stdin/stdout/stderr + exit control — the encoder never touches `Process` directly |
|
||||||
| `Encoder/FfmpegEncoderProcess.cs` | The real process wrapper (`Process` with all three std streams redirected) |
|
| `Encoder/FfmpegEncoderProcess.cs` | The real process wrapper (`Process` with all three std streams redirected) |
|
||||||
| `Encoder/FfmpegArgs.cs` | Pure FFmpeg command-line builder: rawvideo `pipe:0` input, `anullsrc` silent audio (WASAPI replaces it), H.264+AAC, closed GOP (`-g fps×4 -keyint_min -sc_threshold 0 -bf 0`, `yuv420p`), FLV → RTMP |
|
| `Encoder/FfmpegArgs.cs` | Pure FFmpeg command-line builder: rawvideo `pipe:0` input, `anullsrc` silent audio (WASAPI replaces it), H.264+AAC, closed GOP (`-g fps×4 -keyint_min -sc_threshold 0 -bf 0`, `yuv420p`), FLV → RTMP |
|
||||||
| `Encoder/FfmpegProgressParser.cs` | Pure parser for `frame=/fps=/size=/time=/bitrate=` stats lines → `FfmpegProgress` |
|
| `Encoder/FfmpegProgressParser.cs` | Pure parser for `frame=/fps=/size=/time=/bitrate=` stats lines → `FfmpegProgress` |
|
||||||
| `Encoder/FfmpegEncoderPicker.cs` | Pure H.264 encoder picker from `-encoders` output: NVENC → QSV → AMF → OpenH264; **never returns libx264** (GPL) |
|
| `Encoder/FfmpegEncoderPicker.cs` | Pure H.264 encoder picker from `-encoders` output: NVENC → QSV → AMF → OpenH264; **never returns libx264** (GPL) |
|
||||||
| `Encoder/FfmpegLocator.cs` | The default locator: PATH probe wins, then the cache, then pull+extract (`ffmpeg.exe` **+ the `libav*.dll` family** via a staging dir, failures via `AppLog`). Pinned to BtbN `autobuild-2026-08-09-13-03` **lgpl-shared** (dynamic linking = LGPL compliance without static-relink §6 material; no GPL libx264; NVENC/QSV/AMF + libopenh264 + native AAC — see `ai.md` → Licensing). Search dirs / tools dir / downloader constructor-injected for hermetic tests. Not yet constructed by the app (the encoder step wires it) |
|
| `Encoder/FramePump.cs` | **TASK 4 ship step 5**: the live frame producer — while live it snapshots the active scene each tick, resolves every element to its latest frame (`Func<SceneElement, VideoFrame?>` resolver), composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. All collaborators constructor-injected seams; free of WPF and the capture managers. `StartAsync` never throws (failures log + `Failed`); no RTMP URL = encoder skipped; `StopAsync` stops the encoder before awaiting the loop (backpressure deadlock); `ProcessFailed` self-stops. See `ai.md` "Live frame pipeline" |
|
||||||
|
| `Encoder/FfmpegLocator.cs` | The default locator: PATH probe wins, then the cache, then pull+extract (`ffmpeg.exe` **+ the `libav*.dll` family** via a staging dir, failures via `AppLog`). Pinned to BtbN `autobuild-2026-08-09-13-03` **lgpl-shared** (dynamic linking = LGPL compliance without static-relink §6 material; no GPL libx264; NVENC/QSV/AMF + libopenh264 + native AAC — see `ai.md` → Licensing). Search dirs / tools dir / downloader constructor-injected for hermetic tests. Wired into the encoder since ship step 5 |
|
||||||
| `Audio/IAudioSource.cs` | **TASK 4 ship step 4 seam**: live capture source — `Start`/`Stop`/`SampleReady(AudioSample)`/`Failed(Exception)`, `IDisposable`; runs only while live. The app consumes this seam; tests inject hermetic fakes |
|
| `Audio/IAudioSource.cs` | **TASK 4 ship step 4 seam**: live capture source — `Start`/`Stop`/`SampleReady(AudioSample)`/`Failed(Exception)`, `IDisposable`; runs only while live. The app consumes this seam; tests inject hermetic fakes |
|
||||||
| `Audio/AudioSample.cs` | One captured chunk: interleaved PCM float (-1..1) + sample rate + channels |
|
| `Audio/AudioSample.cs` | One captured chunk: interleaved PCM float (-1..1) + sample rate + channels |
|
||||||
| `Audio/WasapiLoopbackAudioSource.cs` | Desktop/game capture: NAudio `WasapiLoopbackCapture` on the default render device — automatic at unity, zero UI |
|
| `Audio/WasapiLoopbackAudioSource.cs` | Desktop/game capture: NAudio `WasapiLoopbackCapture` on the default render device — automatic at unity, zero UI |
|
||||||
|
|||||||
@@ -201,7 +201,7 @@ Preview shows the transition too (WYSIWYG). No wipes/slides/LUTs beyond the four
|
|||||||
2. ✅ **Ship step 2 — the FFmpeg locator SHIPPED** (2026-08-10)
|
2. ✅ **Ship step 2 — the FFmpeg locator SHIPPED** (2026-08-10)
|
||||||
3. ✅ **Encoder + RTMP push SHIPPED** (2026-08-12) — the FFmpeg subprocess: raw BGRA frames via stdin, stderr health parsing, FLV mux + push to the ingestion URL (see the ship step 3 plan below)
|
3. ✅ **Encoder + RTMP push SHIPPED** (2026-08-12) — the FFmpeg subprocess: raw BGRA frames via stdin, stderr health parsing, FLV mux + push to the ingestion URL (see the ship step 3 plan below)
|
||||||
4. ✅ **WASAPI audio capture SHIPPED** (2026-08-12) — NAudio loopback (desktop/game) + the picked mic feeding `AudioLevel`, so the realtime meter comes alive (see the ship step 4 plan below)
|
4. ✅ **WASAPI audio capture SHIPPED** (2026-08-12) — NAudio loopback (desktop/game) + the picked mic feeding `AudioLevel`, so the realtime meter comes alive (see the ship step 4 plan below)
|
||||||
5. ☐ **Frame-pipeline wiring** — `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder
|
5. ✅ **Frame-pipeline wiring SHIPPED** (2026-08-12) — `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven by a paced `FramePump` (see the ship step 5 plan below)
|
||||||
6. ☐ **Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (req 5)
|
6. ☐ **Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (req 5)
|
||||||
7. ☐ **One-click go live + private-only enforcement** — Go Live always creates/updates the broadcast with `privacyStatus = "private"` + PRIVATE badge (req 8, test-verifiable)
|
7. ☐ **One-click go live + private-only enforcement** — Go Live always creates/updates the broadcast with `privacyStatus = "private"` + PRIVATE badge (req 8, test-verifiable)
|
||||||
|
|
||||||
@@ -423,8 +423,57 @@ lifecycle/forwarding/failure against fakes, meter RMS/smoothing/reset, `WaveToFl
|
|||||||
extensible/truncation) — **139 passing**.
|
extensible/truncation) — **139 passing**.
|
||||||
|
|
||||||
**Deferred (later ship steps):** wiring the desktop-capture samples into the encoder's AAC mix (replaces
|
**Deferred (later ship steps):** wiring the desktop-capture samples into the encoder's AAC mix (replaces
|
||||||
the `-f lavfi -i anullsrc` placeholder; ship step 5 owns the encoder construction), WASAPI capture while
|
the `-f lavfi -i anullsrc` placeholder; the encoder construction itself shipped in ship step 5), WASAPI
|
||||||
not live, and any audio UI beyond the existing mic controls.
|
capture while not live, and any audio UI beyond the existing mic controls.
|
||||||
|
|
||||||
|
#### Ship step 5 — Frame-pipeline wiring (the encoder gets a frame source)
|
||||||
|
|
||||||
|
**Goal:** the chain `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven while
|
||||||
|
live by a paced frame pump: snapshot the active scene → resolve each element to its latest frame →
|
||||||
|
composite into the tier's output frame → pace into the encoder's stdin at the tier's FPS.
|
||||||
|
|
||||||
|
**Decisions (locked via user Q&A, 2026-08-12):**
|
||||||
|
1. **Video pipeline first** — the `-f lavfi -i anullsrc` silent track stays; mixing the loopback/mic
|
||||||
|
WASAPI samples into the encoder's AAC track is its own later step.
|
||||||
|
2. **RTMP URL via a provider seam** — `MainViewModel._rtmpUrlProvider` is a `Func<string?>` returning
|
||||||
|
null today (the reusable stream's ingest URL lands with TASK 5); when it yields null the pump logs
|
||||||
|
and skips the encoder entirely, so go-live runs the existing visual flow without pushing.
|
||||||
|
|
||||||
|
**Design:**
|
||||||
|
1. `Services/Encoder/FramePump.cs` — the frame producer. All collaborators constructor-injected seams
|
||||||
|
(`Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
|
||||||
|
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, injectable pacing delay) so it
|
||||||
|
stays free of WPF and of the capture managers and is hermetic in tests. `StartAsync` never throws
|
||||||
|
(failures log + surface via `Failed` — the VM fires-and-forgets from the sync command handler);
|
||||||
|
loop = snapshot → render → `SubmitFrameAsync`, paced at `1/options.Fps` (default `Task.Delay`; tests
|
||||||
|
inject `Task.Yield`). `StopAsync` stops the encoder (closes stdin) BEFORE awaiting the loop — closing
|
||||||
|
stdin unblocks a write stuck on pipe backpressure, so stop can't deadlock on the pump. `ProcessFailed`
|
||||||
|
self-stops the pump. `HealthUpdated` forwards the encoder's stats (ship step 6 binds the bottom bar).
|
||||||
|
2. `ScreenCaptureManager.GetLatestFrame(key)` — mirrors `CameraManager.GetLatestFrame(deviceId)`; the
|
||||||
|
backdrop's live frame for the compositor.
|
||||||
|
3. `MainViewModel` — owns the resolver (`WebcamSceneConfig` → `GetLatestFrame(WebcamId)`;
|
||||||
|
`Source.IsLiveCapture` → `GetLatestFrame(CaptureKey)`; image/background → `StaticPixelCache.Get(AssetId)`),
|
||||||
|
builds `CompositorOptions` from the tier + `OutputRect*` (doubles rounded to ints — the vertical
|
||||||
|
607.5 half-pixel crop rounds to a perfectly-centered 608), builds `EncoderOptions` from the tier when
|
||||||
|
the URL provider returns one, constructs the real `FfmpegEncoder(new FfmpegLocator())`, starts the
|
||||||
|
pump on go-live, stops it on end-stream, disposes in `Shutdown`, and flips `StreamStatus.Error` when
|
||||||
|
the pump fails while live (minimal — detailed health surfacing is ship step 6).
|
||||||
|
|
||||||
|
**Test plan (Good Dog Rule — ONE integration test):** `FramePumpTests.Start_CompositesScene_FeedsEncoder_StopsCleanly`
|
||||||
|
drives the full lifecycle against fakes — real `SceneCompositor` + real `FramePump`, fake `IFfmpegEncoder`
|
||||||
|
— asserting the composited red backdrop frame actually reaches the encoder at the tier size and that stop
|
||||||
|
tears everything down. Units: no-URL start skips the encoder, re-entrant start/stop no-ops, encoder
|
||||||
|
start-failure raises `Failed` + disposes, `ProcessFailed` self-stops the pump, `HealthUpdated` forwards.
|
||||||
|
`ScreenCaptureManagerTests.GetLatestFrame_ReturnsLatestPump_UntilReleased` pins the new accessor.
|
||||||
|
|
||||||
|
**Out of scope (later ship steps):** the loopback/mic → AAC mix (replaces `anullsrc`), health stats in the
|
||||||
|
bottom bar (ship step 6), scene-switching transitions, and any flash-frame wiring.
|
||||||
|
|
||||||
|
**Built (2026-08-12):** `FramePump` shipped in `Services/Encoder/`, `ScreenCaptureManager.GetLatestFrame`
|
||||||
|
added, `MainViewModel` wired end-to-end (resolver + both option builders + pump lifecycle), `FramePumpTests`
|
||||||
|
(7) + `GetLatestFrame` test (1) added — build **0 warnings**, **147 tests passing**. Known consideration:
|
||||||
|
the pump reads the active scene on a background thread while the UI can still edit it; a concurrent-mutation
|
||||||
|
exception is contained (logged + `Failed` + pump stops) rather than crashing.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -15,6 +15,8 @@ using ytLive.Helpers;
|
|||||||
using ytLive.Models;
|
using ytLive.Models;
|
||||||
using ytLive.Services;
|
using ytLive.Services;
|
||||||
using ytLive.Services.Audio;
|
using ytLive.Services.Audio;
|
||||||
|
using ytLive.Services.Compositor;
|
||||||
|
using ytLive.Services.Encoder;
|
||||||
|
|
||||||
namespace ytLive.ViewModels;
|
namespace ytLive.ViewModels;
|
||||||
|
|
||||||
@@ -86,6 +88,12 @@ public class MainViewModel : ViewModelBase
|
|||||||
// encoder mix. Private by design — no UI beyond the existing mic controls.
|
// encoder mix. Private by design — no UI beyond the existing mic controls.
|
||||||
private readonly AudioMixer _audioMixer;
|
private readonly AudioMixer _audioMixer;
|
||||||
|
|
||||||
|
// Live frame pipeline (TASK 4 ship step 5): composites the active scene at the
|
||||||
|
// tier's FPS and paces frames into the encoder. The RTMP URL seam stays null
|
||||||
|
// until the live-stream create flow (TASK 5) supplies the reusable stream URL.
|
||||||
|
private readonly Func<string?> _rtmpUrlProvider;
|
||||||
|
private readonly FramePump _framePump;
|
||||||
|
|
||||||
// Screen backdrop: a permanent live capture (desktop/game) that every scene
|
// Screen backdrop: a permanent live capture (desktop/game) that every scene
|
||||||
// shows at the bottom layer. One shared capture session per key — the
|
// shows at the bottom layer. One shared capture session per key — the
|
||||||
// ScreenCaptureManager refcounts by key, mirroring CameraManager.
|
// ScreenCaptureManager refcounts by key, mirroring CameraManager.
|
||||||
@@ -857,6 +865,16 @@ public class MainViewModel : ViewModelBase
|
|||||||
_screenCaptureManager.CaptureFailed += (key, message) =>
|
_screenCaptureManager.CaptureFailed += (key, message) =>
|
||||||
AppLog.Write($"ScreenCaptureManager: capture '{key}' failed: {message}");
|
AppLog.Write($"ScreenCaptureManager: capture '{key}' failed: {message}");
|
||||||
|
|
||||||
|
_rtmpUrlProvider = () => null; // TASK 5: the reusable stream's ingest URL
|
||||||
|
_framePump = new FramePump(
|
||||||
|
sceneProvider: () => ActiveScene,
|
||||||
|
frameResolver: ResolveOutputFrame,
|
||||||
|
compositorOptions: BuildCompositorOptions,
|
||||||
|
encoderOptions: BuildEncoderOptions,
|
||||||
|
encoderFactory: () => new FfmpegEncoder(new FfmpegLocator()),
|
||||||
|
log: message => AppLog.Write(message));
|
||||||
|
_framePump.Failed += OnFramePumpFailed;
|
||||||
|
|
||||||
LoadLayout();
|
LoadLayout();
|
||||||
_ = LoadSavedSessionAsync();
|
_ = LoadSavedSessionAsync();
|
||||||
AppLog.Write("MainViewModel ctor end");
|
AppLog.Write("MainViewModel ctor end");
|
||||||
@@ -1169,6 +1187,7 @@ public class MainViewModel : ViewModelBase
|
|||||||
{
|
{
|
||||||
_saveDebounce?.Stop();
|
_saveDebounce?.Stop();
|
||||||
SaveLayoutNow();
|
SaveLayoutNow();
|
||||||
|
_framePump.Dispose();
|
||||||
_cameraManager.Dispose();
|
_cameraManager.Dispose();
|
||||||
_screenCaptureManager.Dispose();
|
_screenCaptureManager.Dispose();
|
||||||
_layoutStore.Dispose();
|
_layoutStore.Dispose();
|
||||||
@@ -1739,6 +1758,7 @@ public class MainViewModel : ViewModelBase
|
|||||||
: $"{dialog.StreamTitle} — ytLlive";
|
: $"{dialog.StreamTitle} — ytLlive";
|
||||||
StreamStatus = StreamStatus.Streaming;
|
StreamStatus = StreamStatus.Streaming;
|
||||||
_audioMixer.Start();
|
_audioMixer.Start();
|
||||||
|
_ = _framePump.StartAsync(); // never throws; failures log + surface via Failed
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1747,6 +1767,7 @@ public class MainViewModel : ViewModelBase
|
|||||||
StreamStatus = StreamStatus.Offline;
|
StreamStatus = StreamStatus.Offline;
|
||||||
WindowTitle = "ytLlive";
|
WindowTitle = "ytLlive";
|
||||||
_audioMixer.Stop();
|
_audioMixer.Stop();
|
||||||
|
_ = _framePump.StopAsync();
|
||||||
// Graceful end completes the session = signs out (the DPAPI token is
|
// Graceful end completes the session = signs out (the DPAPI token is
|
||||||
// cleared so the next Start Stream requires a fresh sign-in). A crash
|
// cleared so the next Start Stream requires a fresh sign-in). A crash
|
||||||
// never runs this, so the token survives and the creator stays signed in.
|
// never runs this, so the token survives and the creator stays signed in.
|
||||||
@@ -1768,6 +1789,59 @@ public class MainViewModel : ViewModelBase
|
|||||||
AudioLevel = level;
|
AudioLevel = level;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Scene-element → latest frame, for the live compositor. The map mirrors the
|
||||||
|
// preview: webcam by DeviceId, live captures (the backdrop) by CaptureKey,
|
||||||
|
// images/background by AssetId. A null frame leaves the element transparent.
|
||||||
|
private VideoFrame? ResolveOutputFrame(SceneElement element)
|
||||||
|
{
|
||||||
|
return element switch
|
||||||
|
{
|
||||||
|
WebcamSceneConfig webcam => _cameraManager.GetLatestFrame(webcam.WebcamId),
|
||||||
|
Source { IsLiveCapture: true, CaptureKey: not null } live => _screenCaptureManager.GetLatestFrame(live.CaptureKey),
|
||||||
|
Source { AssetId: not null } image => StaticPixelCache.Get(image.AssetId),
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// The tier's crop rect over the 1920x1080 master, integer-aligned (the VM's
|
||||||
|
// OutputRect* are doubles — the vertical 607.5 half-pixel crop rounds to 608).
|
||||||
|
private CompositorOptions BuildCompositorOptions()
|
||||||
|
{
|
||||||
|
var quality = SelectedQuality;
|
||||||
|
return new CompositorOptions
|
||||||
|
{
|
||||||
|
SourceRectX = (int)Math.Round(OutputRectX),
|
||||||
|
SourceRectY = (int)Math.Round(OutputRectY),
|
||||||
|
SourceRectWidth = (int)Math.Round(OutputRectWidth),
|
||||||
|
SourceRectHeight = (int)Math.Round(OutputRectHeight),
|
||||||
|
OutputWidth = quality.Width,
|
||||||
|
OutputHeight = quality.Height,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Full encoder options for the current tier, or null when no RTMP URL is
|
||||||
|
// available — the pump then skips the encoder entirely (TASK 5 fills the seam).
|
||||||
|
private EncoderOptions? BuildEncoderOptions()
|
||||||
|
{
|
||||||
|
var url = _rtmpUrlProvider();
|
||||||
|
if (string.IsNullOrWhiteSpace(url)) return null;
|
||||||
|
var quality = SelectedQuality;
|
||||||
|
return new EncoderOptions
|
||||||
|
{
|
||||||
|
RtmpUrl = url,
|
||||||
|
Width = quality.Width,
|
||||||
|
Height = quality.Height,
|
||||||
|
Fps = quality.Fps,
|
||||||
|
BitrateKbps = (int)Math.Round(quality.Bitrate * 1000),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnFramePumpFailed(object? sender, string message)
|
||||||
|
{
|
||||||
|
AppLog.Write($"Frame pump failed: {message}");
|
||||||
|
if (IsLive) StreamStatus = StreamStatus.Error;
|
||||||
|
}
|
||||||
|
|
||||||
private void UpdateLiveVisuals()
|
private void UpdateLiveVisuals()
|
||||||
{
|
{
|
||||||
var live = IsLive;
|
var live = IsLive;
|
||||||
|
|||||||
@@ -91,7 +91,7 @@ C# / WPF (.NET 8) following MVVM:
|
|||||||
|------|------|
|
|------|------|
|
||||||
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
|
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
|
||||||
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
|
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
|
||||||
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")** |
|
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")** |
|
||||||
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
|
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
|
||||||
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
|
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
|
||||||
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
|
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
|
||||||
@@ -352,7 +352,8 @@ integration test fakes the whole subprocess (probe + encoder) with a Channel-bac
|
|||||||
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
|
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
|
||||||
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
|
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
|
||||||
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
|
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
|
||||||
`GopSize` = FPS×4. Not yet constructed by the app — the frame-pipeline wiring (ship step 5) owns it.
|
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
|
||||||
|
driven by the `FramePump` below.
|
||||||
|
|
||||||
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12, plan in TASKS.md)
|
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12, plan in TASKS.md)
|
||||||
|
|
||||||
@@ -371,16 +372,44 @@ timers).
|
|||||||
- **`AudioMixer`** owns both sources; `Start`/`Stop` follow go-live (`MainViewModel.BeginGoLive` success
|
- **`AudioMixer`** owns both sources; `Start`/`Stop` follow go-live (`MainViewModel.BeginGoLive` success
|
||||||
→ `_audioMixer.Start()`, `StopStream` → `Stop()`). Mic samples feed a pure **`AudioLevelMeter`** (RMS
|
→ `_audioMixer.Start()`, `StopStream` → `Stop()`). Mic samples feed a pure **`AudioLevelMeter`** (RMS
|
||||||
with 0.2 exponential smoothing) → `MicLevelChanged` → marshalled to the UI thread → `AudioLevel`.
|
with 0.2 exponential smoothing) → `MicLevelChanged` → marshalled to the UI thread → `AudioLevel`.
|
||||||
Desktop samples are currently **dropped** — the future encoder's AAC mix (ship step 5) consumes them,
|
Desktop samples are currently **dropped** — a later step's AAC mix consumes them, replacing the
|
||||||
replacing the `-f lavfi -i anullsrc` placeholder. Failures log via `AppLog`; a mic failure zeroes the
|
`-f lavfi -i anullsrc` placeholder (the encoder construction itself shipped in ship step 5). Failures log via `AppLog`; a mic failure zeroes the
|
||||||
meter, a loopback failure never kills the mic.
|
meter, a loopback failure never kills the mic.
|
||||||
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
|
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
|
||||||
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
|
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
|
||||||
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
|
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
|
||||||
- `FfmpegEncoder.cs:139` pre-existing CS8602 fixed (`process!`) — build **0 warnings**; **139 passing**.
|
- `FfmpegEncoder.cs:139` pre-existing CS8602 fixed (`process!`) — build **0 warnings**; **139 passing**.
|
||||||
|
|
||||||
**Deferred:** loopback→encoder mix wiring and the encoder construction (ship step 5); capture while not
|
**Deferred:** the loopback→encoder AAC mix wiring (a later step — the encoder construction + frame
|
||||||
live is deliberately not shipped (privacy indicator otherwise).
|
pipeline shipped in ship step 5); capture while not live is deliberately not shipped (privacy indicator
|
||||||
|
otherwise).
|
||||||
|
|
||||||
|
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, plan in TASKS.md)
|
||||||
|
|
||||||
|
The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active
|
||||||
|
scene each tick, resolves every element to its latest frame, composites it into the tier's output frame,
|
||||||
|
and paces frames into the encoder at the tier's FPS. **Pattern — everything is a constructor-injected
|
||||||
|
seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
|
||||||
|
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, and an injectable pacing delay
|
||||||
|
(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers.
|
||||||
|
|
||||||
|
- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log
|
||||||
|
+ surface via the `Failed` event. `EncoderOptions == null` means "no RTMP URL": the pump logs and skips
|
||||||
|
the encoder entirely. `MainViewModel._rtmpUrlProvider` is that seam — a `Func<string?>` returning null
|
||||||
|
until TASK 5 supplies the reusable stream's ingest URL, so go-live runs the current visual flow.
|
||||||
|
- **Stop ordering matters:** `StopAsync` stops the encoder (closes stdin → EOF → ffmpeg finalizes+exits)
|
||||||
|
**before** awaiting the loop, because closing stdin unblocks a write stuck on pipe backpressure — the
|
||||||
|
reverse order would deadlock. `ProcessFailed` self-stops the pump. `HealthUpdated` is forwarded
|
||||||
|
(ship step 6 binds it to the bottom bar); `Failed` while live flips `StreamStatus.Error` (minimal).
|
||||||
|
- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` →
|
||||||
|
`CameraManager.GetLatestFrame(WebcamId)`, `Source { IsLiveCapture, CaptureKey }` →
|
||||||
|
`ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/
|
||||||
|
background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*`
|
||||||
|
doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`,
|
||||||
|
ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one.
|
||||||
|
- Known consideration: the pump reads the active scene on a background thread while the UI can still edit
|
||||||
|
it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than
|
||||||
|
crashing. The background thread + video pipeline is the new reality since this step.
|
||||||
|
|
||||||
### Social bar (TASK 14 — shipped 2026-08-12, plan in TASKS.md)
|
### Social bar (TASK 14 — shipped 2026-08-12, plan in TASKS.md)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,221 @@
|
|||||||
|
using Xunit;
|
||||||
|
using ytLive.Models;
|
||||||
|
using ytLive.Services;
|
||||||
|
using ytLive.Services.Compositor;
|
||||||
|
using ytLive.Services.Encoder;
|
||||||
|
|
||||||
|
namespace ytLive.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The live frame producer (TASK 4 ship step 5): composites the active scene at
|
||||||
|
/// the tier's FPS and paces frames into the encoder. The integration test drives
|
||||||
|
/// the full lifecycle against fakes — real SceneCompositor + real FramePump, fake
|
||||||
|
/// IFfmpegEncoder — proving the composite frame actually reaches the encoder and
|
||||||
|
/// that stop tears the pump down cleanly. The units pin the failure edges: the
|
||||||
|
/// no-URL skip (the TASK 5 seam), re-entrancy, and encoder death.
|
||||||
|
/// </summary>
|
||||||
|
public class FramePumpTests
|
||||||
|
{
|
||||||
|
private sealed class FakeEncoder : IFfmpegEncoder
|
||||||
|
{
|
||||||
|
public readonly List<VideoFrame> Frames = new();
|
||||||
|
public int StartCount;
|
||||||
|
public int StopCount;
|
||||||
|
public bool Disposed;
|
||||||
|
public EncoderOptions? LastOptions;
|
||||||
|
public Exception? StartError;
|
||||||
|
public TaskCompletionSource FrameArrived = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
|
||||||
|
public event EventHandler<StreamHealth>? HealthUpdated;
|
||||||
|
public event EventHandler<string>? ProcessFailed;
|
||||||
|
|
||||||
|
public Task StartAsync(EncoderOptions options, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
StartCount++;
|
||||||
|
LastOptions = options;
|
||||||
|
if (StartError != null) throw StartError;
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task SubmitFrameAsync(VideoFrame frame, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
Frames.Add(frame);
|
||||||
|
FrameArrived.TrySetResult();
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task StopAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
StopCount++;
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose() => Disposed = true;
|
||||||
|
|
||||||
|
public void RaiseProcessFailed(string message) => ProcessFailed?.Invoke(this, message);
|
||||||
|
|
||||||
|
public void RaiseHealth(StreamHealth health) => HealthUpdated?.Invoke(this, health);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static Scene BackdropScene()
|
||||||
|
{
|
||||||
|
var scene = new Scene { Name = "Live" };
|
||||||
|
scene.Elements.Add(new Source { Type = SourceType.DisplayCapture, IsBackdrop = true, CaptureKey = "monitor:0" });
|
||||||
|
return scene;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static FramePump NewPump(FakeEncoder encoder, Func<EncoderOptions?>? options = null,
|
||||||
|
Func<Scene?>? scene = null, Func<SceneElement, VideoFrame?>? resolve = null,
|
||||||
|
List<string>? log = null)
|
||||||
|
{
|
||||||
|
return new FramePump(
|
||||||
|
sceneProvider: scene ?? (() => BackdropScene()),
|
||||||
|
frameResolver: resolve ?? (_ => null),
|
||||||
|
compositorOptions: () => new CompositorOptions
|
||||||
|
{
|
||||||
|
SourceRectX = 0, SourceRectY = 0, SourceRectWidth = 64, SourceRectHeight = 48,
|
||||||
|
OutputWidth = 64, OutputHeight = 48,
|
||||||
|
},
|
||||||
|
encoderOptions: options ?? (() => new EncoderOptions
|
||||||
|
{
|
||||||
|
RtmpUrl = "rtmp://a.rtmp.youtube.com/live2/abc",
|
||||||
|
Width = 64, Height = 48, Fps = 60,
|
||||||
|
}),
|
||||||
|
encoderFactory: () => encoder,
|
||||||
|
log: log != null ? m => log.Add(m) : null,
|
||||||
|
pacingDelay: async (_, _) => await Task.Yield()); // deterministic: no real waits
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void AssertColor(VideoFrame frame, int x, int y, byte r, byte g, byte b)
|
||||||
|
{
|
||||||
|
var i = (y * frame.Width + x) * 4;
|
||||||
|
var br = frame.BgraPixels[i + 2];
|
||||||
|
var bg = frame.BgraPixels[i + 1];
|
||||||
|
var bb = frame.BgraPixels[i];
|
||||||
|
Assert.True(
|
||||||
|
Math.Abs(br - r) <= 2 && Math.Abs(bg - g) <= 2 && Math.Abs(bb - b) <= 2,
|
||||||
|
$"pixel ({x},{y}): expected rgb({r},{g},{b}), got rgb({br},{bg},{bb})");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Start_CompositesScene_FeedsEncoder_StopsCleanly()
|
||||||
|
{
|
||||||
|
var red = SceneCompositorTests.Solid(64, 48, 255, 0, 0);
|
||||||
|
var encoder = new FakeEncoder();
|
||||||
|
using var pump = NewPump(encoder,
|
||||||
|
resolve: e => e is Source { IsBackdrop: true } ? red : null);
|
||||||
|
|
||||||
|
await pump.StartAsync();
|
||||||
|
Assert.True(pump.IsRunning);
|
||||||
|
Assert.Equal(1, encoder.StartCount);
|
||||||
|
|
||||||
|
await encoder.FrameArrived.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||||
|
Assert.NotEmpty(encoder.Frames);
|
||||||
|
var frame = encoder.Frames[0];
|
||||||
|
Assert.Equal(64, frame.Width);
|
||||||
|
Assert.Equal(48, frame.Height);
|
||||||
|
AssertColor(frame, 0, 0, 255, 0, 0); // the backdrop really was composited in
|
||||||
|
|
||||||
|
await pump.StopAsync();
|
||||||
|
Assert.False(pump.IsRunning);
|
||||||
|
Assert.Equal(1, encoder.StopCount);
|
||||||
|
Assert.True(encoder.Disposed);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Start_WithoutRtmpUrl_SkipsEncoder()
|
||||||
|
{
|
||||||
|
var encoder = new FakeEncoder();
|
||||||
|
var log = new List<string>();
|
||||||
|
using var pump = NewPump(encoder, options: () => null, log: log);
|
||||||
|
|
||||||
|
await pump.StartAsync();
|
||||||
|
|
||||||
|
Assert.False(pump.IsRunning);
|
||||||
|
Assert.Equal(0, encoder.StartCount);
|
||||||
|
Assert.Contains(log, m => m.Contains("RTMP"));
|
||||||
|
|
||||||
|
await pump.StopAsync(); // no-op after a skipped start
|
||||||
|
Assert.Equal(0, encoder.StopCount);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Start_WhileRunning_IsNoop()
|
||||||
|
{
|
||||||
|
var encoder = new FakeEncoder();
|
||||||
|
using var pump = NewPump(encoder);
|
||||||
|
|
||||||
|
await pump.StartAsync();
|
||||||
|
await pump.StartAsync();
|
||||||
|
|
||||||
|
Assert.Equal(1, encoder.StartCount);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Stop_WithoutStart_IsNoop()
|
||||||
|
{
|
||||||
|
var encoder = new FakeEncoder();
|
||||||
|
using var pump = NewPump(encoder);
|
||||||
|
|
||||||
|
await pump.StopAsync();
|
||||||
|
|
||||||
|
Assert.Equal(0, encoder.StopCount);
|
||||||
|
Assert.False(pump.IsRunning);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Start_EncoderThrows_RaisesFailed_AndDisposes()
|
||||||
|
{
|
||||||
|
var encoder = new FakeEncoder { StartError = new InvalidOperationException("access denied") };
|
||||||
|
var failed = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
using var pump = NewPump(encoder);
|
||||||
|
pump.Failed += (_, m) => failed.TrySetResult(m);
|
||||||
|
|
||||||
|
await pump.StartAsync();
|
||||||
|
|
||||||
|
Assert.False(pump.IsRunning);
|
||||||
|
var message = await failed.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||||
|
Assert.Contains("access denied", message);
|
||||||
|
Assert.True(encoder.Disposed);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ProcessDeath_StopsPump_AndRaisesFailed()
|
||||||
|
{
|
||||||
|
var encoder = new FakeEncoder();
|
||||||
|
var failed = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
using var pump = NewPump(encoder);
|
||||||
|
pump.Failed += (_, m) => failed.TrySetResult(m);
|
||||||
|
|
||||||
|
await pump.StartAsync();
|
||||||
|
encoder.RaiseProcessFailed("FFmpeg exited with code 1");
|
||||||
|
|
||||||
|
var message = await failed.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||||
|
Assert.Contains("1", message);
|
||||||
|
|
||||||
|
// The pump self-stops (fire-and-forget); poll the definitive teardown
|
||||||
|
// marker — the encoder's disposal — rather than the IsRunning flag, which
|
||||||
|
// StopAsync clears before the loop has fully drained.
|
||||||
|
var deadline = DateTime.UtcNow.AddSeconds(5);
|
||||||
|
while (!encoder.Disposed && DateTime.UtcNow < deadline)
|
||||||
|
await Task.Delay(10);
|
||||||
|
Assert.True(encoder.Disposed);
|
||||||
|
Assert.False(pump.IsRunning);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task HealthUpdated_ForwardsEncoderHealth()
|
||||||
|
{
|
||||||
|
var encoder = new FakeEncoder();
|
||||||
|
var health = new TaskCompletionSource<StreamHealth>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
using var pump = NewPump(encoder);
|
||||||
|
pump.HealthUpdated += (_, h) => health.TrySetResult(h);
|
||||||
|
|
||||||
|
await pump.StartAsync();
|
||||||
|
encoder.RaiseHealth(new StreamHealth { Status = StreamStatus.Streaming, FPS = 59.9 });
|
||||||
|
|
||||||
|
var h = await health.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||||
|
Assert.Equal(StreamStatus.Streaming, h.Status);
|
||||||
|
Assert.Equal(59.9, h.FPS, 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -145,6 +145,28 @@ public class ScreenCaptureManagerTests
|
|||||||
Assert.False(await manager.AcquireAsync(" "));
|
Assert.False(await manager.AcquireAsync(" "));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task GetLatestFrame_ReturnsLatestPump_UntilReleased()
|
||||||
|
{
|
||||||
|
FakeScreenSource? captured = null;
|
||||||
|
var manager = new ScreenCaptureManager(key => captured = new FakeScreenSource(key));
|
||||||
|
|
||||||
|
Assert.Null(manager.GetLatestFrame("monitor:0"));
|
||||||
|
|
||||||
|
Assert.True(await manager.AcquireAsync("monitor:0"));
|
||||||
|
Assert.Null(manager.GetLatestFrame("monitor:0")); // nothing pumped yet
|
||||||
|
|
||||||
|
var first = new VideoFrame(2, 2, Pixels(1, 0, 0, 255, 2, 0, 0, 255, 3, 0, 0, 255, 4, 0, 0, 255));
|
||||||
|
var second = new VideoFrame(2, 2, Pixels(5, 0, 0, 255, 6, 0, 0, 255, 7, 0, 0, 255, 8, 0, 0, 255));
|
||||||
|
captured!.Pump(first);
|
||||||
|
captured.Pump(second);
|
||||||
|
|
||||||
|
Assert.Same(second, manager.GetLatestFrame("monitor:0"));
|
||||||
|
|
||||||
|
await manager.ReleaseAsync("monitor:0");
|
||||||
|
Assert.Null(manager.GetLatestFrame("monitor:0"));
|
||||||
|
}
|
||||||
|
|
||||||
// The one integration test for this branch: one target creates one shared
|
// The one integration test for this branch: one target creates one shared
|
||||||
// WriteableBitmap, published once, and back-to-back frames coalesce to the
|
// WriteableBitmap, published once, and back-to-back frames coalesce to the
|
||||||
// latest (a single pending UI copy per session).
|
// latest (a single pending UI copy per session).
|
||||||
|
|||||||
Reference in New Issue
Block a user