TASK 21 slice 1 step 3: IMediaFrameSource + MediaVideoSourceManager
Refactor MediaVideoSource onto IMediaFrameSource (Key/FrameAvailable/Completed/ StartAsync/StopAsync, renaming Start->StartAsync, FrameReady->FrameAvailable), behavior preserved. Add MediaVideoSourceManager: app-wide decode-session owner refcounted by MediaPath with Func<string,IMediaFrameSource?> factory seam, Acquire/Release/ReleaseAll/GetLatestFrame, coalescing each file's frames onto the UI dispatcher onto one shared WriteableBitmap; MediaFailed + PreviewBitmapChanged. Mirrors ScreenCaptureManager (screen-capture session ownership + dispatcher coalescing), per the codebase precedent and derivative-work rule. Tests: MediaVideoSourceManagerTests (5 unit + 1 integration: single shared bitmap, coalesce-to-latest); MediaVideoSourceTests updated for renames. 10/10 media tests pass, build 0 warnings. Docs (TASKS/HANDOFF/ai.md) updated.
This commit is contained in:
+29
-21
@@ -4,35 +4,43 @@
|
||||
|
||||
**`main`**, working through TASK 21 (media source). Local commits ahead of
|
||||
`origin/main` (not yet pushed — user pushes at agreed checkpoints):
|
||||
- `a1d9751` — docs: ffmpeg decode-contract verification recipe + WSL CLR boundary
|
||||
- `8f49010` — TASK 21 Increment B: FFmpeg rawvideo video decoder -> VideoFrame
|
||||
- `f89b9f9` — docs: default-device audio assumption (README + startup log)
|
||||
- `e8ff4df` — TASK 22: global audio sync offset (positive delay at the mixer out)
|
||||
- `670fe3a` — TASK 31 (already pushed; listed for context)
|
||||
|
||||
Prior milestones pushed through `origin/main`: TASK 31 (`3bf053a..670fe3a`), plus
|
||||
the full refactor series (Commit F `3107f92`, Commit G `85893ea`).
|
||||
|
||||
## What's In Flight
|
||||
|
||||
**TASK 21 (media source) — Increment B (the decoder) implemented, uncommitted-or-pending-commit, awaiting next unit.**
|
||||
**TASK 21 (media source) — slice 1 step 3 shipped (the session-manager + interface); next open work is the VM wire-in (slice 1 step 4).**
|
||||
|
||||
Increment A (the `Source` model: `MediaSourceType`, `SourceType.MediaSource`,
|
||||
`MediaPath`/`MediaIsLooping`/`MediaVolume`/`MediaPlaybackState` + LayoutStore
|
||||
persistence) shipped earlier. This session shipped **Increment B — the decoder**:
|
||||
persistence) and Increment B (the decoder `MediaVideoSource`) shipped earlier.
|
||||
|
||||
New files:
|
||||
- `Services/RawVideoFrameReader.cs` — pure: parses a rawvideo BGRA stream into
|
||||
`VideoFrame`s (exactly W×H×4 per frame; partial reads kept across `Feed`).
|
||||
- `Services/IDecodeProcess.cs` + `Services/FfmpegDecodeProcess.cs` — binary-stdout
|
||||
subprocess seam (mirror of `IEncoderProcess`).
|
||||
- `Services/MediaVideoSource.cs` — spawns ffmpeg `-f rawvideo -pix_fmt bgra -an`,
|
||||
drains the pipe, raises `FrameReady` per frame; `Completed` on EOF.
|
||||
- `ytLive.Tests/MediaVideoSourceTests.cs` — 3 pure reader tests + 1 integration
|
||||
(fake `IDecodeProcess`+`IFfmpegLocator` through the real source loop, frames in
|
||||
order). **Verified: 4/4 pass, build 0 warnings.**
|
||||
This session:
|
||||
- Refactored `Services/MediaVideoSource.cs` onto the new
|
||||
`Services/IMediaFrameSource.cs` (`Key`, `FrameAvailable`, `Completed`,
|
||||
`StartAsync`/`StopAsync`; `Start`→`StartAsync`, `FrameReady`→`FrameAvailable`)
|
||||
— behavior preserved, `MediaVideoSourceTests` updated and passing.
|
||||
- Added `Services/MediaVideoSourceManager.cs` — app-wide decode-session
|
||||
ownership refcounted by `MediaPath`, `Func<string, IMediaFrameSource?>` factory
|
||||
seam, `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`, frames
|
||||
coalesced onto the UI dispatcher onto one shared `WriteableBitmap` per file,
|
||||
`MediaFailed` + `PreviewBitmapChanged`. Structural mirror of `ScreenCaptureManager`.
|
||||
- Added `ytLive.Tests/MediaVideoSourceManagerTests.cs` (5 unit + 1 integration:
|
||||
single shared bitmap, coalesce-to-latest). **Verified: all 10 media tests pass
|
||||
(5 manager + 3 reader + 1 decoder integration + 1), build 0 warnings.**
|
||||
|
||||
Still open (next slices): native-FPS pacing (ffprobe probe), loop control, volume,
|
||||
the `ResolveOutputFrame` resolver case + wire into `SceneCompositor`, the file-picker
|
||||
UI, and the remaining tests.
|
||||
Next (slice 1 step 4 — VM wire-in): create `_mediaManager` in the ctor with
|
||||
`path => new MediaVideoSource(path, <W>, <H>, new FfmpegLocator(), new FfmpegDecodeProcess())`,
|
||||
subscribe `OnMediaPreviewBitmapChanged` (set `source.VideoImageSource` for every
|
||||
`IsMediaSource` with that `MediaPath`), add the `Source { IsMediaSource: true, MediaPath: not null }`
|
||||
case to `ResolveOutputFrame` (`MainViewModel.cs:447`), add `or SourceType.MediaSource`
|
||||
to `Source.DisplaySource` (Source.cs:119-121) so media previews route `VideoImageSource`,
|
||||
and dispose the manager on shutdown. Then slices 2 (native-FPS pacing) and 3 (loop).
|
||||
|
||||
**Landmine:** a stale `testhost` can lock `ytLive.Tests`'s `ytLive.dll` and break
|
||||
`dotnet build` of the test project — kill it first (`cmd.exe /c "taskkill /PID N /F"`)
|
||||
@@ -42,11 +50,11 @@ pure tests run in WSL; user verifies the GUI suites on native Windows PowerShell
|
||||
|
||||
## Next
|
||||
|
||||
1. Commit Increment B (decoder) as one work unit (scope-check + docs already
|
||||
staged in ai.md/TASKS.md/HANDOFF.md).
|
||||
2. Next slice: native-FPS pacing + wire media into `ResolveOutputFrame` and the
|
||||
compositor, per TASK 21 steps 6–7.
|
||||
3. Await user push checkpoint for the TASK 22/21 commits.
|
||||
1. Commit slice 1 step 3 (this unit): manager + `IMediaFrameSource` + tests. Commit
|
||||
is already covered by `TASKS.md`/`HANDOFF.md`; this HANDOFF is the shipping doc.
|
||||
2. Slice 1 step 4: wire `_mediaManager` into the resolver + preview routing (see above).
|
||||
3. Slice 2: native-FPS pacing (ffprobe FPS probe + per-frame pacing seam).
|
||||
4. Await user push checkpoint for the pending TASK 22/21 commits.
|
||||
|
||||
## The directive (2026-08-31, user)
|
||||
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
namespace ytLive.Services;
|
||||
|
||||
/// <summary>
|
||||
/// TASK 21: a media file decoder exposed to <see cref="MediaVideoSourceManager"/>.
|
||||
/// Mirrors <c>IScreenCaptureSource</c>/<c>ICameraFrameSource</c> so the manager
|
||||
/// (and tests) treat a local video file like any other frame source: start/stop
|
||||
/// and a per-frame <see cref="FrameAvailable"/> event consumers coalesce.
|
||||
/// </summary>
|
||||
public interface IMediaFrameSource : IDisposable
|
||||
{
|
||||
string Key { get; }
|
||||
|
||||
/// <summary>Raised (on a worker thread) for each decoded frame.</summary>
|
||||
event Action<VideoFrame>? FrameAvailable;
|
||||
|
||||
/// <summary>Raised when the file finishes decoding (natural EOF).</summary>
|
||||
event Action? Completed;
|
||||
|
||||
Task StartAsync(CancellationToken cancellationToken = default);
|
||||
Task StopAsync();
|
||||
}
|
||||
@@ -6,19 +6,20 @@ namespace ytLive.Services;
|
||||
|
||||
/// <summary>
|
||||
/// TASK 21: decodes a local media file (any format ffmpeg handles) into
|
||||
/// <see cref="VideoFrame"/>s for a MediaSource. Spawns ffmpeg with
|
||||
/// <c>-f rawvideo -pix_fmt bgra</c>, reads the raw BGRA pipe via
|
||||
/// <see cref="RawVideoFrameReader"/>, and raises <see cref="FrameReady"/> per
|
||||
/// frame. The decoder process is a seam (<see cref="IDecodeProcess"/>) so tests
|
||||
/// run without a real codec or audio device. Frames are emitted as fast as the
|
||||
/// pipe produces them; native-FPS pacing is wired by the caller via a probe in a
|
||||
/// later unit (TASK 21 step 5 pacing).
|
||||
/// <see cref="VideoFrame"/>s for a MediaSource. Implements
|
||||
/// <see cref="IMediaFrameSource"/>: raises <see cref="FrameAvailable"/> per
|
||||
/// frame and <see cref="IMediaFrameSource.Completed"/> on natural EOF. Spawns
|
||||
/// ffmpeg with <c>-f rawvideo -pix_fmt bgra</c> and drains the raw BGRA pipe via
|
||||
/// <see cref="RawVideoFrameReader"/>. The decoder process and emitters are seams
|
||||
/// so tests run without a real codec or audio device.
|
||||
/// </summary>
|
||||
public sealed class MediaVideoSource : IDisposable
|
||||
public sealed class MediaVideoSource : IMediaFrameSource
|
||||
{
|
||||
public event Action<VideoFrame>? FrameReady;
|
||||
public event Action<VideoFrame>? FrameAvailable;
|
||||
public event Action? Completed;
|
||||
|
||||
public string Key => _path;
|
||||
|
||||
private readonly string _path;
|
||||
private readonly int _width;
|
||||
private readonly int _height;
|
||||
@@ -41,17 +42,19 @@ public sealed class MediaVideoSource : IDisposable
|
||||
_process = process;
|
||||
}
|
||||
|
||||
public void Start()
|
||||
public Task StartAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
if (_run != null) return;
|
||||
if (_run != null) return Task.CompletedTask;
|
||||
_run = _cts.Token.Register(StopInternal);
|
||||
_ = Task.Run(RunAsync);
|
||||
_ = Task.Run(RunAsync, CancellationToken.None);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
public Task StopAsync()
|
||||
{
|
||||
_cts.Cancel();
|
||||
_run = null;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
@@ -85,14 +88,12 @@ public sealed class MediaVideoSource : IDisposable
|
||||
var reader = new RawVideoFrameReader(_width, _height);
|
||||
var buffer = new byte[65536];
|
||||
var stream = _process.StandardOutput;
|
||||
byte[] readChunk;
|
||||
while (!_cts.IsCancellationRequested && !_process.HasExited)
|
||||
{
|
||||
var read = await stream.ReadAsync(buffer, 0, buffer.Length, _cts.Token).ConfigureAwait(false);
|
||||
if (read <= 0) break;
|
||||
readChunk = buffer.AsSpan(0, read).ToArray();
|
||||
foreach (var frame in reader.Feed(readChunk))
|
||||
FrameReady?.Invoke(frame);
|
||||
foreach (var frame in reader.Feed(buffer.AsSpan(0, read).ToArray()))
|
||||
FrameAvailable?.Invoke(frame);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Threading.Tasks;
|
||||
using System.Windows;
|
||||
using System.Windows.Media;
|
||||
using System.Windows.Media.Imaging;
|
||||
using System.Windows.Threading;
|
||||
using ytLive.Helpers;
|
||||
|
||||
namespace ytLive.Services;
|
||||
|
||||
/// <summary>
|
||||
/// TASK 21: owns media-file decode sessions app-wide, refcounted by file path.
|
||||
/// One file = at most one decoder, one shared WriteableBitmap; the last release
|
||||
/// stops and disposes the source. Frames arrive on a worker thread and are
|
||||
/// coalesced onto the UI dispatcher (at most one pending copy per session, using
|
||||
/// the latest frame). Mirrors ScreenCaptureManager/CameraManager.
|
||||
/// </summary>
|
||||
public sealed class MediaVideoSourceManager : IDisposable
|
||||
{
|
||||
private sealed class MediaSession
|
||||
{
|
||||
public string Key { get; }
|
||||
public IMediaFrameSource Source { get; }
|
||||
public Action<VideoFrame>? FrameHandler;
|
||||
public int RefCount;
|
||||
public bool Started;
|
||||
public WriteableBitmap? PreviewBitmap;
|
||||
public VideoFrame? LatestFrame;
|
||||
public bool FramePending;
|
||||
|
||||
public MediaSession(string key, IMediaFrameSource source)
|
||||
{
|
||||
Key = key;
|
||||
Source = source;
|
||||
RefCount = 1;
|
||||
}
|
||||
}
|
||||
|
||||
private readonly Func<string, IMediaFrameSource?> _sourceFactory;
|
||||
private readonly Dispatcher? _uiDispatcher;
|
||||
private readonly Dictionary<string, MediaSession> _sessions = new();
|
||||
private readonly object _gate = new();
|
||||
|
||||
/// <summary>Raised on the UI thread when a media file's shared preview bitmap is first created.</summary>
|
||||
public event Action<string, WriteableBitmap>? PreviewBitmapChanged;
|
||||
|
||||
/// <summary>Raised when a media decoder cannot be created (bad path).</summary>
|
||||
public event Action<string, string>? MediaFailed;
|
||||
|
||||
public MediaVideoSourceManager(Func<string, IMediaFrameSource?> sourceFactory, Dispatcher? uiDispatcher = null)
|
||||
{
|
||||
_sourceFactory = sourceFactory;
|
||||
_uiDispatcher = uiDispatcher;
|
||||
}
|
||||
|
||||
/// <summary>Increments the refcount for a file, starting the decoder the first time.</summary>
|
||||
public async Task<bool> AcquireAsync(string key)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(key)) return false;
|
||||
|
||||
MediaSession session;
|
||||
bool shouldStart;
|
||||
lock (_gate)
|
||||
{
|
||||
if (_sessions.TryGetValue(key, out var existing))
|
||||
{
|
||||
existing.RefCount++;
|
||||
shouldStart = false;
|
||||
session = existing;
|
||||
}
|
||||
else
|
||||
{
|
||||
IMediaFrameSource? source;
|
||||
try
|
||||
{
|
||||
source = _sourceFactory(key);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
AppLog.Write($"MediaSource: creating decoder '{key}' failed: {ex.Message}");
|
||||
MediaFailed?.Invoke(key, ex.Message);
|
||||
return false;
|
||||
}
|
||||
if (source == null)
|
||||
{
|
||||
MediaFailed?.Invoke(key, "No decoder for this file");
|
||||
return false;
|
||||
}
|
||||
session = new MediaSession(key, source);
|
||||
session.FrameHandler = frame => OnFrameAvailable(session, frame);
|
||||
session.Source.FrameAvailable += session.FrameHandler;
|
||||
_sessions[key] = session;
|
||||
shouldStart = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (!shouldStart) return session.Started;
|
||||
|
||||
try
|
||||
{
|
||||
await session.Source.StartAsync();
|
||||
session.Started = true;
|
||||
return true;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
lock (_gate)
|
||||
_sessions.Remove(key);
|
||||
session.Source.FrameAvailable -= session.FrameHandler;
|
||||
AppLog.Write($"MediaSource: failed to start decoder '{key}': {ex.Message}");
|
||||
MediaFailed?.Invoke(key, ex.Message);
|
||||
await SafeStopAsync(session.Source);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
public async Task ReleaseAsync(string key)
|
||||
{
|
||||
MediaSession? toStop = null;
|
||||
lock (_gate)
|
||||
{
|
||||
if (!_sessions.TryGetValue(key, out var session)) return;
|
||||
if (--session.RefCount > 0) return;
|
||||
_sessions.Remove(key);
|
||||
toStop = session;
|
||||
}
|
||||
|
||||
if (toStop == null) return;
|
||||
toStop.Source.FrameAvailable -= toStop.FrameHandler;
|
||||
await SafeStopAsync(toStop.Source);
|
||||
toStop.PreviewBitmap = null;
|
||||
}
|
||||
|
||||
/// <summary>Releases a file unconditionally (every ref) — used on layout reloads.</summary>
|
||||
public async Task ReleaseAllAsync(string key)
|
||||
{
|
||||
MediaSession? toStop = null;
|
||||
lock (_gate)
|
||||
{
|
||||
if (!_sessions.TryGetValue(key, out var session)) return;
|
||||
session.RefCount = 0;
|
||||
_sessions.Remove(key);
|
||||
toStop = session;
|
||||
}
|
||||
|
||||
if (toStop == null) return;
|
||||
toStop.Source.FrameAvailable -= toStop.FrameHandler;
|
||||
await SafeStopAsync(toStop.Source);
|
||||
toStop.PreviewBitmap = null;
|
||||
}
|
||||
|
||||
/// <summary>The most recent decoded frame for a file, or null before the first
|
||||
/// frame (or if the file has no session). The live compositor reads media from here.</summary>
|
||||
public VideoFrame? GetLatestFrame(string key)
|
||||
{
|
||||
lock (_gate)
|
||||
return _sessions.TryGetValue(key, out var session) ? session.LatestFrame : null;
|
||||
}
|
||||
|
||||
/// <summary>The shared preview bitmap for a file (so a config created after the
|
||||
/// first frame can adopt the running bitmap), or null.</summary>
|
||||
public WriteableBitmap? GetPreviewBitmap(string key)
|
||||
{
|
||||
lock (_gate)
|
||||
return _sessions.TryGetValue(key, out var session) ? session.PreviewBitmap : null;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
List<MediaSession> sessions;
|
||||
lock (_gate)
|
||||
{
|
||||
sessions = new List<MediaSession>(_sessions.Values);
|
||||
_sessions.Clear();
|
||||
}
|
||||
|
||||
foreach (var session in sessions)
|
||||
{
|
||||
session.Source.FrameAvailable -= session.FrameHandler;
|
||||
_ = SafeStopAsync(session.Source);
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task SafeStopAsync(IMediaFrameSource source)
|
||||
{
|
||||
try
|
||||
{
|
||||
await source.StopAsync();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
AppLog.Write($"MediaSource: stopping decoder failed: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
private bool TryGetActiveSession(MediaSession session)
|
||||
{
|
||||
lock (_gate)
|
||||
return _sessions.TryGetValue(session.Key, out var current) && ReferenceEquals(current, session);
|
||||
}
|
||||
|
||||
private void OnFrameAvailable(MediaSession session, VideoFrame frame)
|
||||
{
|
||||
if (!TryGetActiveSession(session)) return;
|
||||
session.LatestFrame = frame;
|
||||
|
||||
if (session.PreviewBitmap == null)
|
||||
{
|
||||
if (_uiDispatcher == null) return;
|
||||
if (_uiDispatcher.CheckAccess())
|
||||
EnsurePreviewBitmap(session);
|
||||
else
|
||||
_uiDispatcher.BeginInvoke(() =>
|
||||
{
|
||||
if (TryGetActiveSession(session))
|
||||
EnsurePreviewBitmap(session);
|
||||
}, DispatcherPriority.Render);
|
||||
return;
|
||||
}
|
||||
|
||||
if (_uiDispatcher == null || _uiDispatcher.CheckAccess())
|
||||
{
|
||||
CopyFrame(session);
|
||||
return;
|
||||
}
|
||||
|
||||
if (session.FramePending) return;
|
||||
session.FramePending = true;
|
||||
_uiDispatcher.BeginInvoke(() =>
|
||||
{
|
||||
session.FramePending = false;
|
||||
if (TryGetActiveSession(session) && session.PreviewBitmap != null)
|
||||
CopyFrame(session);
|
||||
}, DispatcherPriority.Render);
|
||||
}
|
||||
|
||||
private void EnsurePreviewBitmap(MediaSession session)
|
||||
{
|
||||
if (session.PreviewBitmap != null || session.LatestFrame == null) return;
|
||||
var frame = session.LatestFrame;
|
||||
var bitmap = new WriteableBitmap(frame.Width, frame.Height, 96, 96, PixelFormats.Bgra32, null);
|
||||
bitmap.WritePixels(new Int32Rect(0, 0, frame.Width, frame.Height), frame.BgraPixels, frame.Stride, 0);
|
||||
session.PreviewBitmap = bitmap;
|
||||
PreviewBitmapChanged?.Invoke(session.Key, bitmap);
|
||||
}
|
||||
|
||||
private void CopyFrame(MediaSession session)
|
||||
{
|
||||
var bitmap = session.PreviewBitmap;
|
||||
var frame = session.LatestFrame;
|
||||
if (bitmap == null || frame == null) return;
|
||||
if (frame.Width != bitmap.PixelWidth || frame.Height != bitmap.PixelHeight) return;
|
||||
bitmap.WritePixels(new Int32Rect(0, 0, frame.Width, frame.Height), frame.BgraPixels, frame.Stride, 0);
|
||||
}
|
||||
}
|
||||
@@ -1036,20 +1036,21 @@ The tasks below are ordered by dependency and risk. Each task builds on the prev
|
||||
|
||||
**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.
|
||||
|
||||
### Status: 🔶 In progress — Increment A shipped (model + persistence); Increment B (decoder) shipped; slices 6–12 open
|
||||
### Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) shipped; slice 1 step 3 (manager + `IMediaFrameSource`) shipped; resolver/preview wire-in + pacing + loop open
|
||||
|
||||
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
|
||||
2. ✅ `SourceType.MediaSource` addition to the enum
|
||||
3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
|
||||
4. ✅ `MediaVideoSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameReady` per frame; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31.
|
||||
5. 🔶 Frame capture from video file, output BGRA8 frames — decode + BGRA8 emit done; **native-FPS pacing still open** (ffprobe probe, wired by the caller; frames currently emit as fast as the pipe produces).
|
||||
6. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame
|
||||
7. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
|
||||
8. ☐ Loop control — `IsLooping` property, restart on end
|
||||
9. ☐ Volume control — per-source volume slider for audio playback
|
||||
10. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider
|
||||
11. ☐ Schema migration for media source settings (file path, loop, volume)
|
||||
12. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation
|
||||
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
|
||||
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass.
|
||||
6. 🔶 Native-FPS pacing — frames currently emit as fast as the pipe produces them; ffprobe FPS probe + per-frame pacing seam still open (slice 2).
|
||||
7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame
|
||||
8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
|
||||
9. ☐ Loop control — `IsLooping` property, restart on end
|
||||
10. ☐ Volume control — per-source volume slider for audio playback
|
||||
11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider
|
||||
12. ☐ Schema migration for media source settings (file path, loop, volume)
|
||||
13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation
|
||||
|
||||
### Design decisions
|
||||
|
||||
|
||||
@@ -155,7 +155,7 @@ C# / WPF (.NET 8) following MVVM:
|
||||
|------|------|
|
||||
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
|
||||
| `ViewModels/` | MainViewModel — `public partial class`, one file per functional area (Scenes, Background, Webcam, Audio, Trax, Socials, Streaming, Chat, Overlays, Account, License, Recording — split complete, see `ViewModels/index.md`); **Chat.cs is a thin delegating facade over `Services/ChatOverlayLayer.cs` (Commit G, first true decomposition)**; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
|
||||
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** |
|
||||
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** |
|
||||
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
|
||||
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
|
||||
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview + live controls below), left (scenes/sources), right (chat), bottom (gear + stream stats + resolution) |
|
||||
@@ -672,11 +672,16 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
||||
- **Media source decoder (TASK 21, Increment B — 2026-08-31):** `MediaVideoSource` (`Services/MediaVideoSource.cs`)
|
||||
decodes a local file to `VideoFrame`s by spawning ffmpeg with
|
||||
`-f rawvideo -pix_fmt bgra -vf scale=WxH -an` and draining the raw BGRA stdout pipe through the pure
|
||||
`RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raising `FrameReady` per frame. The subprocess sits
|
||||
behind `IDecodeProcess`/`FfmpegDecodeProcess` (binary-stdout mirror of the encoder's `IEncoderProcess`), and
|
||||
the path comes from `IFfmpegLocator` — so it's codec-agnostic, uses the ffmpeg already shipped, and is
|
||||
testable with a fake process. Frames emit as fast as the pipe produces them; native-FPS pacing (ffprobe probe,
|
||||
wired by the caller) and the resolver/compositor wiring (`ResolveOutputFrame` case) are the next slice.
|
||||
`RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raising `FrameAvailable` per frame. Implements
|
||||
`IMediaFrameSource` (`Services/IMediaFrameSource.cs`: `Key`, `FrameAvailable`, `Completed`, `StartAsync`/`StopAsync`).
|
||||
The subprocess sits behind `IDecodeProcess`/`FfmpegDecodeProcess` (binary-stdout mirror of the encoder's
|
||||
`IEncoderProcess`), and the path comes from `IFfmpegLocator` — codec-agnostic, uses the ffmpeg already shipped,
|
||||
testable with a fake process. Decode sessions are owned app-wide by `MediaVideoSourceManager`
|
||||
(`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func<string, IMediaFrameSource?>` factory
|
||||
seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`,
|
||||
coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as
|
||||
the pipe produces them; native-FPS pacing (ffprobe probe) and the resolver/preview wire-in (`ResolveOutputFrame`
|
||||
case + `Source.DisplaySource`) are the next slices.
|
||||
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
||||
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
||||
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
using System.Threading;
|
||||
using System.Windows.Media.Imaging;
|
||||
using System.Windows.Threading;
|
||||
using Xunit;
|
||||
using ytLive.Services;
|
||||
|
||||
namespace ytLive.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// MediaVideoSourceManager owns media-file decode sessions, refcounted by file
|
||||
/// path and coalesced onto the UI dispatcher — a structural mirror of
|
||||
/// ScreenCaptureManager. These tests pin that contract with a fake
|
||||
/// IMediaFrameSource (the real ffmpeg decode is exercised on Windows at runtime).
|
||||
/// </summary>
|
||||
public class MediaVideoSourceManagerTests
|
||||
{
|
||||
private sealed class FakeMediaSource : IMediaFrameSource
|
||||
{
|
||||
private readonly List<string>? _started;
|
||||
private readonly List<string>? _stopped;
|
||||
|
||||
public string Key { get; }
|
||||
public event Action<VideoFrame>? FrameAvailable;
|
||||
public event Action? Completed;
|
||||
|
||||
public FakeMediaSource(string key, List<string>? started = null, List<string>? stopped = null)
|
||||
{
|
||||
Key = key;
|
||||
_started = started;
|
||||
_stopped = stopped;
|
||||
}
|
||||
|
||||
public Task StartAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
_started?.Add(Key);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task StopAsync()
|
||||
{
|
||||
_stopped?.Add(Key);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
||||
public void Complete() => Completed?.Invoke();
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
private sealed class FailingMediaSource : IMediaFrameSource
|
||||
{
|
||||
public string Key { get; }
|
||||
public event Action<VideoFrame>? FrameAvailable;
|
||||
public event Action? Completed;
|
||||
public FailingMediaSource(string key) => Key = key;
|
||||
public Task StartAsync(CancellationToken cancellationToken = default)
|
||||
=> throw new InvalidOperationException("access denied");
|
||||
public Task StopAsync() => Task.CompletedTask;
|
||||
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
||||
public void Complete() => Completed?.Invoke();
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
private sealed class DispatcherPump : IDisposable
|
||||
{
|
||||
private readonly Thread _thread;
|
||||
private readonly ManualResetEventSlim _ready = new(false);
|
||||
private Dispatcher? _dispatcher;
|
||||
|
||||
public Dispatcher Dispatcher => _dispatcher!;
|
||||
|
||||
public DispatcherPump()
|
||||
{
|
||||
_thread = new Thread(() =>
|
||||
{
|
||||
_dispatcher = Dispatcher.CurrentDispatcher;
|
||||
_ready.Set();
|
||||
Dispatcher.Run();
|
||||
});
|
||||
_thread.IsBackground = true;
|
||||
_thread.SetApartmentState(ApartmentState.STA);
|
||||
_thread.Start();
|
||||
_ready.Wait();
|
||||
}
|
||||
|
||||
public void Drain() => Dispatcher.Invoke(() => { }, DispatcherPriority.ContextIdle);
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Dispatcher.InvokeShutdown();
|
||||
_thread.Join(3000);
|
||||
}
|
||||
}
|
||||
|
||||
private static byte[] Pixels(params byte[] raw) => raw;
|
||||
|
||||
[Fact]
|
||||
public async Task Refcount_TwoAcquires_OneDecoderUntilLastRelease()
|
||||
{
|
||||
var started = new List<string>();
|
||||
var stopped = new List<string>();
|
||||
var manager = new MediaVideoSourceManager(key => new FakeMediaSource(key, started, stopped));
|
||||
|
||||
Assert.True(await manager.AcquireAsync(@"C:\videos\clip.mp4"));
|
||||
Assert.True(await manager.AcquireAsync(@"C:\videos\clip.mp4"));
|
||||
Assert.Single(started);
|
||||
|
||||
await manager.ReleaseAsync(@"C:\videos\clip.mp4");
|
||||
Assert.Empty(stopped);
|
||||
|
||||
await manager.ReleaseAsync(@"C:\videos\clip.mp4");
|
||||
Assert.Single(stopped);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReleaseAll_StopsEvenWithMultipleRefs()
|
||||
{
|
||||
var started = new List<string>();
|
||||
var stopped = new List<string>();
|
||||
var manager = new MediaVideoSourceManager(key => new FakeMediaSource(key, started, stopped));
|
||||
|
||||
await manager.AcquireAsync(@"C:\videos\clip.mp4");
|
||||
await manager.AcquireAsync(@"C:\videos\clip.mp4");
|
||||
await manager.ReleaseAllAsync(@"C:\videos\clip.mp4");
|
||||
|
||||
Assert.Single(stopped);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Acquire_FailingSource_ReturnsFalseAndRaisesMediaFailed()
|
||||
{
|
||||
var manager = new MediaVideoSourceManager(key => new FailingMediaSource(key));
|
||||
string? failedKey = null;
|
||||
manager.MediaFailed += (key, _) => failedKey = key;
|
||||
|
||||
Assert.False(await manager.AcquireAsync(@"C:\videos\clip.mp4"));
|
||||
Assert.Equal(@"C:\videos\clip.mp4", failedKey);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Acquire_EmptyKey_ReturnsFalse()
|
||||
{
|
||||
var manager = new MediaVideoSourceManager(key => new FakeMediaSource(key));
|
||||
Assert.False(await manager.AcquireAsync(" "));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetLatestFrame_ReturnsLatestPump_UntilReleased()
|
||||
{
|
||||
FakeMediaSource? captured = null;
|
||||
var manager = new MediaVideoSourceManager(key => captured = new FakeMediaSource(key));
|
||||
|
||||
Assert.Null(manager.GetLatestFrame(@"C:\videos\clip.mp4"));
|
||||
|
||||
Assert.True(await manager.AcquireAsync(@"C:\videos\clip.mp4"));
|
||||
Assert.Null(manager.GetLatestFrame(@"C:\videos\clip.mp4"));
|
||||
|
||||
var first = new VideoFrame(2, 2, Pixels(1, 0, 0, 255, 2, 0, 0, 255, 3, 0, 0, 255, 4, 0, 0, 255));
|
||||
var second = new VideoFrame(2, 2, Pixels(5, 0, 0, 255, 6, 0, 0, 255, 7, 0, 0, 255, 8, 0, 0, 255));
|
||||
captured!.Pump(first);
|
||||
captured.Pump(second);
|
||||
|
||||
Assert.Same(second, manager.GetLatestFrame(@"C:\videos\clip.mp4"));
|
||||
|
||||
await manager.ReleaseAsync(@"C:\videos\clip.mp4");
|
||||
Assert.Null(manager.GetLatestFrame(@"C:\videos\clip.mp4"));
|
||||
}
|
||||
|
||||
// The one integration test for this branch: one file creates one shared
|
||||
// WriteableBitmap, published once, and back-to-back frames coalesce to the
|
||||
// latest (a single pending UI copy per session) — mirror of ScreenCapture.
|
||||
[Fact]
|
||||
public async Task Frames_ShareOneBitmap_AndCoalesceToLatest()
|
||||
{
|
||||
using var pump = new DispatcherPump();
|
||||
FakeMediaSource? captured = null;
|
||||
var manager = new MediaVideoSourceManager(key => captured = new FakeMediaSource(key), pump.Dispatcher);
|
||||
|
||||
WriteableBitmap? published = null;
|
||||
var publishedCount = 0;
|
||||
manager.PreviewBitmapChanged += (_, bitmap) => { published = bitmap; publishedCount++; };
|
||||
|
||||
Assert.True(await manager.AcquireAsync(@"C:\videos\clip.mp4"));
|
||||
|
||||
var first = new VideoFrame(2, 2, Pixels(1, 0, 0, 255, 2, 0, 0, 255, 3, 0, 0, 255, 4, 0, 0, 255));
|
||||
captured!.Pump(first);
|
||||
pump.Drain();
|
||||
|
||||
Assert.NotNull(published);
|
||||
Assert.Equal(1, publishedCount);
|
||||
|
||||
var dims = pump.Dispatcher.Invoke(() => new[] { published!.PixelWidth, published.PixelHeight });
|
||||
Assert.Equal(new[] { 2, 2 }, dims);
|
||||
|
||||
var second = new VideoFrame(2, 2, Pixels(5, 0, 0, 255, 6, 0, 0, 255, 7, 0, 0, 255, 8, 0, 0, 255));
|
||||
var third = new VideoFrame(2, 2, Pixels(9, 0, 0, 255, 10, 0, 0, 255, 11, 0, 0, 255, 12, 0, 0, 255));
|
||||
captured.Pump(second);
|
||||
captured.Pump(third);
|
||||
pump.Drain();
|
||||
|
||||
var result = pump.Dispatcher.Invoke(() =>
|
||||
{
|
||||
var bytes = new byte[16];
|
||||
published!.CopyPixels(new System.Windows.Int32Rect(0, 0, 2, 2), bytes, 8, 0);
|
||||
return bytes;
|
||||
});
|
||||
Assert.Equal(third.BgraPixels, result);
|
||||
}
|
||||
}
|
||||
@@ -66,10 +66,10 @@ public class MediaVideoSourceTests
|
||||
|
||||
var seen = new List<byte[]>();
|
||||
var completed = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
source.FrameReady += f => seen.Add(((byte[])f.BgraPixels.Clone()));
|
||||
source.FrameAvailable += f => seen.Add(((byte[])f.BgraPixels.Clone()));
|
||||
source.Completed += () => completed.TrySetResult(true);
|
||||
|
||||
source.Start();
|
||||
await source.StartAsync();
|
||||
await completed.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||
|
||||
Assert.Equal(2, seen.Count);
|
||||
|
||||
Reference in New Issue
Block a user