From 071ec8be0cf9f62223df9c25c7d5bdaa5a3dc1d6 Mon Sep 17 00:00:00 2001 From: gramps Date: Mon, 31 Aug 2026 19:32:41 -0700 Subject: [PATCH] TASK 21 slice 1 step 4: wire media into resolver + preview routing Wire the MediaVideoSourceManager into the live pipeline (new partial ViewModels/MainViewModel.Media.cs mirrors Webcam/Background partials): readonly _mediaManager assigned in the core ctor (decode at 1920x1080 master, dispatcher-coalesced preview), OnMediaPreviewBitmapChanged adopts the shared bitmap onto every IsMediaSource with a matching MediaPath, a Source { IsMediaSource, MediaPath } case in ResolveOutputFrame feeds GetLatestFrame, and the manager is disposed on shutdown. Source.DisplaySource now routes VideoImageSource for MediaSource (Source.cs), so media previews/canvas show decoded video. Compositor needs no change: BlitContent UniformToFill-scales any frame to the element rect. Decoder/manager/seam derived-work mirrors ScreenCaptureManager (citation in commit for slice-step-3). Test (one unit for this change, mirroring the live-capture case): Source_DisplaySource_IsVideoImageSource_WhenMediaSource. 14/14 media + DisplaySource tests pass, build 0 warnings. Docs (TASKS/HANDOFF/ai.md) updated. --- HANDOFF.md | 32 +++++++++++++++++----------- Models/Source.cs | 2 +- TASKS.md | 2 +- ViewModels/MainViewModel.Media.cs | 35 +++++++++++++++++++++++++++++++ ViewModels/MainViewModel.cs | 10 +++++++++ ai.md | 6 ++++-- ytLive.Tests/BackgroundTests.cs | 14 +++++++++++++ 7 files changed, 85 insertions(+), 16 deletions(-) create mode 100644 ViewModels/MainViewModel.Media.cs diff --git a/HANDOFF.md b/HANDOFF.md index 0ac808d..62928d8 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -34,13 +34,21 @@ This session: single shared bitmap, coalesce-to-latest). **Verified: all 10 media tests pass (5 manager + 3 reader + 1 decoder integration + 1), build 0 warnings.** -Next (slice 1 step 4 — VM wire-in): create `_mediaManager` in the ctor with -`path => new MediaVideoSource(path, , , new FfmpegLocator(), new FfmpegDecodeProcess())`, -subscribe `OnMediaPreviewBitmapChanged` (set `source.VideoImageSource` for every -`IsMediaSource` with that `MediaPath`), add the `Source { IsMediaSource: true, MediaPath: not null }` -case to `ResolveOutputFrame` (`MainViewModel.cs:447`), add `or SourceType.MediaSource` -to `Source.DisplaySource` (Source.cs:119-121) so media previews route `VideoImageSource`, -and dispose the manager on shutdown. Then slices 2 (native-FPS pacing) and 3 (loop). +Slice 1 step 4 (wire-in) shipped: `_mediaManager` wired in the core ctor +(`path => new MediaVideoSource(path, 1920, 1080, new FfmpegLocator(), new FfmpegDecodeProcess())`, +dispatched preview), `OnMediaPreviewBitmapChanged` sets `VideoImageSource` on every +`IsMediaSource` with a matching `MediaPath`, a `Source { IsMediaSource, MediaPath }` +case in `ResolveOutputFrame` feeds `_mediaManager.GetLatestFrame`, `Source.DisplaySource` +routes `VideoImageSource` for `MediaSource`, and the manager is disposed on shutdown. +Lives in the new `ViewModels/MainViewModel.Media.cs` partial. **Verified: 14/14 +media + DisplaySource tests pass, build 0 warnings.** + +Next (slice 1 step 4's remaining open end): no session acquisition yet — nothing +`AcquireAsync`s a media path, so no frames flow until the UI picker (a later slice) +starts a session on add and releases on remove. + +Next (slice 2): native-FPS pacing (ffprobe FPS probe + per-frame pacing seam). +Slice 3: loop control (restart decode on `Completed` when `MediaIsLooping`). **Landmine:** a stale `testhost` can lock `ytLive.Tests`'s `ytLive.dll` and break `dotnet build` of the test project — kill it first (`cmd.exe /c "taskkill /PID N /F"`) @@ -50,11 +58,11 @@ pure tests run in WSL; user verifies the GUI suites on native Windows PowerShell ## Next -1. Commit slice 1 step 3 (this unit): manager + `IMediaFrameSource` + tests. Commit - is already covered by `TASKS.md`/`HANDOFF.md`; this HANDOFF is the shipping doc. -2. Slice 1 step 4: wire `_mediaManager` into the resolver + preview routing (see above). -3. Slice 2: native-FPS pacing (ffprobe FPS probe + per-frame pacing seam). -4. Await user push checkpoint for the pending TASK 22/21 commits. +1. Commit slice 1 step 4 (this unit): resolver/preview wire-in + `Source.DisplaySource` media branch + tests. +2. Slice 2: native-FPS pacing (ffprobe FPS probe + per-frame pacing seam). +3. Slice 3: loop control (restart decode on `Completed` when `MediaIsLooping`). +4. UI picker slice: acquire/release media sessions on add/remove (the remaining slice-1-step-4 open end). +5. Await user push checkpoint for the pending TASK 22/21 commits. ## The directive (2026-08-31, user) diff --git a/Models/Source.cs b/Models/Source.cs index eaa34b6..5428a49 100644 --- a/Models/Source.cs +++ b/Models/Source.cs @@ -118,7 +118,7 @@ public class Source : SceneElement public override ImageSource? DisplaySource => IsLiveCapture ? (VideoImageSource ?? _imageSource) - : (Type is SourceType.ChatBox or SourceType.WebSource ? VideoImageSource : _imageSource); + : (Type is SourceType.ChatBox or SourceType.WebSource or SourceType.MediaSource ? VideoImageSource : _imageSource); public override bool IsImageSource => Type == SourceType.Image; diff --git a/TASKS.md b/TASKS.md index 1cf620f..e18e4f8 100644 --- a/TASKS.md +++ b/TASKS.md @@ -1044,7 +1044,7 @@ The tasks below are ordered by dependency and risk. Each task builds on the prev 4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice. 5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass. 6. 🔶 Native-FPS pacing — frames currently emit as fast as the pipe produces them; ffprobe FPS probe + per-frame pacing seam still open (slice 2). -7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame +7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)** 8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size 9. ☐ Loop control — `IsLooping` property, restart on end 10. ☐ Volume control — per-source volume slider for audio playback diff --git a/ViewModels/MainViewModel.Media.cs b/ViewModels/MainViewModel.Media.cs new file mode 100644 index 0000000..3e634a1 --- /dev/null +++ b/ViewModels/MainViewModel.Media.cs @@ -0,0 +1,35 @@ +using System.Linq; +using System.Windows.Media.Imaging; +using ytLive.Models; +using ytLive.Services; + +namespace ytLive.ViewModels; + +// Media source (TASK 21): decode sessions are owned app-wide by +// MediaVideoSourceManager, refcounted by MediaPath. Modeled on the Webcam/Background +// partials: the readonly manager field is assigned in the core constructor, the +// shared WriteableBitmap published per file is adopted by every media Source whose +// MediaPath matches, and ResolveOutputFrame feeds the live compositor. +public partial class MainViewModel +{ + // Decode resolution: the 1920x1080 master. Media frames are scaled to the + // element rect by the compositor (UniformToFill), so this is the capture size. + private const int MediaDecodeWidth = 1920; + private const int MediaDecodeHeight = 1080; + + private readonly MediaVideoSourceManager _mediaManager; + + // MediaVideoSourceManager publishes one shared WriteableBitmap per file; every + // media Source pointing at that path picks it up for the preview/canvas. + private void OnMediaPreviewBitmapChanged(string path, WriteableBitmap bitmap) + { + foreach (var source in Scenes.SelectMany(s => s.Elements.OfType()) + .Where(s => s.IsMediaSource && s.MediaPath == path)) + source.VideoImageSource = bitmap; + } + + // Called from the encoder error path is out of scope (TASK 21 sees the decoder + // fail at startup); log to the debug console for now, surfaced via the UI later. + private static void OnMediaFailed(string path, string message) + => System.Diagnostics.Debug.WriteLine($"MediaSource: '{path}' failed: {message}"); +} diff --git a/ViewModels/MainViewModel.cs b/ViewModels/MainViewModel.cs index 71ca2f7..cd2ad5d 100644 --- a/ViewModels/MainViewModel.cs +++ b/ViewModels/MainViewModel.cs @@ -282,6 +282,14 @@ public partial class MainViewModel : ViewModelBase _screenCaptureManager.CaptureFailed += (key, message) => AppLog.Write($"ScreenCaptureManager: capture '{key}' failed: {message}"); + _mediaManager = new MediaVideoSourceManager( + path => new MediaVideoSource( + path, MediaDecodeWidth, MediaDecodeHeight, + new FfmpegLocator(), new FfmpegDecodeProcess()), + System.Windows.Application.Current?.Dispatcher); + _mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged; + _mediaManager.MediaFailed += OnMediaFailed; + _chatLayer = new ChatOverlayLayer(new ChatBoxRenderer()); _sceneGraph = new SceneGraph(); @@ -397,6 +405,7 @@ public partial class MainViewModel : ViewModelBase _framePump.Dispose(); _cameraManager.Dispose(); _screenCaptureManager.Dispose(); + _mediaManager.Dispose(); _webView2Manager?.Dispose(); _layoutStore.Dispose(); _musicPlayer.Dispose(); @@ -452,6 +461,7 @@ public partial class MainViewModel : ViewModelBase Source { IsLiveCapture: true, CaptureKey: not null } live => _screenCaptureManager.GetLatestFrame(live.CaptureKey), Source { Type: SourceType.ChatBox } chatBox => RenderChatBox(chatBox), Source { Type: SourceType.WebSource } web => _webView2Manager?.GetLatestFrame(web.Id), + Source { IsMediaSource: true, MediaPath: not null } media => _mediaManager.GetLatestFrame(media.MediaPath), Source { AssetId: not null } image => StaticPixelCache.Get(image.AssetId), _ => null, }; diff --git a/ai.md b/ai.md index 69de975..817a9de 100644 --- a/ai.md +++ b/ai.md @@ -680,8 +680,10 @@ seam:** `Func`, `Func` resolver, `Func` factory seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`, coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as - the pipe produces them; native-FPS pacing (ffprobe probe) and the resolver/preview wire-in (`ResolveOutputFrame` - case + `Source.DisplaySource`) are the next slices. + the pipe produces them. Wired into the live pipeline (slice 1 step 4): `ResolveOutputFrame` reads + `_mediaManager.GetLatestFrame(MediaPath)` for a `MediaSource`, and `Source.DisplaySource` + (`Models/Source.cs`) routes `VideoImageSource` for `MediaSource` so the preview/canvas show decoded video. + Still open: session acquisition on add/remove (UI picker), native-FPS pacing (ffprobe probe), loop control. - **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs` diff --git a/ytLive.Tests/BackgroundTests.cs b/ytLive.Tests/BackgroundTests.cs index 4fde6b9..1b8c135 100644 --- a/ytLive.Tests/BackgroundTests.cs +++ b/ytLive.Tests/BackgroundTests.cs @@ -93,6 +93,20 @@ public class BackgroundTests Assert.Same(bitmap, source.DisplaySource); } + // TASK 21: a media source's DisplaySource routes its live video frames (like + // Display/Window capture), so media previews show decoded video, not the stale + // default art. + [Fact] + public void Source_DisplaySource_IsVideoImageSource_WhenMediaSource() + { + var source = new Source { Type = SourceType.MediaSource }; + var bitmap = new System.Windows.Media.Imaging.WriteableBitmap( + 2, 2, 96, 96, System.Windows.Media.PixelFormats.Bgra32, null); + source.VideoImageSource = bitmap; + + Assert.Same(bitmap, source.DisplaySource); + } + [Fact] public void Source_DisplaySource_FallsBackToImage_WhenLiveCaptureWithNoVideoImage() {