TASK 21 slice 1 step 4: wire media into resolver + preview routing
Wire the MediaVideoSourceManager into the live pipeline (new partial
ViewModels/MainViewModel.Media.cs mirrors Webcam/Background partials):
readonly _mediaManager assigned in the core ctor (decode at 1920x1080 master,
dispatcher-coalesced preview), OnMediaPreviewBitmapChanged adopts the shared
bitmap onto every IsMediaSource with a matching MediaPath, a
Source { IsMediaSource, MediaPath } case in ResolveOutputFrame feeds
GetLatestFrame, and the manager is disposed on shutdown.
Source.DisplaySource now routes VideoImageSource for MediaSource (Source.cs),
so media previews/canvas show decoded video.
Compositor needs no change: BlitContent UniformToFill-scales any frame to the
element rect. Decoder/manager/seam derived-work mirrors ScreenCaptureManager
(citation in commit for slice-step-3).
Test (one unit for this change, mirroring the live-capture case):
Source_DisplaySource_IsVideoImageSource_WhenMediaSource. 14/14 media +
DisplaySource tests pass, build 0 warnings. Docs (TASKS/HANDOFF/ai.md) updated.
This commit is contained in:
+20
-12
@@ -34,13 +34,21 @@ This session:
|
|||||||
single shared bitmap, coalesce-to-latest). **Verified: all 10 media tests pass
|
single shared bitmap, coalesce-to-latest). **Verified: all 10 media tests pass
|
||||||
(5 manager + 3 reader + 1 decoder integration + 1), build 0 warnings.**
|
(5 manager + 3 reader + 1 decoder integration + 1), build 0 warnings.**
|
||||||
|
|
||||||
Next (slice 1 step 4 — VM wire-in): create `_mediaManager` in the ctor with
|
Slice 1 step 4 (wire-in) shipped: `_mediaManager` wired in the core ctor
|
||||||
`path => new MediaVideoSource(path, <W>, <H>, new FfmpegLocator(), new FfmpegDecodeProcess())`,
|
(`path => new MediaVideoSource(path, 1920, 1080, new FfmpegLocator(), new FfmpegDecodeProcess())`,
|
||||||
subscribe `OnMediaPreviewBitmapChanged` (set `source.VideoImageSource` for every
|
dispatched preview), `OnMediaPreviewBitmapChanged` sets `VideoImageSource` on every
|
||||||
`IsMediaSource` with that `MediaPath`), add the `Source { IsMediaSource: true, MediaPath: not null }`
|
`IsMediaSource` with a matching `MediaPath`, a `Source { IsMediaSource, MediaPath }`
|
||||||
case to `ResolveOutputFrame` (`MainViewModel.cs:447`), add `or SourceType.MediaSource`
|
case in `ResolveOutputFrame` feeds `_mediaManager.GetLatestFrame`, `Source.DisplaySource`
|
||||||
to `Source.DisplaySource` (Source.cs:119-121) so media previews route `VideoImageSource`,
|
routes `VideoImageSource` for `MediaSource`, and the manager is disposed on shutdown.
|
||||||
and dispose the manager on shutdown. Then slices 2 (native-FPS pacing) and 3 (loop).
|
Lives in the new `ViewModels/MainViewModel.Media.cs` partial. **Verified: 14/14
|
||||||
|
media + DisplaySource tests pass, build 0 warnings.**
|
||||||
|
|
||||||
|
Next (slice 1 step 4's remaining open end): no session acquisition yet — nothing
|
||||||
|
`AcquireAsync`s a media path, so no frames flow until the UI picker (a later slice)
|
||||||
|
starts a session on add and releases on remove.
|
||||||
|
|
||||||
|
Next (slice 2): native-FPS pacing (ffprobe FPS probe + per-frame pacing seam).
|
||||||
|
Slice 3: loop control (restart decode on `Completed` when `MediaIsLooping`).
|
||||||
|
|
||||||
**Landmine:** a stale `testhost` can lock `ytLive.Tests`'s `ytLive.dll` and break
|
**Landmine:** a stale `testhost` can lock `ytLive.Tests`'s `ytLive.dll` and break
|
||||||
`dotnet build` of the test project — kill it first (`cmd.exe /c "taskkill /PID N /F"`)
|
`dotnet build` of the test project — kill it first (`cmd.exe /c "taskkill /PID N /F"`)
|
||||||
@@ -50,11 +58,11 @@ pure tests run in WSL; user verifies the GUI suites on native Windows PowerShell
|
|||||||
|
|
||||||
## Next
|
## Next
|
||||||
|
|
||||||
1. Commit slice 1 step 3 (this unit): manager + `IMediaFrameSource` + tests. Commit
|
1. Commit slice 1 step 4 (this unit): resolver/preview wire-in + `Source.DisplaySource` media branch + tests.
|
||||||
is already covered by `TASKS.md`/`HANDOFF.md`; this HANDOFF is the shipping doc.
|
2. Slice 2: native-FPS pacing (ffprobe FPS probe + per-frame pacing seam).
|
||||||
2. Slice 1 step 4: wire `_mediaManager` into the resolver + preview routing (see above).
|
3. Slice 3: loop control (restart decode on `Completed` when `MediaIsLooping`).
|
||||||
3. Slice 2: native-FPS pacing (ffprobe FPS probe + per-frame pacing seam).
|
4. UI picker slice: acquire/release media sessions on add/remove (the remaining slice-1-step-4 open end).
|
||||||
4. Await user push checkpoint for the pending TASK 22/21 commits.
|
5. Await user push checkpoint for the pending TASK 22/21 commits.
|
||||||
|
|
||||||
## The directive (2026-08-31, user)
|
## The directive (2026-08-31, user)
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -118,7 +118,7 @@ public class Source : SceneElement
|
|||||||
|
|
||||||
public override ImageSource? DisplaySource => IsLiveCapture
|
public override ImageSource? DisplaySource => IsLiveCapture
|
||||||
? (VideoImageSource ?? _imageSource)
|
? (VideoImageSource ?? _imageSource)
|
||||||
: (Type is SourceType.ChatBox or SourceType.WebSource ? VideoImageSource : _imageSource);
|
: (Type is SourceType.ChatBox or SourceType.WebSource or SourceType.MediaSource ? VideoImageSource : _imageSource);
|
||||||
|
|
||||||
public override bool IsImageSource => Type == SourceType.Image;
|
public override bool IsImageSource => Type == SourceType.Image;
|
||||||
|
|
||||||
|
|||||||
@@ -1044,7 +1044,7 @@ The tasks below are ordered by dependency and risk. Each task builds on the prev
|
|||||||
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
|
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
|
||||||
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass.
|
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass.
|
||||||
6. 🔶 Native-FPS pacing — frames currently emit as fast as the pipe produces them; ffprobe FPS probe + per-frame pacing seam still open (slice 2).
|
6. 🔶 Native-FPS pacing — frames currently emit as fast as the pipe produces them; ffprobe FPS probe + per-frame pacing seam still open (slice 2).
|
||||||
7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame
|
7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)**
|
||||||
8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
|
8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
|
||||||
9. ☐ Loop control — `IsLooping` property, restart on end
|
9. ☐ Loop control — `IsLooping` property, restart on end
|
||||||
10. ☐ Volume control — per-source volume slider for audio playback
|
10. ☐ Volume control — per-source volume slider for audio playback
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
using System.Linq;
|
||||||
|
using System.Windows.Media.Imaging;
|
||||||
|
using ytLive.Models;
|
||||||
|
using ytLive.Services;
|
||||||
|
|
||||||
|
namespace ytLive.ViewModels;
|
||||||
|
|
||||||
|
// Media source (TASK 21): decode sessions are owned app-wide by
|
||||||
|
// MediaVideoSourceManager, refcounted by MediaPath. Modeled on the Webcam/Background
|
||||||
|
// partials: the readonly manager field is assigned in the core constructor, the
|
||||||
|
// shared WriteableBitmap published per file is adopted by every media Source whose
|
||||||
|
// MediaPath matches, and ResolveOutputFrame feeds the live compositor.
|
||||||
|
public partial class MainViewModel
|
||||||
|
{
|
||||||
|
// Decode resolution: the 1920x1080 master. Media frames are scaled to the
|
||||||
|
// element rect by the compositor (UniformToFill), so this is the capture size.
|
||||||
|
private const int MediaDecodeWidth = 1920;
|
||||||
|
private const int MediaDecodeHeight = 1080;
|
||||||
|
|
||||||
|
private readonly MediaVideoSourceManager _mediaManager;
|
||||||
|
|
||||||
|
// MediaVideoSourceManager publishes one shared WriteableBitmap per file; every
|
||||||
|
// media Source pointing at that path picks it up for the preview/canvas.
|
||||||
|
private void OnMediaPreviewBitmapChanged(string path, WriteableBitmap bitmap)
|
||||||
|
{
|
||||||
|
foreach (var source in Scenes.SelectMany(s => s.Elements.OfType<Source>())
|
||||||
|
.Where(s => s.IsMediaSource && s.MediaPath == path))
|
||||||
|
source.VideoImageSource = bitmap;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Called from the encoder error path is out of scope (TASK 21 sees the decoder
|
||||||
|
// fail at startup); log to the debug console for now, surfaced via the UI later.
|
||||||
|
private static void OnMediaFailed(string path, string message)
|
||||||
|
=> System.Diagnostics.Debug.WriteLine($"MediaSource: '{path}' failed: {message}");
|
||||||
|
}
|
||||||
@@ -282,6 +282,14 @@ public partial class MainViewModel : ViewModelBase
|
|||||||
_screenCaptureManager.CaptureFailed += (key, message) =>
|
_screenCaptureManager.CaptureFailed += (key, message) =>
|
||||||
AppLog.Write($"ScreenCaptureManager: capture '{key}' failed: {message}");
|
AppLog.Write($"ScreenCaptureManager: capture '{key}' failed: {message}");
|
||||||
|
|
||||||
|
_mediaManager = new MediaVideoSourceManager(
|
||||||
|
path => new MediaVideoSource(
|
||||||
|
path, MediaDecodeWidth, MediaDecodeHeight,
|
||||||
|
new FfmpegLocator(), new FfmpegDecodeProcess()),
|
||||||
|
System.Windows.Application.Current?.Dispatcher);
|
||||||
|
_mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged;
|
||||||
|
_mediaManager.MediaFailed += OnMediaFailed;
|
||||||
|
|
||||||
_chatLayer = new ChatOverlayLayer(new ChatBoxRenderer());
|
_chatLayer = new ChatOverlayLayer(new ChatBoxRenderer());
|
||||||
|
|
||||||
_sceneGraph = new SceneGraph();
|
_sceneGraph = new SceneGraph();
|
||||||
@@ -397,6 +405,7 @@ public partial class MainViewModel : ViewModelBase
|
|||||||
_framePump.Dispose();
|
_framePump.Dispose();
|
||||||
_cameraManager.Dispose();
|
_cameraManager.Dispose();
|
||||||
_screenCaptureManager.Dispose();
|
_screenCaptureManager.Dispose();
|
||||||
|
_mediaManager.Dispose();
|
||||||
_webView2Manager?.Dispose();
|
_webView2Manager?.Dispose();
|
||||||
_layoutStore.Dispose();
|
_layoutStore.Dispose();
|
||||||
_musicPlayer.Dispose();
|
_musicPlayer.Dispose();
|
||||||
@@ -452,6 +461,7 @@ public partial class MainViewModel : ViewModelBase
|
|||||||
Source { IsLiveCapture: true, CaptureKey: not null } live => _screenCaptureManager.GetLatestFrame(live.CaptureKey),
|
Source { IsLiveCapture: true, CaptureKey: not null } live => _screenCaptureManager.GetLatestFrame(live.CaptureKey),
|
||||||
Source { Type: SourceType.ChatBox } chatBox => RenderChatBox(chatBox),
|
Source { Type: SourceType.ChatBox } chatBox => RenderChatBox(chatBox),
|
||||||
Source { Type: SourceType.WebSource } web => _webView2Manager?.GetLatestFrame(web.Id),
|
Source { Type: SourceType.WebSource } web => _webView2Manager?.GetLatestFrame(web.Id),
|
||||||
|
Source { IsMediaSource: true, MediaPath: not null } media => _mediaManager.GetLatestFrame(media.MediaPath),
|
||||||
Source { AssetId: not null } image => StaticPixelCache.Get(image.AssetId),
|
Source { AssetId: not null } image => StaticPixelCache.Get(image.AssetId),
|
||||||
_ => null,
|
_ => null,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -680,8 +680,10 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
|||||||
(`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func<string, IMediaFrameSource?>` factory
|
(`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func<string, IMediaFrameSource?>` factory
|
||||||
seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`,
|
seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`,
|
||||||
coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as
|
coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as
|
||||||
the pipe produces them; native-FPS pacing (ffprobe probe) and the resolver/preview wire-in (`ResolveOutputFrame`
|
the pipe produces them. Wired into the live pipeline (slice 1 step 4): `ResolveOutputFrame` reads
|
||||||
case + `Source.DisplaySource`) are the next slices.
|
`_mediaManager.GetLatestFrame(MediaPath)` for a `MediaSource`, and `Source.DisplaySource`
|
||||||
|
(`Models/Source.cs`) routes `VideoImageSource` for `MediaSource` so the preview/canvas show decoded video.
|
||||||
|
Still open: session acquisition on add/remove (UI picker), native-FPS pacing (ffprobe probe), loop control.
|
||||||
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
||||||
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
||||||
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
||||||
|
|||||||
@@ -93,6 +93,20 @@ public class BackgroundTests
|
|||||||
Assert.Same(bitmap, source.DisplaySource);
|
Assert.Same(bitmap, source.DisplaySource);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TASK 21: a media source's DisplaySource routes its live video frames (like
|
||||||
|
// Display/Window capture), so media previews show decoded video, not the stale
|
||||||
|
// default art.
|
||||||
|
[Fact]
|
||||||
|
public void Source_DisplaySource_IsVideoImageSource_WhenMediaSource()
|
||||||
|
{
|
||||||
|
var source = new Source { Type = SourceType.MediaSource };
|
||||||
|
var bitmap = new System.Windows.Media.Imaging.WriteableBitmap(
|
||||||
|
2, 2, 96, 96, System.Windows.Media.PixelFormats.Bgra32, null);
|
||||||
|
source.VideoImageSource = bitmap;
|
||||||
|
|
||||||
|
Assert.Same(bitmap, source.DisplaySource);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Source_DisplaySource_FallsBackToImage_WhenLiveCaptureWithNoVideoImage()
|
public void Source_DisplaySource_FallsBackToImage_WhenLiveCaptureWithNoVideoImage()
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user