feat(media): add a video file layer that decodes into the scene

TASK 21's picker slice: the decoder, manager, pacing and loop mechanism were
already shipped but nothing ever acquired a session, so a media Source rendered
nothing. Adds "Video File" to the (+) menu (video-filtered OpenFileDialog),
one decode session per distinct path with a HasSession guard, release when the
last scene reference goes, and a post-load reconcile. Loop is a manager-level
per-path answer ("any live reference loops") applied before StartAsync and
updated live via SetLooping. Properties panel shows the file with Change… and
Loop. Volume stays unwired: media audio rides the desktop loopback, TRAX-style.

Reference: OBS's media source (file → decode → composite into the scene) is the
established answer for this feature; the lifecycle shape mirrors this repo's
WebView2/ScreenCapture registration rather than inventing a new one.

Test: ytLive.Tests/MediaSourceAcquisitionTests.cs — add starts one decoder and
frames reach the layer, loop flip reaches the decoder, remove stops it.

Docs: task-21 closed, TASKS row + Day-4 row + stale 3·17 row corrected,
frame-pipeline acquisition note, HANDOFF rewritten. verify.sh green.
This commit is contained in:
2026-11-05 07:17:43 -08:00
parent 785b5ff8f3
commit 77d7208bfa
11 changed files with 429 additions and 68 deletions
+25
View File
@@ -70,6 +70,8 @@
IsEnabled="{Binding CanAddWebcam}"
ToolTip="{Binding WebcamAddToolTip}"/>
<MenuItem Header="Image" Command="{Binding AddImageCommand}"/>
<MenuItem Header="Video File" Command="{Binding AddMediaCommand}"
ToolTip="Play a video file in the scene (mp4, mov, avi, mkv, webm)"/>
<MenuItem Header="Text" Command="{Binding AddSourceCommand}" CommandParameter="{x:Static models:SourceType.TextOverlay}"/>
<MenuItem Header="Countdown" Command="{Binding AddSourceCommand}" CommandParameter="{x:Static models:SourceType.Countdown}"
Visibility="{Binding IsCountdownAvailable, Converter={StaticResource BoolToVis}}"
@@ -725,6 +727,29 @@
</DockPanel>
</StackPanel>
<!-- Media (video file) config — TASK 21 -->
<StackPanel Visibility="{Binding SelectedElement.IsMediaSource, Converter={StaticResource BoolToVis}}" Margin="0,0,0,8">
<TextBlock Text="Video File" Style="{StaticResource PropertyHeader}"/>
<Grid Margin="10,0,0,2">
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"/>
<ColumnDefinition Width="Auto"/>
</Grid.ColumnDefinitions>
<TextBlock Grid.Column="0" Text="{Binding SelectedElement.MediaFileDisplay}"
Style="{StaticResource PropertyFieldLabel}"
TextTrimming="CharacterEllipsis" VerticalAlignment="Center"
ToolTip="{Binding SelectedElement.MediaPath}"/>
<Button Grid.Column="1" Content="Change…" Command="{Binding ChangeMediaFileCommand}"
CommandParameter="{Binding SelectedElement}" Style="{StaticResource PropertyButton}"
Margin="4,0,0,0" Padding="8,2" FontSize="11"/>
</Grid>
<CheckBox Style="{StaticResource PropertyPillCheck}" Margin="10,4,0,4"
Content="Loop" IsChecked="{Binding SelectedElement.MediaIsLooping, Mode=TwoWay}"
ToolTip="Restart from the beginning at the end of the file — for starting-soon and BRB loops"/>
<TextBlock Text="Audio plays through your desktop audio, like TRAX."
Style="{StaticResource PropertyHint}"/>
</StackPanel>
<!-- Chat box config -->
<StackPanel Visibility="{Binding SelectedElement.IsChatBox, Converter={StaticResource BoolToVis}}" Margin="0,0,0,8">
<TextBlock Text="Chat Overlay" Style="{StaticResource PropertyHeader}"/>
+11 -10
View File
@@ -1,19 +1,18 @@
# HANDOFF — 2026-11-04 (TASK 3·17 — static text layers)
# HANDOFF — 2026-11-05 (TASK 21 — video file layer)
## State
- `main`; TASK 53 is complete and TASK 3·17 is complete in three commits: `60c4a00`, `58eb640`,
and the final output/compositing phase.
- Text layers persist independently, edit as static multiline text, and share one cached raster between
preview and output. The frame pump only reads the raster cache; no WPF render occurs on its thread.
- The app clean build has 0 warnings. 524 tests passed with the documented physical mouse-injection test
and known audio test excluded; the new text renderer, preview/output, and pump-cache tests pass.
- The four stray `Assets/llamacasty-*` exports were deleted 2026-11-04 — unreferenced by the csproj,
XAML or scripts (recipe in `MyMistakes.md`).
- `main`; TASK 21 is complete: a video file added from the (+) menu decodes, composites into preview
and stream, loops on toggle, and stops with its last scene reference. 3·17 (text layers) completed
2026-11-04.
- New test `MediaSourceAcquisitionTests` (add → one decoder starts + frames reach the layer + loop flip
reaches the decoder + remove → stops). App build is 0 warnings.
- Volume is deliberately unwired: media audio rides the desktop loopback, so a slider would do nothing.
## Next action
- Next action: select the next queued task from `TASKS.md`.
- Next queued unit: TASK 49 (chat profanity filter). Day 3's TASK 32/33 resilience slices are still
outstanding — 32·4 (fragmented MP4) is the smallest.
- Do not push absent a checkpoint request.
## Landmines
@@ -21,4 +20,6 @@
- `LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` injects physical
mouse input. Do not run it while the desktop is in use.
- Integration tests constructing `MainWindow` must set a temporary `LayoutPathOverride` and clear it in `finally`.
- Media sessions are keyed per **path**, not per layer: acquire through `EnsureMediaSession`, release
through `ReleaseMediaIfUnused`, never `AcquireAsync` twice for one path.
- Bundled FFmpeg stays outside the single-file exe (`ExcludeFromSingleFile="true"`).
+9 -1
View File
@@ -118,7 +118,8 @@ public class Source : SceneElement
/// and sizes it in the preview, so it needs the same selection box, grab and resize handle
/// (creator ruling 2026-11-01).</summary>
public override bool IsDraggable => Type is SourceType.Image or SourceType.WebSource or SourceType.ChatBox
or SourceType.TextOverlay or SourceType.AlertBox or SourceType.Blur or SourceType.Ticker;
or SourceType.TextOverlay or SourceType.AlertBox or SourceType.Blur or SourceType.Ticker
or SourceType.MediaSource;
/// <summary>Live captures, chat, web sources and media are dynamic; static art is fixed.</summary>
public override ElementKind Kind => Type is SourceType.DisplayCapture or SourceType.WindowCapture
@@ -238,10 +239,17 @@ public class Source : SceneElement
set
{
if (Set(ref _mediaPath, value))
{
Raise(nameof(IsMediaSource));
Raise(nameof(MediaFileDisplay));
}
}
}
/// <summary>The chosen file's name for the properties panel (path stays in the model).</summary>
public string MediaFileDisplay =>
string.IsNullOrWhiteSpace(_mediaPath) ? "No video chosen" : System.IO.Path.GetFileName(_mediaPath);
private bool _mediaIsLooping;
public bool MediaIsLooping
{
+36 -1
View File
@@ -39,6 +39,7 @@ public sealed class MediaVideoSourceManager : IDisposable
private readonly Func<string, IMediaFrameSource?> _sourceFactory;
private readonly Dispatcher? _uiDispatcher;
private readonly Func<string, bool>? _loopProvider;
private readonly Dictionary<string, MediaSession> _sessions = new();
private readonly object _gate = new();
@@ -48,10 +49,14 @@ public sealed class MediaVideoSourceManager : IDisposable
/// <summary>Raised when a media decoder cannot be created (bad path).</summary>
public event Action<string, string>? MediaFailed;
public MediaVideoSourceManager(Func<string, IMediaFrameSource?> sourceFactory, Dispatcher? uiDispatcher = null)
public MediaVideoSourceManager(
Func<string, IMediaFrameSource?> sourceFactory,
Dispatcher? uiDispatcher = null,
Func<string, bool>? loopProvider = null)
{
_sourceFactory = sourceFactory;
_uiDispatcher = uiDispatcher;
_loopProvider = loopProvider;
}
/// <summary>Increments the refcount for a file, starting the decoder the first time.</summary>
@@ -90,6 +95,10 @@ public sealed class MediaVideoSourceManager : IDisposable
session = new MediaSession(key, source);
session.FrameHandler = frame => OnFrameAvailable(session, frame);
session.Source.FrameAvailable += session.FrameHandler;
// Loop intent lives on the Source (per scene), so the caller supplies the
// per-path answer ("any live reference loops"). Applied before Start so a
// zero-length or instantly-EOF file cannot slip past a later assignment.
session.Source.Looping = _loopProvider?.Invoke(key) ?? false;
_sessions[key] = session;
shouldStart = true;
}
@@ -158,6 +167,32 @@ public sealed class MediaVideoSourceManager : IDisposable
return _sessions.TryGetValue(key, out var session) ? session.LatestFrame : null;
}
/// <summary>True while a decode session exists for the file — the caller uses this to
/// keep one acquire per path instead of stacking refcounts on every layout pass.</summary>
public bool HasSession(string key)
{
lock (_gate)
return _sessions.ContainsKey(key);
}
/// <summary>The files with a live session, for reconciling a layout load.</summary>
public IReadOnlyList<string> GetActiveKeys()
{
lock (_gate)
return new List<string>(_sessions.Keys);
}
/// <summary>Applies a loop change to a running session (the creator flipped the
/// toggle). No-op when the file has no session.</summary>
public void SetLooping(string key, bool looping)
{
lock (_gate)
{
if (_sessions.TryGetValue(key, out var session))
session.Source.Looping = looping;
}
}
/// <summary>The shared preview bitmap for a file (so a config created after the
/// first frame can adopt the running bitmap), or null.</summary>
public WriteableBitmap? GetPreviewBitmap(string key)
+6 -8
View File
@@ -20,7 +20,7 @@ corrected. No new scope mid-week; anything discovered goes on a list for day 6.
| **1 · Mon** — licensing | Paid-app ownership check (`BIsSubscribedApp`) + Debug fake + app-wide `IsAppOwned` · greyed tier entries · About conversion copy + llamacasty.com link · current-launch Logs tab (replaces the separate system-message strip) · Debug-only `InternalsVisibleTo` | A locked tier opens About without changing the selected free tier or leaving the app; the site opens only from its link |
| **2 · Tue** — release blockers | Visibility flow (creator-selected at release; development/test stays Private) · `OutputProfile` entitlement: 30 fps free / 60 fps paid regardless of resolution · ~~`StreamConfig` defaults~~ retired (no production call sites) · TASK 53 (vertical resolution check ✅; remaining output contract open) · 9·5 webcam identity key · Move transitions (per-layer carry-across-scenes) | Visibility is selectable at creation and live-editable; the 30/60-fps gate is tested; remaining Day 2 items stay queued |
| **3 · Wed** — resilience | TASK 32: blip retry → measured grace → amber RECONNECTING → one-click Back on air → fragmented MP4 → ingest pre-flight · 33 bandwidth step-down | Killing the network mid-broadcast recovers without ending it |
| **4 · Thu** — content & media | 3·17 Text layer · 21 media acquisition wiring + file picker + loop/volume UI · 49 chat profanity filter · 34 scheduled streams | Text layer renders/persists; media file plays, loops, has volume; scheduled stream created |
| **4 · Thu** — content & media | ~~3·17 Text layer~~ ✅ · ~~21 media acquisition wiring~~ ✅ (picker, loop, lifecycle) · 49 chat profanity filter · 34 scheduled streams | Text layer renders/persists; media file plays, loops; scheduled stream created |
| **5 · Fri** — data & settings | 3·20 RewardEvent persistence + `superChatEvents.list` backfill · 39·2 YPP slice 2 · 37 defaults-vs-current split · 35 scene-linked audio · 40·A/C/D settings round · 55·2–3 disk hygiene | Session report reads persisted events; defaults can be reverted; temp clips cleaned |
| **6 · Sat** — finish & prove | 50 publish recording · preview undo (Ctrl+Z) · 14 creator feedback batch · 36 gold pass · full sweep: clean build, whole per-class suite, `publish.sh` artifact, EULA/privacy/notices in the artifact · commit + push | Gold pass list empty; artifact stages; suite green; pushed |
@@ -29,11 +29,9 @@ in the middle.
## Next up (after the surge slice)
**TASK 3·17 — the Text layer** is scheduled in the surge (day 4). Placeholder detail retained for
the unit itself: `AddSource` creates it 0×0, all four compositor paths `continue` past it,
`FramePump`'s hash skips it, it is not draggable, and it has no text/font/colour properties or
persistence. Text content, font family/size/weight/style, colour, alignment, drag/resize,
persisted; renderer mirrors `Services/ChatBoxRenderer.cs`.
**Day 4 remainder:** TASK 49 (chat profanity filter) then TASK 34 (scheduled streams — its two research
flags must be cited first). Day 3's resilience row (TASK 32/33) is still outstanding and is a launch
blocker; the smallest slice is 32·4 (fragmented MP4 on the record block).
## Queue
@@ -43,8 +41,8 @@ persisted; renderer mirrors `Services/ChatBoxRenderer.cs`.
|---|---|---|---|
| 53 | ✅ | Output-format contract — six size/FPS modes, vertical YouTube playback, and recordings visually accepted | [`task-53`](TASKS/task-53-output-formats.md) |
| — | ✅ | Output entitlement code: free at 30 fps / paid at 60 fps regardless of resolution; Steam App ID is still pending | [`commercial`](docs/ai/commercial.md) |
| 3·17 | ☐ **NEXT** | **Text layer** — content, font family/size/weight/style, colour, alignment, drag/resize, persisted | [`task-03`](TASKS/task-03-capture-pipeline.md) |
| 21 | 🔶 | Media source: acquisition wiring + file picker + loop/volume UI (decoder, manager, pacing, loop all shipped — **no `AcquireAsync` caller exists**) | [`task-21`](TASKS/task-21-media-source.md) |
| 3·17 | ✅ | **Text layer** — content, font family/size/weight/style, colour, alignment, drag/resize, persisted (done 2026-11-04) | [`task-03`](TASKS/task-03-capture-pipeline.md) |
| 21 | ✅ | Media source: video file added from the (+) menu decodes, composites, loops, and stops with its last reference (done 2026-11-05) | [`task-21`](TASKS/task-21-media-source.md) |
| — | ☐ | Transitions **Move**: per-layer carry-across-scenes flag, per-scene placement, fixed z band | [`ui-surface`](docs/ai/ui-surface.md) |
| 55·2–3 | ☐ | Disk hygiene: delete `%TEMP%` alert clips on close + sweep at startup; low-disk guard before REC (slice 1 shipped) | [`task-55`](TASKS/task-55-disk-hygiene.md) |
| 40·A/C/D | ☐ | App Settings: active-camera picker; persist output resolution + surface transition defaults + confirm-before-End; accent consolidation | [`task-40`](TASKS/task-40-app-settings-round.md) |
+24 -38
View File
@@ -5,22 +5,33 @@
**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.
### Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) + slice 1 (manager + resolver/preview wire-in) + slice 2a/2b (ffprobe probe + native-FPS pacing) + slice 3 (loop mechanism) shipped. **Remaining: the UI picker slice (AddMedia command + file dialog + Acquire/Release wiring + loop-flag wiring) — GUI, handed off to build/verify natively on Windows (see HANDOFF → "UI picker slice — handoff spec").**
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
reference goes away. Audio is the desktop loopback (TRAX-style); a per-file volume route is not built.
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
2. ✅ `SourceType.MediaSource` addition to the enum
3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
4b. ✅ Slice 2a — native-FPS probe seam: `FfmpegFrameRateParser` (pure, prefers `avg_frame_rate=` then `r_frame_rate=`, rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` from the located ffmpeg dir, reuses the `IDecodeProcess` seam; null if ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive (conditional). Tests: `FfmpegFrameRateParserTests` (6 pure units + 1 probe integration via fake locator/process) + `FfmpegLocatorTests` still green — 15/15.
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass.
6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps` (fake probe + recording delay, one delay per frame ≈ 1ms).
7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)**
8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
9. ✅ Loop control (mechanism) — `IMediaFrameSource.Looping`; `MediaVideoSource` takes a `Func<IDecodeProcess>` process factory and restarts the decode on natural EOF when `Looping` (fresh process per pass, since a `Process` can't be re-`Start()`ed). Test: `MediaVideoSource_LoopsUntilLoopDisabled` (single frame re-emits across passes, `Completed` only after loop cleared). Wiring `Source.MediaIsLooping` into the flag lands with the UI-picker (acquisition) slice.
10. ☐ Volume control — per-source volume slider for audio playback
11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider
12. ☐ Schema migration for media source settings (file path, loop, volume)
13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`/`HasSession`/`GetActiveKeys`/`SetLooping`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`.
6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps`.
7. ✅ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame, and the same
resolver serves preview and thumbnails (`FramePump` defaults `resolver` to the frame resolver).
8. ✅ Wire into `SceneCompositor` — media renders through the generic `frameFor(element)` path at the
element's rect, so no compositor change was needed.
9. ✅ Loop control — `IMediaFrameSource.Looping`; `MediaVideoSource` restarts the decode on natural EOF
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
session loops if **any** live media reference has it on, because one decoder is shared per path.
10. ⏸ Volume control — deliberately not wired: media audio is captured by the desktop loopback like
TRAX, so a per-file slider would be a control with no audible effect. Revisit only with a local
audio route for the file's own stream.
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
("Change…" in the properties panel), loop toggle, video name display.
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
13. ✅ Tests: decode/loop/pacing at the unit level, and `MediaSourceAcquisitionTests` (the Good Dog for the
wiring): add a file → one decoder starts, frames reach the layer, loop flip reaches the decoder, remove → stops.
### Design decisions
@@ -28,31 +39,6 @@
- **Audio plays through the desktop channel** — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed.
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
### UI picker slice — spec (imported from HANDOFF 2026-09-01 so it can't be lost by a rewrite)
Nothing `AcquireAsync`s a media path yet, so **no frames flow in a running app** — decoder, manager,
pacing, and loop mechanism are shipped and unit-tested, but a media Source has no way to start a
session. Deliverables, in order:
1. **File picker + add/remove commands** (mirror `MainViewModel.Trax.cs:68` `OpenFileDialog` usage).
A "Media" source action opens an `OpenFileDialog` filtered to video (mp4/mov/avi); on OK set
`Source.MediaPath`, add to scene, and **`AcquireAsync(MediaPath)`**; on remove
**`ReleaseAsync(MediaPath)`** (refcounted — not `ReleaseAllAsync`, unless the path leaves the
layout entirely). On `StagedScene` layout loads/teardown, release every media path no longer
present and acquire new ones so sessions track the live layout.
2. **Wire `Source.MediaIsLooping` → `IMediaFrameSource.Looping`** — needs a manager-level per-**path**
loop provider. Ambiguity to rule + record: sessions are refcounted per `MediaPath` and shared
across scenes, but `MediaIsLooping` is per-`Source`. Chosen rule: the path's session `Looping` =
**any** live reference has it on (shared behaviour). Record it in the slice commit.
3. **Loop toggle + volume slider** in the source context menu / inspector (items 10/11).
4. `MediaVolume`/`MediaPlaybackState` wiring — audio path for media is future work (desktop loopback
already carries it, TRAX-style; `MediaVolume` slider can drive the local file's audio separately
later — out of this slice).
**Design notes to preserve:** `ResolveOutputFrame` reads `_mediaManager.GetLatestFrame(MediaPath)`;
`OnMediaPreviewBitmapChanged` adopts the shared `WriteableBitmap` per path; `MediaFailed` →
`OnMediaFailed` (currently `Debug.WriteLine` — surface in UI via toast, TASK 24 stack). The compositor
scales any frame size — media renders through the generic `frameFor(element)` path, no compositor
change. First GUI smoke test: add a short real mp4 on native Windows, confirm it plays and previews.
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
the ordinary (+) menu, with no per-scene availability rules. Which scenes may host a video layer is a
later pass, and scene-only layers (an ending credit roll, for instance) are defined after that pass.
+142 -4
View File
@@ -1,5 +1,9 @@
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Linq;
using System.Windows.Media.Imaging;
using Microsoft.Win32;
using ytLive.Models;
using ytLive.Services;
@@ -17,8 +21,15 @@ public partial class MainViewModel
private const int MediaDecodeWidth = 1920;
private const int MediaDecodeHeight = 1080;
private const string VideoFileFilter =
"Video files (*.mp4;*.mov;*.avi;*.mkv;*.webm)|*.mp4;*.mov;*.avi;*.mkv;*.webm|All files (*.*)|*.*";
private readonly MediaVideoSourceManager _mediaManager;
// One subscription per media Source, so removing a layer stops the listener as
// well as the decoder (a scene element outlives its layer list row).
private readonly Dictionary<string, PropertyChangedEventHandler> _mediaHandlers = new();
// MediaVideoSourceManager publishes one shared WriteableBitmap per file; every
// media Source pointing at that path picks it up for the preview/canvas.
private void OnMediaPreviewBitmapChanged(string path, WriteableBitmap bitmap)
@@ -28,8 +39,135 @@ public partial class MainViewModel
source.VideoImageSource = bitmap;
}
// Called from the encoder error path is out of scope (TASK 21 sees the decoder
// fail at startup); log to the debug console for now, surfaced via the UI later.
private static void OnMediaFailed(string path, string message)
=> System.Diagnostics.Debug.WriteLine($"MediaSource: '{path}' failed: {message}");
private void OnMediaFailed(string path, string message)
{
Helpers.AppLog.Write($"MediaSource: '{path}' failed: {message}");
_notifications.Warning("Couldn't play that video", System.IO.Path.GetFileName(path));
}
// ---------- add / change the file ----------
private void AddMedia()
{
if (PickVideoFile() is { } path)
AddMediaSource(path);
}
private string? PickVideoFile(string title = "Choose a video file")
{
var dialog = new OpenFileDialog
{
Title = title,
Filter = VideoFileFilter,
};
return dialog.ShowDialog() == true ? dialog.FileName : null;
}
/// <summary>Adds a MediaSource layer for <paramref name="path"/> to the staged scene.
/// Internal so the Good Dog test drives the real add path without a file dialog.</summary>
internal void AddMediaSource(string path)
{
if (string.IsNullOrWhiteSpace(path)) return;
AddSource(SourceType.MediaSource, path);
}
private void ChangeMediaFile(Source? source)
{
if (source is not { IsMediaSource: true }) return;
if (PickVideoFile("Replace the video") is not { } path) return;
PushPreviewUndo();
source.MediaPath = path;
}
// ---------- session lifecycle (one session per distinct path) ----------
/// <summary>Starts decoding the file if nothing else already is, and subscribes to the
/// layer so a file swap or a loop flip reaches the session.</summary>
private void RegisterMediaSource(Source source)
{
if (_mediaHandlers.ContainsKey(source.Id)) return;
string? lastPath = source.MediaPath;
void OnSourceChanged(object? _, PropertyChangedEventArgs e)
{
if (e.PropertyName == nameof(Source.MediaPath))
{
var previous = lastPath;
lastPath = source.MediaPath;
if (string.Equals(previous, source.MediaPath, StringComparison.OrdinalIgnoreCase)) return;
// The layer now points elsewhere, so the old file may be unreferenced —
// release it before acquiring the replacement.
ReleaseMediaIfUnused(previous);
EnsureMediaSession(source.MediaPath);
}
else if (e.PropertyName == nameof(Source.MediaIsLooping) && source.MediaPath is { } path)
{
_mediaManager.SetLooping(path, AnyMediaReferenceLoops(path));
}
}
_mediaHandlers[source.Id] = OnSourceChanged;
source.PropertyChanged += OnSourceChanged;
EnsureMediaSession(source.MediaPath);
}
private void UnregisterMediaSource(Source source)
{
if (_mediaHandlers.Remove(source.Id, out var handler))
source.PropertyChanged -= handler;
}
private void EnsureMediaSession(string? path)
{
if (string.IsNullOrWhiteSpace(path)) return;
if (_mediaManager.HasSession(path)) return;
_ = AcquireMediaAsync(path);
}
private async System.Threading.Tasks.Task AcquireMediaAsync(string path)
{
if (!await _mediaManager.AcquireAsync(path)) return;
// A session can be running before this layer existed (shared path), in which
// case the bitmap is already published and will not be raised again for us.
if (_mediaManager.GetPreviewBitmap(path) is { } bitmap)
OnMediaPreviewBitmapChanged(path, bitmap);
}
/// <summary>Shared sessions or not, the loop state is the path's: any live media
/// reference asking for a loop makes the one decoder loop (recorded rule, TASK 21).</summary>
private bool AnyMediaReferenceLoops(string path)
=> Scenes.SelectMany(s => s.Elements).OfType<Source>()
.Any(s => s.IsMediaSource && s.MediaPath == path && s.MediaIsLooping);
private bool HasMediaReference(string path)
=> Scenes.SelectMany(s => s.Elements).OfType<Source>()
.Any(s => s.IsMediaSource && s.MediaPath == path);
private void ReleaseMediaIfUnused(string? path)
{
if (string.IsNullOrWhiteSpace(path)) return;
if (!_mediaManager.HasSession(path)) return;
if (HasMediaReference(path)) return;
_ = _mediaManager.ReleaseAsync(path);
}
/// <summary>After a layout load: subscribe every persisted media layer and start one
/// session per distinct file; release sessions for files the layout no longer uses.</summary>
private void ReacquireMediaSources()
{
var sources = Scenes.SelectMany(s => s.Elements).OfType<Source>()
.Where(s => s.IsMediaSource && !string.IsNullOrWhiteSpace(s.MediaPath))
.ToList();
foreach (var source in sources)
RegisterMediaSource(source);
var wanted = sources.Select(s => s.MediaPath!).Distinct(StringComparer.OrdinalIgnoreCase).ToList();
foreach (var stale in _mediaManager.GetActiveKeys()
.Where(key => !wanted.Contains(key, StringComparer.OrdinalIgnoreCase))
.ToList())
_ = _mediaManager.ReleaseAllAsync(stale);
}
}
+26 -3
View File
@@ -15,14 +15,19 @@ namespace ytLive.ViewModels;
public partial class MainViewModel
{
private void AddSource(object? parameter)
{
var sourceType = parameter is SourceType typed
? typed
: Enum.TryParse<SourceType>(parameter?.ToString(), true, out var parsed) ? parsed : SourceType.Image;
AddSource(sourceType);
}
private void AddSource(SourceType sourceType, string? mediaPath = null)
{
var scene = StagedScene;
if (scene == null) return;
PushPreviewUndo();
var sourceType = parameter is SourceType typed
? typed
: Enum.TryParse<SourceType>(parameter?.ToString(), true, out var parsed) ? parsed : SourceType.Image;
// Privacy blur is a Live-scene tool: it exists to hide the creator's desktop, and every
// other scene's art is authored. Rejecting here (not only in the (+) menu) keeps a stale
// menu item or a scripted call from planting one on BRB.
@@ -40,6 +45,7 @@ public partial class MainViewModel
SourceType.AlertBox => "Stream Alerts",
SourceType.Ticker => "Ticker",
SourceType.Scoreboard => "Scoreboard",
SourceType.MediaSource => "Video",
_ => "Layer",
};
@@ -62,6 +68,7 @@ public partial class MainViewModel
? $"{baseName}-{webCount}"
: NextSourceName(scene, baseName),
Type = sourceType,
MediaPath = mediaPath,
X = sourceType == SourceType.TextOverlay ? 160
: sourceType == SourceType.ChatBox ? 1400
: sourceType == SourceType.AlertBox ? 620
@@ -79,6 +86,7 @@ public partial class MainViewModel
: sourceType == SourceType.AlertBox ? 680
: sourceType == SourceType.Scoreboard ? 1920
: sourceType == SourceType.Ticker ? 1920
: sourceType == SourceType.MediaSource ? 1920
: sourceType == SourceType.Blur ? 240 : 0,
Height = sourceType == SourceType.TextOverlay ? 140
: sourceType == SourceType.WebSource ? 480
@@ -86,6 +94,7 @@ public partial class MainViewModel
: sourceType == SourceType.AlertBox ? 200
: sourceType == SourceType.Scoreboard ? 56
: sourceType == SourceType.Ticker ? 48
: sourceType == SourceType.MediaSource ? 1080
: sourceType == SourceType.Blur ? 160 : 0,
};
@@ -104,6 +113,9 @@ public partial class MainViewModel
};
}
if (sourceType == SourceType.MediaSource)
RegisterMediaSource(source);
if (sourceType == SourceType.ChatBox)
{
source.PropertyChanged += (_, e) =>
@@ -286,7 +298,18 @@ public partial class MainViewModel
}
if (element is Source { Type: SourceType.WebSource } webSource)
_webView2Manager?.Unregister(webSource.Id);
// Media: the layer stops listening, and its decode session stops once no scene
// references the file any more (sessions are shared per path).
string? mediaPath = null;
if (element is Source mediaSource && mediaSource.IsMediaSource)
{
UnregisterMediaSource(mediaSource);
mediaPath = mediaSource.MediaPath;
}
_sceneGraph.RemoveElement(scene, element);
ReleaseMediaIfUnused(mediaPath);
if (element is WebcamSceneConfig)
{
+16 -3
View File
@@ -188,6 +188,8 @@ public partial class MainViewModel : ViewModelBase
public ICommand AddSceneCommand { get; }
public ICommand AddSourceCommand { get; }
public ICommand AddImageCommand { get; }
public ICommand AddMediaCommand { get; }
public ICommand ChangeMediaFileCommand { get; }
public ICommand RemoveSourceCommand { get; }
public ICommand EditElementCommand { get; }
public ICommand ToggleElementVisibilityCommand { get; }
@@ -260,6 +262,10 @@ public partial class MainViewModel : ViewModelBase
AddSourceCommand = new RelayCommand(parameter => AddSource(parameter));
AddWebcamCommand = new RelayCommand(_ => _ = AddWebcamToStagedSceneAsync());
AddImageCommand = new RelayCommand(_ => AddImage());
AddMediaCommand = new RelayCommand(_ => AddMedia());
ChangeMediaFileCommand = new RelayCommand(
element => ChangeMediaFile(element as Source),
element => element is Source { IsMediaSource: true });
RemoveSourceCommand = new RelayCommand(element => { PushPreviewUndo(); RemoveElement(element as SceneElement); });
EditElementCommand = new RelayCommand(element => BeginEditElement(element as SceneElement));
ToggleElementVisibilityCommand = new RelayCommand(element => { PushPreviewUndo(); ToggleElementVisibility(element as SceneElement); });
@@ -396,12 +402,14 @@ public partial class MainViewModel : ViewModelBase
_screenCaptureManager.CaptureFailed += OnScreenCaptureFailed;
_mediaManager = new MediaVideoSourceManager(
path => new MediaVideoSource(
MediaFrameSourceFactoryOverride ?? (path => new MediaVideoSource(
path, MediaDecodeWidth, MediaDecodeHeight,
new FfmpegLocator(), () => new FfmpegDecodeProcess(),
frameRateProbe: new FfmpegFrameRateProbe(
new FfmpegLocator(), () => new FfmpegDecodeProcess())),
System.Windows.Application.Current?.Dispatcher);
new FfmpegLocator(), () => new FfmpegDecodeProcess()))),
System.Windows.Application.Current?.Dispatcher,
// Loop intent is per Source; the manager needs the per-*path* answer.
AnyMediaReferenceLoops);
_mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged;
_mediaManager.MediaFailed += OnMediaFailed;
@@ -466,6 +474,10 @@ public partial class MainViewModel : ViewModelBase
/// real ONNX model. Mirrors <see cref="CameraFrameSourceFactoryOverride"/>.</summary>
internal static Services.BackgroundEffects.IPersonSegmenter? PersonSegmenterOverride { get; set; }
/// <summary>Test seam (TASK 21): a fake media decoder so a test can prove add/release
/// lifecycle without spawning ffmpeg. Mirrors <see cref="CameraFrameSourceFactoryOverride"/>.</summary>
internal static Func<string, IMediaFrameSource?>? MediaFrameSourceFactoryOverride { get; set; }
private void LoadLayout()
{
_isLoading = true;
@@ -518,6 +530,7 @@ public partial class MainViewModel : ViewModelBase
UpdateBackgroundImage();
ReacquireWebcam();
ReacquireScreenCaptures();
ReacquireMediaSources();
RegisterLoadedWebSources();
_socials = _layoutStore.Socials;
RenderSocialBarFrame();
+11
View File
@@ -790,6 +790,17 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
() => new FfmpegDecodeProcess())` as the probe.
Still open: session acquisition on add/remove (UI picker) + wiring `Source.MediaIsLooping` into
`IMediaFrameSource.Looping` (needs a manager-level per-path loop provider, comes with the picker slice).
**Acquisition shipped 2026-11-05:** `AddMediaCommand` (`MainViewModel.Media.cs`) opens a video-filtered
`OpenFileDialog` and `AddMediaSource(path)` adds a `MediaSource` layer through the ordinary `AddSource`
path, defaulting to a full-canvas 1920x1080 rect, then `RegisterMediaSource` subscribes the layer and
`EnsureMediaSession` starts one decoder per **distinct path** (`HasSession` guard keeps refcounts at 0/1).
Remove (`RemoveElement`) unsubscribes and releases only when no scene references the file any more;
`ReacquireMediaSources` runs after `LoadLayout` so persisted layers get sessions and stale files are
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
the layer no longer references it. Volume stays unwired on purpose — sound rides the desktop loopback
(TRAX-style), so a per-file slider would be a control with no effect.
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
+123
View File
@@ -0,0 +1,123 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Data.Sqlite;
using Xunit;
using ytLive.Models;
using ytLive.Services;
using ytLive.ViewModels;
namespace ytLive.Tests;
/// <summary>
/// TASK 21 Good Dog (2026-11-05): a MediaSource layer exists but nothing ever started
/// a decode session, so a video file layer rendered nothing in a running app. This
/// drives the real add path through <see cref="MainWindow"/> and asserts the piece the
/// unit adds — the session lifecycle: adding a video file starts exactly one decoder,
/// the decoded frames reach the layer, a loop flip reaches that decoder, and removing
/// the layer stops it.
/// </summary>
[Collection("RealApp")]
public sealed class MediaSourceAcquisitionTests
{
private readonly RealAppHost _app;
public MediaSourceAcquisitionTests(RealAppHost app) => _app = app;
[Fact]
public void AddingAVideoLayer_StartsTheDecoder_AndRemovingItStopsTheDecoder()
{
_app.Run(() =>
{
var tempDb = Path.Combine(Path.GetTempPath(), $"ytLlive-test-{Guid.NewGuid():N}.db");
var videoPath = Path.Combine(Path.GetTempPath(), "starting-soon.mp4");
var decoders = new List<FakeMediaFrameSource>();
MainViewModel.LayoutPathOverride = tempDb;
MainViewModel.MediaFrameSourceFactoryOverride = key =>
{
var decoder = new FakeMediaFrameSource(key);
decoders.Add(decoder);
return decoder;
};
var window = new MainWindow();
var vm = (MainViewModel)window.DataContext;
try
{
window.Show();
window.UpdateLayout();
var scene = vm.StagedScene!;
vm.AddMediaSource(videoPath);
var layer = scene.Elements.OfType<Source>().Single(s => s.IsMediaSource);
Assert.Equal(videoPath, layer.MediaPath);
Assert.Equal(SourceType.MediaSource, layer.Type);
Assert.Single(decoders);
var decoder = decoders[0];
Assert.Equal(videoPath, decoder.Key);
Assert.Equal(1, decoder.StartCount);
// A decoded frame reaches the layer the preview/renderer draws from —
// otherwise the session would be running and still invisible.
Assert.Null(layer.DisplaySource);
decoder.Emit(new VideoFrame(64, 36, new byte[64 * 36 * 4]));
Assert.NotNull(layer.DisplaySource);
// Loop intent is per Source; the shared session is the per-path answer.
Assert.False(decoder.Looping);
layer.MediaIsLooping = true;
Assert.True(decoder.Looping);
vm.RemoveSourceCommand.Execute(layer);
Assert.DoesNotContain(layer, scene.Elements);
Assert.Equal(1, decoder.StopCount);
}
finally
{
window.Close();
MainViewModel.MediaFrameSourceFactoryOverride = null;
MainViewModel.LayoutPathOverride = null;
SqliteConnection.ClearAllPools();
try { File.Delete(tempDb); } catch { /* best-effort cleanup */ }
}
});
}
/// <summary>Stands in for the ffmpeg decode process (the manager's factory seam).</summary>
private sealed class FakeMediaFrameSource : IMediaFrameSource
{
public FakeMediaFrameSource(string key) => Key = key;
public string Key { get; }
public bool Looping { get; set; }
public int StartCount { get; private set; }
public int StopCount { get; private set; }
public event Action<VideoFrame>? FrameAvailable;
public event Action? Completed { add { } remove { } }
public Task StartAsync(CancellationToken cancellationToken = default)
{
StartCount++;
return Task.CompletedTask;
}
public Task StopAsync()
{
StopCount++;
return Task.CompletedTask;
}
public void Emit(VideoFrame frame) => FrameAvailable?.Invoke(frame);
public void Dispose()
{
}
}
}