feat(media): add a video file layer that decodes into the scene

TASK 21's picker slice: the decoder, manager, pacing and loop mechanism were
already shipped but nothing ever acquired a session, so a media Source rendered
nothing. Adds "Video File" to the (+) menu (video-filtered OpenFileDialog),
one decode session per distinct path with a HasSession guard, release when the
last scene reference goes, and a post-load reconcile. Loop is a manager-level
per-path answer ("any live reference loops") applied before StartAsync and
updated live via SetLooping. Properties panel shows the file with Change… and
Loop. Volume stays unwired: media audio rides the desktop loopback, TRAX-style.

Reference: OBS's media source (file → decode → composite into the scene) is the
established answer for this feature; the lifecycle shape mirrors this repo's
WebView2/ScreenCapture registration rather than inventing a new one.

Test: ytLive.Tests/MediaSourceAcquisitionTests.cs — add starts one decoder and
frames reach the layer, loop flip reaches the decoder, remove stops it.

Docs: task-21 closed, TASKS row + Day-4 row + stale 3·17 row corrected,
frame-pipeline acquisition note, HANDOFF rewritten. verify.sh green.
This commit is contained in:
2026-11-05 07:17:43 -08:00
parent 785b5ff8f3
commit 77d7208bfa
11 changed files with 429 additions and 68 deletions
+24 -38
View File
@@ -5,22 +5,33 @@
**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.
### Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) + slice 1 (manager + resolver/preview wire-in) + slice 2a/2b (ffprobe probe + native-FPS pacing) + slice 3 (loop mechanism) shipped. **Remaining: the UI picker slice (AddMedia command + file dialog + Acquire/Release wiring + loop-flag wiring) — GUI, handed off to build/verify natively on Windows (see HANDOFF → "UI picker slice — handoff spec").**
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
reference goes away. Audio is the desktop loopback (TRAX-style); a per-file volume route is not built.
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
2. ✅ `SourceType.MediaSource` addition to the enum
3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
4b. ✅ Slice 2a — native-FPS probe seam: `FfmpegFrameRateParser` (pure, prefers `avg_frame_rate=` then `r_frame_rate=`, rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` from the located ffmpeg dir, reuses the `IDecodeProcess` seam; null if ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive (conditional). Tests: `FfmpegFrameRateParserTests` (6 pure units + 1 probe integration via fake locator/process) + `FfmpegLocatorTests` still green — 15/15.
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass.
6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps` (fake probe + recording delay, one delay per frame ≈ 1ms).
7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)**
8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
9. ✅ Loop control (mechanism) — `IMediaFrameSource.Looping`; `MediaVideoSource` takes a `Func<IDecodeProcess>` process factory and restarts the decode on natural EOF when `Looping` (fresh process per pass, since a `Process` can't be re-`Start()`ed). Test: `MediaVideoSource_LoopsUntilLoopDisabled` (single frame re-emits across passes, `Completed` only after loop cleared). Wiring `Source.MediaIsLooping` into the flag lands with the UI-picker (acquisition) slice.
10. ☐ Volume control — per-source volume slider for audio playback
11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider
12. ☐ Schema migration for media source settings (file path, loop, volume)
13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`/`HasSession`/`GetActiveKeys`/`SetLooping`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`.
6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps`.
7. ✅ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame, and the same
resolver serves preview and thumbnails (`FramePump` defaults `resolver` to the frame resolver).
8. ✅ Wire into `SceneCompositor` — media renders through the generic `frameFor(element)` path at the
element's rect, so no compositor change was needed.
9. ✅ Loop control — `IMediaFrameSource.Looping`; `MediaVideoSource` restarts the decode on natural EOF
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
session loops if **any** live media reference has it on, because one decoder is shared per path.
10. ⏸ Volume control — deliberately not wired: media audio is captured by the desktop loopback like
TRAX, so a per-file slider would be a control with no audible effect. Revisit only with a local
audio route for the file's own stream.
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
("Change…" in the properties panel), loop toggle, video name display.
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
13. ✅ Tests: decode/loop/pacing at the unit level, and `MediaSourceAcquisitionTests` (the Good Dog for the
wiring): add a file → one decoder starts, frames reach the layer, loop flip reaches the decoder, remove → stops.
### Design decisions
@@ -28,31 +39,6 @@
- **Audio plays through the desktop channel** — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed.
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
### UI picker slice — spec (imported from HANDOFF 2026-09-01 so it can't be lost by a rewrite)
Nothing `AcquireAsync`s a media path yet, so **no frames flow in a running app** — decoder, manager,
pacing, and loop mechanism are shipped and unit-tested, but a media Source has no way to start a
session. Deliverables, in order:
1. **File picker + add/remove commands** (mirror `MainViewModel.Trax.cs:68` `OpenFileDialog` usage).
A "Media" source action opens an `OpenFileDialog` filtered to video (mp4/mov/avi); on OK set
`Source.MediaPath`, add to scene, and **`AcquireAsync(MediaPath)`**; on remove
**`ReleaseAsync(MediaPath)`** (refcounted — not `ReleaseAllAsync`, unless the path leaves the
layout entirely). On `StagedScene` layout loads/teardown, release every media path no longer
present and acquire new ones so sessions track the live layout.
2. **Wire `Source.MediaIsLooping` → `IMediaFrameSource.Looping`** — needs a manager-level per-**path**
loop provider. Ambiguity to rule + record: sessions are refcounted per `MediaPath` and shared
across scenes, but `MediaIsLooping` is per-`Source`. Chosen rule: the path's session `Looping` =
**any** live reference has it on (shared behaviour). Record it in the slice commit.
3. **Loop toggle + volume slider** in the source context menu / inspector (items 10/11).
4. `MediaVolume`/`MediaPlaybackState` wiring — audio path for media is future work (desktop loopback
already carries it, TRAX-style; `MediaVolume` slider can drive the local file's audio separately
later — out of this slice).
**Design notes to preserve:** `ResolveOutputFrame` reads `_mediaManager.GetLatestFrame(MediaPath)`;
`OnMediaPreviewBitmapChanged` adopts the shared `WriteableBitmap` per path; `MediaFailed` →
`OnMediaFailed` (currently `Debug.WriteLine` — surface in UI via toast, TASK 24 stack). The compositor
scales any frame size — media renders through the generic `frameFor(element)` path, no compositor
change. First GUI smoke test: add a short real mp4 on native Windows, confirm it plays and previews.
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
the ordinary (+) menu, with no per-scene availability rules. Which scenes may host a video layer is a
later pass, and scene-only layers (an ending credit roll, for instance) are defined after that pass.