LlamaCasty/TASKS/task-21-media-source.md

TASK 21 — Media source (video file playback)

Catalog: TASKS.md — status and requirements live here.

Goal: play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.

Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) + slice 1 (manager + resolver/preview wire-in) + slice 2a/2b (ffprobe probe + native-FPS pacing) + slice 3 (loop mechanism) shipped. Remaining: the UI picker slice (AddMedia command + file dialog + Acquire/Release wiring + loop-flag wiring) — GUI, handed off to build/verify natively on Windows (see HANDOFF → "UI picker slice — handoff spec").

1. ✅ MediaSourceType enum: Video, Audio (audio-only files via media source)
2. ✅ SourceType.MediaSource addition to the enum
3. ✅ MediaSourceModel: MediaPath, MediaIsLooping, MediaVolume (0-1), MediaPlaybackState — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
4. ✅ MediaVideoSource implements IMediaFrameSource: FFmpeg raw video decoder → VideoFrame pipeline — spawns ffmpeg -f rawvideo -pix_fmt bgra, drains the pipe via pure RawVideoFrameReader (Services/RawVideoFrameReader.cs), raises FrameAvailable/Completed; lifecycle is StartAsync/StopAsync; decode process behind IDecodeProcess/FfmpegDecodeProcess seam (binary-stdout mirror of IEncoderProcess). Shipped 2026-08-31; refactored onto IMediaFrameSource with this slice. 4b. ✅ Slice 2a — native-FPS probe seam: FfmpegFrameRateParser (pure, prefers avg_frame_rate= then r_frame_rate=, rational N/N/M, unknown→null) + IFrameRateProbe/FfmpegFrameRateProbe (derives sibling ffprobe.exe from the located ffmpeg dir, reuses the IDecodeProcess seam; null if ffprobe absent) + FfmpegLocator.ProbeFileName now also extracts ffprobe.exe from the pinned archive (conditional). Tests: FfmpegFrameRateParserTests (6 pure units + 1 probe integration via fake locator/process) + FfmpegLocatorTests still green — 15/15.
5. ✅ IMediaFrameSource + MediaVideoSourceManager (Services/IMediaFrameSource.cs, Services/MediaVideoSourceManager.cs): app-wide decode-session ownership refcounted by MediaPath with a Func<string, IMediaFrameSource?> factory seam; AcquireAsync/ReleaseAsync/ReleaseAllAsync/GetLatestFrame; coalesces frames onto the UI dispatcher onto a single shared WriteableBitmap per file (mirror of ScreenCaptureManager); MediaFailed + PreviewBitmapChanged. Unit tests + one integration test (bitmap share/coalesce) — MediaVideoSourceManagerTests, 5/5 pass.
6. ✅ Native-FPS pacing (slice 2b): MediaVideoSource takes optional IFrameRateProbe? + Func<TimeSpan,CancellationToken,Task>? delay seams, probes FPS once in RunAsync, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: MediaVideoSource_PacesFramesByProbedFps (fake probe + recording delay, one delay per frame ≈ 1ms).
7. ☐ Wire into FramePump resolver — Source { Type: MediaSource } → latest video frame — resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)
8. ☐ Wire into SceneCompositor — render media source as an image element at its position/size
9. ✅ Loop control (mechanism) — IMediaFrameSource.Looping; MediaVideoSource takes a Func<IDecodeProcess> process factory and restarts the decode on natural EOF when Looping (fresh process per pass, since a Process can't be re-Start()ed). Test: MediaVideoSource_LoopsUntilLoopDisabled (single frame re-emits across passes, Completed only after loop cleared). Wiring Source.MediaIsLooping into the flag lands with the UI-picker (acquisition) slice.
10. ☐ Volume control — per-source volume slider for audio playback
11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider
12. ☐ Schema migration for media source settings (file path, loop, volume)
13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation

Design decisions

• FFmpeg handles all formats — no codec-specific code. FFmpeg already in the project.
• Audio plays through the desktop channel — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed.
• This pairs with TRAX — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
• Not a full NLE — no trimming, no multi-track, no effects. Just "play this video in the scene."

UI picker slice — spec (imported from HANDOFF 2026-09-01 so it can't be lost by a rewrite)

Nothing AcquireAsyncs a media path yet, so no frames flow in a running app — decoder, manager, pacing, and loop mechanism are shipped and unit-tested, but a media Source has no way to start a session. Deliverables, in order:

1. File picker + add/remove commands (mirror MainViewModel.Trax.cs:68 OpenFileDialog usage). A "Media" source action opens an OpenFileDialog filtered to video (mp4/mov/avi); on OK set Source.MediaPath, add to scene, and AcquireAsync(MediaPath); on remove ReleaseAsync(MediaPath) (refcounted — not ReleaseAllAsync, unless the path leaves the layout entirely). On StagedScene layout loads/teardown, release every media path no longer present and acquire new ones so sessions track the live layout.
2. Wire Source.MediaIsLooping → IMediaFrameSource.Looping — needs a manager-level per-path loop provider. Ambiguity to rule + record: sessions are refcounted per MediaPath and shared across scenes, but MediaIsLooping is per-Source. Chosen rule: the path's session Looping = any live reference has it on (shared behaviour). Record it in the slice commit.
3. Loop toggle + volume slider in the source context menu / inspector (items 10/11).
4. MediaVolume/MediaPlaybackState wiring — audio path for media is future work (desktop loopback already carries it, TRAX-style; MediaVolume slider can drive the local file's audio separately later — out of this slice).

Design notes to preserve: ResolveOutputFrame reads _mediaManager.GetLatestFrame(MediaPath); OnMediaPreviewBitmapChanged adopts the shared WriteableBitmap per path; MediaFailed → OnMediaFailed (currently Debug.WriteLine — surface in UI via toast, TASK 24 stack). The compositor scales any frame size — media renders through the generic frameFor(element) path, no compositor change. First GUI smoke test: add a short real mp4 on native Windows, confirm it plays and previews.