LlamaCasty/TASKS/task-21-media-source.md

TASK 21 — Media source (video file playback)

Catalog: TASKS.md — status and requirements live here.

Goal: play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.

Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the

preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene reference goes away, and the file's own soundtrack plays into the stream (item 10).

1. ✅ MediaSourceType enum: Video, Audio (audio-only files via media source)
2. ✅ SourceType.MediaSource addition to the enum
3. ✅ MediaSourceModel: MediaPath, MediaIsLooping, MediaVolume (0-1), MediaPlaybackState — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
4. ✅ MediaVideoSource implements IMediaFrameSource: FFmpeg raw video decoder → VideoFrame pipeline — spawns ffmpeg -f rawvideo -pix_fmt bgra, drains the pipe via pure RawVideoFrameReader (Services/RawVideoFrameReader.cs), raises FrameAvailable/Completed; lifecycle is StartAsync/StopAsync; decode process behind IDecodeProcess/FfmpegDecodeProcess seam (binary-stdout mirror of IEncoderProcess). Shipped 2026-08-31; refactored onto IMediaFrameSource with this slice. 4b. ✅ Slice 2a — native-FPS probe seam: FfmpegFrameRateParser (pure, prefers avg_frame_rate= then r_frame_rate=, rational N/N/M, unknown→null) + IFrameRateProbe/FfmpegFrameRateProbe (derives sibling ffprobe.exe from the located ffmpeg dir, reuses the IDecodeProcess seam; null if ffprobe absent) + FfmpegLocator.ProbeFileName now also extracts ffprobe.exe from the pinned archive (conditional). Tests: FfmpegFrameRateParserTests (6 pure units + 1 probe integration via fake locator/process) + FfmpegLocatorTests still green — 15/15.
5. ✅ IMediaFrameSource + MediaVideoSourceManager (Services/IMediaFrameSource.cs, Services/MediaVideoSourceManager.cs): app-wide decode-session ownership refcounted by MediaPath with a Func<string, IMediaFrameSource?> factory seam; AcquireAsync/ReleaseAsync/ReleaseAllAsync/GetLatestFrame/HasSession/GetActiveKeys/SetLooping; coalesces frames onto the UI dispatcher onto a single shared WriteableBitmap per file (mirror of ScreenCaptureManager); MediaFailed + PreviewBitmapChanged. Unit tests + one integration test (bitmap share/coalesce) — MediaVideoSourceManagerTests.
6. ✅ Native-FPS pacing (slice 2b): MediaVideoSource takes optional IFrameRateProbe? + Func<TimeSpan,CancellationToken,Task>? delay seams, probes FPS once in RunAsync, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: MediaVideoSource_PacesFramesByProbedFps.
7. ✅ Wire into FramePump resolver — Source { Type: MediaSource } → latest video frame, and the same resolver serves preview and thumbnails (FramePump defaults resolver to the frame resolver).
8. ✅ Wire into SceneCompositor — media renders through the generic frameFor(element) path at the element's rect, so no compositor change was needed.
9. ✅ Loop control — IMediaFrameSource.Looping; MediaVideoSource restarts the decode on natural EOF when Looping (fresh process per pass). MediaIsLooping reaches the session through the manager's per-path loopProvider at start and SetLooping on a live toggle. Rule (recorded): the path's session loops if any live media reference has it on, because one decoder is shared per path.
10. ✅ File audio + volume — the shared clip pipe decodes the soundtrack (f32le -ac 2 -ar 48000 -vn, 50 ms chunks paced to real time, byte-carry across reads) and the layer carries Mute + Level. The manager applies them per session (one decoder per path, so the last layer touched wins — the alternative would play the same file twice) and the audio joins the DESKTOP/GAME group in AudioMixer.EnqueueDesktopMediaAudio, so the desktop fader and the voice ducker behave exactly as they do for game audio and TRAX. Test: MediaSourceAudioTests (chunks, level scaling, mute, and the layer's sound reaching the desktop meter).
11. ✅ UI: AddMediaCommand (+ menu → "Video File", video-filtered OpenFileDialog), ChangeMediaFileCommand ("Change…" in the properties panel), loop toggle, video name display.
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
13. ✅ Tests: decode/loop/pacing at the unit level, and MediaSourceAcquisitionTests (the Good Dog for the wiring): add a file → one decoder starts, frames reach the layer, loop flip reaches the decoder, remove → stops.

Design decisions

• FFmpeg handles all formats — no codec-specific code. FFmpeg already in the project.
• Audio joins the desktop/game group (creator ruling 2026-11-05) — the clip's sound is decoded by the same pipe as its picture, mixed directly into the stream (not played locally and captured), and grouped with game audio so one fader and the ducker cover it. A per-layer Mute/Level sits on top. Chosen over TRAX-style local playback because it works regardless of the creator's desktop-capture state, and because an alert clip needs the same direct route anyway.
• This pairs with TRAX — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
• Not a full NLE — no trimming, no multi-track, no effects. Just "play this video in the scene."
• Layers are defined in Live first (creator 2026-11-05): this unit adds the layer and the picker in the ordinary (+) menu, with no per-scene availability rules. Which scenes may host a video layer is a later pass, and scene-only layers (an ending credit roll, for instance) are defined after that pass.