LlamaCasty/TASKS/task-21-media-source.md

TASK 21 — Media source (video file playback)

Catalog: TASKS.md — status and requirements live here.

Goal: play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.

Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the

preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene reference goes away. Picture only: the file's own audio is not played anywhere (the decode pipe passes -an), so a video with a soundtrack is silent on stream — the audio route is open, item 10.

1. ✅ MediaSourceType enum: Video, Audio (audio-only files via media source)
2. ✅ SourceType.MediaSource addition to the enum
3. ✅ MediaSourceModel: MediaPath, MediaIsLooping, MediaVolume (0-1), MediaPlaybackState — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
4. ✅ MediaVideoSource implements IMediaFrameSource: FFmpeg raw video decoder → VideoFrame pipeline — spawns ffmpeg -f rawvideo -pix_fmt bgra, drains the pipe via pure RawVideoFrameReader (Services/RawVideoFrameReader.cs), raises FrameAvailable/Completed; lifecycle is StartAsync/StopAsync; decode process behind IDecodeProcess/FfmpegDecodeProcess seam (binary-stdout mirror of IEncoderProcess). Shipped 2026-08-31; refactored onto IMediaFrameSource with this slice. 4b. ✅ Slice 2a — native-FPS probe seam: FfmpegFrameRateParser (pure, prefers avg_frame_rate= then r_frame_rate=, rational N/N/M, unknown→null) + IFrameRateProbe/FfmpegFrameRateProbe (derives sibling ffprobe.exe from the located ffmpeg dir, reuses the IDecodeProcess seam; null if ffprobe absent) + FfmpegLocator.ProbeFileName now also extracts ffprobe.exe from the pinned archive (conditional). Tests: FfmpegFrameRateParserTests (6 pure units + 1 probe integration via fake locator/process) + FfmpegLocatorTests still green — 15/15.
5. ✅ IMediaFrameSource + MediaVideoSourceManager (Services/IMediaFrameSource.cs, Services/MediaVideoSourceManager.cs): app-wide decode-session ownership refcounted by MediaPath with a Func<string, IMediaFrameSource?> factory seam; AcquireAsync/ReleaseAsync/ReleaseAllAsync/GetLatestFrame/HasSession/GetActiveKeys/SetLooping; coalesces frames onto the UI dispatcher onto a single shared WriteableBitmap per file (mirror of ScreenCaptureManager); MediaFailed + PreviewBitmapChanged. Unit tests + one integration test (bitmap share/coalesce) — MediaVideoSourceManagerTests.
6. ✅ Native-FPS pacing (slice 2b): MediaVideoSource takes optional IFrameRateProbe? + Func<TimeSpan,CancellationToken,Task>? delay seams, probes FPS once in RunAsync, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: MediaVideoSource_PacesFramesByProbedFps.
7. ✅ Wire into FramePump resolver — Source { Type: MediaSource } → latest video frame, and the same resolver serves preview and thumbnails (FramePump defaults resolver to the frame resolver).
8. ✅ Wire into SceneCompositor — media renders through the generic frameFor(element) path at the element's rect, so no compositor change was needed.
9. ✅ Loop control — IMediaFrameSource.Looping; MediaVideoSource restarts the decode on natural EOF when Looping (fresh process per pass). MediaIsLooping reaches the session through the manager's per-path loopProvider at start and SetLooping on a live toggle. Rule (recorded): the path's session loops if any live media reference has it on, because one decoder is shared per path.
10. ⏸ File audio + volume — the file's soundtrack is silent everywhere today: the decode pipe passes -an, so nothing plays it and the desktop loopback has nothing to carry. Two established routes when this is taken: alert-style (a second f32le pipe into the mixer — AlertClipDecoder.RunAudioAsync is the working model, independent of desktop capture) or TRAX-style (MusicPlayer plays it to the default device, so the loopback, the desktop fader and ducking all apply — at the cost of a second clock to hold in sync with the video). Product call; not started, and no volume slider until one of them exists.
11. ✅ UI: AddMediaCommand (+ menu → "Video File", video-filtered OpenFileDialog), ChangeMediaFileCommand ("Change…" in the properties panel), loop toggle, video name display.
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
13. ✅ Tests: decode/loop/pacing at the unit level, and MediaSourceAcquisitionTests (the Good Dog for the wiring): add a file → one decoder starts, frames reach the layer, loop flip reaches the decoder, remove → stops.

Design decisions

• FFmpeg handles all formats — no codec-specific code. FFmpeg already in the project.
• Audio is NOT wired, and it is not hiding behind the loopback — MediaVideoSource decodes with -an, so a video file's soundtrack is silent on stream today (the loopback only carries what the machine actually plays). Item 10 records the two established routes; do not describe media as audible until one is built.
• This pairs with TRAX — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
• Not a full NLE — no trimming, no multi-track, no effects. Just "play this video in the scene."
• Layers are defined in Live first (creator 2026-11-05): this unit adds the layer and the picker in the ordinary (+) menu, with no per-scene availability rules. Which scenes may host a video layer is a later pass, and scene-only layers (an ending credit roll, for instance) are defined after that pass.