gramps/LlamaCasty
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
db563c86449bb63f06946c5954ae0ba2dec41bcb
LlamaCasty/TASKS/task-21-media-source.md
T
gramps 77d7208bfa feat(media): add a video file layer that decodes into the scene 
TASK 21's picker slice: the decoder, manager, pacing and loop mechanism were
already shipped but nothing ever acquired a session, so a media Source rendered
nothing. Adds "Video File" to the (+) menu (video-filtered OpenFileDialog),
one decode session per distinct path with a HasSession guard, release when the
last scene reference goes, and a post-load reconcile. Loop is a manager-level
per-path answer ("any live reference loops") applied before StartAsync and
updated live via SetLooping. Properties panel shows the file with Change… and
Loop. Volume stays unwired: media audio rides the desktop loopback, TRAX-style.

Reference: OBS's media source (file → decode → composite into the scene) is the
established answer for this feature; the lifecycle shape mirrors this repo's
WebView2/ScreenCapture registration rather than inventing a new one.

Test: ytLive.Tests/MediaSourceAcquisitionTests.cs — add starts one decoder and
frames reach the layer, loop flip reaches the decoder, remove stops it.

Docs: task-21 closed, TASKS row + Day-4 row + stale 3·17 row corrected,
frame-pipeline acquisition note, HANDOFF rewritten. verify.sh green.
2026-11-05 07:17:43 -08:00

5.1 KiB
Raw Blame History

TASK 21 — Media source (video file playback)

Catalog: TASKS.md — status and requirements live here.

Goal: play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.

Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the

preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene reference goes away. Audio is the desktop loopback (TRAX-style); a per-file volume route is not built.

  1. ✅ MediaSourceType enum: Video, Audio (audio-only files via media source)
  2. ✅ SourceType.MediaSource addition to the enum
  3. ✅ MediaSourceModel: MediaPath, MediaIsLooping, MediaVolume (0-1), MediaPlaybackState — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
  4. ✅ MediaVideoSource implements IMediaFrameSource: FFmpeg raw video decoder → VideoFrame pipeline — spawns ffmpeg -f rawvideo -pix_fmt bgra, drains the pipe via pure RawVideoFrameReader (Services/RawVideoFrameReader.cs), raises FrameAvailable/Completed; lifecycle is StartAsync/StopAsync; decode process behind IDecodeProcess/FfmpegDecodeProcess seam (binary-stdout mirror of IEncoderProcess). Shipped 2026-08-31; refactored onto IMediaFrameSource with this slice. 4b. ✅ Slice 2a — native-FPS probe seam: FfmpegFrameRateParser (pure, prefers avg_frame_rate= then r_frame_rate=, rational N/N/M, unknown→null) + IFrameRateProbe/FfmpegFrameRateProbe (derives sibling ffprobe.exe from the located ffmpeg dir, reuses the IDecodeProcess seam; null if ffprobe absent) + FfmpegLocator.ProbeFileName now also extracts ffprobe.exe from the pinned archive (conditional). Tests: FfmpegFrameRateParserTests (6 pure units + 1 probe integration via fake locator/process) + FfmpegLocatorTests still green — 15/15.
  5. ✅ IMediaFrameSource + MediaVideoSourceManager (Services/IMediaFrameSource.cs, Services/MediaVideoSourceManager.cs): app-wide decode-session ownership refcounted by MediaPath with a Func<string, IMediaFrameSource?> factory seam; AcquireAsync/ReleaseAsync/ReleaseAllAsync/GetLatestFrame/HasSession/GetActiveKeys/SetLooping; coalesces frames onto the UI dispatcher onto a single shared WriteableBitmap per file (mirror of ScreenCaptureManager); MediaFailed + PreviewBitmapChanged. Unit tests + one integration test (bitmap share/coalesce) — MediaVideoSourceManagerTests.
  6. ✅ Native-FPS pacing (slice 2b): MediaVideoSource takes optional IFrameRateProbe? + Func<TimeSpan,CancellationToken,Task>? delay seams, probes FPS once in RunAsync, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: MediaVideoSource_PacesFramesByProbedFps.
  7. ✅ Wire into FramePump resolver — Source { Type: MediaSource } → latest video frame, and the same resolver serves preview and thumbnails (FramePump defaults resolver to the frame resolver).
  8. ✅ Wire into SceneCompositor — media renders through the generic frameFor(element) path at the element's rect, so no compositor change was needed.
  9. ✅ Loop control — IMediaFrameSource.Looping; MediaVideoSource restarts the decode on natural EOF when Looping (fresh process per pass). MediaIsLooping reaches the session through the manager's per-path loopProvider at start and SetLooping on a live toggle. Rule (recorded): the path's session loops if any live media reference has it on, because one decoder is shared per path.
  10. ⏸ Volume control — deliberately not wired: media audio is captured by the desktop loopback like TRAX, so a per-file slider would be a control with no audible effect. Revisit only with a local audio route for the file's own stream.
  11. ✅ UI: AddMediaCommand (+ menu → "Video File", video-filtered OpenFileDialog), ChangeMediaFileCommand ("Change…" in the properties panel), loop toggle, video name display.
  12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
  13. ✅ Tests: decode/loop/pacing at the unit level, and MediaSourceAcquisitionTests (the Good Dog for the wiring): add a file → one decoder starts, frames reach the layer, loop flip reaches the decoder, remove → stops.

Design decisions

  • FFmpeg handles all formats — no codec-specific code. FFmpeg already in the project.
  • Audio plays through the desktop channel — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed.
  • This pairs with TRAX — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
  • Not a full NLE — no trimming, no multi-track, no effects. Just "play this video in the scene."
  • Layers are defined in Live first (creator 2026-11-05): this unit adds the layer and the picker in the ordinary (+) menu, with no per-scene availability rules. Which scenes may host a video layer is a later pass, and scene-only layers (an ending credit roll, for instance) are defined after that pass.
Reference in New Issue View Git Blame Copy Permalink