53 lines
5.8 KiB
Markdown
53 lines
5.8 KiB
Markdown
# TASK 21 — Media source (video file playback)
|
|
|
|
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
|
|
|
|
|
**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.
|
|
|
|
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
|
|
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
|
|
reference goes away. **Picture only: the file's own audio is not played anywhere** (the decode pipe
|
|
passes `-an`), so a video with a soundtrack is silent on stream — the audio route is open, item 10.
|
|
|
|
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
|
|
2. ✅ `SourceType.MediaSource` addition to the enum
|
|
3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
|
|
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
|
|
4b. ✅ Slice 2a — native-FPS probe seam: `FfmpegFrameRateParser` (pure, prefers `avg_frame_rate=` then `r_frame_rate=`, rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` from the located ffmpeg dir, reuses the `IDecodeProcess` seam; null if ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive (conditional). Tests: `FfmpegFrameRateParserTests` (6 pure units + 1 probe integration via fake locator/process) + `FfmpegLocatorTests` still green — 15/15.
|
|
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`/`HasSession`/`GetActiveKeys`/`SetLooping`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`.
|
|
6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps`.
|
|
7. ✅ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame, and the same
|
|
resolver serves preview and thumbnails (`FramePump` defaults `resolver` to the frame resolver).
|
|
8. ✅ Wire into `SceneCompositor` — media renders through the generic `frameFor(element)` path at the
|
|
element's rect, so no compositor change was needed.
|
|
9. ✅ Loop control — `IMediaFrameSource.Looping`; `MediaVideoSource` restarts the decode on natural EOF
|
|
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
|
|
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
|
|
session loops if **any** live media reference has it on, because one decoder is shared per path.
|
|
10. ⏸ File audio + volume — **the file's soundtrack is silent everywhere today**: the decode pipe
|
|
passes `-an`, so nothing plays it and the desktop loopback has nothing to carry. Two established
|
|
routes when this is taken: **alert-style** (a second `f32le` pipe into the mixer —
|
|
`AlertClipDecoder.RunAudioAsync` is the working model, independent of desktop capture) or
|
|
**TRAX-style** (`MusicPlayer` plays it to the default device, so the loopback, the desktop fader
|
|
and ducking all apply — at the cost of a second clock to hold in sync with the video). Product
|
|
call; not started, and no volume slider until one of them exists.
|
|
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
|
|
("Change…" in the properties panel), loop toggle, video name display.
|
|
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
|
|
13. ✅ Tests: decode/loop/pacing at the unit level, and `MediaSourceAcquisitionTests` (the Good Dog for the
|
|
wiring): add a file → one decoder starts, frames reach the layer, loop flip reaches the decoder, remove → stops.
|
|
|
|
### Design decisions
|
|
|
|
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
|
|
- **Audio is NOT wired, and it is not hiding behind the loopback** — `MediaVideoSource` decodes with
|
|
`-an`, so a video file's soundtrack is silent on stream today (the loopback only carries what the
|
|
machine actually plays). Item 10 records the two established routes; do not describe media as
|
|
audible until one is built.
|
|
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
|
|
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
|
|
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
|
|
the ordinary (+) menu, with no per-scene availability rules. Which scenes may host a video layer is a
|
|
later pass, and scene-only layers (an ending credit roll, for instance) are defined after that pass.
|