gramps
87509bcf99
TASKS.md is now the index (status table, open items, research pointer). 33 files: 32 task files + 1 research facts file. The full take-saga narrative and all design decisions are preserved verbatim; the catalog makes the queue readable without opening every task body. Schema and AGENTS.md updated to reflect the new layout.
59 lines
6.6 KiB
Markdown
59 lines
6.6 KiB
Markdown
# TASK 21 — Media source (video file playback)
|
|
|
|
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
|
|
|
|
|
**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips.
|
|
|
|
### Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) + slice 1 (manager + resolver/preview wire-in) + slice 2a/2b (ffprobe probe + native-FPS pacing) + slice 3 (loop mechanism) shipped. **Remaining: the UI picker slice (AddMedia command + file dialog + Acquire/Release wiring + loop-flag wiring) — GUI, handed off to build/verify natively on Windows (see HANDOFF → "UI picker slice — handoff spec").**
|
|
|
|
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
|
|
2. ✅ `SourceType.MediaSource` addition to the enum
|
|
3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test)
|
|
4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice.
|
|
4b. ✅ Slice 2a — native-FPS probe seam: `FfmpegFrameRateParser` (pure, prefers `avg_frame_rate=` then `r_frame_rate=`, rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` from the located ffmpeg dir, reuses the `IDecodeProcess` seam; null if ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive (conditional). Tests: `FfmpegFrameRateParserTests` (6 pure units + 1 probe integration via fake locator/process) + `FfmpegLocatorTests` still green — 15/15.
|
|
5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func<string, IMediaFrameSource?>` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass.
|
|
6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps` (fake probe + recording delay, one delay per frame ≈ 1ms).
|
|
7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)**
|
|
8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size
|
|
9. ✅ Loop control (mechanism) — `IMediaFrameSource.Looping`; `MediaVideoSource` takes a `Func<IDecodeProcess>` process factory and restarts the decode on natural EOF when `Looping` (fresh process per pass, since a `Process` can't be re-`Start()`ed). Test: `MediaVideoSource_LoopsUntilLoopDisabled` (single frame re-emits across passes, `Completed` only after loop cleared). Wiring `Source.MediaIsLooping` into the flag lands with the UI-picker (acquisition) slice.
|
|
10. ☐ Volume control — per-source volume slider for audio playback
|
|
11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider
|
|
12. ☐ Schema migration for media source settings (file path, loop, volume)
|
|
13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation
|
|
|
|
### Design decisions
|
|
|
|
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
|
|
- **Audio plays through the desktop channel** — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed.
|
|
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
|
|
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
|
|
|
|
### UI picker slice — spec (imported from HANDOFF 2026-09-01 so it can't be lost by a rewrite)
|
|
|
|
Nothing `AcquireAsync`s a media path yet, so **no frames flow in a running app** — decoder, manager,
|
|
pacing, and loop mechanism are shipped and unit-tested, but a media Source has no way to start a
|
|
session. Deliverables, in order:
|
|
|
|
1. **File picker + add/remove commands** (mirror `MainViewModel.Trax.cs:68` `OpenFileDialog` usage).
|
|
A "Media" source action opens an `OpenFileDialog` filtered to video (mp4/mov/avi); on OK set
|
|
`Source.MediaPath`, add to scene, and **`AcquireAsync(MediaPath)`**; on remove
|
|
**`ReleaseAsync(MediaPath)`** (refcounted — not `ReleaseAllAsync`, unless the path leaves the
|
|
layout entirely). On `StagedScene` layout loads/teardown, release every media path no longer
|
|
present and acquire new ones so sessions track the live layout.
|
|
2. **Wire `Source.MediaIsLooping` → `IMediaFrameSource.Looping`** — needs a manager-level per-**path**
|
|
loop provider. Ambiguity to rule + record: sessions are refcounted per `MediaPath` and shared
|
|
across scenes, but `MediaIsLooping` is per-`Source`. Chosen rule: the path's session `Looping` =
|
|
**any** live reference has it on (shared behaviour). Record it in the slice commit.
|
|
3. **Loop toggle + volume slider** in the source context menu / inspector (items 10/11).
|
|
4. `MediaVolume`/`MediaPlaybackState` wiring — audio path for media is future work (desktop loopback
|
|
already carries it, TRAX-style; `MediaVolume` slider can drive the local file's audio separately
|
|
later — out of this slice).
|
|
|
|
**Design notes to preserve:** `ResolveOutputFrame` reads `_mediaManager.GetLatestFrame(MediaPath)`;
|
|
`OnMediaPreviewBitmapChanged` adopts the shared `WriteableBitmap` per path; `MediaFailed` →
|
|
`OnMediaFailed` (currently `Debug.WriteLine` — surface in UI via toast, TASK 24 stack). The compositor
|
|
scales any frame size — media renders through the generic `frameFor(element)` path, no compositor
|
|
change. First GUI smoke test: add a short real mp4 on native Windows, confirm it plays and previews.
|
|
|