feat(media): play the clip's soundtrack and share one decode pipe

A video file was picture-only: the decode pipe passed `-an`, so a starting-soon
or BRB clip with music played silent everywhere. Adds sound to every clip and
collapses the two decoders into one.

- `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21)
  and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio
  (`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to
  real time, loop restarts both so picture and sound stay together. Alert
  behaviour is unchanged.
- `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises
  `AudioAvailable`; `MainViewModel` feeds it to the new
  `AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the
  desktop fader and the mic auto-ducker apply exactly as they do for game audio
  and TRAX (creator ruling 2026-11-05: audio joins the group).
- Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the
  properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT).
- Level/mute are per SESSION because one decoder is shared per file path — the
  last layer touched wins; one decoder per layer would play the file twice.

Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the
manager scales by level and drops when muted, and a media layer's sound moves the
desktop meter. Alert/decode suites unchanged and green.

Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in
frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed;
the two documented pre-existing failures only).
This commit is contained in:
2026-11-05 07:54:06 -08:00
parent 776f84fb88
commit 2670ac022b
21 changed files with 669 additions and 328 deletions
+13 -13
View File
@@ -7,8 +7,7 @@
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
reference goes away. **Picture only: the file's own audio is not played anywhere** (the decode pipe
passes `-an`), so a video with a soundtrack is silent on stream — the audio route is open, item 10.
reference goes away, and **the file's own soundtrack plays into the stream** (item 10).
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
2. ✅ `SourceType.MediaSource` addition to the enum
@@ -25,13 +24,13 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
session loops if **any** live media reference has it on, because one decoder is shared per path.
10. ⏸ File audio + volume — **the file's soundtrack is silent everywhere today**: the decode pipe
passes `-an`, so nothing plays it and the desktop loopback has nothing to carry. Two established
routes when this is taken: **alert-style** (a second `f32le` pipe into the mixer —
`AlertClipDecoder.RunAudioAsync` is the working model, independent of desktop capture) or
**TRAX-style** (`MusicPlayer` plays it to the default device, so the loopback, the desktop fader
and ducking all apply — at the cost of a second clock to hold in sync with the video). Product
call; not started, and no volume slider until one of them exists.
10. ✅ File audio + volume — the shared clip pipe decodes the soundtrack (`f32le -ac 2 -ar 48000 -vn`,
50 ms chunks paced to real time, byte-carry across reads) and the layer carries **Mute** + **Level**.
The manager applies them per session (one decoder per path, so the last layer touched wins — the
alternative would play the same file twice) and the audio joins the DESKTOP/GAME group in
`AudioMixer.EnqueueDesktopMediaAudio`, so the desktop fader and the voice ducker behave exactly as
they do for game audio and TRAX. Test: `MediaSourceAudioTests` (chunks, level scaling, mute, and the
layer's sound reaching the desktop meter).
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
("Change…" in the properties panel), loop toggle, video name display.
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
@@ -41,10 +40,11 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
### Design decisions
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
- **Audio is NOT wired, and it is not hiding behind the loopback** — `MediaVideoSource` decodes with
`-an`, so a video file's soundtrack is silent on stream today (the loopback only carries what the
machine actually plays). Item 10 records the two established routes; do not describe media as
audible until one is built.
- **Audio joins the desktop/game group** (creator ruling 2026-11-05) — the clip's sound is decoded by
the same pipe as its picture, mixed directly into the stream (not played locally and captured), and
grouped with game audio so one fader and the ducker cover it. A per-layer Mute/Level sits on top.
Chosen over TRAX-style local playback because it works regardless of the creator's desktop-capture
state, and because an alert clip needs the same direct route anyway.
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in