feat(media): play the clip's soundtrack and share one decode pipe
A video file was picture-only: the decode pipe passed `-an`, so a starting-soon or BRB clip with music played silent everywhere. Adds sound to every clip and collapses the two decoders into one. - `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21) and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio (`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to real time, loop restarts both so picture and sound stay together. Alert behaviour is unchanged. - `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises `AudioAvailable`; `MainViewModel` feeds it to the new `AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the desktop fader and the mic auto-ducker apply exactly as they do for game audio and TRAX (creator ruling 2026-11-05: audio joins the group). - Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT). - Level/mute are per SESSION because one decoder is shared per file path — the last layer touched wins; one decoder per layer would play the file twice. Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the manager scales by level and drops when muted, and a media layer's sound moves the desktop meter. Alert/decode suites unchanged and green. Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed; the two documented pre-existing failures only).
This commit is contained in:
@@ -7,8 +7,7 @@
|
||||
|
||||
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
|
||||
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
|
||||
reference goes away. **Picture only: the file's own audio is not played anywhere** (the decode pipe
|
||||
passes `-an`), so a video with a soundtrack is silent on stream — the audio route is open, item 10.
|
||||
reference goes away, and **the file's own soundtrack plays into the stream** (item 10).
|
||||
|
||||
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
|
||||
2. ✅ `SourceType.MediaSource` addition to the enum
|
||||
@@ -25,13 +24,13 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
|
||||
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
|
||||
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
|
||||
session loops if **any** live media reference has it on, because one decoder is shared per path.
|
||||
10. ⏸ File audio + volume — **the file's soundtrack is silent everywhere today**: the decode pipe
|
||||
passes `-an`, so nothing plays it and the desktop loopback has nothing to carry. Two established
|
||||
routes when this is taken: **alert-style** (a second `f32le` pipe into the mixer —
|
||||
`AlertClipDecoder.RunAudioAsync` is the working model, independent of desktop capture) or
|
||||
**TRAX-style** (`MusicPlayer` plays it to the default device, so the loopback, the desktop fader
|
||||
and ducking all apply — at the cost of a second clock to hold in sync with the video). Product
|
||||
call; not started, and no volume slider until one of them exists.
|
||||
10. ✅ File audio + volume — the shared clip pipe decodes the soundtrack (`f32le -ac 2 -ar 48000 -vn`,
|
||||
50 ms chunks paced to real time, byte-carry across reads) and the layer carries **Mute** + **Level**.
|
||||
The manager applies them per session (one decoder per path, so the last layer touched wins — the
|
||||
alternative would play the same file twice) and the audio joins the DESKTOP/GAME group in
|
||||
`AudioMixer.EnqueueDesktopMediaAudio`, so the desktop fader and the voice ducker behave exactly as
|
||||
they do for game audio and TRAX. Test: `MediaSourceAudioTests` (chunks, level scaling, mute, and the
|
||||
layer's sound reaching the desktop meter).
|
||||
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
|
||||
("Change…" in the properties panel), loop toggle, video name display.
|
||||
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
|
||||
@@ -41,10 +40,11 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
|
||||
### Design decisions
|
||||
|
||||
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
|
||||
- **Audio is NOT wired, and it is not hiding behind the loopback** — `MediaVideoSource` decodes with
|
||||
`-an`, so a video file's soundtrack is silent on stream today (the loopback only carries what the
|
||||
machine actually plays). Item 10 records the two established routes; do not describe media as
|
||||
audible until one is built.
|
||||
- **Audio joins the desktop/game group** (creator ruling 2026-11-05) — the clip's sound is decoded by
|
||||
the same pipe as its picture, mixed directly into the stream (not played locally and captured), and
|
||||
grouped with game audio so one fader and the ducker cover it. A per-layer Mute/Level sits on top.
|
||||
Chosen over TRAX-style local playback because it works regardless of the creator's desktop-capture
|
||||
state, and because an alert clip needs the same direct route anyway.
|
||||
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
|
||||
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
|
||||
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
|
||||
|
||||
Reference in New Issue
Block a user