gramps/LlamaCasty
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(media): play the clip's soundtrack and share one decode pipe

Browse Source 
A video file was picture-only: the decode pipe passed `-an`, so a starting-soon
or BRB clip with music played silent everywhere. Adds sound to every clip and
collapses the two decoders into one.

- `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21)
  and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio
  (`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to
  real time, loop restarts both so picture and sound stay together. Alert
  behaviour is unchanged.
- `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises
  `AudioAvailable`; `MainViewModel` feeds it to the new
  `AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the
  desktop fader and the mic auto-ducker apply exactly as they do for game audio
  and TRAX (creator ruling 2026-11-05: audio joins the group).
- Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the
  properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT).
- Level/mute are per SESSION because one decoder is shared per file path — the
  last layer touched wins; one decoder per layer would play the file twice.

Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the
manager scales by level and drops when muted, and a media layer's sound moves the
desktop meter. Alert/decode suites unchanged and green.

Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in
frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed;
the two documented pre-existing failures only).
This commit is contained in:
gramps
2026-11-05 07:54:06 -08:00
parent 776f84fb88
commit 2670ac022b
21 changed files with 669 additions and 328 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ai/frame-pipeline.md
+6 -3
View File
@@ -799,9 +799,12 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
the layer no longer references it. **A video file's own soundtrack is silent today** — the decode pipe
is `-an`, so nothing plays it and the loopback has nothing to carry; wiring it (alert-style mixer feed
or TRAX-style local playback) is open, and no volume slider exists until then.
the layer no longer references it. **Sound ships with it (2026-11-05):** both `MediaVideoSource` and
`AlertClipDecoder` are now thin wrappers over one `Services/VideoClipPipe.cs` (video pipe + `f32le`
audio pipe, both real-time paced, loop restarts both), so a clip is a clip. The manager applies the
layer's Level/Mute per session and raises `AudioAvailable`, and `MainViewModel` feeds it to
`AudioMixer.EnqueueDesktopMediaAudio` — the DESKTOP/GAME ring, so the desktop fader and the voice
ducker apply. The alert clip keeps its own unity-gain ring (`EnqueueAlertAudio`, never ducked).
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`