feat(media): play the clip's soundtrack and share one decode pipe
A video file was picture-only: the decode pipe passed `-an`, so a starting-soon or BRB clip with music played silent everywhere. Adds sound to every clip and collapses the two decoders into one. - `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21) and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio (`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to real time, loop restarts both so picture and sound stay together. Alert behaviour is unchanged. - `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises `AudioAvailable`; `MainViewModel` feeds it to the new `AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the desktop fader and the mic auto-ducker apply exactly as they do for game audio and TRAX (creator ruling 2026-11-05: audio joins the group). - Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT). - Level/mute are per SESSION because one decoder is shared per file path — the last layer touched wins; one decoder per layer would play the file twice. Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the manager scales by level and drops when muted, and a media layer's sound moves the desktop meter. Alert/decode suites unchanged and green. Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed; the two documented pre-existing failures only).
This commit is contained in:
@@ -799,9 +799,12 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
|
||||
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
|
||||
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
|
||||
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
|
||||
the layer no longer references it. **A video file's own soundtrack is silent today** — the decode pipe
|
||||
is `-an`, so nothing plays it and the loopback has nothing to carry; wiring it (alert-style mixer feed
|
||||
or TRAX-style local playback) is open, and no volume slider exists until then.
|
||||
the layer no longer references it. **Sound ships with it (2026-11-05):** both `MediaVideoSource` and
|
||||
`AlertClipDecoder` are now thin wrappers over one `Services/VideoClipPipe.cs` (video pipe + `f32le`
|
||||
audio pipe, both real-time paced, loop restarts both), so a clip is a clip. The manager applies the
|
||||
layer's Level/Mute per session and raises `AudioAvailable`, and `MainViewModel` feeds it to
|
||||
`AudioMixer.EnqueueDesktopMediaAudio` — the DESKTOP/GAME ring, so the desktop fader and the voice
|
||||
ducker apply. The alert clip keeps its own unity-gain ring (`EnqueueAlertAudio`, never ducked).
|
||||
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
||||
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
||||
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
||||
|
||||
@@ -250,6 +250,13 @@ devices, no timers).
|
||||
`StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) —
|
||||
the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the
|
||||
pipe lifecycle.
|
||||
**Two clip-audio rings join that mix (2026-11-05):** `EnqueueAlertAudio` (TASK 47) is pre-scaled by the
|
||||
alert layer and mixed at **unity, never ducked** — an alert must be heard over the mic.
|
||||
`EnqueueDesktopMediaAudio` (TASK 21's audio slice) is the opposite on purpose: a video layer's
|
||||
soundtrack joins the **desktop/game ring**, so the desktop fader and the auto-ducker apply exactly as
|
||||
they do for game audio and TRAX. Both are fed by the shared `Services/VideoClipPipe.cs`; the layer's
|
||||
own Mute/Level are applied upstream by `MediaVideoSourceManager` (level/mute are per **session**, since
|
||||
one decoder is shared per file path).
|
||||
- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo
|
||||
mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose
|
||||
offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500
|
||||
|
||||
+5
-6
@@ -97,12 +97,11 @@ published decisions (recorded in `TASKS/task-43-native-alerts.md`):
|
||||
`AlertClipDecoder.RunVideoAsync` and `MediaVideoSource.RunAsync` are near-duplicates —
|
||||
same ffmpeg args (`-f rawvideo -pix_fmt bgra -vf scale=W:H -an -`), same
|
||||
`RawVideoFrameReader` feed loop, same `IFrameRateProbe` + delay pacing, both behind
|
||||
`IDecodeProcess`. What differs is ownership and role, not decoding: alerts are per-play
|
||||
and disposed at drain, media is a refcounted per-path session; alerts also run an
|
||||
`f32le` audio pipe into the mixer, media runs none at all (`-an`) — so a media file's
|
||||
soundtrack is silent today, and task-21 item 10 holds the two routes for changing that;
|
||||
alerts play once, media loops. A shared raw-video pump
|
||||
is the obvious consolidation if a third consumer appears — do not do it for two.
|
||||
`IDecodeProcess`. Consolidated 2026-11-05 with the media audio slice: both are now thin wrappers over
|
||||
one `Services/VideoClipPipe.cs` (video pipe + `f32le` audio pipe, real-time paced, loop restarts both),
|
||||
so decode behaviour cannot drift between them again. What still differs is ownership and role: alerts
|
||||
are per-play and disposed at drain, media is a refcounted per-path session; alerts play once, media
|
||||
loops; alert audio is mixed at unity and never ducked, media audio joins the desktop/game group.
|
||||
Fades in/out via **`FadeDurationSeconds = 0.30`**
|
||||
(`AlertOverlayLayer` const): fade-in by `elapsed` on straight-alpha frame copies;
|
||||
EOF → freeze-frame → fade-out → drain (the clip branch of `Advance` **clears
|
||||
|
||||
Reference in New Issue
Block a user