feat(media): play the clip's soundtrack and share one decode pipe

A video file was picture-only: the decode pipe passed `-an`, so a starting-soon
or BRB clip with music played silent everywhere. Adds sound to every clip and
collapses the two decoders into one.

- `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21)
  and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio
  (`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to
  real time, loop restarts both so picture and sound stay together. Alert
  behaviour is unchanged.
- `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises
  `AudioAvailable`; `MainViewModel` feeds it to the new
  `AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the
  desktop fader and the mic auto-ducker apply exactly as they do for game audio
  and TRAX (creator ruling 2026-11-05: audio joins the group).
- Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the
  properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT).
- Level/mute are per SESSION because one decoder is shared per file path — the
  last layer touched wins; one decoder per layer would play the file twice.

Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the
manager scales by level and drops when muted, and a media layer's sound moves the
desktop meter. Alert/decode suites unchanged and green.

Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in
frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed;
the two documented pre-existing failures only).
This commit is contained in:
2026-11-05 07:54:06 -08:00
parent 776f84fb88
commit 2670ac022b
21 changed files with 669 additions and 328 deletions
+14 -1
View File
@@ -746,7 +746,20 @@
<CheckBox Style="{StaticResource PropertyPillCheck}" Margin="10,4,0,4" <CheckBox Style="{StaticResource PropertyPillCheck}" Margin="10,4,0,4"
Content="Loop" IsChecked="{Binding SelectedElement.MediaIsLooping, Mode=TwoWay}" Content="Loop" IsChecked="{Binding SelectedElement.MediaIsLooping, Mode=TwoWay}"
ToolTip="Restart from the beginning at the end of the file — for starting-soon and BRB loops"/> ToolTip="Restart from the beginning at the end of the file — for starting-soon and BRB loops"/>
<TextBlock Text="Picture only — a video's own sound is not played yet." <DockPanel Margin="10,0,0,4">
<CheckBox DockPanel.Dock="Left" Style="{StaticResource PropertyPillCheck}"
Content="Mute" IsChecked="{Binding SelectedElement.MediaMuted, Mode=TwoWay}"
ToolTip="Silence this layer's sound. The picture keeps playing."/>
</DockPanel>
<DockPanel Margin="10,0,0,4">
<TextBlock DockPanel.Dock="Left" Text="Level" Style="{StaticResource PropertyFieldLabel}" Width="40"/>
<TextBlock DockPanel.Dock="Right" Text="{Binding SelectedElement.MediaVolume, StringFormat={}{0:P0}}"
Style="{StaticResource PropertyFieldLabel}" Width="34" TextAlignment="Right"/>
<Slider Minimum="0" Maximum="1" Width="120"
Value="{Binding SelectedElement.MediaVolume, Mode=TwoWay}"
VerticalAlignment="Center"/>
</DockPanel>
<TextBlock Text="Sound plays through the desktop audio controls, like game audio."
Style="{StaticResource PropertyHint}"/> Style="{StaticResource PropertyHint}"/>
</StackPanel> </StackPanel>
+12 -10
View File
@@ -1,20 +1,21 @@
# HANDOFF — 2026-11-05 (TASK 21 — video file layer) # HANDOFF — 2026-11-05 (TASK 21 — video file layer + its audio)
## State ## State
- `main`; TASK 21 is complete: a video file added from the (+) menu decodes, composites into preview - `main`; TASK 21 is complete. A video file added from the (+) menu decodes, composites into preview and
and stream, loops on toggle, and stops with its last scene reference. 3·17 (text layers) completed stream, loops on toggle, and **plays its own soundtrack** (Mute + Level per layer, mixed into the
2026-11-04. desktop/game group so the desktop fader and the ducker apply). Removing it stops the decoder.
- New test `MediaSourceAcquisitionTests` (add → one decoder starts + frames reach the layer + loop flip - Decoding is now ONE pipe: `Services/VideoClipPipe.cs` behind both `MediaVideoSource` (TASK 21) and
reaches the decoder + remove → stops). App build is 0 warnings. `AlertClipDecoder` (TASK 47) — "a clip is a clip". Alert behaviour is unchanged and its tests stay green.
- **Media is picture-only:** `MediaVideoSource` decodes with `-an`, so a video file's own soundtrack - Tests: `MediaSourceAcquisitionTests` (add → decoder starts, frames, loop, remove) and
is silent on stream — the loopback does not carry it because nothing plays it. Two routes recorded `MediaSourceAudioTests` (chunks, level scaling, mute, sound reaching the desktop meter). App build 0 warnings.
in `task-21` item 10 (alert-style mixer feed vs TRAX-style local play); no volume slider until one exists.
## Next action ## Next action
- Next queued unit: TASK 49 (chat profanity filter). Day 3's TASK 32/33 resilience slices are still - Next queued unit: TASK 49 (chat profanity filter). Day 3's TASK 32/33 resilience slices are still
outstanding — 32·4 (fragmented MP4) is the smallest. outstanding — 32·4 (fragmented MP4) is the smallest.
- Agreed next feature work (designed, not started): green/blue screen detection + keying through the clip
pipe — suggest-only by default, an opt-in "Auto Green/Blue Screen" setting, alert clips included.
- Do not push absent a checkpoint request. - Do not push absent a checkpoint request.
## Landmines ## Landmines
@@ -23,5 +24,6 @@
mouse input. Do not run it while the desktop is in use. mouse input. Do not run it while the desktop is in use.
- Integration tests constructing `MainWindow` must set a temporary `LayoutPathOverride` and clear it in `finally`. - Integration tests constructing `MainWindow` must set a temporary `LayoutPathOverride` and clear it in `finally`.
- Media sessions are keyed per **path**, not per layer: acquire through `EnsureMediaSession`, release - Media sessions are keyed per **path**, not per layer: acquire through `EnsureMediaSession`, release
through `ReleaseMediaIfUnused`, never `AcquireAsync` twice for one path. through `ReleaseMediaIfUnused`, never `AcquireAsync` twice for one path. Level/mute are therefore
per session too.
- Bundled FFmpeg stays outside the single-file exe (`ExcludeFromSingleFile="true"`). - Bundled FFmpeg stays outside the single-file exe (`ExcludeFromSingleFile="true"`).
+9
View File
@@ -264,6 +264,15 @@ public class Source : SceneElement
set => Set(ref _mediaVolume, Math.Clamp(value, 0.0, 1.0)); set => Set(ref _mediaVolume, Math.Clamp(value, 0.0, 1.0));
} }
private bool _mediaMuted;
/// <summary>Silences this layer's soundtrack (the picture keeps playing). One decoder is
/// shared per file, so the level/mute are the file's — the layer is the control.</summary>
public bool MediaMuted
{
get => _mediaMuted;
set => Set(ref _mediaMuted, value);
}
private MediaPlaybackState _mediaPlaybackState = MediaPlaybackState.Stopped; private MediaPlaybackState _mediaPlaybackState = MediaPlaybackState.Stopped;
public MediaPlaybackState MediaPlaybackState public MediaPlaybackState MediaPlaybackState
{ {
+16 -176
View File
@@ -1,6 +1,5 @@
using System.Diagnostics; using System;
using System.IO; using System.Threading;
using ytLive.Helpers;
using ytLive.Services.Audio; using ytLive.Services.Audio;
using ytLive.Services.Encoder; using ytLive.Services.Encoder;
@@ -11,10 +10,10 @@ namespace ytLive.Services;
/// same file, both paced to real time. The alert box owns a decoder per event /// same file, both paced to real time. The alert box owns a decoder per event
/// (start at play, dispose at drain) — unlike MediaSource's app-wide refcounted /// (start at play, dispose at drain) — unlike MediaSource's app-wide refcounted
/// session, an alert REPLAYS the same clip each event, so a fresh decode per /// session, an alert REPLAYS the same clip each event, so a fresh decode per
/// play is required. Video is rawvideo BGRA scaled to the box rect (the /// play is required. The fade is applied by the alert layer, not here.
/// <see cref="MediaVideoSource"/> shape); audio is ffmpeg <c>f32le</c> at the /// <para>Since TASK 21's audio slice this is a thin wrapper over the shared
/// mixer's native 48 kHz stereo (so the mixer ring needs no resampler). The /// <see cref="VideoClipPipe"/> (the same pipe a media layer uses); the alert case
/// fade is applied by the alert layer, not here. /// simply never sets <c>Looping</c>.</para>
/// </summary> /// </summary>
public interface IAlertClipDecoder : IDisposable public interface IAlertClipDecoder : IDisposable
{ {
@@ -35,30 +34,14 @@ public interface IAlertClipDecoder : IDisposable
event Action? Completed; event Action? Completed;
} }
/// <summary>Default <see cref="IAlertClipDecoder"/>: two ffmpeg subprocesses on /// <summary>Default <see cref="IAlertClipDecoder"/>: the shared clip pipe with looping off.</summary>
/// the same file. Video via <see cref="RawVideoFrameReader"/> on a <c>rawvideo
/// bgra</c> pipe (-an), audio via an <c>f32le -ac 2 -ar 48000</c> pipe (-vn).
/// The seams (locator, process factory, frame-rate probe, delay) let tests drive
/// a fake decoder without a real codec or Windows.</summary>
public sealed class AlertClipDecoder : IAlertClipDecoder public sealed class AlertClipDecoder : IAlertClipDecoder
{ {
public event Action<VideoFrame>? FrameAvailable; public event Action<VideoFrame>? FrameAvailable;
public event Action<AudioSample>? AudioReady; public event Action<AudioSample>? AudioReady;
public event Action? Completed; public event Action? Completed;
private const int AudioSampleRate = 48000; private readonly VideoClipPipe _clip;
private const int AudioChannels = 2;
private const int AudioChunkFloats = 4800; // 50ms of stereo
private readonly string _path;
private readonly int _width;
private readonly int _height;
private readonly IFfmpegLocator _locator;
private readonly Func<IDecodeProcess> _processFactory;
private readonly IFrameRateProbe? _probe;
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
private readonly CancellationTokenSource _cts = new();
private Task? _run;
public AlertClipDecoder( public AlertClipDecoder(
string path, string path,
@@ -69,159 +52,16 @@ public sealed class AlertClipDecoder : IAlertClipDecoder
IFrameRateProbe? frameRateProbe = null, IFrameRateProbe? frameRateProbe = null,
Func<TimeSpan, CancellationToken, Task>? delay = null) Func<TimeSpan, CancellationToken, Task>? delay = null)
{ {
_path = path; _clip = new VideoClipPipe(
_width = width; path, width, height, locator, processFactory, frameRateProbe, delay, audio: true);
_height = height; _clip.FrameAvailable += frame => FrameAvailable?.Invoke(frame);
_locator = locator; _clip.AudioAvailable += sample => AudioReady?.Invoke(sample);
_processFactory = processFactory; _clip.Completed += () => Completed?.Invoke();
_probe = frameRateProbe;
_delay = delay ?? ((t, ct) => Task.Delay(t, ct));
} }
public void Start() public void Start() => _ = _clip.StartAsync();
{
if (_run != null) return;
_run = Task.Run(RunAsync, CancellationToken.None);
}
public void Stop() => _cts.Cancel(); public void Stop() => _ = _clip.StopAsync();
public void Dispose() public void Dispose() => _clip.Dispose();
{
_cts.Cancel();
_cts.Dispose();
}
private async Task RunAsync()
{
var ct = _cts.Token;
var videoFrames = 0;
var audioChunks = 0;
var failed = false;
try
{
double? frameRate = null;
if (_probe != null)
frameRate = await _probe.ProbeAsync(_path, ct).ConfigureAwait(false);
var frameDuration = frameRate is > 0 ? TimeSpan.FromSeconds(1.0 / frameRate.Value) : TimeSpan.Zero;
var ffmpegPath = await _locator.LocateAsync(ct).ConfigureAwait(false);
var videoTask = RunVideoAsync(ffmpegPath, frameDuration, ct);
var audioTask = RunAudioAsync(ffmpegPath, ct);
videoFrames = await videoTask.ConfigureAwait(false);
audioChunks = await audioTask.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
failed = true;
return;
}
catch (Exception ex)
{
// A failed decode is not fatal: the alert layer falls back to the six
// animations (or the frame it already holds) exactly as a missing file.
// Logged so a silent blank box traces back to a decode failure.
failed = true;
AppLog.Write(ex, $"Alert clip decode failed ({Path.GetFileName(_path)})", AppLogErrorClass.Class2);
}
if (!ct.IsCancellationRequested)
{
AppLog.Write($"Alert clip end: {Path.GetFileName(_path)} frames={videoFrames} audioChunks={audioChunks} failed={failed}");
Completed?.Invoke();
}
}
private async Task<int> RunVideoAsync(string ffmpegPath, TimeSpan frameDuration, CancellationToken ct)
{
var args =
$"-hide_banner -loglevel error -i \"{_path}\" " +
$"-f rawvideo -pix_fmt bgra -vf scale={_width}:{_height} -an -";
using var process = _processFactory();
process.Start(new ProcessStartInfo
{
FileName = ffmpegPath,
Arguments = args,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
});
var reader = new RawVideoFrameReader(_width, _height);
var buffer = new byte[65536];
var stream = process.StandardOutput;
var frames = 0;
while (!ct.IsCancellationRequested && !process.HasExited)
{
var read = await stream.ReadAsync(buffer, 0, buffer.Length, ct).ConfigureAwait(false);
if (read <= 0) break;
var chunk = new byte[read];
Buffer.BlockCopy(buffer, 0, chunk, 0, read);
foreach (var frame in reader.Feed(chunk))
{
frames++;
FrameAvailable?.Invoke(frame);
if (frameDuration > TimeSpan.Zero)
await _delay(frameDuration, ct).ConfigureAwait(false);
}
}
return frames;
}
private async Task<int> RunAudioAsync(string ffmpegPath, CancellationToken ct)
{
var args =
$"-hide_banner -loglevel error -i \"{_path}\" " +
$"-f f32le -ac {AudioChannels} -ar {AudioSampleRate} -vn -";
using var process = _processFactory();
process.Start(new ProcessStartInfo
{
FileName = ffmpegPath,
Arguments = args,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
});
var stream = process.StandardOutput;
var raw = new byte[65536];
var carry = new byte[0];
var floatScratch = new float[AudioChunkFloats];
var floatIndex = 0;
var chunks = 0;
// The pipe is byte-delimited only: a read may split a float, so leftover
// bytes are carried into the next read instead of dropped (a dropped
// half-float would desync the entire stream).
while (!ct.IsCancellationRequested && !process.HasExited)
{
var read = await stream.ReadAsync(raw, 0, raw.Length, ct).ConfigureAwait(false);
if (read <= 0) break;
var total = carry.Length + read;
var bytes = total <= raw.Length ? raw : new byte[total];
if (carry.Length > 0)
{
Buffer.BlockCopy(carry, 0, bytes, 0, carry.Length);
Buffer.BlockCopy(raw, 0, bytes, carry.Length, read);
}
var floatsTo = total - (total % 4);
for (var i = 0; i < floatsTo && floatIndex < floatScratch.Length; i += 4)
floatScratch[floatIndex++] = BitConverter.ToSingle(bytes, i);
carry = total > floatsTo ? bytes[floatsTo..total] : new byte[0];
while (floatIndex == floatScratch.Length)
{
chunks++;
AudioReady?.Invoke(new AudioSample(floatScratch, AudioSampleRate, AudioChannels));
// Pace the NEXT chunk so the whole clip's audio lands at real-time
// cadence (the fade in the alert layer tracks the same wall clock).
await _delay(TimeSpan.FromSeconds(AudioChunkFloats / (double)(AudioSampleRate * AudioChannels)), ct).ConfigureAwait(false);
floatScratch = new float[AudioChunkFloats];
floatIndex = 0;
}
}
return chunks;
}
} }
+19
View File
@@ -283,6 +283,25 @@ public sealed class AudioMixer : IDisposable
LoopbackLevelChanged?.Invoke(level); LoopbackLevelChanged?.Invoke(level);
} }
/// <summary>Drops a MEDIA clip's audio into the desktop/game ring (TASK 21's audio slice).
/// It joins that group on purpose: the desktop fader and the mic auto-ducker then apply
/// exactly as they do to game audio and TRAX, so a BRB bed ducks under the creator's
/// voice without a second audio knob. The layer's own level/mute are applied upstream
/// by <see cref="MediaVideoSourceManager"/>.</summary>
public void EnqueueDesktopMediaAudio(AudioSample sample)
{
var data = sample.Samples;
if (data.Length == 0) return;
if (sample.SampleRate != OutputSampleRate)
data = ResampleLoopback(data, sample.SampleRate);
if (sample.Channels == 1)
data = UpmixToStereo(data);
_loopbackBuffer.Write(data);
var level = _loopbackMeter.Push(new AudioSample(data, OutputSampleRate, sample.Channels));
LoopbackLevelChanged?.Invoke(level);
}
/// <summary>Drops the alert clip's audio into its own ring (TASK 47 — the /// <summary>Drops the alert clip's audio into its own ring (TASK 47 — the
/// stream-alert video's soundtrack, pre-scaled by the alert layer to /// stream-alert video's soundtrack, pre-scaled by the alert layer to
/// volume × fade). Mixed at unity into the live output — NO auto-duck: an /// volume × fade). Mixed at unity into the live output — NO auto-duck: an
+7
View File
@@ -1,3 +1,5 @@
using ytLive.Services.Audio;
namespace ytLive.Services; namespace ytLive.Services;
/// <summary> /// <summary>
@@ -17,6 +19,11 @@ public interface IMediaFrameSource : IDisposable
/// <summary>Raised (on a worker thread) for each decoded frame.</summary> /// <summary>Raised (on a worker thread) for each decoded frame.</summary>
event Action<VideoFrame>? FrameAvailable; event Action<VideoFrame>? FrameAvailable;
/// <summary>Raised (on a worker thread) per chunk of the file's own soundtrack
/// (48 kHz stereo float). The manager applies the layer's level/mute and hands it
/// to the mixer; a source with no audio simply never raises it.</summary>
event Action<AudioSample>? AudioAvailable;
/// <summary>Raised when the file finishes decoding (natural EOF) — not raised /// <summary>Raised when the file finishes decoding (natural EOF) — not raised
/// while <see cref="Looping"/> is true.</summary> /// while <see cref="Looping"/> is true.</summary>
event Action? Completed; event Action? Completed;
+2 -1
View File
@@ -113,7 +113,7 @@ public partial class LayoutStore : IDisposable
AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume, SortOrder, AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume, SortOrder,
AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop, AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop,
TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic, TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic,
TextUnderline, TextAlignment, TextVerticalAlignment TextUnderline, TextAlignment, TextVerticalAlignment, MediaMuted
FROM Source ORDER BY SortOrder FROM Source ORDER BY SortOrder
"""; """;
using var reader = cmd.ExecuteReader(); using var reader = cmd.ExecuteReader();
@@ -183,6 +183,7 @@ public partial class LayoutStore : IDisposable
|| !Enum.IsDefined((TextLayerVerticalAlignment)reader.GetInt32(49)) || !Enum.IsDefined((TextLayerVerticalAlignment)reader.GetInt32(49))
? TextLayerVerticalAlignment.Top ? TextLayerVerticalAlignment.Top
: (TextLayerVerticalAlignment)reader.GetInt32(49), : (TextLayerVerticalAlignment)reader.GetInt32(49),
MediaMuted = reader.GetInt32(reader.GetOrdinal("MediaMuted")) != 0,
}; };
if (!sourcesByScene.TryGetValue(sceneId, out var list)) if (!sourcesByScene.TryGetValue(sceneId, out var list))
sourcesByScene[sceneId] = list = new List<(Source, int)>(); sourcesByScene[sceneId] = list = new List<(Source, int)>();
+6
View File
@@ -378,6 +378,12 @@ public partial class LayoutStore : IDisposable
cmd.CommandText = "ALTER TABLE Source ADD COLUMN MediaPlaybackState TEXT NOT NULL DEFAULT 'Stopped';"; cmd.CommandText = "ALTER TABLE Source ADD COLUMN MediaPlaybackState TEXT NOT NULL DEFAULT 'Stopped';";
cmd.ExecuteNonQuery(); cmd.ExecuteNonQuery();
} }
if (!columns.Contains("MediaMuted"))
{
using var cmd = _connection.CreateCommand();
cmd.CommandText = "ALTER TABLE Source ADD COLUMN MediaMuted INTEGER NOT NULL DEFAULT 0;";
cmd.ExecuteNonQuery();
}
// TASK 47: alert video config. The default clip asset id + the creator's // TASK 47: alert video config. The default clip asset id + the creator's
// replacement path (a FILE reference, never a stored BLOB) + the pill, // replacement path (a FILE reference, never a stored BLOB) + the pill,
+4 -2
View File
@@ -94,7 +94,7 @@ public partial class LayoutStore : IDisposable
AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume, AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume,
AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop, AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop,
TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic, TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic,
TextUnderline, TextAlignment, TextVerticalAlignment) TextUnderline, TextAlignment, TextVerticalAlignment, MediaMuted)
VALUES ($id, $sceneId, $assetId, $type, $name, $isEnabled, VALUES ($id, $sceneId, $assetId, $type, $name, $isEnabled,
$x, $y, $w, $h, $opacity, $monitor, $x, $y, $w, $h, $opacity, $monitor,
$clip, $mirrored, $isBackground, $captureKey, $showDesktop, $sort, $clip, $mirrored, $isBackground, $captureKey, $showDesktop, $sort,
@@ -104,7 +104,7 @@ public partial class LayoutStore : IDisposable
$alertVideoAssetId, $alertVideoPath, $alertUseDefaultVideo, $alertShowTicker, $alertVideoVolume, $alertVideoAssetId, $alertVideoPath, $alertUseDefaultVideo, $alertShowTicker, $alertVideoVolume,
$alertDisplayMethod, $isBrandLogo, $blurAmount, $alertAudioPath, $alertAudioLoop, $alertDisplayMethod, $isBrandLogo, $blurAmount, $alertAudioPath, $alertAudioLoop,
$textContent, $textFontFamily, $textFontSize, $textColor, $textBold, $textItalic, $textContent, $textFontFamily, $textFontSize, $textColor, $textBold, $textItalic,
$textUnderline, $textAlignment, $textVerticalAlignment) $textUnderline, $textAlignment, $textVerticalAlignment, $mediaMuted)
"""; """;
cmd.Transaction = tx; cmd.Transaction = tx;
var idP = cmd.Parameters.Add("$id", SqliteType.Text); var idP = cmd.Parameters.Add("$id", SqliteType.Text);
@@ -137,6 +137,7 @@ public partial class LayoutStore : IDisposable
var mediaPathP = cmd.Parameters.Add("$mediaPath", SqliteType.Text); var mediaPathP = cmd.Parameters.Add("$mediaPath", SqliteType.Text);
var mediaIsLoopingP = cmd.Parameters.Add("$mediaIsLooping", SqliteType.Integer); var mediaIsLoopingP = cmd.Parameters.Add("$mediaIsLooping", SqliteType.Integer);
var mediaVolumeP = cmd.Parameters.Add("$mediaVolume", SqliteType.Real); var mediaVolumeP = cmd.Parameters.Add("$mediaVolume", SqliteType.Real);
var mediaMutedP = cmd.Parameters.Add("$mediaMuted", SqliteType.Integer);
var mediaPlaybackStateP = cmd.Parameters.Add("$mediaPlaybackState", SqliteType.Text); var mediaPlaybackStateP = cmd.Parameters.Add("$mediaPlaybackState", SqliteType.Text);
var alertVideoAssetIdP = cmd.Parameters.Add("$alertVideoAssetId", SqliteType.Text); var alertVideoAssetIdP = cmd.Parameters.Add("$alertVideoAssetId", SqliteType.Text);
var alertVideoPathP = cmd.Parameters.Add("$alertVideoPath", SqliteType.Text); var alertVideoPathP = cmd.Parameters.Add("$alertVideoPath", SqliteType.Text);
@@ -198,6 +199,7 @@ public partial class LayoutStore : IDisposable
mediaPathP.Value = (object?)source.MediaPath ?? DBNull.Value; mediaPathP.Value = (object?)source.MediaPath ?? DBNull.Value;
mediaIsLoopingP.Value = source.MediaIsLooping ? 1 : 0; mediaIsLoopingP.Value = source.MediaIsLooping ? 1 : 0;
mediaVolumeP.Value = source.MediaVolume; mediaVolumeP.Value = source.MediaVolume;
mediaMutedP.Value = source.MediaMuted ? 1 : 0;
mediaPlaybackStateP.Value = source.MediaPlaybackState.ToString(); mediaPlaybackStateP.Value = source.MediaPlaybackState.ToString();
alertVideoAssetIdP.Value = (object?)source.AlertVideoAssetId ?? DBNull.Value; alertVideoAssetIdP.Value = (object?)source.AlertVideoAssetId ?? DBNull.Value;
alertVideoPathP.Value = (object?)source.AlertVideoPath ?? DBNull.Value; alertVideoPathP.Value = (object?)source.AlertVideoPath ?? DBNull.Value;
+21 -115
View File
@@ -1,38 +1,32 @@
using System.Diagnostics; using ytLive.Services.Audio;
using System.IO;
using ytLive.Services.Encoder; using ytLive.Services.Encoder;
namespace ytLive.Services; namespace ytLive.Services;
/// <summary> /// <summary>
/// TASK 21: decodes a local media file (any format ffmpeg handles) into /// TASK 21: decodes a local media file (any format ffmpeg handles) into
/// <see cref="VideoFrame"/>s for a MediaSource. Implements /// <see cref="VideoFrame"/>s for a MediaSource. This is the <see cref="IMediaFrameSource"/>
/// <see cref="IMediaFrameSource"/>: raises <see cref="FrameAvailable"/> per /// shape over the shared <see cref="VideoClipPipe"/> — the same pipe an alert clip uses
/// frame and <see cref="IMediaFrameSource.Completed"/> on natural EOF (unless /// (TASK 47), so the two features cannot drift apart again in how they decode.
/// <see cref="IMediaFrameSource.Looping"/>). Spawns ffmpeg with /// <para>Video AND the file's own soundtrack are decoded. The manager scales the audio by
/// <c>-f rawvideo -pix_fmt bgra</c> and drains the raw BGRA pipe via /// the layer's level/mute and hands it to the mixer's desktop group.</para>
/// <see cref="RawVideoFrameReader"/>. The decoder process and emitters are seams
/// so tests run without a real codec or audio device.
/// </summary> /// </summary>
public sealed class MediaVideoSource : IMediaFrameSource public sealed class MediaVideoSource : IMediaFrameSource
{ {
public event Action<VideoFrame>? FrameAvailable; public event Action<VideoFrame>? FrameAvailable;
public event Action<AudioSample>? AudioAvailable;
public event Action? Completed; public event Action? Completed;
public string Key => _path; public string Key => _path;
public bool Looping { get; set; } public bool Looping
{
get => _clip.Looping;
set => _clip.Looping = value;
}
private readonly string _path; private readonly string _path;
private readonly int _width; private readonly VideoClipPipe _clip;
private readonly int _height;
private readonly IFfmpegLocator _locator;
private readonly Func<IDecodeProcess> _processFactory;
private readonly IFrameRateProbe? _probe;
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
private readonly CancellationTokenSource _cts = new();
private IDisposable? _run;
private IDecodeProcess? _active;
public MediaVideoSource( public MediaVideoSource(
string path, string path,
@@ -44,104 +38,16 @@ public sealed class MediaVideoSource : IMediaFrameSource
Func<TimeSpan, CancellationToken, Task>? delay = null) Func<TimeSpan, CancellationToken, Task>? delay = null)
{ {
_path = path; _path = path;
_width = width; _clip = new VideoClipPipe(
_height = height; path, width, height, locator, processFactory, frameRateProbe, delay, audio: true);
_locator = locator; _clip.FrameAvailable += frame => FrameAvailable?.Invoke(frame);
_processFactory = processFactory; _clip.AudioAvailable += sample => AudioAvailable?.Invoke(sample);
_probe = frameRateProbe; _clip.Completed += () => Completed?.Invoke();
_delay = delay ?? ((t, ct) => Task.Delay(t, ct));
} }
public Task StartAsync(CancellationToken cancellationToken = default) public Task StartAsync(CancellationToken cancellationToken = default) => _clip.StartAsync(cancellationToken);
{
if (_run != null) return Task.CompletedTask;
_run = _cts.Token.Register(StopInternal);
_ = Task.Run(RunAsync, CancellationToken.None);
return Task.CompletedTask;
}
public Task StopAsync() public Task StopAsync() => _clip.StopAsync();
{
_cts.Cancel();
_run = null;
return Task.CompletedTask;
}
public void Dispose() public void Dispose() => _clip.Dispose();
{
_cts.Cancel();
_cts.Dispose();
_active?.Dispose();
}
private void StopInternal() => KillProcess();
private async Task RunAsync()
{
try
{
double? frameRate = null;
if (_probe != null)
frameRate = await _probe.ProbeAsync(_path, _cts.Token).ConfigureAwait(false);
var frameDuration = frameRate is > 0 ? TimeSpan.FromSeconds(1.0 / frameRate.Value) : TimeSpan.Zero;
var ffmpegPath = await _locator.LocateAsync(_cts.Token).ConfigureAwait(false);
var args =
$"-hide_banner -loglevel error -i \"{_path}\" " +
$"-f rawvideo -pix_fmt bgra -vf scale={_width}:{_height} -an -";
var startInfo = new ProcessStartInfo
{
FileName = ffmpegPath,
Arguments = args,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
};
do
{
using (var process = _processFactory())
{
_active = process;
process.Start(startInfo);
var reader = new RawVideoFrameReader(_width, _height);
var buffer = new byte[65536];
var stream = process.StandardOutput;
while (!_cts.IsCancellationRequested && !process.HasExited)
{
var read = await stream.ReadAsync(buffer, 0, buffer.Length, _cts.Token).ConfigureAwait(false);
if (read <= 0) break;
foreach (var frame in reader.Feed(buffer.AsSpan(0, read).ToArray()))
{
FrameAvailable?.Invoke(frame);
if (frameDuration > TimeSpan.Zero)
await _delay(frameDuration, _cts.Token).ConfigureAwait(false);
}
}
}
_active = null;
}
while (Looping && !_cts.IsCancellationRequested);
}
catch (OperationCanceledException)
{
}
catch (Exception ex)
{
Fail($"Media decode failed: {ex.Message}");
return;
}
if (!_cts.IsCancellationRequested)
Completed?.Invoke();
}
private void Fail(string message) => Console.Error.WriteLine(message);
private void KillProcess()
{
try { _active?.Kill(); } catch { /* already exited */ }
}
} }
+54
View File
@@ -6,6 +6,7 @@ using System.Windows.Media;
using System.Windows.Media.Imaging; using System.Windows.Media.Imaging;
using System.Windows.Threading; using System.Windows.Threading;
using ytLive.Helpers; using ytLive.Helpers;
using ytLive.Services.Audio;
namespace ytLive.Services; namespace ytLive.Services;
@@ -23,8 +24,11 @@ public sealed class MediaVideoSourceManager : IDisposable
public string Key { get; } public string Key { get; }
public IMediaFrameSource Source { get; } public IMediaFrameSource Source { get; }
public Action<VideoFrame>? FrameHandler; public Action<VideoFrame>? FrameHandler;
public Action<AudioSample>? AudioHandler;
public int RefCount; public int RefCount;
public bool Started; public bool Started;
public bool AudioMuted;
public double AudioVolume = 1.0;
public WriteableBitmap? PreviewBitmap; public WriteableBitmap? PreviewBitmap;
public VideoFrame? LatestFrame; public VideoFrame? LatestFrame;
public bool FramePending; public bool FramePending;
@@ -49,6 +53,10 @@ public sealed class MediaVideoSourceManager : IDisposable
/// <summary>Raised when a media decoder cannot be created (bad path).</summary> /// <summary>Raised when a media decoder cannot be created (bad path).</summary>
public event Action<string, string>? MediaFailed; public event Action<string, string>? MediaFailed;
/// <summary>Raised (worker thread) per audio chunk of a file's own soundtrack, already
/// scaled by the session's level and dropped when muted.</summary>
public event Action<string, AudioSample>? AudioAvailable;
public MediaVideoSourceManager( public MediaVideoSourceManager(
Func<string, IMediaFrameSource?> sourceFactory, Func<string, IMediaFrameSource?> sourceFactory,
Dispatcher? uiDispatcher = null, Dispatcher? uiDispatcher = null,
@@ -95,6 +103,8 @@ public sealed class MediaVideoSourceManager : IDisposable
session = new MediaSession(key, source); session = new MediaSession(key, source);
session.FrameHandler = frame => OnFrameAvailable(session, frame); session.FrameHandler = frame => OnFrameAvailable(session, frame);
session.Source.FrameAvailable += session.FrameHandler; session.Source.FrameAvailable += session.FrameHandler;
session.AudioHandler = sample => OnAudioAvailable(session, sample);
session.Source.AudioAvailable += session.AudioHandler;
// Loop intent lives on the Source (per scene), so the caller supplies the // Loop intent lives on the Source (per scene), so the caller supplies the
// per-path answer ("any live reference loops"). Applied before Start so a // per-path answer ("any live reference loops"). Applied before Start so a
// zero-length or instantly-EOF file cannot slip past a later assignment. // zero-length or instantly-EOF file cannot slip past a later assignment.
@@ -117,6 +127,7 @@ public sealed class MediaVideoSourceManager : IDisposable
lock (_gate) lock (_gate)
_sessions.Remove(key); _sessions.Remove(key);
session.Source.FrameAvailable -= session.FrameHandler; session.Source.FrameAvailable -= session.FrameHandler;
if (session.AudioHandler != null) session.Source.AudioAvailable -= session.AudioHandler;
AppLog.Write($"MediaSource: failed to start decoder '{key}': {ex.Message}"); AppLog.Write($"MediaSource: failed to start decoder '{key}': {ex.Message}");
MediaFailed?.Invoke(key, ex.Message); MediaFailed?.Invoke(key, ex.Message);
await SafeStopAsync(session.Source); await SafeStopAsync(session.Source);
@@ -137,6 +148,7 @@ public sealed class MediaVideoSourceManager : IDisposable
if (toStop == null) return; if (toStop == null) return;
toStop.Source.FrameAvailable -= toStop.FrameHandler; toStop.Source.FrameAvailable -= toStop.FrameHandler;
if (toStop.AudioHandler != null) toStop.Source.AudioAvailable -= toStop.AudioHandler;
await SafeStopAsync(toStop.Source); await SafeStopAsync(toStop.Source);
toStop.PreviewBitmap = null; toStop.PreviewBitmap = null;
} }
@@ -155,6 +167,7 @@ public sealed class MediaVideoSourceManager : IDisposable
if (toStop == null) return; if (toStop == null) return;
toStop.Source.FrameAvailable -= toStop.FrameHandler; toStop.Source.FrameAvailable -= toStop.FrameHandler;
if (toStop.AudioHandler != null) toStop.Source.AudioAvailable -= toStop.AudioHandler;
await SafeStopAsync(toStop.Source); await SafeStopAsync(toStop.Source);
toStop.PreviewBitmap = null; toStop.PreviewBitmap = null;
} }
@@ -193,6 +206,46 @@ public sealed class MediaVideoSourceManager : IDisposable
} }
} }
/// <summary>The layer's sound controls for this file. One decoder is shared per path,
/// so the level/mute are per SESSION: whichever layer the creator last touched wins —
/// the alternative (one decoder per layer) would play the same file twice.</summary>
public void SetAudioSettings(string key, double volume, bool muted)
{
lock (_gate)
{
if (!_sessions.TryGetValue(key, out var session)) return;
session.AudioVolume = Math.Clamp(volume, 0, 1);
session.AudioMuted = muted;
}
}
private void OnAudioAvailable(MediaSession session, AudioSample sample)
{
if (!TryGetActiveSession(session)) return;
double volume;
bool muted;
lock (_gate)
{
volume = session.AudioVolume;
muted = session.AudioMuted;
}
if (muted || volume <= 0) return;
if (volume < 0.999)
{
// Copy before scaling: the source owns its buffer and may recycle it.
var scaled = new float[sample.Samples.Length];
var gain = (float)volume;
for (var i = 0; i < scaled.Length; i++)
scaled[i] = sample.Samples[i] * gain;
AudioAvailable?.Invoke(session.Key, new AudioSample(scaled, sample.SampleRate, sample.Channels));
return;
}
AudioAvailable?.Invoke(session.Key, sample);
}
/// <summary>The shared preview bitmap for a file (so a config created after the /// <summary>The shared preview bitmap for a file (so a config created after the
/// first frame can adopt the running bitmap), or null.</summary> /// first frame can adopt the running bitmap), or null.</summary>
public WriteableBitmap? GetPreviewBitmap(string key) public WriteableBitmap? GetPreviewBitmap(string key)
@@ -213,6 +266,7 @@ public sealed class MediaVideoSourceManager : IDisposable
foreach (var session in sessions) foreach (var session in sessions)
{ {
session.Source.FrameAvailable -= session.FrameHandler; session.Source.FrameAvailable -= session.FrameHandler;
if (session.AudioHandler != null) session.Source.AudioAvailable -= session.AudioHandler;
_ = SafeStopAsync(session.Source); _ = SafeStopAsync(session.Source);
} }
} }
+248
View File
@@ -0,0 +1,248 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using ytLive.Helpers;
using ytLive.Services.Audio;
using ytLive.Services.Encoder;
namespace ytLive.Services;
/// <summary>
/// One video FILE played through ffmpeg — the shared pipe behind every clip, so a media
/// layer (TASK 21) and an alert clip (TASK 47) decode identically. Owns two subprocesses
/// on the same file: video (<c>rawvideo bgra</c>, scaled to the target rect) and audio
/// (<c>f32le -ac 2 -ar 48000</c>, matching the mixer's native format so no resampler is
/// needed). Both are paced to real time, so the picture and the sound stay together even
/// though they arrive on separate pipes.
/// <para><b>Looping is the media case</b>: on natural EOF the whole pass restarts — a fresh
/// pair of processes, because a <see cref="Process"/> cannot be re-started. An alert clip
/// never loops, so its wrapper simply never sets <see cref="Looping"/>.</para>
/// <para>The seams (locator, process factory, frame-rate probe, delay) exist so tests drive
/// a fake decoder with no ffmpeg, no codec, and no audio device.</para>
/// </summary>
public sealed class VideoClipPipe : IDisposable
{
public const int AudioSampleRate = 48000;
public const int AudioChannels = 2;
private const int AudioChunkFloats = 4800; // 50 ms of stereo
public event Action<VideoFrame>? FrameAvailable;
public event Action<AudioSample>? AudioAvailable;
public event Action? Completed;
/// <summary>Restart the pass on natural EOF (media/BRB loops).</summary>
public bool Looping { get; set; }
private readonly string _path;
private readonly int _width;
private readonly int _height;
private readonly IFfmpegLocator _locator;
private readonly Func<IDecodeProcess> _processFactory;
private readonly IFrameRateProbe? _probe;
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
private readonly bool _audio;
private readonly CancellationTokenSource _cts = new();
private IDisposable? _run;
private IDecodeProcess? _activeVideo;
private IDecodeProcess? _activeAudio;
public VideoClipPipe(
string path,
int width,
int height,
IFfmpegLocator locator,
Func<IDecodeProcess> processFactory,
IFrameRateProbe? frameRateProbe = null,
Func<TimeSpan, CancellationToken, Task>? delay = null,
bool audio = true)
{
_path = path;
_width = width;
_height = height;
_locator = locator;
_processFactory = processFactory;
_probe = frameRateProbe;
_delay = delay ?? ((t, ct) => Task.Delay(t, ct));
_audio = audio;
}
public Task StartAsync(CancellationToken cancellationToken = default)
{
if (_run != null) return Task.CompletedTask;
_run = _cts.Token.Register(StopInternal);
_ = Task.Run(RunAsync, CancellationToken.None);
return Task.CompletedTask;
}
public Task StopAsync()
{
_cts.Cancel();
_run = null;
return Task.CompletedTask;
}
public void Dispose()
{
_cts.Cancel();
_cts.Dispose();
_activeVideo?.Dispose();
_activeAudio?.Dispose();
}
private void StopInternal()
{
Kill(_activeVideo);
Kill(_activeAudio);
}
private async Task RunAsync()
{
var ct = _cts.Token;
var frames = 0;
var chunks = 0;
try
{
double? frameRate = null;
if (_probe != null)
frameRate = await _probe.ProbeAsync(_path, ct).ConfigureAwait(false);
var frameDuration = frameRate is > 0 ? TimeSpan.FromSeconds(1.0 / frameRate.Value) : TimeSpan.Zero;
var ffmpegPath = await _locator.LocateAsync(ct).ConfigureAwait(false);
do
{
// Both pipes per pass, awaited together: a restart must begin a new
// video AND a new audio stream, or the picture drifts from the sound
// after the first loop.
var video = RunVideoAsync(ffmpegPath, frameDuration, ct);
var audio = _audio ? RunAudioAsync(ffmpegPath, ct) : Task.FromResult(0);
frames += await video.ConfigureAwait(false);
chunks += await audio.ConfigureAwait(false);
}
while (Looping && !ct.IsCancellationRequested);
}
catch (OperationCanceledException)
{
return;
}
catch (Exception ex)
{
// A failed decode is not fatal: the alert layer falls back to its animations,
// and a media layer just holds its last frame.
AppLog.Write($"Video clip decode failed ({Path.GetFileName(_path)}): {ex.Message}");
return;
}
if (!ct.IsCancellationRequested)
{
AppLog.Write($"Video clip end: {Path.GetFileName(_path)} frames={frames} audioChunks={chunks}");
Completed?.Invoke();
}
}
private async Task<int> RunVideoAsync(string ffmpegPath, TimeSpan frameDuration, CancellationToken ct)
{
var args =
$"-hide_banner -loglevel error -i \"{_path}\" " +
$"-f rawvideo -pix_fmt bgra -vf scale={_width}:{_height} -an -";
using var process = _processFactory();
_activeVideo = process;
process.Start(new ProcessStartInfo
{
FileName = ffmpegPath,
Arguments = args,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
});
var reader = new RawVideoFrameReader(_width, _height);
var buffer = new byte[65536];
var stream = process.StandardOutput;
var frames = 0;
while (!ct.IsCancellationRequested && !process.HasExited)
{
var read = await stream.ReadAsync(buffer, 0, buffer.Length, ct).ConfigureAwait(false);
if (read <= 0) break;
var chunk = new byte[read];
Buffer.BlockCopy(buffer, 0, chunk, 0, read);
foreach (var frame in reader.Feed(chunk))
{
frames++;
FrameAvailable?.Invoke(frame);
if (frameDuration > TimeSpan.Zero)
await _delay(frameDuration, ct).ConfigureAwait(false);
}
}
_activeVideo = null;
return frames;
}
private async Task<int> RunAudioAsync(string ffmpegPath, CancellationToken ct)
{
var args =
$"-hide_banner -loglevel error -i \"{_path}\" " +
$"-f f32le -ac {AudioChannels} -ar {AudioSampleRate} -vn -";
using var process = _processFactory();
_activeAudio = process;
process.Start(new ProcessStartInfo
{
FileName = ffmpegPath,
Arguments = args,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
});
var stream = process.StandardOutput;
var raw = new byte[65536];
var carry = Array.Empty<byte>();
var floatScratch = new float[AudioChunkFloats];
var floatIndex = 0;
var chunks = 0;
// The pipe is byte-delimited only: a read may split a float, so leftover
// bytes are carried into the next read instead of dropped (a dropped
// half-float would desync the entire stream).
while (!ct.IsCancellationRequested && !process.HasExited)
{
var read = await stream.ReadAsync(raw, 0, raw.Length, ct).ConfigureAwait(false);
if (read <= 0) break;
var total = carry.Length + read;
var bytes = total <= raw.Length ? raw : new byte[total];
if (carry.Length > 0)
{
Buffer.BlockCopy(carry, 0, bytes, 0, carry.Length);
Buffer.BlockCopy(raw, 0, bytes, carry.Length, read);
}
var floatsTo = total - (total % 4);
for (var i = 0; i < floatsTo && floatIndex < floatScratch.Length; i += 4)
floatScratch[floatIndex++] = BitConverter.ToSingle(bytes, i);
carry = total > floatsTo ? bytes[floatsTo..total] : Array.Empty<byte>();
while (floatIndex == floatScratch.Length)
{
chunks++;
AudioAvailable?.Invoke(new AudioSample(floatScratch, AudioSampleRate, AudioChannels));
// Pace the NEXT chunk so the clip's audio lands at real-time cadence,
// matching how the video pipe is paced by frame rate.
await _delay(
TimeSpan.FromSeconds(AudioChunkFloats / (double)(AudioSampleRate * AudioChannels)), ct)
.ConfigureAwait(false);
floatScratch = new float[AudioChunkFloats];
floatIndex = 0;
}
}
_activeAudio = null;
return chunks;
}
private static void Kill(IDecodeProcess? process)
{
try { process?.Kill(); } catch { /* already exited */ }
}
}
+13 -13
View File
@@ -7,8 +7,7 @@
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the ### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
reference goes away. **Picture only: the file's own audio is not played anywhere** (the decode pipe reference goes away, and **the file's own soundtrack plays into the stream** (item 10).
passes `-an`), so a video with a soundtrack is silent on stream — the audio route is open, item 10.
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source) 1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
2. ✅ `SourceType.MediaSource` addition to the enum 2. ✅ `SourceType.MediaSource` addition to the enum
@@ -25,13 +24,13 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
session loops if **any** live media reference has it on, because one decoder is shared per path. session loops if **any** live media reference has it on, because one decoder is shared per path.
10. ⏸ File audio + volume — **the file's soundtrack is silent everywhere today**: the decode pipe 10. ✅ File audio + volume — the shared clip pipe decodes the soundtrack (`f32le -ac 2 -ar 48000 -vn`,
passes `-an`, so nothing plays it and the desktop loopback has nothing to carry. Two established 50 ms chunks paced to real time, byte-carry across reads) and the layer carries **Mute** + **Level**.
routes when this is taken: **alert-style** (a second `f32le` pipe into the mixer — The manager applies them per session (one decoder per path, so the last layer touched wins — the
`AlertClipDecoder.RunAudioAsync` is the working model, independent of desktop capture) or alternative would play the same file twice) and the audio joins the DESKTOP/GAME group in
**TRAX-style** (`MusicPlayer` plays it to the default device, so the loopback, the desktop fader `AudioMixer.EnqueueDesktopMediaAudio`, so the desktop fader and the voice ducker behave exactly as
and ducking all apply — at the cost of a second clock to hold in sync with the video). Product they do for game audio and TRAX. Test: `MediaSourceAudioTests` (chunks, level scaling, mute, and the
call; not started, and no volume slider until one of them exists. layer's sound reaching the desktop meter).
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand` 11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
("Change…" in the properties panel), loop toggle, video name display. ("Change…" in the properties panel), loop toggle, video name display.
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3. 12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
@@ -41,10 +40,11 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
### Design decisions ### Design decisions
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project. - **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
- **Audio is NOT wired, and it is not hiding behind the loopback** — `MediaVideoSource` decodes with - **Audio joins the desktop/game group** (creator ruling 2026-11-05) — the clip's sound is decoded by
`-an`, so a video file's soundtrack is silent on stream today (the loopback only carries what the the same pipe as its picture, mixed directly into the stream (not played locally and captured), and
machine actually plays). Item 10 records the two established routes; do not describe media as grouped with game audio so one fader and the ducker cover it. A per-layer Mute/Level sits on top.
audible until one is built. Chosen over TRAX-style local playback because it works regardless of the creator's desktop-capture
state, and because an alert clip needs the same direct route anyway.
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished. - **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene." - **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in - **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
+28 -1
View File
@@ -6,6 +6,7 @@ using System.Windows.Media.Imaging;
using Microsoft.Win32; using Microsoft.Win32;
using ytLive.Models; using ytLive.Models;
using ytLive.Services; using ytLive.Services;
using ytLive.Services.Audio;
namespace ytLive.ViewModels; namespace ytLive.ViewModels;
@@ -105,6 +106,11 @@ public partial class MainViewModel
{ {
_mediaManager.SetLooping(path, AnyMediaReferenceLoops(path)); _mediaManager.SetLooping(path, AnyMediaReferenceLoops(path));
} }
else if (e.PropertyName is nameof(Source.MediaVolume) or nameof(Source.MediaMuted)
&& source.MediaPath is { } audioPath)
{
ApplyMediaAudioSettings(audioPath);
}
} }
_mediaHandlers[source.Id] = OnSourceChanged; _mediaHandlers[source.Id] = OnSourceChanged;
@@ -121,7 +127,11 @@ public partial class MainViewModel
private void EnsureMediaSession(string? path) private void EnsureMediaSession(string? path)
{ {
if (string.IsNullOrWhiteSpace(path)) return; if (string.IsNullOrWhiteSpace(path)) return;
if (_mediaManager.HasSession(path)) return; if (_mediaManager.HasSession(path))
{
ApplyMediaAudioSettings(path);
return;
}
_ = AcquireMediaAsync(path); _ = AcquireMediaAsync(path);
} }
@@ -133,8 +143,25 @@ public partial class MainViewModel
// case the bitmap is already published and will not be raised again for us. // case the bitmap is already published and will not be raised again for us.
if (_mediaManager.GetPreviewBitmap(path) is { } bitmap) if (_mediaManager.GetPreviewBitmap(path) is { } bitmap)
OnMediaPreviewBitmapChanged(path, bitmap); OnMediaPreviewBitmapChanged(path, bitmap);
ApplyMediaAudioSettings(path);
} }
/// <summary>The file's sound controls: one decoder is shared per path, so the level and
/// mute are per session — the layer that was last touched is the one that wins.</summary>
private void ApplyMediaAudioSettings(string path)
{
var source = Scenes.SelectMany(s => s.Elements).OfType<Source>()
.FirstOrDefault(s => s.IsMediaSource && s.MediaPath == path);
if (source == null) return;
_mediaManager.SetAudioSettings(path, source.MediaVolume, source.MediaMuted);
}
/// <summary>A clip's own soundtrack joins the desktop/game group, so the desktop fader and
/// the voice ducker treat it exactly like game audio (TASK 21 audio slice).</summary>
private void OnMediaAudioAvailable(string path, AudioSample sample)
=> _audioMixer.EnqueueDesktopMediaAudio(sample);
/// <summary>Shared sessions or not, the loop state is the path's: any live media /// <summary>Shared sessions or not, the loop state is the path's: any live media
/// reference asking for a loop makes the one decoder loop (recorded rule, TASK 21).</summary> /// reference asking for a loop makes the one decoder loop (recorded rule, TASK 21).</summary>
private bool AnyMediaReferenceLoops(string path) private bool AnyMediaReferenceLoops(string path)
+1
View File
@@ -412,6 +412,7 @@ public partial class MainViewModel : ViewModelBase
AnyMediaReferenceLoops); AnyMediaReferenceLoops);
_mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged; _mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged;
_mediaManager.MediaFailed += OnMediaFailed; _mediaManager.MediaFailed += OnMediaFailed;
_mediaManager.AudioAvailable += OnMediaAudioAvailable;
_chatLayer = new ChatOverlayLayer(new ChatBoxRenderer()); _chatLayer = new ChatOverlayLayer(new ChatBoxRenderer());
_alertLayer = new AlertOverlayLayer( _alertLayer = new AlertOverlayLayer(
+6 -3
View File
@@ -799,9 +799,12 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before `Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after `StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
the layer no longer references it. **A video file's own soundtrack is silent today** — the decode pipe the layer no longer references it. **Sound ships with it (2026-11-05):** both `MediaVideoSource` and
is `-an`, so nothing plays it and the loopback has nothing to carry; wiring it (alert-style mixer feed `AlertClipDecoder` are now thin wrappers over one `Services/VideoClipPipe.cs` (video pipe + `f32le`
or TRAX-style local playback) is open, and no volume slider exists until then. audio pipe, both real-time paced, loop restarts both), so a clip is a clip. The manager applies the
layer's Level/Mute per session and raises `AudioAvailable`, and `MainViewModel` feeds it to
`AudioMixer.EnqueueDesktopMediaAudio` — the DESKTOP/GAME ring, so the desktop fader and the voice
ducker apply. The alert clip keeps its own unity-gain ring (`EnqueueAlertAudio`, never ducked).
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional - **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs` mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
+7
View File
@@ -250,6 +250,13 @@ devices, no timers).
`StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) — `StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) —
the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the
pipe lifecycle. pipe lifecycle.
**Two clip-audio rings join that mix (2026-11-05):** `EnqueueAlertAudio` (TASK 47) is pre-scaled by the
alert layer and mixed at **unity, never ducked** — an alert must be heard over the mic.
`EnqueueDesktopMediaAudio` (TASK 21's audio slice) is the opposite on purpose: a video layer's
soundtrack joins the **desktop/game ring**, so the desktop fader and the auto-ducker apply exactly as
they do for game audio and TRAX. Both are fed by the shared `Services/VideoClipPipe.cs`; the layer's
own Mute/Level are applied upstream by `MediaVideoSourceManager` (level/mute are per **session**, since
one decoder is shared per file path).
- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo - **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo
mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose
offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500 offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500
+5 -6
View File
@@ -97,12 +97,11 @@ published decisions (recorded in `TASKS/task-43-native-alerts.md`):
`AlertClipDecoder.RunVideoAsync` and `MediaVideoSource.RunAsync` are near-duplicates — `AlertClipDecoder.RunVideoAsync` and `MediaVideoSource.RunAsync` are near-duplicates —
same ffmpeg args (`-f rawvideo -pix_fmt bgra -vf scale=W:H -an -`), same same ffmpeg args (`-f rawvideo -pix_fmt bgra -vf scale=W:H -an -`), same
`RawVideoFrameReader` feed loop, same `IFrameRateProbe` + delay pacing, both behind `RawVideoFrameReader` feed loop, same `IFrameRateProbe` + delay pacing, both behind
`IDecodeProcess`. What differs is ownership and role, not decoding: alerts are per-play `IDecodeProcess`. Consolidated 2026-11-05 with the media audio slice: both are now thin wrappers over
and disposed at drain, media is a refcounted per-path session; alerts also run an one `Services/VideoClipPipe.cs` (video pipe + `f32le` audio pipe, real-time paced, loop restarts both),
`f32le` audio pipe into the mixer, media runs none at all (`-an`) — so a media file's so decode behaviour cannot drift between them again. What still differs is ownership and role: alerts
soundtrack is silent today, and task-21 item 10 holds the two routes for changing that; are per-play and disposed at drain, media is a refcounted per-path session; alerts play once, media
alerts play once, media loops. A shared raw-video pump loops; alert audio is mixed at unity and never ducked, media audio joins the desktop/game group.
is the obvious consolidation if a third consumer appears — do not do it for two.
Fades in/out via **`FadeDurationSeconds = 0.30`** Fades in/out via **`FadeDurationSeconds = 0.30`**
(`AlertOverlayLayer` const): fade-in by `elapsed` on straight-alpha frame copies; (`AlertOverlayLayer` const): fade-in by `elapsed` on straight-alpha frame copies;
EOF → freeze-frame → fade-out → drain (the clip branch of `Advance` **clears EOF → freeze-frame → fade-out → drain (the clip branch of `Advance` **clears
@@ -100,6 +100,7 @@ public sealed class MediaSourceAcquisitionTests
public int StopCount { get; private set; } public int StopCount { get; private set; }
public event Action<VideoFrame>? FrameAvailable; public event Action<VideoFrame>? FrameAvailable;
public event Action<ytLive.Services.Audio.AudioSample>? AudioAvailable { add { } remove { } }
public event Action? Completed { add { } remove { } } public event Action? Completed { add { } remove { } }
public Task StartAsync(CancellationToken cancellationToken = default) public Task StartAsync(CancellationToken cancellationToken = default)
+191
View File
@@ -0,0 +1,191 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Data.Sqlite;
using Xunit;
using ytLive.Services;
using ytLive.Services.Audio;
using ytLive.Services.Encoder;
using ytLive.ViewModels;
namespace ytLive.Tests;
/// <summary>
/// TASK 21 audio slice (2026-11-05): a video file's own soundtrack used to be thrown away
/// (`-an` on the decode pipe), so a "starting soon" clip with music played silent. This pins
/// the replacement: the shared clip pipe decodes the sound, the manager applies the layer's
/// level/mute, and a media layer's audio lands in the mixer's desktop/game group.
/// </summary>
[Collection("RealApp")]
public sealed class MediaSourceAudioTests
{
private readonly RealAppHost _app;
public MediaSourceAudioTests(RealAppHost app) => _app = app;
// ─── the pipe decodes the soundtrack ───
[Fact]
public async Task ClipPipe_EmitsTheSoundtrackInPacedChunks()
{
// 4800 stereo floats = one 50 ms chunk.
var audioBytes = new byte[4800 * 4];
for (var i = 0; i < 4800; i++)
BitConverter.TryWriteBytes(audioBytes.AsSpan(i * 4, 4), 0.5f);
using var pipe = new VideoClipPipe(
"clip.mp4", 1, 1,
new FakeLocator(),
() => new FakeDecodeProcess(args => args.Contains("-vn") ? audioBytes : Array.Empty<byte>()),
delay: (_, _) => Task.CompletedTask);
var chunks = new List<AudioSample>();
var completed = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
pipe.AudioAvailable += sample => chunks.Add(sample);
pipe.Completed += () => completed.TrySetResult(true);
await pipe.StartAsync();
await completed.Task.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Single(chunks);
Assert.Equal(4800, chunks[0].Samples.Length);
Assert.Equal(48000, chunks[0].SampleRate);
Assert.Equal(2, chunks[0].Channels);
}
// ─── the manager applies the layer's controls ───
[Fact]
public async Task Manager_ScalesTheSoundtrackByTheSessionsLevel_AndDropsItWhenMuted()
{
var source = new FakeMediaSource("clip.mp4");
var manager = new MediaVideoSourceManager(_ => source, uiDispatcher: null);
var heard = new List<AudioSample>();
manager.AudioAvailable += (_, sample) => heard.Add(sample);
Assert.True(await manager.AcquireAsync("clip.mp4"));
manager.SetAudioSettings("clip.mp4", 0.5, muted: false);
source.EmitAudio(new AudioSample(new[] { 1.0f, -1.0f }, 48000, 2));
Assert.Single(heard);
Assert.Equal(0.5f, heard[0].Samples[0], 3);
Assert.Equal(-0.5f, heard[0].Samples[1], 3);
manager.SetAudioSettings("clip.mp4", 1.0, muted: true);
source.EmitAudio(new AudioSample(new[] { 1.0f, -1.0f }, 48000, 2));
Assert.Single(heard); // muted: nothing reaches the mixer
manager.Dispose();
}
// ─── the layer's audio reaches the desktop/game group ───
[Fact]
public void MediaLayerSound_ReachesTheDesktopAudioGroup()
{
_app.Run(() =>
{
var tempDb = Path.Combine(Path.GetTempPath(), $"ytLlive-test-{Guid.NewGuid():N}.db");
var videoPath = Path.Combine(Path.GetTempPath(), "brb-bed.mp4");
var decoders = new List<FakeMediaSource>();
MainViewModel.LayoutPathOverride = tempDb;
MainViewModel.MediaFrameSourceFactoryOverride = key =>
{
var decoder = new FakeMediaSource(key);
decoders.Add(decoder);
return decoder;
};
var window = new MainWindow();
var vm = (MainViewModel)window.DataContext;
try
{
window.Show();
window.UpdateLayout();
vm.AddMediaSource(videoPath);
var decoder = decoders.Single();
// The desktop/game meter is what the creator watches. It has to move for a
// media layer's sound, or the soundtrack is in the mix but invisible.
Assert.Equal(0.0, vm.GameAudioLevel);
var loud = Enumerable.Repeat(0.8f, 9600).ToArray();
for (var i = 0; i < 6; i++)
decoder.EmitAudio(new AudioSample(loud, 48000, 2));
Assert.True(vm.GameAudioLevel > 0.05,
$"the media soundtrack never reached the desktop group (level {vm.GameAudioLevel})");
vm.StagedScene!.Elements.OfType<ytLive.Models.Source>()
.Single(s => s.IsMediaSource).MediaMuted = true;
var afterMute = vm.GameAudioLevel;
decoder.EmitAudio(new AudioSample(loud, 48000, 2));
Assert.Equal(afterMute, vm.GameAudioLevel); // muted layer stops feeding the meter
}
finally
{
window.Close();
MainViewModel.MediaFrameSourceFactoryOverride = null;
MainViewModel.LayoutPathOverride = null;
SqliteConnection.ClearAllPools();
try { File.Delete(tempDb); } catch { /* best-effort cleanup */ }
}
});
}
private sealed class FakeMediaSource : IMediaFrameSource
{
public FakeMediaSource(string key) => Key = key;
public string Key { get; }
public bool Looping { get; set; }
public event Action<VideoFrame>? FrameAvailable { add { } remove { } }
public event Action<AudioSample>? AudioAvailable;
public event Action? Completed { add { } remove { } }
public void EmitAudio(AudioSample sample) => AudioAvailable?.Invoke(sample);
public Task StartAsync(CancellationToken cancellationToken = default) => Task.CompletedTask;
public Task StopAsync() => Task.CompletedTask;
public void Dispose()
{
}
}
private sealed class FakeDecodeProcess : IDecodeProcess
{
private readonly Func<string, byte[]> _pick;
private readonly MemoryStream _stream = new();
public FakeDecodeProcess(Func<string, byte[]> pick) => _pick = pick;
public void Start(ProcessStartInfo startInfo)
{
// The pipe runs a video process and an audio process; the payload depends on
// which one this is, so the fake reads the arguments.
var bytes = _pick(startInfo.Arguments);
_stream.SetLength(0);
_stream.Write(bytes, 0, bytes.Length);
_stream.Position = 0;
}
public Stream StandardOutput => _stream;
public bool HasExited => _stream.Position >= _stream.Length;
public int ExitCode => 0;
public void Kill() { }
public Task WaitForExitAsync(CancellationToken ct = default) => Task.CompletedTask;
public void Dispose() => _stream.Dispose();
}
private sealed class FakeLocator : IFfmpegLocator
{
public Task<string> LocateAsync(CancellationToken ct = default) => Task.FromResult("ffmpeg.exe");
}
}
@@ -3,6 +3,7 @@ using System.Windows.Media.Imaging;
using System.Windows.Threading; using System.Windows.Threading;
using Xunit; using Xunit;
using ytLive.Services; using ytLive.Services;
using ytLive.Services.Audio;
namespace ytLive.Tests; namespace ytLive.Tests;
@@ -22,6 +23,7 @@ public class MediaVideoSourceManagerTests
public string Key { get; } public string Key { get; }
public bool Looping { get; set; } public bool Looping { get; set; }
public event Action<VideoFrame>? FrameAvailable; public event Action<VideoFrame>? FrameAvailable;
public event Action<AudioSample>? AudioAvailable;
public event Action? Completed; public event Action? Completed;
public FakeMediaSource(string key, List<string>? started = null, List<string>? stopped = null) public FakeMediaSource(string key, List<string>? started = null, List<string>? stopped = null)
@@ -44,6 +46,7 @@ public class MediaVideoSourceManagerTests
} }
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame); public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
public void EmitAudio(AudioSample sample) => AudioAvailable?.Invoke(sample);
public void Complete() => Completed?.Invoke(); public void Complete() => Completed?.Invoke();
public void Dispose() { } public void Dispose() { }
} }
@@ -53,12 +56,14 @@ public class MediaVideoSourceManagerTests
public string Key { get; } public string Key { get; }
public bool Looping { get; set; } public bool Looping { get; set; }
public event Action<VideoFrame>? FrameAvailable; public event Action<VideoFrame>? FrameAvailable;
public event Action<AudioSample>? AudioAvailable;
public event Action? Completed; public event Action? Completed;
public FailingMediaSource(string key) => Key = key; public FailingMediaSource(string key) => Key = key;
public Task StartAsync(CancellationToken cancellationToken = default) public Task StartAsync(CancellationToken cancellationToken = default)
=> throw new InvalidOperationException("access denied"); => throw new InvalidOperationException("access denied");
public Task StopAsync() => Task.CompletedTask; public Task StopAsync() => Task.CompletedTask;
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame); public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
public void EmitAudio(AudioSample sample) => AudioAvailable?.Invoke(sample);
public void Complete() => Completed?.Invoke(); public void Complete() => Completed?.Invoke();
public void Dispose() { } public void Dispose() { }
} }