feat(media): play the clip's soundtrack and share one decode pipe
A video file was picture-only: the decode pipe passed `-an`, so a starting-soon or BRB clip with music played silent everywhere. Adds sound to every clip and collapses the two decoders into one. - `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21) and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio (`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to real time, loop restarts both so picture and sound stay together. Alert behaviour is unchanged. - `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises `AudioAvailable`; `MainViewModel` feeds it to the new `AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the desktop fader and the mic auto-ducker apply exactly as they do for game audio and TRAX (creator ruling 2026-11-05: audio joins the group). - Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT). - Level/mute are per SESSION because one decoder is shared per file path — the last layer touched wins; one decoder per layer would play the file twice. Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the manager scales by level and drops when muted, and a media layer's sound moves the desktop meter. Alert/decode suites unchanged and green. Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed; the two documented pre-existing failures only).
This commit is contained in:
+14
-1
@@ -746,7 +746,20 @@
|
|||||||
<CheckBox Style="{StaticResource PropertyPillCheck}" Margin="10,4,0,4"
|
<CheckBox Style="{StaticResource PropertyPillCheck}" Margin="10,4,0,4"
|
||||||
Content="Loop" IsChecked="{Binding SelectedElement.MediaIsLooping, Mode=TwoWay}"
|
Content="Loop" IsChecked="{Binding SelectedElement.MediaIsLooping, Mode=TwoWay}"
|
||||||
ToolTip="Restart from the beginning at the end of the file — for starting-soon and BRB loops"/>
|
ToolTip="Restart from the beginning at the end of the file — for starting-soon and BRB loops"/>
|
||||||
<TextBlock Text="Picture only — a video's own sound is not played yet."
|
<DockPanel Margin="10,0,0,4">
|
||||||
|
<CheckBox DockPanel.Dock="Left" Style="{StaticResource PropertyPillCheck}"
|
||||||
|
Content="Mute" IsChecked="{Binding SelectedElement.MediaMuted, Mode=TwoWay}"
|
||||||
|
ToolTip="Silence this layer's sound. The picture keeps playing."/>
|
||||||
|
</DockPanel>
|
||||||
|
<DockPanel Margin="10,0,0,4">
|
||||||
|
<TextBlock DockPanel.Dock="Left" Text="Level" Style="{StaticResource PropertyFieldLabel}" Width="40"/>
|
||||||
|
<TextBlock DockPanel.Dock="Right" Text="{Binding SelectedElement.MediaVolume, StringFormat={}{0:P0}}"
|
||||||
|
Style="{StaticResource PropertyFieldLabel}" Width="34" TextAlignment="Right"/>
|
||||||
|
<Slider Minimum="0" Maximum="1" Width="120"
|
||||||
|
Value="{Binding SelectedElement.MediaVolume, Mode=TwoWay}"
|
||||||
|
VerticalAlignment="Center"/>
|
||||||
|
</DockPanel>
|
||||||
|
<TextBlock Text="Sound plays through the desktop audio controls, like game audio."
|
||||||
Style="{StaticResource PropertyHint}"/>
|
Style="{StaticResource PropertyHint}"/>
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
|
|
||||||
|
|||||||
+12
-10
@@ -1,20 +1,21 @@
|
|||||||
# HANDOFF — 2026-11-05 (TASK 21 — video file layer)
|
# HANDOFF — 2026-11-05 (TASK 21 — video file layer + its audio)
|
||||||
|
|
||||||
## State
|
## State
|
||||||
|
|
||||||
- `main`; TASK 21 is complete: a video file added from the (+) menu decodes, composites into preview
|
- `main`; TASK 21 is complete. A video file added from the (+) menu decodes, composites into preview and
|
||||||
and stream, loops on toggle, and stops with its last scene reference. 3·17 (text layers) completed
|
stream, loops on toggle, and **plays its own soundtrack** (Mute + Level per layer, mixed into the
|
||||||
2026-11-04.
|
desktop/game group so the desktop fader and the ducker apply). Removing it stops the decoder.
|
||||||
- New test `MediaSourceAcquisitionTests` (add → one decoder starts + frames reach the layer + loop flip
|
- Decoding is now ONE pipe: `Services/VideoClipPipe.cs` behind both `MediaVideoSource` (TASK 21) and
|
||||||
reaches the decoder + remove → stops). App build is 0 warnings.
|
`AlertClipDecoder` (TASK 47) — "a clip is a clip". Alert behaviour is unchanged and its tests stay green.
|
||||||
- **Media is picture-only:** `MediaVideoSource` decodes with `-an`, so a video file's own soundtrack
|
- Tests: `MediaSourceAcquisitionTests` (add → decoder starts, frames, loop, remove) and
|
||||||
is silent on stream — the loopback does not carry it because nothing plays it. Two routes recorded
|
`MediaSourceAudioTests` (chunks, level scaling, mute, sound reaching the desktop meter). App build 0 warnings.
|
||||||
in `task-21` item 10 (alert-style mixer feed vs TRAX-style local play); no volume slider until one exists.
|
|
||||||
|
|
||||||
## Next action
|
## Next action
|
||||||
|
|
||||||
- Next queued unit: TASK 49 (chat profanity filter). Day 3's TASK 32/33 resilience slices are still
|
- Next queued unit: TASK 49 (chat profanity filter). Day 3's TASK 32/33 resilience slices are still
|
||||||
outstanding — 32·4 (fragmented MP4) is the smallest.
|
outstanding — 32·4 (fragmented MP4) is the smallest.
|
||||||
|
- Agreed next feature work (designed, not started): green/blue screen detection + keying through the clip
|
||||||
|
pipe — suggest-only by default, an opt-in "Auto Green/Blue Screen" setting, alert clips included.
|
||||||
- Do not push absent a checkpoint request.
|
- Do not push absent a checkpoint request.
|
||||||
|
|
||||||
## Landmines
|
## Landmines
|
||||||
@@ -23,5 +24,6 @@
|
|||||||
mouse input. Do not run it while the desktop is in use.
|
mouse input. Do not run it while the desktop is in use.
|
||||||
- Integration tests constructing `MainWindow` must set a temporary `LayoutPathOverride` and clear it in `finally`.
|
- Integration tests constructing `MainWindow` must set a temporary `LayoutPathOverride` and clear it in `finally`.
|
||||||
- Media sessions are keyed per **path**, not per layer: acquire through `EnsureMediaSession`, release
|
- Media sessions are keyed per **path**, not per layer: acquire through `EnsureMediaSession`, release
|
||||||
through `ReleaseMediaIfUnused`, never `AcquireAsync` twice for one path.
|
through `ReleaseMediaIfUnused`, never `AcquireAsync` twice for one path. Level/mute are therefore
|
||||||
|
per session too.
|
||||||
- Bundled FFmpeg stays outside the single-file exe (`ExcludeFromSingleFile="true"`).
|
- Bundled FFmpeg stays outside the single-file exe (`ExcludeFromSingleFile="true"`).
|
||||||
|
|||||||
@@ -264,6 +264,15 @@ public class Source : SceneElement
|
|||||||
set => Set(ref _mediaVolume, Math.Clamp(value, 0.0, 1.0));
|
set => Set(ref _mediaVolume, Math.Clamp(value, 0.0, 1.0));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private bool _mediaMuted;
|
||||||
|
/// <summary>Silences this layer's soundtrack (the picture keeps playing). One decoder is
|
||||||
|
/// shared per file, so the level/mute are the file's — the layer is the control.</summary>
|
||||||
|
public bool MediaMuted
|
||||||
|
{
|
||||||
|
get => _mediaMuted;
|
||||||
|
set => Set(ref _mediaMuted, value);
|
||||||
|
}
|
||||||
|
|
||||||
private MediaPlaybackState _mediaPlaybackState = MediaPlaybackState.Stopped;
|
private MediaPlaybackState _mediaPlaybackState = MediaPlaybackState.Stopped;
|
||||||
public MediaPlaybackState MediaPlaybackState
|
public MediaPlaybackState MediaPlaybackState
|
||||||
{
|
{
|
||||||
|
|||||||
+16
-176
@@ -1,6 +1,5 @@
|
|||||||
using System.Diagnostics;
|
using System;
|
||||||
using System.IO;
|
using System.Threading;
|
||||||
using ytLive.Helpers;
|
|
||||||
using ytLive.Services.Audio;
|
using ytLive.Services.Audio;
|
||||||
using ytLive.Services.Encoder;
|
using ytLive.Services.Encoder;
|
||||||
|
|
||||||
@@ -11,10 +10,10 @@ namespace ytLive.Services;
|
|||||||
/// same file, both paced to real time. The alert box owns a decoder per event
|
/// same file, both paced to real time. The alert box owns a decoder per event
|
||||||
/// (start at play, dispose at drain) — unlike MediaSource's app-wide refcounted
|
/// (start at play, dispose at drain) — unlike MediaSource's app-wide refcounted
|
||||||
/// session, an alert REPLAYS the same clip each event, so a fresh decode per
|
/// session, an alert REPLAYS the same clip each event, so a fresh decode per
|
||||||
/// play is required. Video is rawvideo BGRA scaled to the box rect (the
|
/// play is required. The fade is applied by the alert layer, not here.
|
||||||
/// <see cref="MediaVideoSource"/> shape); audio is ffmpeg <c>f32le</c> at the
|
/// <para>Since TASK 21's audio slice this is a thin wrapper over the shared
|
||||||
/// mixer's native 48 kHz stereo (so the mixer ring needs no resampler). The
|
/// <see cref="VideoClipPipe"/> (the same pipe a media layer uses); the alert case
|
||||||
/// fade is applied by the alert layer, not here.
|
/// simply never sets <c>Looping</c>.</para>
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public interface IAlertClipDecoder : IDisposable
|
public interface IAlertClipDecoder : IDisposable
|
||||||
{
|
{
|
||||||
@@ -35,30 +34,14 @@ public interface IAlertClipDecoder : IDisposable
|
|||||||
event Action? Completed;
|
event Action? Completed;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Default <see cref="IAlertClipDecoder"/>: two ffmpeg subprocesses on
|
/// <summary>Default <see cref="IAlertClipDecoder"/>: the shared clip pipe with looping off.</summary>
|
||||||
/// the same file. Video via <see cref="RawVideoFrameReader"/> on a <c>rawvideo
|
|
||||||
/// bgra</c> pipe (-an), audio via an <c>f32le -ac 2 -ar 48000</c> pipe (-vn).
|
|
||||||
/// The seams (locator, process factory, frame-rate probe, delay) let tests drive
|
|
||||||
/// a fake decoder without a real codec or Windows.</summary>
|
|
||||||
public sealed class AlertClipDecoder : IAlertClipDecoder
|
public sealed class AlertClipDecoder : IAlertClipDecoder
|
||||||
{
|
{
|
||||||
public event Action<VideoFrame>? FrameAvailable;
|
public event Action<VideoFrame>? FrameAvailable;
|
||||||
public event Action<AudioSample>? AudioReady;
|
public event Action<AudioSample>? AudioReady;
|
||||||
public event Action? Completed;
|
public event Action? Completed;
|
||||||
|
|
||||||
private const int AudioSampleRate = 48000;
|
private readonly VideoClipPipe _clip;
|
||||||
private const int AudioChannels = 2;
|
|
||||||
private const int AudioChunkFloats = 4800; // 50ms of stereo
|
|
||||||
|
|
||||||
private readonly string _path;
|
|
||||||
private readonly int _width;
|
|
||||||
private readonly int _height;
|
|
||||||
private readonly IFfmpegLocator _locator;
|
|
||||||
private readonly Func<IDecodeProcess> _processFactory;
|
|
||||||
private readonly IFrameRateProbe? _probe;
|
|
||||||
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
|
|
||||||
private readonly CancellationTokenSource _cts = new();
|
|
||||||
private Task? _run;
|
|
||||||
|
|
||||||
public AlertClipDecoder(
|
public AlertClipDecoder(
|
||||||
string path,
|
string path,
|
||||||
@@ -69,159 +52,16 @@ public sealed class AlertClipDecoder : IAlertClipDecoder
|
|||||||
IFrameRateProbe? frameRateProbe = null,
|
IFrameRateProbe? frameRateProbe = null,
|
||||||
Func<TimeSpan, CancellationToken, Task>? delay = null)
|
Func<TimeSpan, CancellationToken, Task>? delay = null)
|
||||||
{
|
{
|
||||||
_path = path;
|
_clip = new VideoClipPipe(
|
||||||
_width = width;
|
path, width, height, locator, processFactory, frameRateProbe, delay, audio: true);
|
||||||
_height = height;
|
_clip.FrameAvailable += frame => FrameAvailable?.Invoke(frame);
|
||||||
_locator = locator;
|
_clip.AudioAvailable += sample => AudioReady?.Invoke(sample);
|
||||||
_processFactory = processFactory;
|
_clip.Completed += () => Completed?.Invoke();
|
||||||
_probe = frameRateProbe;
|
|
||||||
_delay = delay ?? ((t, ct) => Task.Delay(t, ct));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public void Start()
|
public void Start() => _ = _clip.StartAsync();
|
||||||
{
|
|
||||||
if (_run != null) return;
|
|
||||||
_run = Task.Run(RunAsync, CancellationToken.None);
|
|
||||||
}
|
|
||||||
|
|
||||||
public void Stop() => _cts.Cancel();
|
public void Stop() => _ = _clip.StopAsync();
|
||||||
|
|
||||||
public void Dispose()
|
public void Dispose() => _clip.Dispose();
|
||||||
{
|
|
||||||
_cts.Cancel();
|
|
||||||
_cts.Dispose();
|
|
||||||
}
|
|
||||||
|
|
||||||
private async Task RunAsync()
|
|
||||||
{
|
|
||||||
var ct = _cts.Token;
|
|
||||||
var videoFrames = 0;
|
|
||||||
var audioChunks = 0;
|
|
||||||
var failed = false;
|
|
||||||
try
|
|
||||||
{
|
|
||||||
double? frameRate = null;
|
|
||||||
if (_probe != null)
|
|
||||||
frameRate = await _probe.ProbeAsync(_path, ct).ConfigureAwait(false);
|
|
||||||
var frameDuration = frameRate is > 0 ? TimeSpan.FromSeconds(1.0 / frameRate.Value) : TimeSpan.Zero;
|
|
||||||
|
|
||||||
var ffmpegPath = await _locator.LocateAsync(ct).ConfigureAwait(false);
|
|
||||||
|
|
||||||
var videoTask = RunVideoAsync(ffmpegPath, frameDuration, ct);
|
|
||||||
var audioTask = RunAudioAsync(ffmpegPath, ct);
|
|
||||||
videoFrames = await videoTask.ConfigureAwait(false);
|
|
||||||
audioChunks = await audioTask.ConfigureAwait(false);
|
|
||||||
}
|
|
||||||
catch (OperationCanceledException)
|
|
||||||
{
|
|
||||||
failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
catch (Exception ex)
|
|
||||||
{
|
|
||||||
// A failed decode is not fatal: the alert layer falls back to the six
|
|
||||||
// animations (or the frame it already holds) exactly as a missing file.
|
|
||||||
// Logged so a silent blank box traces back to a decode failure.
|
|
||||||
failed = true;
|
|
||||||
AppLog.Write(ex, $"Alert clip decode failed ({Path.GetFileName(_path)})", AppLogErrorClass.Class2);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!ct.IsCancellationRequested)
|
|
||||||
{
|
|
||||||
AppLog.Write($"Alert clip end: {Path.GetFileName(_path)} frames={videoFrames} audioChunks={audioChunks} failed={failed}");
|
|
||||||
Completed?.Invoke();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private async Task<int> RunVideoAsync(string ffmpegPath, TimeSpan frameDuration, CancellationToken ct)
|
|
||||||
{
|
|
||||||
var args =
|
|
||||||
$"-hide_banner -loglevel error -i \"{_path}\" " +
|
|
||||||
$"-f rawvideo -pix_fmt bgra -vf scale={_width}:{_height} -an -";
|
|
||||||
using var process = _processFactory();
|
|
||||||
process.Start(new ProcessStartInfo
|
|
||||||
{
|
|
||||||
FileName = ffmpegPath,
|
|
||||||
Arguments = args,
|
|
||||||
UseShellExecute = false,
|
|
||||||
RedirectStandardOutput = true,
|
|
||||||
RedirectStandardError = true,
|
|
||||||
CreateNoWindow = true,
|
|
||||||
});
|
|
||||||
|
|
||||||
var reader = new RawVideoFrameReader(_width, _height);
|
|
||||||
var buffer = new byte[65536];
|
|
||||||
var stream = process.StandardOutput;
|
|
||||||
var frames = 0;
|
|
||||||
while (!ct.IsCancellationRequested && !process.HasExited)
|
|
||||||
{
|
|
||||||
var read = await stream.ReadAsync(buffer, 0, buffer.Length, ct).ConfigureAwait(false);
|
|
||||||
if (read <= 0) break;
|
|
||||||
var chunk = new byte[read];
|
|
||||||
Buffer.BlockCopy(buffer, 0, chunk, 0, read);
|
|
||||||
foreach (var frame in reader.Feed(chunk))
|
|
||||||
{
|
|
||||||
frames++;
|
|
||||||
FrameAvailable?.Invoke(frame);
|
|
||||||
if (frameDuration > TimeSpan.Zero)
|
|
||||||
await _delay(frameDuration, ct).ConfigureAwait(false);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return frames;
|
|
||||||
}
|
|
||||||
|
|
||||||
private async Task<int> RunAudioAsync(string ffmpegPath, CancellationToken ct)
|
|
||||||
{
|
|
||||||
var args =
|
|
||||||
$"-hide_banner -loglevel error -i \"{_path}\" " +
|
|
||||||
$"-f f32le -ac {AudioChannels} -ar {AudioSampleRate} -vn -";
|
|
||||||
using var process = _processFactory();
|
|
||||||
process.Start(new ProcessStartInfo
|
|
||||||
{
|
|
||||||
FileName = ffmpegPath,
|
|
||||||
Arguments = args,
|
|
||||||
UseShellExecute = false,
|
|
||||||
RedirectStandardOutput = true,
|
|
||||||
RedirectStandardError = true,
|
|
||||||
CreateNoWindow = true,
|
|
||||||
});
|
|
||||||
|
|
||||||
var stream = process.StandardOutput;
|
|
||||||
var raw = new byte[65536];
|
|
||||||
var carry = new byte[0];
|
|
||||||
var floatScratch = new float[AudioChunkFloats];
|
|
||||||
var floatIndex = 0;
|
|
||||||
var chunks = 0;
|
|
||||||
// The pipe is byte-delimited only: a read may split a float, so leftover
|
|
||||||
// bytes are carried into the next read instead of dropped (a dropped
|
|
||||||
// half-float would desync the entire stream).
|
|
||||||
while (!ct.IsCancellationRequested && !process.HasExited)
|
|
||||||
{
|
|
||||||
var read = await stream.ReadAsync(raw, 0, raw.Length, ct).ConfigureAwait(false);
|
|
||||||
if (read <= 0) break;
|
|
||||||
var total = carry.Length + read;
|
|
||||||
var bytes = total <= raw.Length ? raw : new byte[total];
|
|
||||||
if (carry.Length > 0)
|
|
||||||
{
|
|
||||||
Buffer.BlockCopy(carry, 0, bytes, 0, carry.Length);
|
|
||||||
Buffer.BlockCopy(raw, 0, bytes, carry.Length, read);
|
|
||||||
}
|
|
||||||
var floatsTo = total - (total % 4);
|
|
||||||
for (var i = 0; i < floatsTo && floatIndex < floatScratch.Length; i += 4)
|
|
||||||
floatScratch[floatIndex++] = BitConverter.ToSingle(bytes, i);
|
|
||||||
carry = total > floatsTo ? bytes[floatsTo..total] : new byte[0];
|
|
||||||
|
|
||||||
while (floatIndex == floatScratch.Length)
|
|
||||||
{
|
|
||||||
chunks++;
|
|
||||||
AudioReady?.Invoke(new AudioSample(floatScratch, AudioSampleRate, AudioChannels));
|
|
||||||
// Pace the NEXT chunk so the whole clip's audio lands at real-time
|
|
||||||
// cadence (the fade in the alert layer tracks the same wall clock).
|
|
||||||
await _delay(TimeSpan.FromSeconds(AudioChunkFloats / (double)(AudioSampleRate * AudioChannels)), ct).ConfigureAwait(false);
|
|
||||||
floatScratch = new float[AudioChunkFloats];
|
|
||||||
floatIndex = 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return chunks;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -283,6 +283,25 @@ public sealed class AudioMixer : IDisposable
|
|||||||
LoopbackLevelChanged?.Invoke(level);
|
LoopbackLevelChanged?.Invoke(level);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Drops a MEDIA clip's audio into the desktop/game ring (TASK 21's audio slice).
|
||||||
|
/// It joins that group on purpose: the desktop fader and the mic auto-ducker then apply
|
||||||
|
/// exactly as they do to game audio and TRAX, so a BRB bed ducks under the creator's
|
||||||
|
/// voice without a second audio knob. The layer's own level/mute are applied upstream
|
||||||
|
/// by <see cref="MediaVideoSourceManager"/>.</summary>
|
||||||
|
public void EnqueueDesktopMediaAudio(AudioSample sample)
|
||||||
|
{
|
||||||
|
var data = sample.Samples;
|
||||||
|
if (data.Length == 0) return;
|
||||||
|
if (sample.SampleRate != OutputSampleRate)
|
||||||
|
data = ResampleLoopback(data, sample.SampleRate);
|
||||||
|
if (sample.Channels == 1)
|
||||||
|
data = UpmixToStereo(data);
|
||||||
|
_loopbackBuffer.Write(data);
|
||||||
|
|
||||||
|
var level = _loopbackMeter.Push(new AudioSample(data, OutputSampleRate, sample.Channels));
|
||||||
|
LoopbackLevelChanged?.Invoke(level);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Drops the alert clip's audio into its own ring (TASK 47 — the
|
/// <summary>Drops the alert clip's audio into its own ring (TASK 47 — the
|
||||||
/// stream-alert video's soundtrack, pre-scaled by the alert layer to
|
/// stream-alert video's soundtrack, pre-scaled by the alert layer to
|
||||||
/// volume × fade). Mixed at unity into the live output — NO auto-duck: an
|
/// volume × fade). Mixed at unity into the live output — NO auto-duck: an
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
using ytLive.Services.Audio;
|
||||||
|
|
||||||
namespace ytLive.Services;
|
namespace ytLive.Services;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
@@ -17,6 +19,11 @@ public interface IMediaFrameSource : IDisposable
|
|||||||
/// <summary>Raised (on a worker thread) for each decoded frame.</summary>
|
/// <summary>Raised (on a worker thread) for each decoded frame.</summary>
|
||||||
event Action<VideoFrame>? FrameAvailable;
|
event Action<VideoFrame>? FrameAvailable;
|
||||||
|
|
||||||
|
/// <summary>Raised (on a worker thread) per chunk of the file's own soundtrack
|
||||||
|
/// (48 kHz stereo float). The manager applies the layer's level/mute and hands it
|
||||||
|
/// to the mixer; a source with no audio simply never raises it.</summary>
|
||||||
|
event Action<AudioSample>? AudioAvailable;
|
||||||
|
|
||||||
/// <summary>Raised when the file finishes decoding (natural EOF) — not raised
|
/// <summary>Raised when the file finishes decoding (natural EOF) — not raised
|
||||||
/// while <see cref="Looping"/> is true.</summary>
|
/// while <see cref="Looping"/> is true.</summary>
|
||||||
event Action? Completed;
|
event Action? Completed;
|
||||||
|
|||||||
@@ -113,7 +113,7 @@ public partial class LayoutStore : IDisposable
|
|||||||
AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume, SortOrder,
|
AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume, SortOrder,
|
||||||
AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop,
|
AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop,
|
||||||
TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic,
|
TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic,
|
||||||
TextUnderline, TextAlignment, TextVerticalAlignment
|
TextUnderline, TextAlignment, TextVerticalAlignment, MediaMuted
|
||||||
FROM Source ORDER BY SortOrder
|
FROM Source ORDER BY SortOrder
|
||||||
""";
|
""";
|
||||||
using var reader = cmd.ExecuteReader();
|
using var reader = cmd.ExecuteReader();
|
||||||
@@ -183,6 +183,7 @@ public partial class LayoutStore : IDisposable
|
|||||||
|| !Enum.IsDefined((TextLayerVerticalAlignment)reader.GetInt32(49))
|
|| !Enum.IsDefined((TextLayerVerticalAlignment)reader.GetInt32(49))
|
||||||
? TextLayerVerticalAlignment.Top
|
? TextLayerVerticalAlignment.Top
|
||||||
: (TextLayerVerticalAlignment)reader.GetInt32(49),
|
: (TextLayerVerticalAlignment)reader.GetInt32(49),
|
||||||
|
MediaMuted = reader.GetInt32(reader.GetOrdinal("MediaMuted")) != 0,
|
||||||
};
|
};
|
||||||
if (!sourcesByScene.TryGetValue(sceneId, out var list))
|
if (!sourcesByScene.TryGetValue(sceneId, out var list))
|
||||||
sourcesByScene[sceneId] = list = new List<(Source, int)>();
|
sourcesByScene[sceneId] = list = new List<(Source, int)>();
|
||||||
|
|||||||
@@ -378,6 +378,12 @@ public partial class LayoutStore : IDisposable
|
|||||||
cmd.CommandText = "ALTER TABLE Source ADD COLUMN MediaPlaybackState TEXT NOT NULL DEFAULT 'Stopped';";
|
cmd.CommandText = "ALTER TABLE Source ADD COLUMN MediaPlaybackState TEXT NOT NULL DEFAULT 'Stopped';";
|
||||||
cmd.ExecuteNonQuery();
|
cmd.ExecuteNonQuery();
|
||||||
}
|
}
|
||||||
|
if (!columns.Contains("MediaMuted"))
|
||||||
|
{
|
||||||
|
using var cmd = _connection.CreateCommand();
|
||||||
|
cmd.CommandText = "ALTER TABLE Source ADD COLUMN MediaMuted INTEGER NOT NULL DEFAULT 0;";
|
||||||
|
cmd.ExecuteNonQuery();
|
||||||
|
}
|
||||||
|
|
||||||
// TASK 47: alert video config. The default clip asset id + the creator's
|
// TASK 47: alert video config. The default clip asset id + the creator's
|
||||||
// replacement path (a FILE reference, never a stored BLOB) + the pill,
|
// replacement path (a FILE reference, never a stored BLOB) + the pill,
|
||||||
|
|||||||
@@ -94,7 +94,7 @@ public partial class LayoutStore : IDisposable
|
|||||||
AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume,
|
AlertVideoAssetId, AlertVideoPath, AlertUseDefaultVideo, AlertShowTicker, AlertVideoVolume,
|
||||||
AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop,
|
AlertDisplayMethod, IsBrandLogo, BlurAmount, AlertAudioPath, AlertAudioLoop,
|
||||||
TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic,
|
TextContent, TextFontFamily, TextFontSize, TextColor, TextBold, TextItalic,
|
||||||
TextUnderline, TextAlignment, TextVerticalAlignment)
|
TextUnderline, TextAlignment, TextVerticalAlignment, MediaMuted)
|
||||||
VALUES ($id, $sceneId, $assetId, $type, $name, $isEnabled,
|
VALUES ($id, $sceneId, $assetId, $type, $name, $isEnabled,
|
||||||
$x, $y, $w, $h, $opacity, $monitor,
|
$x, $y, $w, $h, $opacity, $monitor,
|
||||||
$clip, $mirrored, $isBackground, $captureKey, $showDesktop, $sort,
|
$clip, $mirrored, $isBackground, $captureKey, $showDesktop, $sort,
|
||||||
@@ -104,7 +104,7 @@ public partial class LayoutStore : IDisposable
|
|||||||
$alertVideoAssetId, $alertVideoPath, $alertUseDefaultVideo, $alertShowTicker, $alertVideoVolume,
|
$alertVideoAssetId, $alertVideoPath, $alertUseDefaultVideo, $alertShowTicker, $alertVideoVolume,
|
||||||
$alertDisplayMethod, $isBrandLogo, $blurAmount, $alertAudioPath, $alertAudioLoop,
|
$alertDisplayMethod, $isBrandLogo, $blurAmount, $alertAudioPath, $alertAudioLoop,
|
||||||
$textContent, $textFontFamily, $textFontSize, $textColor, $textBold, $textItalic,
|
$textContent, $textFontFamily, $textFontSize, $textColor, $textBold, $textItalic,
|
||||||
$textUnderline, $textAlignment, $textVerticalAlignment)
|
$textUnderline, $textAlignment, $textVerticalAlignment, $mediaMuted)
|
||||||
""";
|
""";
|
||||||
cmd.Transaction = tx;
|
cmd.Transaction = tx;
|
||||||
var idP = cmd.Parameters.Add("$id", SqliteType.Text);
|
var idP = cmd.Parameters.Add("$id", SqliteType.Text);
|
||||||
@@ -137,6 +137,7 @@ public partial class LayoutStore : IDisposable
|
|||||||
var mediaPathP = cmd.Parameters.Add("$mediaPath", SqliteType.Text);
|
var mediaPathP = cmd.Parameters.Add("$mediaPath", SqliteType.Text);
|
||||||
var mediaIsLoopingP = cmd.Parameters.Add("$mediaIsLooping", SqliteType.Integer);
|
var mediaIsLoopingP = cmd.Parameters.Add("$mediaIsLooping", SqliteType.Integer);
|
||||||
var mediaVolumeP = cmd.Parameters.Add("$mediaVolume", SqliteType.Real);
|
var mediaVolumeP = cmd.Parameters.Add("$mediaVolume", SqliteType.Real);
|
||||||
|
var mediaMutedP = cmd.Parameters.Add("$mediaMuted", SqliteType.Integer);
|
||||||
var mediaPlaybackStateP = cmd.Parameters.Add("$mediaPlaybackState", SqliteType.Text);
|
var mediaPlaybackStateP = cmd.Parameters.Add("$mediaPlaybackState", SqliteType.Text);
|
||||||
var alertVideoAssetIdP = cmd.Parameters.Add("$alertVideoAssetId", SqliteType.Text);
|
var alertVideoAssetIdP = cmd.Parameters.Add("$alertVideoAssetId", SqliteType.Text);
|
||||||
var alertVideoPathP = cmd.Parameters.Add("$alertVideoPath", SqliteType.Text);
|
var alertVideoPathP = cmd.Parameters.Add("$alertVideoPath", SqliteType.Text);
|
||||||
@@ -198,6 +199,7 @@ public partial class LayoutStore : IDisposable
|
|||||||
mediaPathP.Value = (object?)source.MediaPath ?? DBNull.Value;
|
mediaPathP.Value = (object?)source.MediaPath ?? DBNull.Value;
|
||||||
mediaIsLoopingP.Value = source.MediaIsLooping ? 1 : 0;
|
mediaIsLoopingP.Value = source.MediaIsLooping ? 1 : 0;
|
||||||
mediaVolumeP.Value = source.MediaVolume;
|
mediaVolumeP.Value = source.MediaVolume;
|
||||||
|
mediaMutedP.Value = source.MediaMuted ? 1 : 0;
|
||||||
mediaPlaybackStateP.Value = source.MediaPlaybackState.ToString();
|
mediaPlaybackStateP.Value = source.MediaPlaybackState.ToString();
|
||||||
alertVideoAssetIdP.Value = (object?)source.AlertVideoAssetId ?? DBNull.Value;
|
alertVideoAssetIdP.Value = (object?)source.AlertVideoAssetId ?? DBNull.Value;
|
||||||
alertVideoPathP.Value = (object?)source.AlertVideoPath ?? DBNull.Value;
|
alertVideoPathP.Value = (object?)source.AlertVideoPath ?? DBNull.Value;
|
||||||
|
|||||||
+21
-115
@@ -1,38 +1,32 @@
|
|||||||
using System.Diagnostics;
|
using ytLive.Services.Audio;
|
||||||
using System.IO;
|
|
||||||
using ytLive.Services.Encoder;
|
using ytLive.Services.Encoder;
|
||||||
|
|
||||||
namespace ytLive.Services;
|
namespace ytLive.Services;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// TASK 21: decodes a local media file (any format ffmpeg handles) into
|
/// TASK 21: decodes a local media file (any format ffmpeg handles) into
|
||||||
/// <see cref="VideoFrame"/>s for a MediaSource. Implements
|
/// <see cref="VideoFrame"/>s for a MediaSource. This is the <see cref="IMediaFrameSource"/>
|
||||||
/// <see cref="IMediaFrameSource"/>: raises <see cref="FrameAvailable"/> per
|
/// shape over the shared <see cref="VideoClipPipe"/> — the same pipe an alert clip uses
|
||||||
/// frame and <see cref="IMediaFrameSource.Completed"/> on natural EOF (unless
|
/// (TASK 47), so the two features cannot drift apart again in how they decode.
|
||||||
/// <see cref="IMediaFrameSource.Looping"/>). Spawns ffmpeg with
|
/// <para>Video AND the file's own soundtrack are decoded. The manager scales the audio by
|
||||||
/// <c>-f rawvideo -pix_fmt bgra</c> and drains the raw BGRA pipe via
|
/// the layer's level/mute and hands it to the mixer's desktop group.</para>
|
||||||
/// <see cref="RawVideoFrameReader"/>. The decoder process and emitters are seams
|
|
||||||
/// so tests run without a real codec or audio device.
|
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public sealed class MediaVideoSource : IMediaFrameSource
|
public sealed class MediaVideoSource : IMediaFrameSource
|
||||||
{
|
{
|
||||||
public event Action<VideoFrame>? FrameAvailable;
|
public event Action<VideoFrame>? FrameAvailable;
|
||||||
|
public event Action<AudioSample>? AudioAvailable;
|
||||||
public event Action? Completed;
|
public event Action? Completed;
|
||||||
|
|
||||||
public string Key => _path;
|
public string Key => _path;
|
||||||
|
|
||||||
public bool Looping { get; set; }
|
public bool Looping
|
||||||
|
{
|
||||||
|
get => _clip.Looping;
|
||||||
|
set => _clip.Looping = value;
|
||||||
|
}
|
||||||
|
|
||||||
private readonly string _path;
|
private readonly string _path;
|
||||||
private readonly int _width;
|
private readonly VideoClipPipe _clip;
|
||||||
private readonly int _height;
|
|
||||||
private readonly IFfmpegLocator _locator;
|
|
||||||
private readonly Func<IDecodeProcess> _processFactory;
|
|
||||||
private readonly IFrameRateProbe? _probe;
|
|
||||||
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
|
|
||||||
private readonly CancellationTokenSource _cts = new();
|
|
||||||
private IDisposable? _run;
|
|
||||||
private IDecodeProcess? _active;
|
|
||||||
|
|
||||||
public MediaVideoSource(
|
public MediaVideoSource(
|
||||||
string path,
|
string path,
|
||||||
@@ -44,104 +38,16 @@ public sealed class MediaVideoSource : IMediaFrameSource
|
|||||||
Func<TimeSpan, CancellationToken, Task>? delay = null)
|
Func<TimeSpan, CancellationToken, Task>? delay = null)
|
||||||
{
|
{
|
||||||
_path = path;
|
_path = path;
|
||||||
_width = width;
|
_clip = new VideoClipPipe(
|
||||||
_height = height;
|
path, width, height, locator, processFactory, frameRateProbe, delay, audio: true);
|
||||||
_locator = locator;
|
_clip.FrameAvailable += frame => FrameAvailable?.Invoke(frame);
|
||||||
_processFactory = processFactory;
|
_clip.AudioAvailable += sample => AudioAvailable?.Invoke(sample);
|
||||||
_probe = frameRateProbe;
|
_clip.Completed += () => Completed?.Invoke();
|
||||||
_delay = delay ?? ((t, ct) => Task.Delay(t, ct));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task StartAsync(CancellationToken cancellationToken = default)
|
public Task StartAsync(CancellationToken cancellationToken = default) => _clip.StartAsync(cancellationToken);
|
||||||
{
|
|
||||||
if (_run != null) return Task.CompletedTask;
|
|
||||||
_run = _cts.Token.Register(StopInternal);
|
|
||||||
_ = Task.Run(RunAsync, CancellationToken.None);
|
|
||||||
return Task.CompletedTask;
|
|
||||||
}
|
|
||||||
|
|
||||||
public Task StopAsync()
|
public Task StopAsync() => _clip.StopAsync();
|
||||||
{
|
|
||||||
_cts.Cancel();
|
|
||||||
_run = null;
|
|
||||||
return Task.CompletedTask;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void Dispose()
|
public void Dispose() => _clip.Dispose();
|
||||||
{
|
|
||||||
_cts.Cancel();
|
|
||||||
_cts.Dispose();
|
|
||||||
_active?.Dispose();
|
|
||||||
}
|
|
||||||
|
|
||||||
private void StopInternal() => KillProcess();
|
|
||||||
|
|
||||||
private async Task RunAsync()
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
|
||||||
double? frameRate = null;
|
|
||||||
if (_probe != null)
|
|
||||||
frameRate = await _probe.ProbeAsync(_path, _cts.Token).ConfigureAwait(false);
|
|
||||||
var frameDuration = frameRate is > 0 ? TimeSpan.FromSeconds(1.0 / frameRate.Value) : TimeSpan.Zero;
|
|
||||||
|
|
||||||
var ffmpegPath = await _locator.LocateAsync(_cts.Token).ConfigureAwait(false);
|
|
||||||
var args =
|
|
||||||
$"-hide_banner -loglevel error -i \"{_path}\" " +
|
|
||||||
$"-f rawvideo -pix_fmt bgra -vf scale={_width}:{_height} -an -";
|
|
||||||
var startInfo = new ProcessStartInfo
|
|
||||||
{
|
|
||||||
FileName = ffmpegPath,
|
|
||||||
Arguments = args,
|
|
||||||
UseShellExecute = false,
|
|
||||||
RedirectStandardOutput = true,
|
|
||||||
RedirectStandardError = true,
|
|
||||||
CreateNoWindow = true,
|
|
||||||
};
|
|
||||||
|
|
||||||
do
|
|
||||||
{
|
|
||||||
using (var process = _processFactory())
|
|
||||||
{
|
|
||||||
_active = process;
|
|
||||||
process.Start(startInfo);
|
|
||||||
|
|
||||||
var reader = new RawVideoFrameReader(_width, _height);
|
|
||||||
var buffer = new byte[65536];
|
|
||||||
var stream = process.StandardOutput;
|
|
||||||
while (!_cts.IsCancellationRequested && !process.HasExited)
|
|
||||||
{
|
|
||||||
var read = await stream.ReadAsync(buffer, 0, buffer.Length, _cts.Token).ConfigureAwait(false);
|
|
||||||
if (read <= 0) break;
|
|
||||||
foreach (var frame in reader.Feed(buffer.AsSpan(0, read).ToArray()))
|
|
||||||
{
|
|
||||||
FrameAvailable?.Invoke(frame);
|
|
||||||
if (frameDuration > TimeSpan.Zero)
|
|
||||||
await _delay(frameDuration, _cts.Token).ConfigureAwait(false);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
_active = null;
|
|
||||||
}
|
|
||||||
while (Looping && !_cts.IsCancellationRequested);
|
|
||||||
}
|
|
||||||
catch (OperationCanceledException)
|
|
||||||
{
|
|
||||||
}
|
|
||||||
catch (Exception ex)
|
|
||||||
{
|
|
||||||
Fail($"Media decode failed: {ex.Message}");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!_cts.IsCancellationRequested)
|
|
||||||
Completed?.Invoke();
|
|
||||||
}
|
|
||||||
|
|
||||||
private void Fail(string message) => Console.Error.WriteLine(message);
|
|
||||||
|
|
||||||
private void KillProcess()
|
|
||||||
{
|
|
||||||
try { _active?.Kill(); } catch { /* already exited */ }
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ using System.Windows.Media;
|
|||||||
using System.Windows.Media.Imaging;
|
using System.Windows.Media.Imaging;
|
||||||
using System.Windows.Threading;
|
using System.Windows.Threading;
|
||||||
using ytLive.Helpers;
|
using ytLive.Helpers;
|
||||||
|
using ytLive.Services.Audio;
|
||||||
|
|
||||||
namespace ytLive.Services;
|
namespace ytLive.Services;
|
||||||
|
|
||||||
@@ -23,8 +24,11 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
public string Key { get; }
|
public string Key { get; }
|
||||||
public IMediaFrameSource Source { get; }
|
public IMediaFrameSource Source { get; }
|
||||||
public Action<VideoFrame>? FrameHandler;
|
public Action<VideoFrame>? FrameHandler;
|
||||||
|
public Action<AudioSample>? AudioHandler;
|
||||||
public int RefCount;
|
public int RefCount;
|
||||||
public bool Started;
|
public bool Started;
|
||||||
|
public bool AudioMuted;
|
||||||
|
public double AudioVolume = 1.0;
|
||||||
public WriteableBitmap? PreviewBitmap;
|
public WriteableBitmap? PreviewBitmap;
|
||||||
public VideoFrame? LatestFrame;
|
public VideoFrame? LatestFrame;
|
||||||
public bool FramePending;
|
public bool FramePending;
|
||||||
@@ -49,6 +53,10 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
/// <summary>Raised when a media decoder cannot be created (bad path).</summary>
|
/// <summary>Raised when a media decoder cannot be created (bad path).</summary>
|
||||||
public event Action<string, string>? MediaFailed;
|
public event Action<string, string>? MediaFailed;
|
||||||
|
|
||||||
|
/// <summary>Raised (worker thread) per audio chunk of a file's own soundtrack, already
|
||||||
|
/// scaled by the session's level and dropped when muted.</summary>
|
||||||
|
public event Action<string, AudioSample>? AudioAvailable;
|
||||||
|
|
||||||
public MediaVideoSourceManager(
|
public MediaVideoSourceManager(
|
||||||
Func<string, IMediaFrameSource?> sourceFactory,
|
Func<string, IMediaFrameSource?> sourceFactory,
|
||||||
Dispatcher? uiDispatcher = null,
|
Dispatcher? uiDispatcher = null,
|
||||||
@@ -95,6 +103,8 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
session = new MediaSession(key, source);
|
session = new MediaSession(key, source);
|
||||||
session.FrameHandler = frame => OnFrameAvailable(session, frame);
|
session.FrameHandler = frame => OnFrameAvailable(session, frame);
|
||||||
session.Source.FrameAvailable += session.FrameHandler;
|
session.Source.FrameAvailable += session.FrameHandler;
|
||||||
|
session.AudioHandler = sample => OnAudioAvailable(session, sample);
|
||||||
|
session.Source.AudioAvailable += session.AudioHandler;
|
||||||
// Loop intent lives on the Source (per scene), so the caller supplies the
|
// Loop intent lives on the Source (per scene), so the caller supplies the
|
||||||
// per-path answer ("any live reference loops"). Applied before Start so a
|
// per-path answer ("any live reference loops"). Applied before Start so a
|
||||||
// zero-length or instantly-EOF file cannot slip past a later assignment.
|
// zero-length or instantly-EOF file cannot slip past a later assignment.
|
||||||
@@ -117,6 +127,7 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
lock (_gate)
|
lock (_gate)
|
||||||
_sessions.Remove(key);
|
_sessions.Remove(key);
|
||||||
session.Source.FrameAvailable -= session.FrameHandler;
|
session.Source.FrameAvailable -= session.FrameHandler;
|
||||||
|
if (session.AudioHandler != null) session.Source.AudioAvailable -= session.AudioHandler;
|
||||||
AppLog.Write($"MediaSource: failed to start decoder '{key}': {ex.Message}");
|
AppLog.Write($"MediaSource: failed to start decoder '{key}': {ex.Message}");
|
||||||
MediaFailed?.Invoke(key, ex.Message);
|
MediaFailed?.Invoke(key, ex.Message);
|
||||||
await SafeStopAsync(session.Source);
|
await SafeStopAsync(session.Source);
|
||||||
@@ -137,6 +148,7 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
|
|
||||||
if (toStop == null) return;
|
if (toStop == null) return;
|
||||||
toStop.Source.FrameAvailable -= toStop.FrameHandler;
|
toStop.Source.FrameAvailable -= toStop.FrameHandler;
|
||||||
|
if (toStop.AudioHandler != null) toStop.Source.AudioAvailable -= toStop.AudioHandler;
|
||||||
await SafeStopAsync(toStop.Source);
|
await SafeStopAsync(toStop.Source);
|
||||||
toStop.PreviewBitmap = null;
|
toStop.PreviewBitmap = null;
|
||||||
}
|
}
|
||||||
@@ -155,6 +167,7 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
|
|
||||||
if (toStop == null) return;
|
if (toStop == null) return;
|
||||||
toStop.Source.FrameAvailable -= toStop.FrameHandler;
|
toStop.Source.FrameAvailable -= toStop.FrameHandler;
|
||||||
|
if (toStop.AudioHandler != null) toStop.Source.AudioAvailable -= toStop.AudioHandler;
|
||||||
await SafeStopAsync(toStop.Source);
|
await SafeStopAsync(toStop.Source);
|
||||||
toStop.PreviewBitmap = null;
|
toStop.PreviewBitmap = null;
|
||||||
}
|
}
|
||||||
@@ -193,6 +206,46 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>The layer's sound controls for this file. One decoder is shared per path,
|
||||||
|
/// so the level/mute are per SESSION: whichever layer the creator last touched wins —
|
||||||
|
/// the alternative (one decoder per layer) would play the same file twice.</summary>
|
||||||
|
public void SetAudioSettings(string key, double volume, bool muted)
|
||||||
|
{
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
if (!_sessions.TryGetValue(key, out var session)) return;
|
||||||
|
session.AudioVolume = Math.Clamp(volume, 0, 1);
|
||||||
|
session.AudioMuted = muted;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnAudioAvailable(MediaSession session, AudioSample sample)
|
||||||
|
{
|
||||||
|
if (!TryGetActiveSession(session)) return;
|
||||||
|
|
||||||
|
double volume;
|
||||||
|
bool muted;
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
volume = session.AudioVolume;
|
||||||
|
muted = session.AudioMuted;
|
||||||
|
}
|
||||||
|
if (muted || volume <= 0) return;
|
||||||
|
|
||||||
|
if (volume < 0.999)
|
||||||
|
{
|
||||||
|
// Copy before scaling: the source owns its buffer and may recycle it.
|
||||||
|
var scaled = new float[sample.Samples.Length];
|
||||||
|
var gain = (float)volume;
|
||||||
|
for (var i = 0; i < scaled.Length; i++)
|
||||||
|
scaled[i] = sample.Samples[i] * gain;
|
||||||
|
AudioAvailable?.Invoke(session.Key, new AudioSample(scaled, sample.SampleRate, sample.Channels));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
AudioAvailable?.Invoke(session.Key, sample);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>The shared preview bitmap for a file (so a config created after the
|
/// <summary>The shared preview bitmap for a file (so a config created after the
|
||||||
/// first frame can adopt the running bitmap), or null.</summary>
|
/// first frame can adopt the running bitmap), or null.</summary>
|
||||||
public WriteableBitmap? GetPreviewBitmap(string key)
|
public WriteableBitmap? GetPreviewBitmap(string key)
|
||||||
@@ -213,6 +266,7 @@ public sealed class MediaVideoSourceManager : IDisposable
|
|||||||
foreach (var session in sessions)
|
foreach (var session in sessions)
|
||||||
{
|
{
|
||||||
session.Source.FrameAvailable -= session.FrameHandler;
|
session.Source.FrameAvailable -= session.FrameHandler;
|
||||||
|
if (session.AudioHandler != null) session.Source.AudioAvailable -= session.AudioHandler;
|
||||||
_ = SafeStopAsync(session.Source);
|
_ = SafeStopAsync(session.Source);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,248 @@
|
|||||||
|
using System;
|
||||||
|
using System.Collections.Generic;
|
||||||
|
using System.Diagnostics;
|
||||||
|
using System.IO;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using ytLive.Helpers;
|
||||||
|
using ytLive.Services.Audio;
|
||||||
|
using ytLive.Services.Encoder;
|
||||||
|
|
||||||
|
namespace ytLive.Services;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// One video FILE played through ffmpeg — the shared pipe behind every clip, so a media
|
||||||
|
/// layer (TASK 21) and an alert clip (TASK 47) decode identically. Owns two subprocesses
|
||||||
|
/// on the same file: video (<c>rawvideo bgra</c>, scaled to the target rect) and audio
|
||||||
|
/// (<c>f32le -ac 2 -ar 48000</c>, matching the mixer's native format so no resampler is
|
||||||
|
/// needed). Both are paced to real time, so the picture and the sound stay together even
|
||||||
|
/// though they arrive on separate pipes.
|
||||||
|
/// <para><b>Looping is the media case</b>: on natural EOF the whole pass restarts — a fresh
|
||||||
|
/// pair of processes, because a <see cref="Process"/> cannot be re-started. An alert clip
|
||||||
|
/// never loops, so its wrapper simply never sets <see cref="Looping"/>.</para>
|
||||||
|
/// <para>The seams (locator, process factory, frame-rate probe, delay) exist so tests drive
|
||||||
|
/// a fake decoder with no ffmpeg, no codec, and no audio device.</para>
|
||||||
|
/// </summary>
|
||||||
|
public sealed class VideoClipPipe : IDisposable
|
||||||
|
{
|
||||||
|
public const int AudioSampleRate = 48000;
|
||||||
|
public const int AudioChannels = 2;
|
||||||
|
private const int AudioChunkFloats = 4800; // 50 ms of stereo
|
||||||
|
|
||||||
|
public event Action<VideoFrame>? FrameAvailable;
|
||||||
|
public event Action<AudioSample>? AudioAvailable;
|
||||||
|
public event Action? Completed;
|
||||||
|
|
||||||
|
/// <summary>Restart the pass on natural EOF (media/BRB loops).</summary>
|
||||||
|
public bool Looping { get; set; }
|
||||||
|
|
||||||
|
private readonly string _path;
|
||||||
|
private readonly int _width;
|
||||||
|
private readonly int _height;
|
||||||
|
private readonly IFfmpegLocator _locator;
|
||||||
|
private readonly Func<IDecodeProcess> _processFactory;
|
||||||
|
private readonly IFrameRateProbe? _probe;
|
||||||
|
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
|
||||||
|
private readonly bool _audio;
|
||||||
|
private readonly CancellationTokenSource _cts = new();
|
||||||
|
private IDisposable? _run;
|
||||||
|
private IDecodeProcess? _activeVideo;
|
||||||
|
private IDecodeProcess? _activeAudio;
|
||||||
|
|
||||||
|
public VideoClipPipe(
|
||||||
|
string path,
|
||||||
|
int width,
|
||||||
|
int height,
|
||||||
|
IFfmpegLocator locator,
|
||||||
|
Func<IDecodeProcess> processFactory,
|
||||||
|
IFrameRateProbe? frameRateProbe = null,
|
||||||
|
Func<TimeSpan, CancellationToken, Task>? delay = null,
|
||||||
|
bool audio = true)
|
||||||
|
{
|
||||||
|
_path = path;
|
||||||
|
_width = width;
|
||||||
|
_height = height;
|
||||||
|
_locator = locator;
|
||||||
|
_processFactory = processFactory;
|
||||||
|
_probe = frameRateProbe;
|
||||||
|
_delay = delay ?? ((t, ct) => Task.Delay(t, ct));
|
||||||
|
_audio = audio;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task StartAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
if (_run != null) return Task.CompletedTask;
|
||||||
|
_run = _cts.Token.Register(StopInternal);
|
||||||
|
_ = Task.Run(RunAsync, CancellationToken.None);
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task StopAsync()
|
||||||
|
{
|
||||||
|
_cts.Cancel();
|
||||||
|
_run = null;
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
_cts.Cancel();
|
||||||
|
_cts.Dispose();
|
||||||
|
_activeVideo?.Dispose();
|
||||||
|
_activeAudio?.Dispose();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void StopInternal()
|
||||||
|
{
|
||||||
|
Kill(_activeVideo);
|
||||||
|
Kill(_activeAudio);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task RunAsync()
|
||||||
|
{
|
||||||
|
var ct = _cts.Token;
|
||||||
|
var frames = 0;
|
||||||
|
var chunks = 0;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
double? frameRate = null;
|
||||||
|
if (_probe != null)
|
||||||
|
frameRate = await _probe.ProbeAsync(_path, ct).ConfigureAwait(false);
|
||||||
|
var frameDuration = frameRate is > 0 ? TimeSpan.FromSeconds(1.0 / frameRate.Value) : TimeSpan.Zero;
|
||||||
|
|
||||||
|
var ffmpegPath = await _locator.LocateAsync(ct).ConfigureAwait(false);
|
||||||
|
|
||||||
|
do
|
||||||
|
{
|
||||||
|
// Both pipes per pass, awaited together: a restart must begin a new
|
||||||
|
// video AND a new audio stream, or the picture drifts from the sound
|
||||||
|
// after the first loop.
|
||||||
|
var video = RunVideoAsync(ffmpegPath, frameDuration, ct);
|
||||||
|
var audio = _audio ? RunAudioAsync(ffmpegPath, ct) : Task.FromResult(0);
|
||||||
|
frames += await video.ConfigureAwait(false);
|
||||||
|
chunks += await audio.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
while (Looping && !ct.IsCancellationRequested);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
// A failed decode is not fatal: the alert layer falls back to its animations,
|
||||||
|
// and a media layer just holds its last frame.
|
||||||
|
AppLog.Write($"Video clip decode failed ({Path.GetFileName(_path)}): {ex.Message}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!ct.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
AppLog.Write($"Video clip end: {Path.GetFileName(_path)} frames={frames} audioChunks={chunks}");
|
||||||
|
Completed?.Invoke();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<int> RunVideoAsync(string ffmpegPath, TimeSpan frameDuration, CancellationToken ct)
|
||||||
|
{
|
||||||
|
var args =
|
||||||
|
$"-hide_banner -loglevel error -i \"{_path}\" " +
|
||||||
|
$"-f rawvideo -pix_fmt bgra -vf scale={_width}:{_height} -an -";
|
||||||
|
using var process = _processFactory();
|
||||||
|
_activeVideo = process;
|
||||||
|
process.Start(new ProcessStartInfo
|
||||||
|
{
|
||||||
|
FileName = ffmpegPath,
|
||||||
|
Arguments = args,
|
||||||
|
UseShellExecute = false,
|
||||||
|
RedirectStandardOutput = true,
|
||||||
|
RedirectStandardError = true,
|
||||||
|
CreateNoWindow = true,
|
||||||
|
});
|
||||||
|
|
||||||
|
var reader = new RawVideoFrameReader(_width, _height);
|
||||||
|
var buffer = new byte[65536];
|
||||||
|
var stream = process.StandardOutput;
|
||||||
|
var frames = 0;
|
||||||
|
while (!ct.IsCancellationRequested && !process.HasExited)
|
||||||
|
{
|
||||||
|
var read = await stream.ReadAsync(buffer, 0, buffer.Length, ct).ConfigureAwait(false);
|
||||||
|
if (read <= 0) break;
|
||||||
|
var chunk = new byte[read];
|
||||||
|
Buffer.BlockCopy(buffer, 0, chunk, 0, read);
|
||||||
|
foreach (var frame in reader.Feed(chunk))
|
||||||
|
{
|
||||||
|
frames++;
|
||||||
|
FrameAvailable?.Invoke(frame);
|
||||||
|
if (frameDuration > TimeSpan.Zero)
|
||||||
|
await _delay(frameDuration, ct).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_activeVideo = null;
|
||||||
|
return frames;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<int> RunAudioAsync(string ffmpegPath, CancellationToken ct)
|
||||||
|
{
|
||||||
|
var args =
|
||||||
|
$"-hide_banner -loglevel error -i \"{_path}\" " +
|
||||||
|
$"-f f32le -ac {AudioChannels} -ar {AudioSampleRate} -vn -";
|
||||||
|
using var process = _processFactory();
|
||||||
|
_activeAudio = process;
|
||||||
|
process.Start(new ProcessStartInfo
|
||||||
|
{
|
||||||
|
FileName = ffmpegPath,
|
||||||
|
Arguments = args,
|
||||||
|
UseShellExecute = false,
|
||||||
|
RedirectStandardOutput = true,
|
||||||
|
RedirectStandardError = true,
|
||||||
|
CreateNoWindow = true,
|
||||||
|
});
|
||||||
|
|
||||||
|
var stream = process.StandardOutput;
|
||||||
|
var raw = new byte[65536];
|
||||||
|
var carry = Array.Empty<byte>();
|
||||||
|
var floatScratch = new float[AudioChunkFloats];
|
||||||
|
var floatIndex = 0;
|
||||||
|
var chunks = 0;
|
||||||
|
// The pipe is byte-delimited only: a read may split a float, so leftover
|
||||||
|
// bytes are carried into the next read instead of dropped (a dropped
|
||||||
|
// half-float would desync the entire stream).
|
||||||
|
while (!ct.IsCancellationRequested && !process.HasExited)
|
||||||
|
{
|
||||||
|
var read = await stream.ReadAsync(raw, 0, raw.Length, ct).ConfigureAwait(false);
|
||||||
|
if (read <= 0) break;
|
||||||
|
var total = carry.Length + read;
|
||||||
|
var bytes = total <= raw.Length ? raw : new byte[total];
|
||||||
|
if (carry.Length > 0)
|
||||||
|
{
|
||||||
|
Buffer.BlockCopy(carry, 0, bytes, 0, carry.Length);
|
||||||
|
Buffer.BlockCopy(raw, 0, bytes, carry.Length, read);
|
||||||
|
}
|
||||||
|
var floatsTo = total - (total % 4);
|
||||||
|
for (var i = 0; i < floatsTo && floatIndex < floatScratch.Length; i += 4)
|
||||||
|
floatScratch[floatIndex++] = BitConverter.ToSingle(bytes, i);
|
||||||
|
carry = total > floatsTo ? bytes[floatsTo..total] : Array.Empty<byte>();
|
||||||
|
|
||||||
|
while (floatIndex == floatScratch.Length)
|
||||||
|
{
|
||||||
|
chunks++;
|
||||||
|
AudioAvailable?.Invoke(new AudioSample(floatScratch, AudioSampleRate, AudioChannels));
|
||||||
|
// Pace the NEXT chunk so the clip's audio lands at real-time cadence,
|
||||||
|
// matching how the video pipe is paced by frame rate.
|
||||||
|
await _delay(
|
||||||
|
TimeSpan.FromSeconds(AudioChunkFloats / (double)(AudioSampleRate * AudioChannels)), ct)
|
||||||
|
.ConfigureAwait(false);
|
||||||
|
floatScratch = new float[AudioChunkFloats];
|
||||||
|
floatIndex = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_activeAudio = null;
|
||||||
|
return chunks;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void Kill(IDecodeProcess? process)
|
||||||
|
{
|
||||||
|
try { process?.Kill(); } catch { /* already exited */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,8 +7,7 @@
|
|||||||
|
|
||||||
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
|
### Status: ✅ Complete 2026-11-05 — a video file added in the (+) menu decodes, composites into the
|
||||||
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
|
preview and the stream, loops on the creator's toggle, and stops when the layer or its last scene
|
||||||
reference goes away. **Picture only: the file's own audio is not played anywhere** (the decode pipe
|
reference goes away, and **the file's own soundtrack plays into the stream** (item 10).
|
||||||
passes `-an`), so a video with a soundtrack is silent on stream — the audio route is open, item 10.
|
|
||||||
|
|
||||||
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
|
1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source)
|
||||||
2. ✅ `SourceType.MediaSource` addition to the enum
|
2. ✅ `SourceType.MediaSource` addition to the enum
|
||||||
@@ -25,13 +24,13 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
|
|||||||
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
|
when `Looping` (fresh process per pass). `MediaIsLooping` reaches the session through the manager's
|
||||||
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
|
per-path `loopProvider` at start and `SetLooping` on a live toggle. Rule (recorded): the path's
|
||||||
session loops if **any** live media reference has it on, because one decoder is shared per path.
|
session loops if **any** live media reference has it on, because one decoder is shared per path.
|
||||||
10. ⏸ File audio + volume — **the file's soundtrack is silent everywhere today**: the decode pipe
|
10. ✅ File audio + volume — the shared clip pipe decodes the soundtrack (`f32le -ac 2 -ar 48000 -vn`,
|
||||||
passes `-an`, so nothing plays it and the desktop loopback has nothing to carry. Two established
|
50 ms chunks paced to real time, byte-carry across reads) and the layer carries **Mute** + **Level**.
|
||||||
routes when this is taken: **alert-style** (a second `f32le` pipe into the mixer —
|
The manager applies them per session (one decoder per path, so the last layer touched wins — the
|
||||||
`AlertClipDecoder.RunAudioAsync` is the working model, independent of desktop capture) or
|
alternative would play the same file twice) and the audio joins the DESKTOP/GAME group in
|
||||||
**TRAX-style** (`MusicPlayer` plays it to the default device, so the loopback, the desktop fader
|
`AudioMixer.EnqueueDesktopMediaAudio`, so the desktop fader and the voice ducker behave exactly as
|
||||||
and ducking all apply — at the cost of a second clock to hold in sync with the video). Product
|
they do for game audio and TRAX. Test: `MediaSourceAudioTests` (chunks, level scaling, mute, and the
|
||||||
call; not started, and no volume slider until one of them exists.
|
layer's sound reaching the desktop meter).
|
||||||
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
|
11. ✅ UI: `AddMediaCommand` (+ menu → "Video File", video-filtered `OpenFileDialog`), `ChangeMediaFileCommand`
|
||||||
("Change…" in the properties panel), loop toggle, video name display.
|
("Change…" in the properties panel), loop toggle, video name display.
|
||||||
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
|
12. ✅ Schema migration for media source settings (file path, loop, volume) — shipped with item 3.
|
||||||
@@ -41,10 +40,11 @@ passes `-an`), so a video with a soundtrack is silent on stream — the audio ro
|
|||||||
### Design decisions
|
### Design decisions
|
||||||
|
|
||||||
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
|
- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project.
|
||||||
- **Audio is NOT wired, and it is not hiding behind the loopback** — `MediaVideoSource` decodes with
|
- **Audio joins the desktop/game group** (creator ruling 2026-11-05) — the clip's sound is decoded by
|
||||||
`-an`, so a video file's soundtrack is silent on stream today (the loopback only carries what the
|
the same pipe as its picture, mixed directly into the stream (not played locally and captured), and
|
||||||
machine actually plays). Item 10 records the two established routes; do not describe media as
|
grouped with game audio so one fader and the ducker cover it. A per-layer Mute/Level sits on top.
|
||||||
audible until one is built.
|
Chosen over TRAX-style local playback because it works regardless of the creator's desktop-capture
|
||||||
|
state, and because an alert clip needs the same direct route anyway.
|
||||||
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
|
- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished.
|
||||||
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
|
- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene."
|
||||||
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
|
- **Layers are defined in Live first** (creator 2026-11-05): this unit adds the layer and the picker in
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ using System.Windows.Media.Imaging;
|
|||||||
using Microsoft.Win32;
|
using Microsoft.Win32;
|
||||||
using ytLive.Models;
|
using ytLive.Models;
|
||||||
using ytLive.Services;
|
using ytLive.Services;
|
||||||
|
using ytLive.Services.Audio;
|
||||||
|
|
||||||
namespace ytLive.ViewModels;
|
namespace ytLive.ViewModels;
|
||||||
|
|
||||||
@@ -105,6 +106,11 @@ public partial class MainViewModel
|
|||||||
{
|
{
|
||||||
_mediaManager.SetLooping(path, AnyMediaReferenceLoops(path));
|
_mediaManager.SetLooping(path, AnyMediaReferenceLoops(path));
|
||||||
}
|
}
|
||||||
|
else if (e.PropertyName is nameof(Source.MediaVolume) or nameof(Source.MediaMuted)
|
||||||
|
&& source.MediaPath is { } audioPath)
|
||||||
|
{
|
||||||
|
ApplyMediaAudioSettings(audioPath);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
_mediaHandlers[source.Id] = OnSourceChanged;
|
_mediaHandlers[source.Id] = OnSourceChanged;
|
||||||
@@ -121,7 +127,11 @@ public partial class MainViewModel
|
|||||||
private void EnsureMediaSession(string? path)
|
private void EnsureMediaSession(string? path)
|
||||||
{
|
{
|
||||||
if (string.IsNullOrWhiteSpace(path)) return;
|
if (string.IsNullOrWhiteSpace(path)) return;
|
||||||
if (_mediaManager.HasSession(path)) return;
|
if (_mediaManager.HasSession(path))
|
||||||
|
{
|
||||||
|
ApplyMediaAudioSettings(path);
|
||||||
|
return;
|
||||||
|
}
|
||||||
_ = AcquireMediaAsync(path);
|
_ = AcquireMediaAsync(path);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -133,8 +143,25 @@ public partial class MainViewModel
|
|||||||
// case the bitmap is already published and will not be raised again for us.
|
// case the bitmap is already published and will not be raised again for us.
|
||||||
if (_mediaManager.GetPreviewBitmap(path) is { } bitmap)
|
if (_mediaManager.GetPreviewBitmap(path) is { } bitmap)
|
||||||
OnMediaPreviewBitmapChanged(path, bitmap);
|
OnMediaPreviewBitmapChanged(path, bitmap);
|
||||||
|
|
||||||
|
ApplyMediaAudioSettings(path);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>The file's sound controls: one decoder is shared per path, so the level and
|
||||||
|
/// mute are per session — the layer that was last touched is the one that wins.</summary>
|
||||||
|
private void ApplyMediaAudioSettings(string path)
|
||||||
|
{
|
||||||
|
var source = Scenes.SelectMany(s => s.Elements).OfType<Source>()
|
||||||
|
.FirstOrDefault(s => s.IsMediaSource && s.MediaPath == path);
|
||||||
|
if (source == null) return;
|
||||||
|
_mediaManager.SetAudioSettings(path, source.MediaVolume, source.MediaMuted);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A clip's own soundtrack joins the desktop/game group, so the desktop fader and
|
||||||
|
/// the voice ducker treat it exactly like game audio (TASK 21 audio slice).</summary>
|
||||||
|
private void OnMediaAudioAvailable(string path, AudioSample sample)
|
||||||
|
=> _audioMixer.EnqueueDesktopMediaAudio(sample);
|
||||||
|
|
||||||
/// <summary>Shared sessions or not, the loop state is the path's: any live media
|
/// <summary>Shared sessions or not, the loop state is the path's: any live media
|
||||||
/// reference asking for a loop makes the one decoder loop (recorded rule, TASK 21).</summary>
|
/// reference asking for a loop makes the one decoder loop (recorded rule, TASK 21).</summary>
|
||||||
private bool AnyMediaReferenceLoops(string path)
|
private bool AnyMediaReferenceLoops(string path)
|
||||||
|
|||||||
@@ -412,6 +412,7 @@ public partial class MainViewModel : ViewModelBase
|
|||||||
AnyMediaReferenceLoops);
|
AnyMediaReferenceLoops);
|
||||||
_mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged;
|
_mediaManager.PreviewBitmapChanged += OnMediaPreviewBitmapChanged;
|
||||||
_mediaManager.MediaFailed += OnMediaFailed;
|
_mediaManager.MediaFailed += OnMediaFailed;
|
||||||
|
_mediaManager.AudioAvailable += OnMediaAudioAvailable;
|
||||||
|
|
||||||
_chatLayer = new ChatOverlayLayer(new ChatBoxRenderer());
|
_chatLayer = new ChatOverlayLayer(new ChatBoxRenderer());
|
||||||
_alertLayer = new AlertOverlayLayer(
|
_alertLayer = new AlertOverlayLayer(
|
||||||
|
|||||||
@@ -799,9 +799,12 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
|
|||||||
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
|
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
|
||||||
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
|
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
|
||||||
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
|
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
|
||||||
the layer no longer references it. **A video file's own soundtrack is silent today** — the decode pipe
|
the layer no longer references it. **Sound ships with it (2026-11-05):** both `MediaVideoSource` and
|
||||||
is `-an`, so nothing plays it and the loopback has nothing to carry; wiring it (alert-style mixer feed
|
`AlertClipDecoder` are now thin wrappers over one `Services/VideoClipPipe.cs` (video pipe + `f32le`
|
||||||
or TRAX-style local playback) is open, and no volume slider exists until then.
|
audio pipe, both real-time paced, loop restarts both), so a clip is a clip. The manager applies the
|
||||||
|
layer's Level/Mute per session and raises `AudioAvailable`, and `MainViewModel` feeds it to
|
||||||
|
`AudioMixer.EnqueueDesktopMediaAudio` — the DESKTOP/GAME ring, so the desktop fader and the voice
|
||||||
|
ducker apply. The alert clip keeps its own unity-gain ring (`EnqueueAlertAudio`, never ducked).
|
||||||
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
||||||
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
||||||
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
||||||
|
|||||||
@@ -250,6 +250,13 @@ devices, no timers).
|
|||||||
`StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) —
|
`StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) —
|
||||||
the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the
|
the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the
|
||||||
pipe lifecycle.
|
pipe lifecycle.
|
||||||
|
**Two clip-audio rings join that mix (2026-11-05):** `EnqueueAlertAudio` (TASK 47) is pre-scaled by the
|
||||||
|
alert layer and mixed at **unity, never ducked** — an alert must be heard over the mic.
|
||||||
|
`EnqueueDesktopMediaAudio` (TASK 21's audio slice) is the opposite on purpose: a video layer's
|
||||||
|
soundtrack joins the **desktop/game ring**, so the desktop fader and the auto-ducker apply exactly as
|
||||||
|
they do for game audio and TRAX. Both are fed by the shared `Services/VideoClipPipe.cs`; the layer's
|
||||||
|
own Mute/Level are applied upstream by `MediaVideoSourceManager` (level/mute are per **session**, since
|
||||||
|
one decoder is shared per file path).
|
||||||
- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo
|
- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo
|
||||||
mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose
|
mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose
|
||||||
offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500
|
offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500
|
||||||
|
|||||||
+5
-6
@@ -97,12 +97,11 @@ published decisions (recorded in `TASKS/task-43-native-alerts.md`):
|
|||||||
`AlertClipDecoder.RunVideoAsync` and `MediaVideoSource.RunAsync` are near-duplicates —
|
`AlertClipDecoder.RunVideoAsync` and `MediaVideoSource.RunAsync` are near-duplicates —
|
||||||
same ffmpeg args (`-f rawvideo -pix_fmt bgra -vf scale=W:H -an -`), same
|
same ffmpeg args (`-f rawvideo -pix_fmt bgra -vf scale=W:H -an -`), same
|
||||||
`RawVideoFrameReader` feed loop, same `IFrameRateProbe` + delay pacing, both behind
|
`RawVideoFrameReader` feed loop, same `IFrameRateProbe` + delay pacing, both behind
|
||||||
`IDecodeProcess`. What differs is ownership and role, not decoding: alerts are per-play
|
`IDecodeProcess`. Consolidated 2026-11-05 with the media audio slice: both are now thin wrappers over
|
||||||
and disposed at drain, media is a refcounted per-path session; alerts also run an
|
one `Services/VideoClipPipe.cs` (video pipe + `f32le` audio pipe, real-time paced, loop restarts both),
|
||||||
`f32le` audio pipe into the mixer, media runs none at all (`-an`) — so a media file's
|
so decode behaviour cannot drift between them again. What still differs is ownership and role: alerts
|
||||||
soundtrack is silent today, and task-21 item 10 holds the two routes for changing that;
|
are per-play and disposed at drain, media is a refcounted per-path session; alerts play once, media
|
||||||
alerts play once, media loops. A shared raw-video pump
|
loops; alert audio is mixed at unity and never ducked, media audio joins the desktop/game group.
|
||||||
is the obvious consolidation if a third consumer appears — do not do it for two.
|
|
||||||
Fades in/out via **`FadeDurationSeconds = 0.30`**
|
Fades in/out via **`FadeDurationSeconds = 0.30`**
|
||||||
(`AlertOverlayLayer` const): fade-in by `elapsed` on straight-alpha frame copies;
|
(`AlertOverlayLayer` const): fade-in by `elapsed` on straight-alpha frame copies;
|
||||||
EOF → freeze-frame → fade-out → drain (the clip branch of `Advance` **clears
|
EOF → freeze-frame → fade-out → drain (the clip branch of `Advance` **clears
|
||||||
|
|||||||
@@ -100,6 +100,7 @@ public sealed class MediaSourceAcquisitionTests
|
|||||||
public int StopCount { get; private set; }
|
public int StopCount { get; private set; }
|
||||||
|
|
||||||
public event Action<VideoFrame>? FrameAvailable;
|
public event Action<VideoFrame>? FrameAvailable;
|
||||||
|
public event Action<ytLive.Services.Audio.AudioSample>? AudioAvailable { add { } remove { } }
|
||||||
public event Action? Completed { add { } remove { } }
|
public event Action? Completed { add { } remove { } }
|
||||||
|
|
||||||
public Task StartAsync(CancellationToken cancellationToken = default)
|
public Task StartAsync(CancellationToken cancellationToken = default)
|
||||||
|
|||||||
@@ -0,0 +1,191 @@
|
|||||||
|
using System;
|
||||||
|
using System.Collections.Generic;
|
||||||
|
using System.Diagnostics;
|
||||||
|
using System.IO;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using Microsoft.Data.Sqlite;
|
||||||
|
using Xunit;
|
||||||
|
using ytLive.Services;
|
||||||
|
using ytLive.Services.Audio;
|
||||||
|
using ytLive.Services.Encoder;
|
||||||
|
using ytLive.ViewModels;
|
||||||
|
|
||||||
|
namespace ytLive.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// TASK 21 audio slice (2026-11-05): a video file's own soundtrack used to be thrown away
|
||||||
|
/// (`-an` on the decode pipe), so a "starting soon" clip with music played silent. This pins
|
||||||
|
/// the replacement: the shared clip pipe decodes the sound, the manager applies the layer's
|
||||||
|
/// level/mute, and a media layer's audio lands in the mixer's desktop/game group.
|
||||||
|
/// </summary>
|
||||||
|
[Collection("RealApp")]
|
||||||
|
public sealed class MediaSourceAudioTests
|
||||||
|
{
|
||||||
|
private readonly RealAppHost _app;
|
||||||
|
|
||||||
|
public MediaSourceAudioTests(RealAppHost app) => _app = app;
|
||||||
|
|
||||||
|
// ─── the pipe decodes the soundtrack ───
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ClipPipe_EmitsTheSoundtrackInPacedChunks()
|
||||||
|
{
|
||||||
|
// 4800 stereo floats = one 50 ms chunk.
|
||||||
|
var audioBytes = new byte[4800 * 4];
|
||||||
|
for (var i = 0; i < 4800; i++)
|
||||||
|
BitConverter.TryWriteBytes(audioBytes.AsSpan(i * 4, 4), 0.5f);
|
||||||
|
|
||||||
|
using var pipe = new VideoClipPipe(
|
||||||
|
"clip.mp4", 1, 1,
|
||||||
|
new FakeLocator(),
|
||||||
|
() => new FakeDecodeProcess(args => args.Contains("-vn") ? audioBytes : Array.Empty<byte>()),
|
||||||
|
delay: (_, _) => Task.CompletedTask);
|
||||||
|
|
||||||
|
var chunks = new List<AudioSample>();
|
||||||
|
var completed = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
pipe.AudioAvailable += sample => chunks.Add(sample);
|
||||||
|
pipe.Completed += () => completed.TrySetResult(true);
|
||||||
|
|
||||||
|
await pipe.StartAsync();
|
||||||
|
await completed.Task.WaitAsync(TimeSpan.FromSeconds(5));
|
||||||
|
|
||||||
|
Assert.Single(chunks);
|
||||||
|
Assert.Equal(4800, chunks[0].Samples.Length);
|
||||||
|
Assert.Equal(48000, chunks[0].SampleRate);
|
||||||
|
Assert.Equal(2, chunks[0].Channels);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── the manager applies the layer's controls ───
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Manager_ScalesTheSoundtrackByTheSessionsLevel_AndDropsItWhenMuted()
|
||||||
|
{
|
||||||
|
var source = new FakeMediaSource("clip.mp4");
|
||||||
|
var manager = new MediaVideoSourceManager(_ => source, uiDispatcher: null);
|
||||||
|
var heard = new List<AudioSample>();
|
||||||
|
manager.AudioAvailable += (_, sample) => heard.Add(sample);
|
||||||
|
|
||||||
|
Assert.True(await manager.AcquireAsync("clip.mp4"));
|
||||||
|
|
||||||
|
manager.SetAudioSettings("clip.mp4", 0.5, muted: false);
|
||||||
|
source.EmitAudio(new AudioSample(new[] { 1.0f, -1.0f }, 48000, 2));
|
||||||
|
Assert.Single(heard);
|
||||||
|
Assert.Equal(0.5f, heard[0].Samples[0], 3);
|
||||||
|
Assert.Equal(-0.5f, heard[0].Samples[1], 3);
|
||||||
|
|
||||||
|
manager.SetAudioSettings("clip.mp4", 1.0, muted: true);
|
||||||
|
source.EmitAudio(new AudioSample(new[] { 1.0f, -1.0f }, 48000, 2));
|
||||||
|
Assert.Single(heard); // muted: nothing reaches the mixer
|
||||||
|
|
||||||
|
manager.Dispose();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── the layer's audio reaches the desktop/game group ───
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void MediaLayerSound_ReachesTheDesktopAudioGroup()
|
||||||
|
{
|
||||||
|
_app.Run(() =>
|
||||||
|
{
|
||||||
|
var tempDb = Path.Combine(Path.GetTempPath(), $"ytLlive-test-{Guid.NewGuid():N}.db");
|
||||||
|
var videoPath = Path.Combine(Path.GetTempPath(), "brb-bed.mp4");
|
||||||
|
var decoders = new List<FakeMediaSource>();
|
||||||
|
MainViewModel.LayoutPathOverride = tempDb;
|
||||||
|
MainViewModel.MediaFrameSourceFactoryOverride = key =>
|
||||||
|
{
|
||||||
|
var decoder = new FakeMediaSource(key);
|
||||||
|
decoders.Add(decoder);
|
||||||
|
return decoder;
|
||||||
|
};
|
||||||
|
|
||||||
|
var window = new MainWindow();
|
||||||
|
var vm = (MainViewModel)window.DataContext;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
window.Show();
|
||||||
|
window.UpdateLayout();
|
||||||
|
|
||||||
|
vm.AddMediaSource(videoPath);
|
||||||
|
var decoder = decoders.Single();
|
||||||
|
|
||||||
|
// The desktop/game meter is what the creator watches. It has to move for a
|
||||||
|
// media layer's sound, or the soundtrack is in the mix but invisible.
|
||||||
|
Assert.Equal(0.0, vm.GameAudioLevel);
|
||||||
|
|
||||||
|
var loud = Enumerable.Repeat(0.8f, 9600).ToArray();
|
||||||
|
for (var i = 0; i < 6; i++)
|
||||||
|
decoder.EmitAudio(new AudioSample(loud, 48000, 2));
|
||||||
|
|
||||||
|
Assert.True(vm.GameAudioLevel > 0.05,
|
||||||
|
$"the media soundtrack never reached the desktop group (level {vm.GameAudioLevel})");
|
||||||
|
|
||||||
|
vm.StagedScene!.Elements.OfType<ytLive.Models.Source>()
|
||||||
|
.Single(s => s.IsMediaSource).MediaMuted = true;
|
||||||
|
var afterMute = vm.GameAudioLevel;
|
||||||
|
|
||||||
|
decoder.EmitAudio(new AudioSample(loud, 48000, 2));
|
||||||
|
Assert.Equal(afterMute, vm.GameAudioLevel); // muted layer stops feeding the meter
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
window.Close();
|
||||||
|
MainViewModel.MediaFrameSourceFactoryOverride = null;
|
||||||
|
MainViewModel.LayoutPathOverride = null;
|
||||||
|
SqliteConnection.ClearAllPools();
|
||||||
|
try { File.Delete(tempDb); } catch { /* best-effort cleanup */ }
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FakeMediaSource : IMediaFrameSource
|
||||||
|
{
|
||||||
|
public FakeMediaSource(string key) => Key = key;
|
||||||
|
|
||||||
|
public string Key { get; }
|
||||||
|
public bool Looping { get; set; }
|
||||||
|
|
||||||
|
public event Action<VideoFrame>? FrameAvailable { add { } remove { } }
|
||||||
|
public event Action<AudioSample>? AudioAvailable;
|
||||||
|
public event Action? Completed { add { } remove { } }
|
||||||
|
|
||||||
|
public void EmitAudio(AudioSample sample) => AudioAvailable?.Invoke(sample);
|
||||||
|
|
||||||
|
public Task StartAsync(CancellationToken cancellationToken = default) => Task.CompletedTask;
|
||||||
|
public Task StopAsync() => Task.CompletedTask;
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FakeDecodeProcess : IDecodeProcess
|
||||||
|
{
|
||||||
|
private readonly Func<string, byte[]> _pick;
|
||||||
|
private readonly MemoryStream _stream = new();
|
||||||
|
|
||||||
|
public FakeDecodeProcess(Func<string, byte[]> pick) => _pick = pick;
|
||||||
|
|
||||||
|
public void Start(ProcessStartInfo startInfo)
|
||||||
|
{
|
||||||
|
// The pipe runs a video process and an audio process; the payload depends on
|
||||||
|
// which one this is, so the fake reads the arguments.
|
||||||
|
var bytes = _pick(startInfo.Arguments);
|
||||||
|
_stream.SetLength(0);
|
||||||
|
_stream.Write(bytes, 0, bytes.Length);
|
||||||
|
_stream.Position = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Stream StandardOutput => _stream;
|
||||||
|
public bool HasExited => _stream.Position >= _stream.Length;
|
||||||
|
public int ExitCode => 0;
|
||||||
|
public void Kill() { }
|
||||||
|
public Task WaitForExitAsync(CancellationToken ct = default) => Task.CompletedTask;
|
||||||
|
public void Dispose() => _stream.Dispose();
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FakeLocator : IFfmpegLocator
|
||||||
|
{
|
||||||
|
public Task<string> LocateAsync(CancellationToken ct = default) => Task.FromResult("ffmpeg.exe");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ using System.Windows.Media.Imaging;
|
|||||||
using System.Windows.Threading;
|
using System.Windows.Threading;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
using ytLive.Services;
|
using ytLive.Services;
|
||||||
|
using ytLive.Services.Audio;
|
||||||
|
|
||||||
namespace ytLive.Tests;
|
namespace ytLive.Tests;
|
||||||
|
|
||||||
@@ -22,6 +23,7 @@ public class MediaVideoSourceManagerTests
|
|||||||
public string Key { get; }
|
public string Key { get; }
|
||||||
public bool Looping { get; set; }
|
public bool Looping { get; set; }
|
||||||
public event Action<VideoFrame>? FrameAvailable;
|
public event Action<VideoFrame>? FrameAvailable;
|
||||||
|
public event Action<AudioSample>? AudioAvailable;
|
||||||
public event Action? Completed;
|
public event Action? Completed;
|
||||||
|
|
||||||
public FakeMediaSource(string key, List<string>? started = null, List<string>? stopped = null)
|
public FakeMediaSource(string key, List<string>? started = null, List<string>? stopped = null)
|
||||||
@@ -44,6 +46,7 @@ public class MediaVideoSourceManagerTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
||||||
|
public void EmitAudio(AudioSample sample) => AudioAvailable?.Invoke(sample);
|
||||||
public void Complete() => Completed?.Invoke();
|
public void Complete() => Completed?.Invoke();
|
||||||
public void Dispose() { }
|
public void Dispose() { }
|
||||||
}
|
}
|
||||||
@@ -53,12 +56,14 @@ public class MediaVideoSourceManagerTests
|
|||||||
public string Key { get; }
|
public string Key { get; }
|
||||||
public bool Looping { get; set; }
|
public bool Looping { get; set; }
|
||||||
public event Action<VideoFrame>? FrameAvailable;
|
public event Action<VideoFrame>? FrameAvailable;
|
||||||
|
public event Action<AudioSample>? AudioAvailable;
|
||||||
public event Action? Completed;
|
public event Action? Completed;
|
||||||
public FailingMediaSource(string key) => Key = key;
|
public FailingMediaSource(string key) => Key = key;
|
||||||
public Task StartAsync(CancellationToken cancellationToken = default)
|
public Task StartAsync(CancellationToken cancellationToken = default)
|
||||||
=> throw new InvalidOperationException("access denied");
|
=> throw new InvalidOperationException("access denied");
|
||||||
public Task StopAsync() => Task.CompletedTask;
|
public Task StopAsync() => Task.CompletedTask;
|
||||||
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
||||||
|
public void EmitAudio(AudioSample sample) => AudioAvailable?.Invoke(sample);
|
||||||
public void Complete() => Completed?.Invoke();
|
public void Complete() => Completed?.Invoke();
|
||||||
public void Dispose() { }
|
public void Dispose() { }
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user