TASK 32·4. A plain MP4 keeps its index (moov) at the end, so a power cut or a killed process left the entire recording unreadable even though every video byte was on disk. The record block now carries `-movflags +frag_keyframe+empty_moov+default_base_moof`: index first, a playable fragment flushed at every keyframe, so an interrupted take plays to the last fragment. Record block only — the FLV stream output has no index and never gets the flags. Rename-on-stop is untouched. Test: `Args_RecordBlock_IsFragmentedMp4_SoACrashKeepsTheFile` asserts the flags on the record block and their absence from the streaming block. Not yet proven on a real interrupted take (kill ffmpeg mid-recording, play the result); the args test is by construction. Docs: task-32 slice 4 ticked with that caveat, TASKS row, output.md record note, HANDOFF. verify.sh: 550 passed (mouse-injection test excluded, desktop in use), 0 warnings.
343 lines
29 KiB
Markdown
343 lines
29 KiB
Markdown
# Output — the ffmpeg bundle, the encoder, recording, and audio
|
||
|
||
> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches;
|
||
> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md).
|
||
|
||
Everything that leaves the app: locating and shipping `ffmpeg.exe`, how the encoder subprocess is
|
||
driven (video stdin + audio named pipe), local recording as a first-class mode, and the WASAPI capture
|
||
/ filter / mix chain that feeds it. Content is the decisions plus the failures that produced them;
|
||
superseded designs are marked inline.
|
||
|
||
---
|
||
|
||
## The encoder
|
||
|
||
### FFmpeg locator (TASK 4 ship step 2 — shipped 2026-08-10; **rewritten to BUNDLED 2026-09-28**, plan in TASKS.md)
|
||
|
||
The encoder's one external dependency is `ffmpeg.exe`. **It is BUNDLED — never downloaded, never
|
||
fetched at runtime.** `IFfmpegLocator` resolves an absolute path on demand: **the bundle first**
|
||
(`<AppContext.BaseDirectory>\ffmpeg\ffmpeg.exe`), then **PATH** as the escape hatch for someone who
|
||
deliberately installed their own. The order is the opposite of the original ("the user's own install
|
||
wins") on purpose: a PATH `ffmpeg` may be a different major version or a **GPL** build, and the
|
||
bundle is the one that was pinned, license-vetted and tested. If neither hits, it throws a loud
|
||
`IOException` naming the missing folder and the fetch script — the old "found something on disk"
|
||
cache path is gone deliberately, because silently satisfying a missing bundle with a stale copy is
|
||
exactly the silent drift a certification-gated dependency must not have.
|
||
|
||
**The bundle is not in git** (135 MB of incompressible DLLs would tax every commit on this
|
||
9p-bridged tree). `scripts/fetch-ffmpeg.sh` populates `./ffmpeg` from a pinned month-end BtbN
|
||
LGPL-**shared** win64 artifact, **verifies SHA-256 before extracting**, and is idempotent; set
|
||
`FFMPEG_ZIP=<path>` to reuse a pre-downloaded archive. `ytLive.csproj` copies `ffmpeg\**` to the
|
||
output (`CopyToPublishDirectory="Always"` for the MSIX package) and the `GuardBundledFfmpeg` target
|
||
**fails the build** if the folder is absent, so a fresh clone finds out before the creator presses
|
||
Record rather than during a take. `ffmpeg.exe` + `ffprobe.exe` + the `libav*.dll` family must stay in
|
||
one folder — Windows resolves a shared build's DLLs from the executable's own directory.
|
||
`ffplay.exe` (18 MB) is deliberately **not** shipped; nothing calls it.
|
||
|
||
BtbN LGPL-shared (not gyan.dev, not static): it drops GPL-only libx264/x265 while keeping
|
||
NVENC/QSV/AMF + libopenh264 + native AAC, and dynamic linking means LGPL compliance is "license text +
|
||
source offer" with no static-relink (§6) material — see the Licensing guardrails below. libopenh264
|
||
is **compiled into `avcodec-63.dll`** (verified 2026-09-28: the encoder is registered there, and there
|
||
is no separate OpenH264 binary to ship), so Cisco's BSD terms + patent grant ride along inside item 1
|
||
of `THIRD-PARTY-NOTICES.txt`. Pin: `autobuild-2026-08-31-13-27` (month-end on purpose — BtbN keeps
|
||
~14 dailies but month-end builds ~2 years; a daily pin is what 404'd, see below).
|
||
|
||
**⛔ Why this was rewritten (2026-09-27 → shipped 2026-09-28).** The pinned-download design was a
|
||
defect, not a cold-start edge case. It **failed in production on 2026-09-01** — the previous daily
|
||
pin aged out of BtbN's 14-day retention and the download 404'd on the creator's first real recording
|
||
attempt. Two facts made "the pin is a const, just bump it" wrong:
|
||
- **Microsoft Store policy 10.2.2 forbids dynamic code inclusion.** Downloading an executable
|
||
from the internet and running it is the exact pattern the policy names. An MSIX submission
|
||
is rejected over it.
|
||
- Bumping the pin is whack-a-mole against a retention window we do not control.
|
||
|
||
Bundling deletes the startup network dependency entirely. The LGPL-**shared** build choice is
|
||
**unaffected** — dynamic linking was chosen for LGPL §6 compliance ("license text + source offer"),
|
||
not to enable downloading. Analysis: `TASKS/research-store-certification.md` §4.
|
||
**Guardrail:** `FfmpegLocatorTests.TheLocatorCarriesNoNetworkCapability` asserts the locator's
|
||
source contains no `HttpClient`/URL/downloader at all, so the policy cannot rot back.
|
||
|
||
**Every other third-party binary already ships via NuGet — ffmpeg was the only download-and-execute.**
|
||
Verified 2026-09-28: `e_sqlite3.dll` and `WebView2Loader.dll` land in `runtimes\win-x64\native\`
|
||
(SQLitePCLRaw + WebView2 packages) and NAudio/Notification.Wpf/Microsoft.Data.Sqlite/ProtectedData
|
||
are managed. Two gaps remain and are **not** fixed by bundling: the **WebView2 Evergreen runtime** is
|
||
a machine prerequisite (a manifest framework-dependency + graceful-failure path, TASK 48 item 2), and
|
||
**Velopack's updater downloads and executes** — its teardown is a compliance action, not just cleanup.
|
||
|
||
### Live encoder + RTMP push (TASK 4 ship step 3 — shipped 2026-08-12, plan in TASKS.md)
|
||
|
||
The encoder is a **thin orchestrator over `ffmpeg.exe`** — no H.264/AAC code in the app. It spawns the
|
||
subprocess (path from `IFfmpegLocator`), feeds raw BGRA compositor-crop frames into stdin, lets FFmpeg
|
||
scale them to the selected output tier, and parses `-stats`
|
||
stderr lines into `StreamHealth` (bitrate/FPS/duration, dropped-from-frame-count). `FfmpegEncoder`
|
||
(`IFfmpegEncoder` seam) holds: `StartAsync` (locate → probe `-encoders` → spawn → stderr loop →
|
||
drain loop), `SubmitFrameAsync` (slice 10: bounded-queue ENQUEUE — never a pipe write; the drain task
|
||
owns stdin writes), `StopAsync` (flush queue → stdin EOF → ffmpeg
|
||
finalizes + exits by itself; a 10s watchdog kills it), `Dispose` (force-kill + wait), and the
|
||
`HealthUpdated`/`ProcessFailed` events. **Pattern:** the encoder never touches `Process` — it drives the
|
||
`IEncoderProcess` seam (`FfmpegEncoderProcess` wraps the real `Process`, redirected stdin/stdout/stderr
|
||
+ exit control); a `Func<IEncoderProcess>` factory + the locator are constructor-injected, so the
|
||
integration test fakes the whole subprocess (probe + encoder) with a Channel-backed `TextReader` whose
|
||
`Complete()` is EOF (`null`), never a `ChannelClosedException`.
|
||
|
||
**Decisions (locked):** args are pure (`FfmpegArgs.Build`, no string building in the encoder):
|
||
`-f rawvideo -pix_fmt bgra -video_size inputWxH -framerate FPS -i pipe:0` (**NO `-re`** — the FramePump
|
||
is the pacer since slice 9, 2026-09-10; `-re` added a second, fighting clock on the rawvideo demux).
|
||
When input and output dimensions differ, each output block adds `-vf scale=outputW:outputH:flags=bilinear,setsar=1`;
|
||
otherwise frames pass through at their existing dimensions.
|
||
+ a **real audio input — the
|
||
mixer writes IEEE-float stereo to a Windows named pipe** (`-f f32le -ar 48000 -ac 2
|
||
-i \\.\pipe\ytllive_audio`, name via `EncoderOptions.AudioPipeName`; replaced the old `-f lavfi -i
|
||
anullsrc` silence in the TASK 8 audio milestone) + explicit `-map 0:v -map 1:a` + `-c:v <enc> -b:v K
|
||
-maxrate K -bufsize 2K` + **`-g fps×4 -keyint_min fps×4 -sc_threshold 0 -bf 0 -pix_fmt yuv420p`**
|
||
(≤4s keyframes, closed GOP, H.264 compliance) + `-c:a aac -ar 48000 -ac 2 -f flv <rtmpUrl>`.
|
||
**Record output (TASK 18):** `FfmpegArgs.Build` emits one self-contained block per output, each its own
|
||
`-map 0:v -map 1:a` + codec tags (`AddVideoTags` helper). Stream block = `-f flv <rtmpUrl>`; record block =
|
||
`-f mp4 <RecordPath>`. `EncoderOptions.StreamEnabled`/`RecordEnabled`/`RecordPath` gate each block, so
|
||
the engine runs record-only (no RTMP) or stream-only — **stream+record simultaneously is OUT by creator
|
||
ruling (2026-09-01): the VOD is already the copy, and dual-encoding drags mid-range chassis and degrades
|
||
BOTH outputs ("we're not them"). The old "cheap on NVENC" claim was an unverified assumption; the
|
||
UI constraint is landed (2026-09-20): the REC/ON-AIR pills are radio-exclusive — arming one clears the
|
||
other — while the dual-block machinery stays generic.**
|
||
`FfmpegEncoder.StartAsync` throws unless at least one output is enabled.
|
||
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
|
||
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
|
||
forces one and skips the probe. `EncoderOptions` carries raw input W×H from the compositor crop, final
|
||
output W×H/FPS/bitrate from the quality tier, and
|
||
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
|
||
driven by the `FramePump` below.
|
||
|
||
---
|
||
|
||
## Recording
|
||
|
||
### Local recording (TASK 18 — shipped 2026-08-29, VM + top-bar UX; plan in TASKS.md)
|
||
|
||
Recording is **independent from streaming** and needs no YouTube sign-in. A single segmented
|
||
**REC|ON-AIR switch** in the top bar picks the mode (redesigned 2026-09-22 from the old two pills):
|
||
**REC** (local file, works signed-out) / **ON-AIR** (streaming; **armable signed-out** — intent may
|
||
light while offline, 2026-09-20: a greyed pill was the dead end the guard shipped to forbid). The switch
|
||
selection is intent; `IsRecording`/`IsLive` are reality — the reality line (dot + word + elapsed) only
|
||
lights when a session is actually running.
|
||
|
||
- **State model (`MainViewModel`):** pills `RecordPillOn`/`OnAirPillOn` stay the **routing model**
|
||
(radio-exclusive — arming one clears the other, record-OR-live 2026-09-01; the ON-AIR setter stays
|
||
armable signed-out (2026-09-20: intent must be able to light while offline)). **Top bar redesign
|
||
(2026-09-22):** the bar is ONE surface rendered by the FIRST decision — Record or Stream — and each
|
||
world is the whole bar (`ShowIdleWorld`/`ShowRecordAction`/`ShowStreamActions`, wired by
|
||
`RaiseTopBarModes()`). The old sign-lights + two pill toggles + status dot + PRIVATE/TEST chips are
|
||
gone; the mode is one **segmented REC|ON-AIR switch** (`SegmentToggle`/`SegmentLabel` styles), and
|
||
"going back" is one tap on the other segment. Idle worlds: Record = switch + "Start Recording"
|
||
(`PrimaryStartButtonLabel` names the armed world); Stream = switch
|
||
+ "Go Live" + Test (Test is a **child of Stream** — visible only stream-armed AND signed-in) plus the
|
||
account zone on the right. **The account zone is WORLD-INDEPENDENT (2026-09-23):**
|
||
`ShowSignInButton` = signed-out and `ShowAvatarButton` = the signed-in avatar, in Record and Stream
|
||
worlds alike — it's app-level identity, and keeping it put pins the bar height (it no longer pops in
|
||
and out between worlds). Right-click the avatar = Change Account/Logout. Running retires the switch
|
||
and worlds; one reality line
|
||
`● word elapsed` + End replace them (`RunningVisible`/`RunningText`/`RunningDotBrush` — TEST/LIVE/REC,
|
||
gold/red/green). The account **status dot is gone** — the avatar/Sign In pair carries connectedness.
|
||
`ShowPrimaryStartButton` remains the always-idle face. `StopStream` clears both pills (see "Stop ends
|
||
everything; failures roll back" below).
|
||
- **Filenames** (pure `Services/RecordingFile`): auto-name `ty-<yyyyMMdd>-<HHmm start>-<fff>-0000.mp4` at
|
||
start; **rename-on-stop** to `ty-…-<hh2mm2 actual length>.mp4` (`FinalizeRecordingAsync` after the pump
|
||
stops & the file closes), numeric `-2`/`-3` suffix on collision (`UniquePath`). Length comes from
|
||
`_liveElapsed`, which the session timer walks while `IsLive || IsRecording`. The **`<fff>` millisecond
|
||
segment is load-bearing, not decoration** (2026-09-27): both names derive from `_recordStartTime`, so it
|
||
survives the rename, and without it two instances started in the same minute compute the *same* start
|
||
path and both ffmpeg processes write one file — which is exactly the dev multi-instance case
|
||
(`Helpers/InstanceProfile`, one instance recording the other).
|
||
- **Private broadcasts end the same way (creator ruling 2026-10-09).** After `StopStream`'s
|
||
transition(complete), `OfferToKeepPrivateBroadcastAsync` reads the broadcast's privacy **from YouTube**
|
||
(`GetBroadcastPrivacyAsync` — visibility can change mid-stream, local state isn't trusted) and, only if
|
||
it is `private`, shows `KeepBroadcastDialog` (repo root): **Enter = Keep** (`IsDefault`), **Escape /
|
||
Delete / the X = delete** via `DeleteFinishedBroadcastAsync` — `videos.delete?id=<broadcastId>` (a live
|
||
broadcast's id IS its video id; 204), falling back to `liveBroadcasts.delete` on 404. An unreadable
|
||
status asks nothing and deletes nothing; a failed delete toasts that the video is still on YouTube.
|
||
Decision logic = `MainViewModel.ClosePrivateBroadcastAsync` (internal static, tested in
|
||
`PrivateBroadcastCloseOutTests`); `KeepBroadcastPromptOverride` is the dialog seam.
|
||
- **Save dialog: Cancel DISCARDS (creator ruling 2026-09-26).** The `RenameRecordingDialog` result is a
|
||
two-outcome decision, and it lives in the extracted `MainViewModel.CompleteRecordingSave(startPath, dir,
|
||
autoStem, chosenStem, creatorSaved)` (internal, so `ytLive.Tests` can drive it against real files without
|
||
a frame pump — see `ytLive.Tests/RecordingSaveDialogTests.cs`):
|
||
- `creatorSaved == false` (**Cancel / Escape / the X**) → **DELETE the temp file.** Nothing is moved,
|
||
nothing is left in the videos folder. A failed delete returns `DiscardFailed` and the toast NAMES the
|
||
file + folder, because a "discarded" recording still on disk is worse than no report.
|
||
- `creatorSaved == true` (Save, or **Enter on the pre-filled default**) → `File.Move` to the final name.
|
||
A blank/whitespace box still means "keep the auto name" — but only on an explicit Save.
|
||
- **The bug this fixed:** the caller used to `File.Move` UNCONDITIONALLY and only overwrite `stem` when
|
||
the dialog returned true, so `ShowDialog() == false` fell straight through into the save — Cancel
|
||
silently kept the recording under the default name. The modal itself was always correct; the
|
||
side effect was in the wrong place. **Any modal whose result gates a side effect must have every
|
||
branch handled explicitly — a falsy result must never fall through into the effect.**
|
||
- **Folder:** default `%APPDATA%\ytLlive\recordings\`, user-overridable via `ChooseRecordFolderCommand`
|
||
(`OpenFolderDialog`), persisted through `LayoutStore.Load/SaveRecordFolder` (`RecordFolder` key).
|
||
- **Explicit sign-out only:** `StopStream` no longer clears the session/token. Sign out via **Logout
|
||
only** — Change Account re-authenticates *in place* and deliberately does not sign out first
|
||
(2026-09-27; signing out first deleted `ytLlive.auth` before the browser even opened, so a
|
||
cancelled switch lost a working session). Stopping a recording leaves the creator signed in.
|
||
- **Stop ends everything; failures roll back (2026-09-01):** `StopStream` also clears both intent
|
||
pills (`RecordPillOn`/`OnAirPillOn`) — a lit pill with no session behind it is a lie. Every
|
||
frame-pump death and every go-live prep failure now runs the full `StopStream` teardown instead of
|
||
leaving a limbo (`StreamStatus.Error` with `IsRecording=true` over a dead encoder — the first-launch
|
||
zombie recording). Toast copy distinguishes "Recording stopped" from "stream pipeline stopped";
|
||
`AudioMixer.StopLive` is idempotent-safe for rollbacks that never reached StartLive. Seams:
|
||
`OnFramePumpFailed` + `IsRecording` setter made internal (test-only, InternalsVisibleTo).
|
||
Test: `SessionTeardownTests`.
|
||
- `EncoderOptions.StreamEnabled/RecordEnabled/RecordPath` gate outputs; record+stream is one ffmpeg with two
|
||
output blocks (the dual-block machinery is unreachable from the UI since the pills are radio-exclusive).
|
||
Rest of the engine (FramePump/encoder) is unchanged — `BuildEncoderOptions` always returns an options
|
||
(flags from the pills + `_activeRecordPath`), so record-only reaches the pump.
|
||
|
||
---
|
||
|
||
## Audio
|
||
|
||
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12; game audio bar 2026-08-13; **TASK 8 audio milestone: real stream audio + filters + duck + TRAX, shipped 2026-08-14**, plan in TASKS.md)
|
||
|
||
**Capture runs for the app's lifetime and is KISS by rule**: desktop/game audio is automatic (WASAPI
|
||
loopback), the mic is the creator's only audio control (meter/mute/volume already shipped). The whole
|
||
layer sits behind an **`IAudioSource` seam** (`Services/Audio/`: `Start`/`Stop`/`Started`/`SampleReady`/
|
||
`Failed`, IDisposable) so the app never touches NAudio directly and tests inject hermetic fakes (no
|
||
devices, no timers).
|
||
|
||
- **Sources (NAudio 2.2.1 — `NAudio.Wasapi` + `NAudio.WinMM` (for `WaveOutEvent`), MIT — item 9 in
|
||
`THIRD-PARTY-NOTICES.txt`):** `WasapiLoopbackAudioSource` = `WasapiLoopbackCapture` on the default
|
||
render device; `WasapiMicAudioSource` = `WasapiCapture` with the device resolved by `FriendlyName`
|
||
matching `MicSourceName` (the app only persists the **DisplayName**), falling back to the default
|
||
capture endpoint. Both run at the **device's own mix format** — NAudio 2.2.1 exposes no overridable
|
||
`GetDefaultMixFormat` on `WasapiCapture`, so there is no forced 48 kHz path; the mixer's
|
||
`TinyResampler` normalizes any rate/channel layout to 48 kHz stereo (this replaced the earlier
|
||
"force IEEE-float 48k" plan — the resampler IS the design). Mic device resolution re-reads the name
|
||
provider `Func<string?>` at each `Start`, so a mic picked mid-session takes effect **immediately**
|
||
(the mixer restarts the mic on pick). Both sources raise `Started` once their capture loop actually
|
||
begins — the mixer turns that into `MicConnected`.
|
||
- **`AudioMixer`** owns both sources; **`StartMicCaptureAsync` starts the mixer once at startup** and
|
||
`Shutdown` disposes it — NOT go-live — so the meters preview live. Mic samples: downmix to mono →
|
||
`TinyResampler` → **`VoiceFilterChain` per sample** (bass `LowShelfFilter` 120 Hz +4 dB → treble
|
||
`HighShelfFilter` 8 kHz +3 dB → `NoiseGate` (threshold 0.005, hysteresis 0.5, attack 0.5, release
|
||
0.0005) → **`SpeechLeveler`** (2026-10-09: automatic gain steering speech RMS to −20 dBFS over a
|
||
0.3 s window, gain −6..+20 dB, rises ≤10 dB/s / falls ≤30 dB/s, learns only while the gate is open
|
||
and above −54 dBFS so pauses/hum are never pumped) → `Compressor` (threshold 0.5, ratio 4:1); all
|
||
pure DSP) → ring buffer → the
|
||
**post-filter** `AudioLevelMeter` (RMS, 0.2 exponential smoothing) → `MicLevelChanged` →
|
||
marshalled to the UI thread → `AudioLevel`. Loopback samples: resample → ring buffer → second meter
|
||
→ `LoopbackLevelChanged` → the game bar's `GameAudioLevel`. The raw linear level maps to the meter's
|
||
display scale by `AudioLevelMeter.ToDisplay` (−60..0 dBFS → 0..1, **honest** — the +10 dB display boost
|
||
was removed 2026-10-09; the mic meter is `ToDisplay(AudioLevel × MicVolume)`, true dB math, no ×1.2).
|
||
**Why (2026-10-09):** a creator's YouTube upload measured −28.5 LUFS / −8.9 dBFS peak against YouTube's
|
||
−14 LUFS reference (YouTube only turns loud videos down, never quiet ones up) while the boosted meter
|
||
looked "dialed in". Nothing in the chain added gain — the compressor folds only above −6 dBFS and the
|
||
limiter only caps. Re-running that same audio through the new chain (mic volume 0.8): **−17.9 LUFS,
|
||
−0.7 dBFS peak**. Recording and streaming share this one mix, so both were (and now are) the same.
|
||
Measure with the bundled ffmpeg: `ffmpeg -i file -af ebur128=peak=true:framelog=quiet -f null -`.
|
||
**Mic connection state surfaces as events:** `MicConnected` (source `Started`), `MicFailed` (source
|
||
`Failed`), and `RestartMic()` re-resolves + restarts just the mic (loopback keeps running). Failures
|
||
log via `AppLog`; a mic failure zeroes the meter, a loopback failure never kills the mic. Meter `Push`
|
||
is unconditional (a `?.` on the event would skip the meter update when nothing is subscribed yet).
|
||
- **Go-live audio (TASK 8):** `BeginGoLive` → `StartLive(pipeName)` (idempotent) creates the named pipe
|
||
server (`NamedPipeAudioWriter`, behind the `IAudioPipeWriter` seam) and a live loop task; every
|
||
`mixInterval` (default 10 ms) `FillAndMix` reads a chunk from each ring buffer (silence-fills
|
||
underruns), computes the post-filter mic RMS → `AutoDucker` (threshold 0.02, duck ×0.25 / −12 dB,
|
||
attack 0.05, release 0.005, always on), applies the **honest gains** via `Func<double>` seams
|
||
(`micGain = MicMuted ? 0 : MicVolume`; `loopbackGain = GameMuted ? 0 : GameAudioVolume`, × duck),
|
||
mixes mono mic + stereo loopback into interleaved stereo, runs the
|
||
**`MasterLimiter`** (a **−1 dBFS ceiling** so the honest gains can never sum past 0 dBFS and clip the
|
||
AAC encode — instant attack per frame, smoothed release toward unity), and writes the floats to the pipe. **The gains are wired through `AudioGainProvider`**
|
||
(`Services/Audio/AudioGainProvider.cs`) — the VM hands its `MicGain`/`LoopbackGain` Funcs to the
|
||
mixer at construction, read live each tick, so the volume sliders and mute buttons are stream-honest.
|
||
(Before this wiring the seams defaulted to unity and the controls were decorative.) `StopStream` →
|
||
`StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) —
|
||
the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the
|
||
pipe lifecycle. **Recordings are fragmented MP4 (TASK 32·4, 2026-11-05):** the record block carries
|
||
`-movflags +frag_keyframe+empty_moov+default_base_moof`, so the index is written first and a playable
|
||
fragment is flushed at every keyframe. A plain MP4 puts its index at the end, which is why a power cut
|
||
or a killed process previously left the whole take unreadable. The FLV stream block never gets it.
|
||
**Two clip-audio rings join that mix (2026-11-05):** `EnqueueAlertAudio` (TASK 47) is pre-scaled by the
|
||
alert layer and mixed at **unity, never ducked** — an alert must be heard over the mic.
|
||
`EnqueueDesktopMediaAudio` (TASK 21's audio slice) is the opposite on purpose: a video layer's
|
||
soundtrack joins the **desktop/game ring**, so the desktop fader and the auto-ducker apply exactly as
|
||
they do for game audio and TRAX. Both are fed by the shared `Services/VideoClipPipe.cs`; the layer's
|
||
own Mute/Level are applied upstream by `MediaVideoSourceManager` (level/mute are per **session**, since
|
||
one decoder is shared per file path).
|
||
- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo
|
||
mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose
|
||
offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500
|
||
ms**, persisted as `Audio.SyncOffsetMs`). This is OBS's documented fix for lip-sync: **positive**
|
||
delays audio when it runs ahead of video; **negative** advances audio when it runs behind by eating
|
||
the first `|N|` ms of the live stream head (see `AudioMixer.StartLive`; the delay line stays
|
||
clamped 0..500 internally — negative bypasses it entirely). The advance is armed once at `StartLive`
|
||
(eating the head mid-stream is impossible); positive is live-reactive (re-read per tick). A slider
|
||
on the mic bar (`"AUDIO SYNC"`, locked while live/recording via `IsEditMode`) surfaces it
|
||
(a one-time status dot next to it was removed 2026-09-23 per creator feedback — the slider +
|
||
numeric tooltip carry the state). Changing the delay flushes the line (a live change clicks
|
||
rather than smears).
|
||
- **TRAX — free background music (TASK 8):** `MusicPlayer` = NAudio `MediaFoundationReader`
|
||
(mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** bed (no slider) → `WaveOutEvent` on
|
||
the default device, **looping on any clean natural end** (`PlaybackStopped` with `e.Exception == null`
|
||
→ rewind + replay; the earlier `Position >= Length` check was unreliable for `MediaFoundationReader`
|
||
and let tracks play once then stop). The desktop/game bar's mute toggles `LocalGain` (0 or 1;
|
||
`UpdateTraxLocalGain` runs on every `GameMuted` change and on TRAX load), so muting kills TRAX
|
||
in the headphones — matching the stream, where it rides the loopback channel. The volume slider
|
||
drives system volume (`PushSystemVolume`) — which scales the HEADPHONES; the loopback capture is
|
||
pre-endpoint-volume (disproven-then-fixed 2026-09-01), so the STREAM follows the knob via
|
||
`AudioGainProvider.LoopbackGain = GameMuted ? 0 : GameAudioVolume`, and the game meter multiplies
|
||
its display by the same. One knob, three honest paths. No third mixer input. The footer
|
||
**TRAX** button lives **inside the game audio bar** (in the preview overlay), **left of the
|
||
"Desktop Audio" label** (the creator's pick — it rides the desktop channel, so the music control
|
||
sits with the desktop audio controls): status dot (**red** no track / **yellow** loaded stopped /
|
||
**green** playing) + "TRAX"; **left-click** toggles play/pause (opens the in-app picker via `OpenFileDialog` when no track is loaded); **right-click** always opens
|
||
the picker (`TraxButton_PreviewMouseRightButtonUp`, `e.Handled = true`, code-behind pattern); tooltip
|
||
shows the loaded/playing track name or "No track — right-click to choose background music". The picked
|
||
track persists via **schema v9** single-row `Music` (`TrackPath`/`IsEnabled`) in `LayoutStore`. Known
|
||
wrinkle (out of scope, future feature): YouTube mutes VODs carrying copyrighted music — "music on
|
||
live, off VOD" is tabled.
|
||
- **⛔ The load is behind a bounded probe, because a share that is gone must never
|
||
freeze startup (2026-10-04).** `LoadLayout` → `LoadTrax` used to call `MusicPlayer.Load`
|
||
synchronously on the UI thread, and `new MediaFoundationReader(path)` does **not** fail
|
||
fast on a dead SMB session — it burns the whole ARP/TCP/SMB timeout (~40s, measured)
|
||
and *then* throws `0x80070035`. Measured: `OnStartup` 08:42:59 → `MainWindow loaded`
|
||
08:43:43, with 42 of those 44s inside `LoadLayout` on that one track. The rule now:
|
||
**a probe may time out; only a bind is allowed to be on the creator's path.**
|
||
`IsTrackReachableAsync(path, budget, File.Exists)` runs the probe on a thread-pool
|
||
thread under a `Task.WhenAny` ceiling (`TraxProbeBudget` = **2s**) and reports
|
||
offline/missing/slow as a normal state, not an error. Only a probe that passed gets
|
||
the real bind, and **that bind stays on the UI thread** with every other
|
||
`MusicPlayer` mutation, so no new cross-thread race is introduced. The orphan probe
|
||
thread stays parked until SMB gives up — deliberate: it is not the thread anyone
|
||
waits on. Same principle covers the webcam: a device held by another app fails fast
|
||
(sharing violation) but a virtual-camera driver can stall enumeration, so device
|
||
probes belong off the startup path too, not behind a long timeout.
|
||
- **Mic status dot (`Models/MicStatus.cs`)** on the preview-bottom MIC button: green = `MicConnected`,
|
||
yellow = `MicFailed` (in use/unplugged), red = no mic device at startup (the mixer is never started,
|
||
so loopback and the game bar can't run either — no capture devices at all). **The picked mic persists:
|
||
`LayoutStore`'s `Settings` key/value table stores `MicSourceName` (saved on pick via
|
||
`SaveMicSourceName`), and the VM restores it at construction BEFORE the mixer's first `Start` — so a
|
||
restart reconnects the same already-vetted device (green) or reports it missing (yellow), instead of
|
||
silently falling back to the default endpoint.**
|
||
- **Game audio bar** (desktop/game, **always visible** — TASK 4's `IGameAudioDetector` show/hide gating was **removed 2026-08-15**: the bar used to appear only while a full-screen game with sound was up, which kept hiding the creator's desktop meter; the detector stack is gone, desktop audio is just automatic WASAPI loopback): overlaid **at the bottom of the preview window** (bottom-center, dark translucent
|
||
chip, a mirror of the mic bar: meter + mute + volume slider). It's monitoring UI **in the preview
|
||
grid** — the meter is display-only (fill = `ToDisplay(GameAudioLevel) × GameAudioVolume` — the ×volume
|
||
term was missing until 2026-09-01: loopback capture does NOT shrink with the endpoint, so an
|
||
unscaled meter left it pegged at 20% volume; `GameMeterHonestyTests` guards it), but its **volume
|
||
slider drives system volume** (`PushSystemVolume` — live headphones follow the knob; the capture
|
||
side is scaled in the mix, see TRAX note above) **and mute kills TRAX** (`MusicPlayer.LocalGain` = 0). Relabelled
|
||
**"Desktop Audio"** (TASK 8) since TRAX rides the same channel.
|
||
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
|
||
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
|
||
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
|
||
- Build **0 warnings**; **197 passing** (DSP/ring-buffer/ducker/resampler/pipe + mixer/hysteresis/
|
||
game-detector/meter-scale unit tests + the TASK 8 integration test
|
||
`AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe` + the polish-batch integration
|
||
test `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` proving the provider gains reach the
|
||
pipe and mute silences the loopback).
|
||
---
|
||
|
||
## Related files
|
||
|
||
- [`frame-pipeline.md`](frame-pipeline.md) — the `FramePump` that feeds this encoder (pacing, backpressure, teardown order).
|
||
- [`ui-surface.md`](ui-surface.md) — resolution tiers (W×H/FPS/bitrate) and toast notifications.
|
||
- [`youtube-api.md`](youtube-api.md) — go-live/prep ordering and the broadcast lifecycle.
|
||
- [`product.md`](product.md) — licensing guardrails (the ffmpeg pin, LGPL obligations).
|
||
- [`TASKS/research-store-certification.md`](../../TASKS/research-store-certification.md) — §4, the ffmpeg bundling analysis.
|