diff --git a/MyMistakes.md b/MyMistakes.md index 1c2d351..bbbd9d3 100644 --- a/MyMistakes.md +++ b/MyMistakes.md @@ -1227,7 +1227,7 @@ clothes, and the failure mode is invisible because the sentence still *sounds* a Corollary: **a stale external fact inside a shipped document is worse than a missing one** — it is trusted, acted on, and costed. When a doc makes a factual claim about the outside world (certificates, platform behaviour, policy text), re-verify it on touch, not on memory. This is -the same class as the dead-Pin incident in `FfmpegLocator` (`ai.md` → FFmpeg locator): an +the same class as the dead-Pin incident in `FfmpegLocator` ([`docs/ai/output.md`](docs/ai/output.md) → FFmpeg locator): an external thing changed, our record of it did not, and the mismatch was discovered by a user rather than by a test. **Records of the outside world rot silently and are believed until they cost money.** diff --git a/ai.md b/ai.md index 3c0700d..adf6337 100644 --- a/ai.md +++ b/ai.md @@ -264,7 +264,7 @@ C# / WPF (.NET 8) following MVVM: |------|------| | `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** | | `ViewModels/` | MainViewModel — `public partial class`, one file per functional area (Scenes, Background, Webcam, Audio, Trax, Socials, Streaming, Chat, Overlays, Account, License, Recording — split complete, see `ViewModels/index.md`); **Chat.cs is a thin delegating facade over `Services/ChatOverlayLayer.cs` (Commit G, first true decomposition), and now also over `Services/AlertOverlayLayer.cs` (TASK 43 — the native alert box, same true-component pattern)**; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** | -| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Scene compositor")**, **alerts: `ChatOverlayLayer` + `AlertOverlayLayer`/`AlertRenderer` (the native celebration-zone overlay, TASK 43 — see "Native events & alerts")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `ITraxPlayer` (TRAX toggle seam — FakePlayer in tests; the dot is green/red/gray and the error scrolls terse in the desktop-audio bar; MusicVolume fixed 0.5) + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** | +| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Scene compositor")**, **alerts: `ChatOverlayLayer` + `AlertOverlayLayer`/`AlertRenderer` (the native celebration-zone overlay, TASK 43 — see "Native events & alerts")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `ITraxPlayer` (TRAX toggle seam — FakePlayer in tests; the dot is green/red/gray and the error scrolls terse in the desktop-audio bar; MusicVolume fixed 0.5) + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see [`docs/ai/output.md`](docs/ai/output.md) → "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see [`docs/ai/output.md`](docs/ai/output.md) → "Live encoder" + [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** | | `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, **BuildStamp (wordmark release counter `#N`, +1 per commit, generated by `GenerateBuildStamp` in ytLive.csproj via git rev-list; the per-build GUID now logs to startup.log only — 2026-09-04)**, visibility converters | | `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) | | `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview + live controls below), left (scenes/sources), right (chat), bottom (gear + stream stats + resolution) | @@ -455,304 +455,35 @@ Invariants that hold even if you never open that file: - **Read render numbers in Release only.** Debug pixel paths are several× slower; a Release 1080p composite measures ~1.0–1.1ms against the 16.67ms budget, and only Release takes are evidence. -### FFmpeg locator (TASK 4 ship step 2 — shipped 2026-08-10; **rewritten to BUNDLED 2026-09-28**, plan in TASKS.md) +### Output — ffmpeg, encoder, recording & audio — moved -The encoder's one external dependency is `ffmpeg.exe`. **It is BUNDLED — never downloaded, never -fetched at runtime.** `IFfmpegLocator` resolves an absolute path on demand: **the bundle first** -(`\ffmpeg\ffmpeg.exe`), then **PATH** as the escape hatch for someone who -deliberately installed their own. The order is the opposite of the original ("the user's own install -wins") on purpose: a PATH `ffmpeg` may be a different major version or a **GPL** build, and the -bundle is the one that was pinned, license-vetted and tested. If neither hits, it throws a loud -`IOException` naming the missing folder and the fetch script — the old "found something on disk" -cache path is gone deliberately, because silently satisfying a missing bundle with a stale copy is -exactly the silent drift a certification-gated dependency must not have. +The bundled `ffmpeg.exe` decision, the encoder subprocess (`FfmpegEncoder`/`IEncoderProcess`), local +recording, and the whole WASAPI capture → filter → mix → named-pipe audio chain live in +**[`docs/ai/output.md`](docs/ai/output.md)** — read it before touching ffmpeg args, the encoder +seams, REC/ON-AIR, or the audio mix. -**The bundle is not in git** (135 MB of incompressible DLLs would tax every commit on this -9p-bridged tree). `scripts/fetch-ffmpeg.sh` populates `./ffmpeg` from a pinned month-end BtbN -LGPL-**shared** win64 artifact, **verifies SHA-256 before extracting**, and is idempotent; set -`FFMPEG_ZIP=` to reuse a pre-downloaded archive. `ytLive.csproj` copies `ffmpeg\**` to the -output (`CopyToPublishDirectory="Always"` for the MSIX package) and the `GuardBundledFfmpeg` target -**fails the build** if the folder is absent, so a fresh clone finds out before the creator presses -Record rather than during a take. `ffmpeg.exe` + `ffprobe.exe` + the `libav*.dll` family must stay in -one folder — Windows resolves a shared build's DLLs from the executable's own directory. -`ffplay.exe` (18 MB) is deliberately **not** shipped; nothing calls it. +Invariants that hold even if you never open that file: -BtbN LGPL-shared (not gyan.dev, not static): it drops GPL-only libx264/x265 while keeping -NVENC/QSV/AMF + libopenh264 + native AAC, and dynamic linking means LGPL compliance is "license text + -source offer" with no static-relink (§6) material — see the Licensing guardrails below. libopenh264 -is **compiled into `avcodec-63.dll`** (verified 2026-09-28: the encoder is registered there, and there -is no separate OpenH264 binary to ship), so Cisco's BSD terms + patent grant ride along inside item 1 -of `THIRD-PARTY-NOTICES.txt`. Pin: `autobuild-2026-08-31-13-27` (month-end on purpose — BtbN keeps -~14 dailies but month-end builds ~2 years; a daily pin is what 404'd, see below). - -**⛔ Why this was rewritten (2026-09-27 → shipped 2026-09-28).** The pinned-download design was a -defect, not a cold-start edge case. It **failed in production on 2026-09-01** — the previous daily -pin aged out of BtbN's 14-day retention and the download 404'd on the creator's first real recording -attempt. Two facts made "the pin is a const, just bump it" wrong: -- **Microsoft Store policy 10.2.2 forbids dynamic code inclusion.** Downloading an executable - from the internet and running it is the exact pattern the policy names. An MSIX submission - is rejected over it. -- Bumping the pin is whack-a-mole against a retention window we do not control. - -Bundling deletes the startup network dependency entirely. The LGPL-**shared** build choice is -**unaffected** — dynamic linking was chosen for LGPL §6 compliance ("license text + source offer"), -not to enable downloading. Analysis: `TASKS/research-store-certification.md` §4. -**Guardrail:** `FfmpegLocatorTests.TheLocatorCarriesNoNetworkCapability` asserts the locator's -source contains no `HttpClient`/URL/downloader at all, so the policy cannot rot back. - -**Every other third-party binary already ships via NuGet — ffmpeg was the only download-and-execute.** -Verified 2026-09-28: `e_sqlite3.dll` and `WebView2Loader.dll` land in `runtimes\win-x64\native\` -(SQLitePCLRaw + WebView2 packages) and NAudio/Notification.Wpf/Microsoft.Data.Sqlite/ProtectedData -are managed. Two gaps remain and are **not** fixed by bundling: the **WebView2 Evergreen runtime** is -a machine prerequisite (a manifest framework-dependency + graceful-failure path, TASK 48 item 2), and -**Velopack's updater downloads and executes** — its teardown is a compliance action, not just cleanup. - - -### Live encoder + RTMP push (TASK 4 ship step 3 — shipped 2026-08-12, plan in TASKS.md) - -The encoder is a **thin orchestrator over `ffmpeg.exe`** — no H.264/AAC code in the app. It spawns the -subprocess (path from `IFfmpegLocator`), feeds raw BGRA master frames into stdin, and parses `-stats` -stderr lines into `StreamHealth` (bitrate/FPS/duration, dropped-from-frame-count). `FfmpegEncoder` -(`IFfmpegEncoder` seam) holds: `StartAsync` (locate → probe `-encoders` → spawn → stderr loop → -drain loop), `SubmitFrameAsync` (slice 10: bounded-queue ENQUEUE — never a pipe write; the drain task -owns stdin writes), `StopAsync` (flush queue → stdin EOF → ffmpeg -finalizes + exits by itself; a 10s watchdog kills it), `Dispose` (force-kill + wait), and the -`HealthUpdated`/`ProcessFailed` events. **Pattern:** the encoder never touches `Process` — it drives the -`IEncoderProcess` seam (`FfmpegEncoderProcess` wraps the real `Process`, redirected stdin/stdout/stderr -+ exit control); a `Func` factory + the locator are constructor-injected, so the -integration test fakes the whole subprocess (probe + encoder) with a Channel-backed `TextReader` whose -`Complete()` is EOF (`null`), never a `ChannelClosedException`. - -**Decisions (locked):** args are pure (`FfmpegArgs.Build`, no string building in the encoder): -`-f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS -i pipe:0` (**NO `-re`** — the FramePump -is the pacer since slice 9, 2026-09-10; `-re` added a second, fighting clock on the rawvideo demux) -+ a **real audio input — the -mixer writes IEEE-float stereo to a Windows named pipe** (`-f f32le -ar 48000 -ac 2 --i \\.\pipe\ytllive_audio`, name via `EncoderOptions.AudioPipeName`; replaced the old `-f lavfi -i -anullsrc` silence in the TASK 8 audio milestone) + explicit `-map 0:v -map 1:a` + `-c:v -b:v K --maxrate K -bufsize 2K` + **`-g fps×4 -keyint_min fps×4 -sc_threshold 0 -bf 0 -pix_fmt yuv420p`** - (≤4s keyframes, closed GOP, H.264 compliance) + `-c:a aac -ar 48000 -ac 2 -f flv `. -**Record output (TASK 18):** `FfmpegArgs.Build` emits one self-contained block per output, each its own -`-map 0:v -map 1:a` + codec tags (`AddVideoTags` helper). Stream block = `-f flv `; record block = -`-f mp4 `. `EncoderOptions.StreamEnabled`/`RecordEnabled`/`RecordPath` gate each block, so -the engine runs record-only (no RTMP) or stream-only — **stream+record simultaneously is OUT by creator -ruling (2026-09-01): the VOD is already the copy, and dual-encoding drags mid-range chassis and degrades -BOTH outputs ("we're not them"). The old "cheap on NVENC" claim was an unverified assumption; the -UI constraint is landed (2026-09-20): the REC/ON-AIR pills are radio-exclusive — arming one clears the -other — while the dual-block machinery stays generic.** -`FfmpegEncoder.StartAsync` throws unless at least one output is enabled. -**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure): -NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder` -forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and -`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`), -driven by the `FramePump` below. - -### Local recording (TASK 18 — shipped 2026-08-29, VM + top-bar UX; plan in TASKS.md) - -Recording is **independent from streaming** and needs no YouTube sign-in. A single segmented -**REC|ON-AIR switch** in the top bar picks the mode (redesigned 2026-09-22 from the old two pills): -**REC** (local file, works signed-out) / **ON-AIR** (streaming; **armable signed-out** — intent may -light while offline, 2026-09-20: a greyed pill was the dead end the guard shipped to forbid). The switch -selection is intent; `IsRecording`/`IsLive` are reality — the reality line (dot + word + elapsed) only -lights when a session is actually running. - -- **State model (`MainViewModel`):** pills `RecordPillOn`/`OnAirPillOn` stay the **routing model** - (radio-exclusive — arming one clears the other, record-OR-live 2026-09-01; the ON-AIR setter stays - armable signed-out (2026-09-20: intent must be able to light while offline)). **Top bar redesign - (2026-09-22):** the bar is ONE surface rendered by the FIRST decision — Record or Stream — and each - world is the whole bar (`ShowIdleWorld`/`ShowRecordAction`/`ShowStreamActions`, wired by - `RaiseTopBarModes()`). The old sign-lights + two pill toggles + status dot + PRIVATE/TEST chips are - gone; the mode is one **segmented REC|ON-AIR switch** (`SegmentToggle`/`SegmentLabel` styles), and - "going back" is one tap on the other segment. Idle worlds: Record = switch + "Start Recording" - (`PrimaryStartButtonLabel` names the armed world); Stream = switch - + "Go Live" + Test (Test is a **child of Stream** — visible only stream-armed AND signed-in) plus the - account zone on the right. **The account zone is WORLD-INDEPENDENT (2026-09-23):** - `ShowSignInButton` = signed-out and `ShowAvatarButton` = the signed-in avatar, in Record and Stream - worlds alike — it's app-level identity, and keeping it put pins the bar height (it no longer pops in - and out between worlds). Right-click the avatar = Change Account/Logout. Running retires the switch - and worlds; one reality line - `● word elapsed` + End replace them (`RunningVisible`/`RunningText`/`RunningDotBrush` — TEST/LIVE/REC, - gold/red/green). The account **status dot is gone** — the avatar/Sign In pair carries connectedness. - `ShowPrimaryStartButton` remains the always-idle face. `StopStream` clears both pills (see "Stop ends - everything; failures roll back" below). -- **Filenames** (pure `Services/RecordingFile`): auto-name `ty----0000.mp4` at - start; **rename-on-stop** to `ty-…-.mp4` (`FinalizeRecordingAsync` after the pump - stops & the file closes), numeric `-2`/`-3` suffix on collision (`UniquePath`). Length comes from - `_liveElapsed`, which the session timer walks while `IsLive || IsRecording`. The **`` millisecond - segment is load-bearing, not decoration** (2026-09-27): both names derive from `_recordStartTime`, so it - survives the rename, and without it two instances started in the same minute compute the *same* start - path and both ffmpeg processes write one file — which is exactly the dev multi-instance case - (`Helpers/InstanceProfile`, one instance recording the other). -- **Private broadcasts end the same way (creator ruling 2026-10-09).** After `StopStream`'s - transition(complete), `OfferToKeepPrivateBroadcastAsync` reads the broadcast's privacy **from YouTube** - (`GetBroadcastPrivacyAsync` — visibility can change mid-stream, local state isn't trusted) and, only if - it is `private`, shows `KeepBroadcastDialog` (repo root): **Enter = Keep** (`IsDefault`), **Escape / - Delete / the X = delete** via `DeleteFinishedBroadcastAsync` — `videos.delete?id=` (a live - broadcast's id IS its video id; 204), falling back to `liveBroadcasts.delete` on 404. An unreadable - status asks nothing and deletes nothing; a failed delete toasts that the video is still on YouTube. - Decision logic = `MainViewModel.ClosePrivateBroadcastAsync` (internal static, tested in - `PrivateBroadcastCloseOutTests`); `KeepBroadcastPromptOverride` is the dialog seam. -- **Save dialog: Cancel DISCARDS (creator ruling 2026-09-26).** The `RenameRecordingDialog` result is a - two-outcome decision, and it lives in the extracted `MainViewModel.CompleteRecordingSave(startPath, dir, - autoStem, chosenStem, creatorSaved)` (internal, so `ytLive.Tests` can drive it against real files without - a frame pump — see `ytLive.Tests/RecordingSaveDialogTests.cs`): - - `creatorSaved == false` (**Cancel / Escape / the X**) → **DELETE the temp file.** Nothing is moved, - nothing is left in the videos folder. A failed delete returns `DiscardFailed` and the toast NAMES the - file + folder, because a "discarded" recording still on disk is worse than no report. - - `creatorSaved == true` (Save, or **Enter on the pre-filled default**) → `File.Move` to the final name. - A blank/whitespace box still means "keep the auto name" — but only on an explicit Save. - - **The bug this fixed:** the caller used to `File.Move` UNCONDITIONALLY and only overwrite `stem` when - the dialog returned true, so `ShowDialog() == false` fell straight through into the save — Cancel - silently kept the recording under the default name. The modal itself was always correct; the - side effect was in the wrong place. **Any modal whose result gates a side effect must have every - branch handled explicitly — a falsy result must never fall through into the effect.** -- **Folder:** default `%APPDATA%\ytLlive\recordings\`, user-overridable via `ChooseRecordFolderCommand` - (`OpenFolderDialog`), persisted through `LayoutStore.Load/SaveRecordFolder` (`RecordFolder` key). -- **Explicit sign-out only:** `StopStream` no longer clears the session/token. Sign out via **Logout - only** — Change Account re-authenticates *in place* and deliberately does not sign out first - (2026-09-27; signing out first deleted `ytLlive.auth` before the browser even opened, so a - cancelled switch lost a working session). Stopping a recording leaves the creator signed in. -- **Stop ends everything; failures roll back (2026-09-01):** `StopStream` also clears both intent - pills (`RecordPillOn`/`OnAirPillOn`) — a lit pill with no session behind it is a lie. Every - frame-pump death and every go-live prep failure now runs the full `StopStream` teardown instead of - leaving a limbo (`StreamStatus.Error` with `IsRecording=true` over a dead encoder — the first-launch - zombie recording). Toast copy distinguishes "Recording stopped" from "stream pipeline stopped"; - `AudioMixer.StopLive` is idempotent-safe for rollbacks that never reached StartLive. Seams: - `OnFramePumpFailed` + `IsRecording` setter made internal (test-only, InternalsVisibleTo). - Test: `SessionTeardownTests`. -- `EncoderOptions.StreamEnabled/RecordEnabled/RecordPath` gate outputs; record+stream is one ffmpeg with two - output blocks (the dual-block machinery is unreachable from the UI since the pills are radio-exclusive). - Rest of the engine (FramePump/encoder) is unchanged — `BuildEncoderOptions` always returns an options - (flags from the pills + `_activeRecordPath`), so record-only reaches the pump. - -### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12; game audio bar 2026-08-13; **TASK 8 audio milestone: real stream audio + filters + duck + TRAX, shipped 2026-08-14**, plan in TASKS.md) - -**Capture runs for the app's lifetime and is KISS by rule**: desktop/game audio is automatic (WASAPI -loopback), the mic is the creator's only audio control (meter/mute/volume already shipped). The whole -layer sits behind an **`IAudioSource` seam** (`Services/Audio/`: `Start`/`Stop`/`Started`/`SampleReady`/ -`Failed`, IDisposable) so the app never touches NAudio directly and tests inject hermetic fakes (no -devices, no timers). - -- **Sources (NAudio 2.2.1 — `NAudio.Wasapi` + `NAudio.WinMM` (for `WaveOutEvent`), MIT — item 9 in - `THIRD-PARTY-NOTICES.txt`):** `WasapiLoopbackAudioSource` = `WasapiLoopbackCapture` on the default - render device; `WasapiMicAudioSource` = `WasapiCapture` with the device resolved by `FriendlyName` - matching `MicSourceName` (the app only persists the **DisplayName**), falling back to the default - capture endpoint. Both run at the **device's own mix format** — NAudio 2.2.1 exposes no overridable - `GetDefaultMixFormat` on `WasapiCapture`, so there is no forced 48 kHz path; the mixer's - `TinyResampler` normalizes any rate/channel layout to 48 kHz stereo (this replaced the earlier - "force IEEE-float 48k" plan — the resampler IS the design). Mic device resolution re-reads the name - provider `Func` at each `Start`, so a mic picked mid-session takes effect **immediately** - (the mixer restarts the mic on pick). Both sources raise `Started` once their capture loop actually - begins — the mixer turns that into `MicConnected`. -- **`AudioMixer`** owns both sources; **`StartMicCaptureAsync` starts the mixer once at startup** and - `Shutdown` disposes it — NOT go-live — so the meters preview live. Mic samples: downmix to mono → - `TinyResampler` → **`VoiceFilterChain` per sample** (bass `LowShelfFilter` 120 Hz +4 dB → treble - `HighShelfFilter` 8 kHz +3 dB → `NoiseGate` (threshold 0.005, hysteresis 0.5, attack 0.5, release - 0.0005) → **`SpeechLeveler`** (2026-10-09: automatic gain steering speech RMS to −20 dBFS over a - 0.3 s window, gain −6..+20 dB, rises ≤10 dB/s / falls ≤30 dB/s, learns only while the gate is open - and above −54 dBFS so pauses/hum are never pumped) → `Compressor` (threshold 0.5, ratio 4:1); all - pure DSP) → ring buffer → the - **post-filter** `AudioLevelMeter` (RMS, 0.2 exponential smoothing) → `MicLevelChanged` → - marshalled to the UI thread → `AudioLevel`. Loopback samples: resample → ring buffer → second meter - → `LoopbackLevelChanged` → the game bar's `GameAudioLevel`. The raw linear level maps to the meter's - display scale by `AudioLevelMeter.ToDisplay` (−60..0 dBFS → 0..1, **honest** — the +10 dB display boost - was removed 2026-10-09; the mic meter is `ToDisplay(AudioLevel × MicVolume)`, true dB math, no ×1.2). - **Why (2026-10-09):** a creator's YouTube upload measured −28.5 LUFS / −8.9 dBFS peak against YouTube's - −14 LUFS reference (YouTube only turns loud videos down, never quiet ones up) while the boosted meter - looked "dialed in". Nothing in the chain added gain — the compressor folds only above −6 dBFS and the - limiter only caps. Re-running that same audio through the new chain (mic volume 0.8): **−17.9 LUFS, - −0.7 dBFS peak**. Recording and streaming share this one mix, so both were (and now are) the same. - Measure with the bundled ffmpeg: `ffmpeg -i file -af ebur128=peak=true:framelog=quiet -f null -`. - **Mic connection state surfaces as events:** `MicConnected` (source `Started`), `MicFailed` (source - `Failed`), and `RestartMic()` re-resolves + restarts just the mic (loopback keeps running). Failures - log via `AppLog`; a mic failure zeroes the meter, a loopback failure never kills the mic. Meter `Push` - is unconditional (a `?.` on the event would skip the meter update when nothing is subscribed yet). -- **Go-live audio (TASK 8):** `BeginGoLive` → `StartLive(pipeName)` (idempotent) creates the named pipe - server (`NamedPipeAudioWriter`, behind the `IAudioPipeWriter` seam) and a live loop task; every - `mixInterval` (default 10 ms) `FillAndMix` reads a chunk from each ring buffer (silence-fills - underruns), computes the post-filter mic RMS → `AutoDucker` (threshold 0.02, duck ×0.25 / −12 dB, - attack 0.05, release 0.005, always on), applies the **honest gains** via `Func` seams - (`micGain = MicMuted ? 0 : MicVolume`; `loopbackGain = GameMuted ? 0 : GameAudioVolume`, × duck), - mixes mono mic + stereo loopback into interleaved stereo, runs the - **`MasterLimiter`** (a **−1 dBFS ceiling** so the honest gains can never sum past 0 dBFS and clip the - AAC encode — instant attack per frame, smoothed release toward unity), and writes the floats to the pipe. **The gains are wired through `AudioGainProvider`** - (`Services/Audio/AudioGainProvider.cs`) — the VM hands its `MicGain`/`LoopbackGain` Funcs to the - mixer at construction, read live each tick, so the volume sliders and mute buttons are stream-honest. - (Before this wiring the seams defaulted to unity and the controls were decorative.) `StopStream` → - `StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) — - the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the - pipe lifecycle. -- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo - mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose - offset comes from a `Func syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500 - ms**, persisted as `Audio.SyncOffsetMs`). This is OBS's documented fix for lip-sync: **positive** - delays audio when it runs ahead of video; **negative** advances audio when it runs behind by eating - the first `|N|` ms of the live stream head (see `AudioMixer.StartLive`; the delay line stays - clamped 0..500 internally — negative bypasses it entirely). The advance is armed once at `StartLive` - (eating the head mid-stream is impossible); positive is live-reactive (re-read per tick). A slider - on the mic bar (`"AUDIO SYNC"`, locked while live/recording via `IsEditMode`) surfaces it - (a one-time status dot next to it was removed 2026-09-23 per creator feedback — the slider + - numeric tooltip carry the state). Changing the delay flushes the line (a live change clicks - rather than smears). -- **TRAX — free background music (TASK 8):** `MusicPlayer` = NAudio `MediaFoundationReader` - (mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** bed (no slider) → `WaveOutEvent` on - the default device, **looping on any clean natural end** (`PlaybackStopped` with `e.Exception == null` - → rewind + replay; the earlier `Position >= Length` check was unreliable for `MediaFoundationReader` - and let tracks play once then stop). The desktop/game bar's mute toggles `LocalGain` (0 or 1; - `UpdateTraxLocalGain` runs on every `GameMuted` change and on TRAX load), so muting kills TRAX - in the headphones — matching the stream, where it rides the loopback channel. The volume slider - drives system volume (`PushSystemVolume`) — which scales the HEADPHONES; the loopback capture is - pre-endpoint-volume (disproven-then-fixed 2026-09-01), so the STREAM follows the knob via - `AudioGainProvider.LoopbackGain = GameMuted ? 0 : GameAudioVolume`, and the game meter multiplies - its display by the same. One knob, three honest paths. No third mixer input. The footer - **TRAX** button lives **inside the game audio bar** (in the preview overlay), **left of the - "Desktop Audio" label** (the creator's pick — it rides the desktop channel, so the music control - sits with the desktop audio controls): status dot (**red** no track / **yellow** loaded stopped / - **green** playing) + "TRAX"; **left-click** toggles play/pause (opens the in-app picker via `OpenFileDialog` when no track is loaded); **right-click** always opens - the picker (`TraxButton_PreviewMouseRightButtonUp`, `e.Handled = true`, code-behind pattern); tooltip - shows the loaded/playing track name or "No track — right-click to choose background music". The picked - track persists via **schema v9** single-row `Music` (`TrackPath`/`IsEnabled`) in `LayoutStore`. Known - wrinkle (out of scope, future feature): YouTube mutes VODs carrying copyrighted music — "music on - live, off VOD" is tabled. - - **⛔ The load is behind a bounded probe, because a share that is gone must never - freeze startup (2026-10-04).** `LoadLayout` → `LoadTrax` used to call `MusicPlayer.Load` - synchronously on the UI thread, and `new MediaFoundationReader(path)` does **not** fail - fast on a dead SMB session — it burns the whole ARP/TCP/SMB timeout (~40s, measured) - and *then* throws `0x80070035`. Measured: `OnStartup` 08:42:59 → `MainWindow loaded` - 08:43:43, with 42 of those 44s inside `LoadLayout` on that one track. The rule now: - **a probe may time out; only a bind is allowed to be on the creator's path.** - `IsTrackReachableAsync(path, budget, File.Exists)` runs the probe on a thread-pool - thread under a `Task.WhenAny` ceiling (`TraxProbeBudget` = **2s**) and reports - offline/missing/slow as a normal state, not an error. Only a probe that passed gets - the real bind, and **that bind stays on the UI thread** with every other - `MusicPlayer` mutation, so no new cross-thread race is introduced. The orphan probe - thread stays parked until SMB gives up — deliberate: it is not the thread anyone - waits on. Same principle covers the webcam: a device held by another app fails fast - (sharing violation) but a virtual-camera driver can stall enumeration, so device - probes belong off the startup path too, not behind a long timeout. -- **Mic status dot (`Models/MicStatus.cs`)** on the preview-bottom MIC button: green = `MicConnected`, - yellow = `MicFailed` (in use/unplugged), red = no mic device at startup (the mixer is never started, - so loopback and the game bar can't run either — no capture devices at all). **The picked mic persists: - `LayoutStore`'s `Settings` key/value table stores `MicSourceName` (saved on pick via - `SaveMicSourceName`), and the VM restores it at construction BEFORE the mixer's first `Start` — so a - restart reconnects the same already-vetted device (green) or reports it missing (yellow), instead of - silently falling back to the default endpoint.** -- **Game audio bar** (desktop/game, **always visible** — TASK 4's `IGameAudioDetector` show/hide gating was **removed 2026-08-15**: the bar used to appear only while a full-screen game with sound was up, which kept hiding the creator's desktop meter; the detector stack is gone, desktop audio is just automatic WASAPI loopback): overlaid **at the bottom of the preview window** (bottom-center, dark translucent - chip, a mirror of the mic bar: meter + mute + volume slider). It's monitoring UI **in the preview - grid** — the meter is display-only (fill = `ToDisplay(GameAudioLevel) × GameAudioVolume` — the ×volume - term was missing until 2026-09-01: loopback capture does NOT shrink with the endpoint, so an - unscaled meter left it pegged at 20% volume; `GameMeterHonestyTests` guards it), but its **volume - slider drives system volume** (`PushSystemVolume` — live headphones follow the knob; the capture - side is scaled in the mix, see TRAX note above) **and mute kills TRAX** (`MusicPlayer.LocalGain` = 0). Relabelled - **"Desktop Audio"** (TASK 8) since TRAX rides the same channel. -- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct, - PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID - (`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored. -- Build **0 warnings**; **197 passing** (DSP/ring-buffer/ducker/resampler/pipe + mixer/hysteresis/ - game-detector/meter-scale unit tests + the TASK 8 integration test - `AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe` + the polish-batch integration - test `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` proving the provider gains reach the - pipe and mute silences the loopback). +- **ffmpeg is BUNDLED, never downloaded or fetched at runtime.** `IFfmpegLocator` checks the bundle + first, PATH second, and throws loudly if neither exists; `GuardBundledFfmpeg` fails the build when + the folder is missing. `TheLocatorCarriesNoNetworkCapability` prevents the old download design + (Store policy 10.2.2, and it 404'd in production on 2026-09-01) from coming back. +- **Args stay pure** (`FfmpegArgs.Build`) and **never use `-re`** — the pump is the pacer; a second + clock on the rawvideo demux fights it. Keyframes `-g fps×4` with a closed GOP, `-bf 0`, yuv420p. +- **Encoder choice is probed, never libx264** (GPL): NVENC → QSV → AMF → OpenH264. Never add a static + or GPL codec path — see the licensing guardrails in `ai.md`. +- **REC and ON-AIR are radio-exclusive by creator ruling.** The dual-output machinery stays generic, + but the UI can never arm both: one ffmpeg, one output, "we're not them". +- **A modal's falsy result must never fall through into its side effect.** Cancel on the recording + save dialog DISCARDS the temp file explicitly (the bug that kept recordings under the default name). +- **Torn-down sessions must leave no lit pill:** every pump death / go-live prep failure runs the full + `StopStream` teardown, and `AudioMixer.StopLive` is idempotent for rollbacks that never started. +- **Audio stop order:** `StopStream` closes the audio pipe (`StopLive` → audio EOF) **BEFORE** stopping + the frame pump (video EOF); the reverse order stalls on pipe backpressure. +- **Audio is honest by measurement, not by ear:** meters show true dBFS (`ToDisplay`), the mix runs + `SpeechLeveler` + `MasterLimiter` (−1 dBFS ceiling), and gains are `Func` seams read live each tick + (`AudioGainProvider`) — a decorative volume control is a bug. Judge loudness with the bundled + ffmpeg (`ebur128`), never the in-app meter alone. ### The live frame pipeline — moved @@ -1462,7 +1193,7 @@ the repo. `runFullTrust` + webcam + microphone. `Square310x310Logo`/`Wide150x150Logo` traps are in `MyMistakes.md`. Logos come from the app's own artwork via `scripts/make-store-logos.py`. - **⛔ Bundle ffmpeg — no runtime downloads** (was Store policy 10.2.2; still right for a direct - download, and it fixes the 2026-09-01 404). See the FFmpeg locator section above. + download, and it fixes the 2026-09-01 404). See [`docs/ai/output.md`](docs/ai/output.md) → "FFmpeg locator". - **⛔ 64-bit Windows only (creator ruling 2026-10-09).** Both csproj files set `x64` (the test project too — an AnyCPU test project referencing an x64 app raises MSB3270 and breaks the 0-warning gate). Why: the bundled ffmpeg diff --git a/docs/ai/index.md b/docs/ai/index.md index db00bf6..5a69633 100644 --- a/docs/ai/index.md +++ b/docs/ai/index.md @@ -11,6 +11,7 @@ Read the file whose row matches the task. There is no required reading order. |------|--------|-----------| | [`frame-pipeline.md`](frame-pipeline.md) | Render budget & measurement (`FrameCadence`, per-element composite timing, scene dump), `SceneCompositor` design + `StripPool`, and `FramePump` from scene snapshot to encoder (pacing, pool/paste caches, backpressure, teardown, health stats) | Touching the compositor, the render loop, per-frame cost, `Services/Encoder/*`, or debugging dropped/juddering frames | | [`capture.md`](capture.md) | The locked Background layer (screen/game/window capture, WinRT GraphicsCapture interop), the webcam seam + `CameraManager` sessions, MediaPipe background effects, and the vertical 9:16 tier's per-scene camera placement | Touching any frame source, the Background row, camera acquisition/conflicts, background effects, or vertical-tier camera geometry | +| [`output.md`](output.md) | The bundled `ffmpeg.exe` and the encoder subprocess (args, probe order, seams), local recording (REC/ON-AIR, filenames, private-broadcast close-out), and the WASAPI capture → filter → mix → named-pipe audio chain (TRAX, sync offset, ducking, limiter) | Touching ffmpeg args/bundling, encoder or recording behaviour, REC/ON-AIR, the audio mix and its meters, or loudness/sync problems | ## Split-out yet to come diff --git a/docs/ai/output.md b/docs/ai/output.md new file mode 100644 index 0000000..32e1fbb --- /dev/null +++ b/docs/ai/output.md @@ -0,0 +1,326 @@ +# Output — the ffmpeg bundle, the encoder, recording, and audio + +> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches; +> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md). + +Everything that leaves the app: locating and shipping `ffmpeg.exe`, how the encoder subprocess is +driven (video stdin + audio named pipe), local recording as a first-class mode, and the WASAPI capture +/ filter / mix chain that feeds it. Content is the decisions plus the failures that produced them; +superseded designs are marked inline. + +--- + +## The encoder + +### FFmpeg locator (TASK 4 ship step 2 — shipped 2026-08-10; **rewritten to BUNDLED 2026-09-28**, plan in TASKS.md) + +The encoder's one external dependency is `ffmpeg.exe`. **It is BUNDLED — never downloaded, never +fetched at runtime.** `IFfmpegLocator` resolves an absolute path on demand: **the bundle first** +(`\ffmpeg\ffmpeg.exe`), then **PATH** as the escape hatch for someone who +deliberately installed their own. The order is the opposite of the original ("the user's own install +wins") on purpose: a PATH `ffmpeg` may be a different major version or a **GPL** build, and the +bundle is the one that was pinned, license-vetted and tested. If neither hits, it throws a loud +`IOException` naming the missing folder and the fetch script — the old "found something on disk" +cache path is gone deliberately, because silently satisfying a missing bundle with a stale copy is +exactly the silent drift a certification-gated dependency must not have. + +**The bundle is not in git** (135 MB of incompressible DLLs would tax every commit on this +9p-bridged tree). `scripts/fetch-ffmpeg.sh` populates `./ffmpeg` from a pinned month-end BtbN +LGPL-**shared** win64 artifact, **verifies SHA-256 before extracting**, and is idempotent; set +`FFMPEG_ZIP=` to reuse a pre-downloaded archive. `ytLive.csproj` copies `ffmpeg\**` to the +output (`CopyToPublishDirectory="Always"` for the MSIX package) and the `GuardBundledFfmpeg` target +**fails the build** if the folder is absent, so a fresh clone finds out before the creator presses +Record rather than during a take. `ffmpeg.exe` + `ffprobe.exe` + the `libav*.dll` family must stay in +one folder — Windows resolves a shared build's DLLs from the executable's own directory. +`ffplay.exe` (18 MB) is deliberately **not** shipped; nothing calls it. + +BtbN LGPL-shared (not gyan.dev, not static): it drops GPL-only libx264/x265 while keeping +NVENC/QSV/AMF + libopenh264 + native AAC, and dynamic linking means LGPL compliance is "license text + +source offer" with no static-relink (§6) material — see the Licensing guardrails below. libopenh264 +is **compiled into `avcodec-63.dll`** (verified 2026-09-28: the encoder is registered there, and there +is no separate OpenH264 binary to ship), so Cisco's BSD terms + patent grant ride along inside item 1 +of `THIRD-PARTY-NOTICES.txt`. Pin: `autobuild-2026-08-31-13-27` (month-end on purpose — BtbN keeps +~14 dailies but month-end builds ~2 years; a daily pin is what 404'd, see below). + +**⛔ Why this was rewritten (2026-09-27 → shipped 2026-09-28).** The pinned-download design was a +defect, not a cold-start edge case. It **failed in production on 2026-09-01** — the previous daily +pin aged out of BtbN's 14-day retention and the download 404'd on the creator's first real recording +attempt. Two facts made "the pin is a const, just bump it" wrong: +- **Microsoft Store policy 10.2.2 forbids dynamic code inclusion.** Downloading an executable + from the internet and running it is the exact pattern the policy names. An MSIX submission + is rejected over it. +- Bumping the pin is whack-a-mole against a retention window we do not control. + +Bundling deletes the startup network dependency entirely. The LGPL-**shared** build choice is +**unaffected** — dynamic linking was chosen for LGPL §6 compliance ("license text + source offer"), +not to enable downloading. Analysis: `TASKS/research-store-certification.md` §4. +**Guardrail:** `FfmpegLocatorTests.TheLocatorCarriesNoNetworkCapability` asserts the locator's +source contains no `HttpClient`/URL/downloader at all, so the policy cannot rot back. + +**Every other third-party binary already ships via NuGet — ffmpeg was the only download-and-execute.** +Verified 2026-09-28: `e_sqlite3.dll` and `WebView2Loader.dll` land in `runtimes\win-x64\native\` +(SQLitePCLRaw + WebView2 packages) and NAudio/Notification.Wpf/Microsoft.Data.Sqlite/ProtectedData +are managed. Two gaps remain and are **not** fixed by bundling: the **WebView2 Evergreen runtime** is +a machine prerequisite (a manifest framework-dependency + graceful-failure path, TASK 48 item 2), and +**Velopack's updater downloads and executes** — its teardown is a compliance action, not just cleanup. + +### Live encoder + RTMP push (TASK 4 ship step 3 — shipped 2026-08-12, plan in TASKS.md) + +The encoder is a **thin orchestrator over `ffmpeg.exe`** — no H.264/AAC code in the app. It spawns the +subprocess (path from `IFfmpegLocator`), feeds raw BGRA master frames into stdin, and parses `-stats` +stderr lines into `StreamHealth` (bitrate/FPS/duration, dropped-from-frame-count). `FfmpegEncoder` +(`IFfmpegEncoder` seam) holds: `StartAsync` (locate → probe `-encoders` → spawn → stderr loop → +drain loop), `SubmitFrameAsync` (slice 10: bounded-queue ENQUEUE — never a pipe write; the drain task +owns stdin writes), `StopAsync` (flush queue → stdin EOF → ffmpeg +finalizes + exits by itself; a 10s watchdog kills it), `Dispose` (force-kill + wait), and the +`HealthUpdated`/`ProcessFailed` events. **Pattern:** the encoder never touches `Process` — it drives the +`IEncoderProcess` seam (`FfmpegEncoderProcess` wraps the real `Process`, redirected stdin/stdout/stderr ++ exit control); a `Func` factory + the locator are constructor-injected, so the +integration test fakes the whole subprocess (probe + encoder) with a Channel-backed `TextReader` whose +`Complete()` is EOF (`null`), never a `ChannelClosedException`. + +**Decisions (locked):** args are pure (`FfmpegArgs.Build`, no string building in the encoder): +`-f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS -i pipe:0` (**NO `-re`** — the FramePump +is the pacer since slice 9, 2026-09-10; `-re` added a second, fighting clock on the rawvideo demux) ++ a **real audio input — the +mixer writes IEEE-float stereo to a Windows named pipe** (`-f f32le -ar 48000 -ac 2 +-i \\.\pipe\ytllive_audio`, name via `EncoderOptions.AudioPipeName`; replaced the old `-f lavfi -i +anullsrc` silence in the TASK 8 audio milestone) + explicit `-map 0:v -map 1:a` + `-c:v -b:v K +-maxrate K -bufsize 2K` + **`-g fps×4 -keyint_min fps×4 -sc_threshold 0 -bf 0 -pix_fmt yuv420p`** + (≤4s keyframes, closed GOP, H.264 compliance) + `-c:a aac -ar 48000 -ac 2 -f flv `. +**Record output (TASK 18):** `FfmpegArgs.Build` emits one self-contained block per output, each its own +`-map 0:v -map 1:a` + codec tags (`AddVideoTags` helper). Stream block = `-f flv `; record block = +`-f mp4 `. `EncoderOptions.StreamEnabled`/`RecordEnabled`/`RecordPath` gate each block, so +the engine runs record-only (no RTMP) or stream-only — **stream+record simultaneously is OUT by creator +ruling (2026-09-01): the VOD is already the copy, and dual-encoding drags mid-range chassis and degrades +BOTH outputs ("we're not them"). The old "cheap on NVENC" claim was an unverified assumption; the +UI constraint is landed (2026-09-20): the REC/ON-AIR pills are radio-exclusive — arming one clears the +other — while the dual-block machinery stays generic.** +`FfmpegEncoder.StartAsync` throws unless at least one output is enabled. +**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure): +NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder` +forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and +`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`), +driven by the `FramePump` below. + +--- + +## Recording + +### Local recording (TASK 18 — shipped 2026-08-29, VM + top-bar UX; plan in TASKS.md) + +Recording is **independent from streaming** and needs no YouTube sign-in. A single segmented +**REC|ON-AIR switch** in the top bar picks the mode (redesigned 2026-09-22 from the old two pills): +**REC** (local file, works signed-out) / **ON-AIR** (streaming; **armable signed-out** — intent may +light while offline, 2026-09-20: a greyed pill was the dead end the guard shipped to forbid). The switch +selection is intent; `IsRecording`/`IsLive` are reality — the reality line (dot + word + elapsed) only +lights when a session is actually running. + +- **State model (`MainViewModel`):** pills `RecordPillOn`/`OnAirPillOn` stay the **routing model** + (radio-exclusive — arming one clears the other, record-OR-live 2026-09-01; the ON-AIR setter stays + armable signed-out (2026-09-20: intent must be able to light while offline)). **Top bar redesign + (2026-09-22):** the bar is ONE surface rendered by the FIRST decision — Record or Stream — and each + world is the whole bar (`ShowIdleWorld`/`ShowRecordAction`/`ShowStreamActions`, wired by + `RaiseTopBarModes()`). The old sign-lights + two pill toggles + status dot + PRIVATE/TEST chips are + gone; the mode is one **segmented REC|ON-AIR switch** (`SegmentToggle`/`SegmentLabel` styles), and + "going back" is one tap on the other segment. Idle worlds: Record = switch + "Start Recording" + (`PrimaryStartButtonLabel` names the armed world); Stream = switch + + "Go Live" + Test (Test is a **child of Stream** — visible only stream-armed AND signed-in) plus the + account zone on the right. **The account zone is WORLD-INDEPENDENT (2026-09-23):** + `ShowSignInButton` = signed-out and `ShowAvatarButton` = the signed-in avatar, in Record and Stream + worlds alike — it's app-level identity, and keeping it put pins the bar height (it no longer pops in + and out between worlds). Right-click the avatar = Change Account/Logout. Running retires the switch + and worlds; one reality line + `● word elapsed` + End replace them (`RunningVisible`/`RunningText`/`RunningDotBrush` — TEST/LIVE/REC, + gold/red/green). The account **status dot is gone** — the avatar/Sign In pair carries connectedness. + `ShowPrimaryStartButton` remains the always-idle face. `StopStream` clears both pills (see "Stop ends + everything; failures roll back" below). +- **Filenames** (pure `Services/RecordingFile`): auto-name `ty----0000.mp4` at + start; **rename-on-stop** to `ty-…-.mp4` (`FinalizeRecordingAsync` after the pump + stops & the file closes), numeric `-2`/`-3` suffix on collision (`UniquePath`). Length comes from + `_liveElapsed`, which the session timer walks while `IsLive || IsRecording`. The **`` millisecond + segment is load-bearing, not decoration** (2026-09-27): both names derive from `_recordStartTime`, so it + survives the rename, and without it two instances started in the same minute compute the *same* start + path and both ffmpeg processes write one file — which is exactly the dev multi-instance case + (`Helpers/InstanceProfile`, one instance recording the other). +- **Private broadcasts end the same way (creator ruling 2026-10-09).** After `StopStream`'s + transition(complete), `OfferToKeepPrivateBroadcastAsync` reads the broadcast's privacy **from YouTube** + (`GetBroadcastPrivacyAsync` — visibility can change mid-stream, local state isn't trusted) and, only if + it is `private`, shows `KeepBroadcastDialog` (repo root): **Enter = Keep** (`IsDefault`), **Escape / + Delete / the X = delete** via `DeleteFinishedBroadcastAsync` — `videos.delete?id=` (a live + broadcast's id IS its video id; 204), falling back to `liveBroadcasts.delete` on 404. An unreadable + status asks nothing and deletes nothing; a failed delete toasts that the video is still on YouTube. + Decision logic = `MainViewModel.ClosePrivateBroadcastAsync` (internal static, tested in + `PrivateBroadcastCloseOutTests`); `KeepBroadcastPromptOverride` is the dialog seam. +- **Save dialog: Cancel DISCARDS (creator ruling 2026-09-26).** The `RenameRecordingDialog` result is a + two-outcome decision, and it lives in the extracted `MainViewModel.CompleteRecordingSave(startPath, dir, + autoStem, chosenStem, creatorSaved)` (internal, so `ytLive.Tests` can drive it against real files without + a frame pump — see `ytLive.Tests/RecordingSaveDialogTests.cs`): + - `creatorSaved == false` (**Cancel / Escape / the X**) → **DELETE the temp file.** Nothing is moved, + nothing is left in the videos folder. A failed delete returns `DiscardFailed` and the toast NAMES the + file + folder, because a "discarded" recording still on disk is worse than no report. + - `creatorSaved == true` (Save, or **Enter on the pre-filled default**) → `File.Move` to the final name. + A blank/whitespace box still means "keep the auto name" — but only on an explicit Save. + - **The bug this fixed:** the caller used to `File.Move` UNCONDITIONALLY and only overwrite `stem` when + the dialog returned true, so `ShowDialog() == false` fell straight through into the save — Cancel + silently kept the recording under the default name. The modal itself was always correct; the + side effect was in the wrong place. **Any modal whose result gates a side effect must have every + branch handled explicitly — a falsy result must never fall through into the effect.** +- **Folder:** default `%APPDATA%\ytLlive\recordings\`, user-overridable via `ChooseRecordFolderCommand` + (`OpenFolderDialog`), persisted through `LayoutStore.Load/SaveRecordFolder` (`RecordFolder` key). +- **Explicit sign-out only:** `StopStream` no longer clears the session/token. Sign out via **Logout + only** — Change Account re-authenticates *in place* and deliberately does not sign out first + (2026-09-27; signing out first deleted `ytLlive.auth` before the browser even opened, so a + cancelled switch lost a working session). Stopping a recording leaves the creator signed in. +- **Stop ends everything; failures roll back (2026-09-01):** `StopStream` also clears both intent + pills (`RecordPillOn`/`OnAirPillOn`) — a lit pill with no session behind it is a lie. Every + frame-pump death and every go-live prep failure now runs the full `StopStream` teardown instead of + leaving a limbo (`StreamStatus.Error` with `IsRecording=true` over a dead encoder — the first-launch + zombie recording). Toast copy distinguishes "Recording stopped" from "stream pipeline stopped"; + `AudioMixer.StopLive` is idempotent-safe for rollbacks that never reached StartLive. Seams: + `OnFramePumpFailed` + `IsRecording` setter made internal (test-only, InternalsVisibleTo). + Test: `SessionTeardownTests`. +- `EncoderOptions.StreamEnabled/RecordEnabled/RecordPath` gate outputs; record+stream is one ffmpeg with two + output blocks (the dual-block machinery is unreachable from the UI since the pills are radio-exclusive). + Rest of the engine (FramePump/encoder) is unchanged — `BuildEncoderOptions` always returns an options + (flags from the pills + `_activeRecordPath`), so record-only reaches the pump. + +--- + +## Audio + +### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12; game audio bar 2026-08-13; **TASK 8 audio milestone: real stream audio + filters + duck + TRAX, shipped 2026-08-14**, plan in TASKS.md) + +**Capture runs for the app's lifetime and is KISS by rule**: desktop/game audio is automatic (WASAPI +loopback), the mic is the creator's only audio control (meter/mute/volume already shipped). The whole +layer sits behind an **`IAudioSource` seam** (`Services/Audio/`: `Start`/`Stop`/`Started`/`SampleReady`/ +`Failed`, IDisposable) so the app never touches NAudio directly and tests inject hermetic fakes (no +devices, no timers). + +- **Sources (NAudio 2.2.1 — `NAudio.Wasapi` + `NAudio.WinMM` (for `WaveOutEvent`), MIT — item 9 in + `THIRD-PARTY-NOTICES.txt`):** `WasapiLoopbackAudioSource` = `WasapiLoopbackCapture` on the default + render device; `WasapiMicAudioSource` = `WasapiCapture` with the device resolved by `FriendlyName` + matching `MicSourceName` (the app only persists the **DisplayName**), falling back to the default + capture endpoint. Both run at the **device's own mix format** — NAudio 2.2.1 exposes no overridable + `GetDefaultMixFormat` on `WasapiCapture`, so there is no forced 48 kHz path; the mixer's + `TinyResampler` normalizes any rate/channel layout to 48 kHz stereo (this replaced the earlier + "force IEEE-float 48k" plan — the resampler IS the design). Mic device resolution re-reads the name + provider `Func` at each `Start`, so a mic picked mid-session takes effect **immediately** + (the mixer restarts the mic on pick). Both sources raise `Started` once their capture loop actually + begins — the mixer turns that into `MicConnected`. +- **`AudioMixer`** owns both sources; **`StartMicCaptureAsync` starts the mixer once at startup** and + `Shutdown` disposes it — NOT go-live — so the meters preview live. Mic samples: downmix to mono → + `TinyResampler` → **`VoiceFilterChain` per sample** (bass `LowShelfFilter` 120 Hz +4 dB → treble + `HighShelfFilter` 8 kHz +3 dB → `NoiseGate` (threshold 0.005, hysteresis 0.5, attack 0.5, release + 0.0005) → **`SpeechLeveler`** (2026-10-09: automatic gain steering speech RMS to −20 dBFS over a + 0.3 s window, gain −6..+20 dB, rises ≤10 dB/s / falls ≤30 dB/s, learns only while the gate is open + and above −54 dBFS so pauses/hum are never pumped) → `Compressor` (threshold 0.5, ratio 4:1); all + pure DSP) → ring buffer → the + **post-filter** `AudioLevelMeter` (RMS, 0.2 exponential smoothing) → `MicLevelChanged` → + marshalled to the UI thread → `AudioLevel`. Loopback samples: resample → ring buffer → second meter + → `LoopbackLevelChanged` → the game bar's `GameAudioLevel`. The raw linear level maps to the meter's + display scale by `AudioLevelMeter.ToDisplay` (−60..0 dBFS → 0..1, **honest** — the +10 dB display boost + was removed 2026-10-09; the mic meter is `ToDisplay(AudioLevel × MicVolume)`, true dB math, no ×1.2). + **Why (2026-10-09):** a creator's YouTube upload measured −28.5 LUFS / −8.9 dBFS peak against YouTube's + −14 LUFS reference (YouTube only turns loud videos down, never quiet ones up) while the boosted meter + looked "dialed in". Nothing in the chain added gain — the compressor folds only above −6 dBFS and the + limiter only caps. Re-running that same audio through the new chain (mic volume 0.8): **−17.9 LUFS, + −0.7 dBFS peak**. Recording and streaming share this one mix, so both were (and now are) the same. + Measure with the bundled ffmpeg: `ffmpeg -i file -af ebur128=peak=true:framelog=quiet -f null -`. + **Mic connection state surfaces as events:** `MicConnected` (source `Started`), `MicFailed` (source + `Failed`), and `RestartMic()` re-resolves + restarts just the mic (loopback keeps running). Failures + log via `AppLog`; a mic failure zeroes the meter, a loopback failure never kills the mic. Meter `Push` + is unconditional (a `?.` on the event would skip the meter update when nothing is subscribed yet). +- **Go-live audio (TASK 8):** `BeginGoLive` → `StartLive(pipeName)` (idempotent) creates the named pipe + server (`NamedPipeAudioWriter`, behind the `IAudioPipeWriter` seam) and a live loop task; every + `mixInterval` (default 10 ms) `FillAndMix` reads a chunk from each ring buffer (silence-fills + underruns), computes the post-filter mic RMS → `AutoDucker` (threshold 0.02, duck ×0.25 / −12 dB, + attack 0.05, release 0.005, always on), applies the **honest gains** via `Func` seams + (`micGain = MicMuted ? 0 : MicVolume`; `loopbackGain = GameMuted ? 0 : GameAudioVolume`, × duck), + mixes mono mic + stereo loopback into interleaved stereo, runs the + **`MasterLimiter`** (a **−1 dBFS ceiling** so the honest gains can never sum past 0 dBFS and clip the + AAC encode — instant attack per frame, smoothed release toward unity), and writes the floats to the pipe. **The gains are wired through `AudioGainProvider`** + (`Services/Audio/AudioGainProvider.cs`) — the VM hands its `MicGain`/`LoopbackGain` Funcs to the + mixer at construction, read live each tick, so the volume sliders and mute buttons are stream-honest. + (Before this wiring the seams defaulted to unity and the controls were decorative.) `StopStream` → + `StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) — + the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the + pipe lifecycle. +- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo + mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose + offset comes from a `Func syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500 + ms**, persisted as `Audio.SyncOffsetMs`). This is OBS's documented fix for lip-sync: **positive** + delays audio when it runs ahead of video; **negative** advances audio when it runs behind by eating + the first `|N|` ms of the live stream head (see `AudioMixer.StartLive`; the delay line stays + clamped 0..500 internally — negative bypasses it entirely). The advance is armed once at `StartLive` + (eating the head mid-stream is impossible); positive is live-reactive (re-read per tick). A slider + on the mic bar (`"AUDIO SYNC"`, locked while live/recording via `IsEditMode`) surfaces it + (a one-time status dot next to it was removed 2026-09-23 per creator feedback — the slider + + numeric tooltip carry the state). Changing the delay flushes the line (a live change clicks + rather than smears). +- **TRAX — free background music (TASK 8):** `MusicPlayer` = NAudio `MediaFoundationReader` + (mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** bed (no slider) → `WaveOutEvent` on + the default device, **looping on any clean natural end** (`PlaybackStopped` with `e.Exception == null` + → rewind + replay; the earlier `Position >= Length` check was unreliable for `MediaFoundationReader` + and let tracks play once then stop). The desktop/game bar's mute toggles `LocalGain` (0 or 1; + `UpdateTraxLocalGain` runs on every `GameMuted` change and on TRAX load), so muting kills TRAX + in the headphones — matching the stream, where it rides the loopback channel. The volume slider + drives system volume (`PushSystemVolume`) — which scales the HEADPHONES; the loopback capture is + pre-endpoint-volume (disproven-then-fixed 2026-09-01), so the STREAM follows the knob via + `AudioGainProvider.LoopbackGain = GameMuted ? 0 : GameAudioVolume`, and the game meter multiplies + its display by the same. One knob, three honest paths. No third mixer input. The footer + **TRAX** button lives **inside the game audio bar** (in the preview overlay), **left of the + "Desktop Audio" label** (the creator's pick — it rides the desktop channel, so the music control + sits with the desktop audio controls): status dot (**red** no track / **yellow** loaded stopped / + **green** playing) + "TRAX"; **left-click** toggles play/pause (opens the in-app picker via `OpenFileDialog` when no track is loaded); **right-click** always opens + the picker (`TraxButton_PreviewMouseRightButtonUp`, `e.Handled = true`, code-behind pattern); tooltip + shows the loaded/playing track name or "No track — right-click to choose background music". The picked + track persists via **schema v9** single-row `Music` (`TrackPath`/`IsEnabled`) in `LayoutStore`. Known + wrinkle (out of scope, future feature): YouTube mutes VODs carrying copyrighted music — "music on + live, off VOD" is tabled. + - **⛔ The load is behind a bounded probe, because a share that is gone must never + freeze startup (2026-10-04).** `LoadLayout` → `LoadTrax` used to call `MusicPlayer.Load` + synchronously on the UI thread, and `new MediaFoundationReader(path)` does **not** fail + fast on a dead SMB session — it burns the whole ARP/TCP/SMB timeout (~40s, measured) + and *then* throws `0x80070035`. Measured: `OnStartup` 08:42:59 → `MainWindow loaded` + 08:43:43, with 42 of those 44s inside `LoadLayout` on that one track. The rule now: + **a probe may time out; only a bind is allowed to be on the creator's path.** + `IsTrackReachableAsync(path, budget, File.Exists)` runs the probe on a thread-pool + thread under a `Task.WhenAny` ceiling (`TraxProbeBudget` = **2s**) and reports + offline/missing/slow as a normal state, not an error. Only a probe that passed gets + the real bind, and **that bind stays on the UI thread** with every other + `MusicPlayer` mutation, so no new cross-thread race is introduced. The orphan probe + thread stays parked until SMB gives up — deliberate: it is not the thread anyone + waits on. Same principle covers the webcam: a device held by another app fails fast + (sharing violation) but a virtual-camera driver can stall enumeration, so device + probes belong off the startup path too, not behind a long timeout. +- **Mic status dot (`Models/MicStatus.cs`)** on the preview-bottom MIC button: green = `MicConnected`, + yellow = `MicFailed` (in use/unplugged), red = no mic device at startup (the mixer is never started, + so loopback and the game bar can't run either — no capture devices at all). **The picked mic persists: + `LayoutStore`'s `Settings` key/value table stores `MicSourceName` (saved on pick via + `SaveMicSourceName`), and the VM restores it at construction BEFORE the mixer's first `Start` — so a + restart reconnects the same already-vetted device (green) or reports it missing (yellow), instead of + silently falling back to the default endpoint.** +- **Game audio bar** (desktop/game, **always visible** — TASK 4's `IGameAudioDetector` show/hide gating was **removed 2026-08-15**: the bar used to appear only while a full-screen game with sound was up, which kept hiding the creator's desktop meter; the detector stack is gone, desktop audio is just automatic WASAPI loopback): overlaid **at the bottom of the preview window** (bottom-center, dark translucent + chip, a mirror of the mic bar: meter + mute + volume slider). It's monitoring UI **in the preview + grid** — the meter is display-only (fill = `ToDisplay(GameAudioLevel) × GameAudioVolume` — the ×volume + term was missing until 2026-09-01: loopback capture does NOT shrink with the endpoint, so an + unscaled meter left it pegged at 20% volume; `GameMeterHonestyTests` guards it), but its **volume + slider drives system volume** (`PushSystemVolume` — live headphones follow the knob; the capture + side is scaled in the mix, see TRAX note above) **and mute kills TRAX** (`MusicPlayer.LocalGain` = 0). Relabelled + **"Desktop Audio"** (TASK 8) since TRAX rides the same channel. +- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct, + PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID + (`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored. +- Build **0 warnings**; **197 passing** (DSP/ring-buffer/ducker/resampler/pipe + mixer/hysteresis/ + game-detector/meter-scale unit tests + the TASK 8 integration test + `AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe` + the polish-batch integration + test `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` proving the provider gains reach the + pipe and mute silences the loopback). +--- + +## Related in [`ai.md`](../../ai.md) and [`frame-pipeline.md`](frame-pipeline.md) + +- [`frame-pipeline.md`](frame-pipeline.md) — the `FramePump` that feeds this encoder (pacing, backpressure, teardown order). +- **ai.md** — resolution tiers (W×H/FPS/bitrate), go-live/prep ordering, toast notifications, licensing guardrails. +- [`TASKS/research-store-certification.md`](../../TASKS/research-store-certification.md) — §4, the ffmpeg bundling analysis.