LlamaCasty/ai.md

# ytLlive — AI Guide

> Memory map entry point. Conventions live in [`schema.md`](schema.md); task
> status and YouTube API research in [`TASKS.md`](TASKS.md); brand/marketing
> palette and launch strategy in [`MARCOM.md`](MARCOM.md) (gitignored — read
> it before any branding or marketing work); directory maps in each folder's
> `index.md`. Reading order: this file → `TASKS.md` → `MARCOM.md` → `<dir>/index.md` → source.
>
> **Subsystem detail is split out:** large blocks of this file now live under
> [`docs/ai/`](docs/ai/index.md) and this file keeps a short invariant stub + link where each was
> removed. Open [`docs/ai/index.md`](docs/ai/index.md) to find the right one, or ignore it entirely
> until your task touches a moved subsystem.

## Response style

No default "Plans & Pitfalls" / planning boilerplate. Respond directly and
concisely: **do the queued work, then report what changed and what's next.**
Skip feature pitch, step-by-step implementation plans, pros/cons tables, and
"potential pitfalls" sections unless the user explicitly asks for a plan first.
A short diff-style summary beats a proposal document every time.

## Derivative work (spin guard mandated in AGENTS.md)

Nothing we build is novel — streaming/overlay/WebView2 problems were solved by
OBS, CEV, and WPF ecosystems long before us. Two consequences:
1. **Proactive:** new features start with a quick external scan, not head-first
   design (citation goes in the commit message).
2. **Objectively triggered spin guard:** a second failed fix for the same symptom
   means STOP theorizing and research the established answer externally; there is
   no unique bug in this repo that the wider ecosystem hasn't hit.
This project's hard lesson: the web-source bounding saga (2026-08) burned 9
commits rediscovering that a browser source is a *fixed canvas* — OBS keeps it at
a stable page size, crops/hugs content via the box, and clips at the edge. The
apparent "gap" that ended the saga was a widget's own CSS glow effect, not a bug.

**Derived-solution rule (2026-08-29, driven by the image-shrink incident):** when
you work out any one-off, reusable solution (recipe / workaround / how-to), write
it into `MyMistakes.md` → **Recipes registry that same session**, and grep it
before ever re-deriving. A solution recorded once ends the loop; an un-recorded
one guarantees the user hears it derived again (see `AGENTS.md` → Working rules → 🔬).


## No-Fluff Mode (on demand)

Invoke with "no-fluff mode" (or similar) when you want ruthless review instead
of reassurance. In that mode:

- Strip all polite pleasantries, emojis, transitions, and conversational padding.
- Treat the user's input as a draft to be methodically deconstructed or
  strengthened — argue, correct, and sharpen rather than agree.
- Give unvarnished truth, not reassurance.

This is an occasional, explicitly-invoked mode — **never the default**. The
default response style above stays in effect unless invoked.


## Architecture

C# / WPF (.NET 8) following MVVM:

| Path | Role |
|------|------|
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
| `ViewModels/` | MainViewModel — `public partial class`, one file per functional area (Scenes, Background, Webcam, Audio, Trax, Socials, Streaming, Chat, Overlays, Account, License, Recording — split complete, see `ViewModels/index.md`); **Chat.cs is a thin delegating facade over `Services/ChatOverlayLayer.cs` (Commit G, first true decomposition), and now also over `Services/AlertOverlayLayer.cs` (TASK 43 — the native alert box, same true-component pattern)**; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Scene compositor")**, **alerts: `ChatOverlayLayer` + `AlertOverlayLayer`/`AlertRenderer` (the native celebration-zone overlay, TASK 43 — see "Native events & alerts")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `ITraxPlayer` (TRAX toggle seam — FakePlayer in tests; the dot is green/red/gray and the error scrolls terse in the desktop-audio bar; MusicVolume fixed 0.5) + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see [`docs/ai/output.md`](docs/ai/output.md) → "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see [`docs/ai/output.md`](docs/ai/output.md) → "Live encoder" + [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** |
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, **BuildStamp (wordmark release counter `#N`, incremented by `GenerateBuildStamp` from the local `BuildCounter.txt`; per-build GUID/time are logged to startup.log)**, visibility converters |
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
| `MainWindow.xaml` | Dark theme; top bar + health banner; work area has a fixed 360px left column and preview column, top-left readouts/resolution and scene strip above preview, lower left panel + preview; bottom bar is an empty spacer |


### Environment, LAN, build & tests — moved

Server IPs and SSH, the Windows `dotnet.exe` build commands, the test suite map and the
hermetic real-`MainWindow` test rule live in
**[`docs/ai/environment.md`](docs/ai/environment.md)** — read it before building, running, or
adding a test.

### Product, brand, licensing & distribution — moved

The brand/naming split, AI transparency, the GPL licensing prohibitions (do not violate), the
free-forever monetization posture, auth/account assumptions, and packaging/distribution live
in **[`docs/ai/product.md`](docs/ai/product.md)**.

### YouTube Live API — moved

Go-live ordering, the broadcast-binding fix, the private-only lock, health parsing, end-stream
pre-flight, Test mode and chat-insert shape live in
**[`docs/ai/youtube-api.md`](docs/ai/youtube-api.md)**.

### Overlays: social bar, alerts, chat — moved

The social bar, the native alert box (six animations, alert clip, ticker) and the chat box
conventions live in **[`docs/ai/overlays.md`](docs/ai/overlays.md)**.

### UI surface — moved

The control surface, scene strip, preview rail, hotkeys, broadcast drawer, toasts, transitions,
brand logo, launch geometry and the theme/properties conventions live in
**[`docs/ai/ui-surface.md`](docs/ai/ui-surface.md)**.

### Capture — backdrop, webcam, effects & the vertical tier — moved

Screen/game/window backdrop capture, the webcam seam and sessions, the MediaPipe background effects
and the vertical tier's per-scene camera placement live in
**[`docs/ai/capture.md`](docs/ai/capture.md)** — read it before touching capture, cameras, the
Background row, or anything the 9:16 tier re-places.

Invariants that hold even if you never open that file:

- **The Background is a locked per-screen layer:** exactly one per canonical scene, named
  "Background", pinned at index 0, never addable/removable/reorderable — Live is `DisplayCapture`,
  every other screen is static art. `NormalizeBackgrounds` heals any layout to that model on load.
- **Never feed a capture row to the `ActiveBackgroundImage` overlay.** That layer is fed by
  static-art rows only; a flavor-blind `IsBackground` lookup there permanently covers the live
  desktop/game feed (the 2026-08-23 regression).
- **`window:`/`picker:` capture keys are session-scoped and stripped on load** (a recycled HWND
  would capture the wrong window), and an explicit window pin beats auto game/desktop detection
  while that window is alive.
- **The WinRT interop paths are settled:** read frame bytes with `WindowsRuntimeMarshal.TryGetDataUnsafe`
  (the `IMemoryBufferByteAccess` ComImport throws `Invalid cast` on every frame under CsWinRT) and
  build the D3D device via `CreateDirect3D11DeviceFromDXGIDevice` (the
  QI-for-`IDirect3DDxgiInterfaceAccess` trick returns E_NOINTERFACE on current Windows).
- **`VideoFrame` + the capture interfaces are the seam** — nothing above the WinRT layer leaks WinRT
  types; capture is CPU-first (`MemoryPreference.Cpu`, `SharedReadOnly`), the reader-subtype ladder
  tries each candidate under its OWN catch (one rejected format must never abort acquisition), and
  there is deliberately **no fallback camera** — the user's pick is never overridden.
- **One webcam identity app-wide, placements per scene.** `IsWebcamAttainable` needs a live lock, not
  just a saved id; the app keeps its own base ref, so removing every placement leaves the session up.
- **`CameraManager` is the ONE background-effects processing point** — segment once per raw frame,
  render per-scene variants keyed by effect+strength, mask smoothed per camera session (0.75 new /
  0.25 previous). The raw path stays untouched so a slow model never stalls the webcam, and the
  bundled model's provenance is fixed (do not swap in PINTO #109).
- **The vertical tier is geometry only:** `BeginVerticalLayout`/`EndVerticalLayout` park and swap the
  live placement, so the compositor and `PreviewPane.xaml` never learn about tiers, and the stored
  X/Y/W/H/ClipShape columns stay the **16:9** layout. The Chat camera is rect-only through the MODEL
  (`VerticalShapeLocked`), not merely through a hidden button.

### Rendering budget, measurement & the compositor — moved

Frame cadence, per-element composite timing, the scene dump, the render budget and the
`SceneCompositor` design live in **[`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md)** —
read it before changing the render loop, the compositor, or anything that moves per-frame cost.

Invariants that hold even if you never open that file:

- **Every global overlay must be composited on EVERY render path** (pump, preview/thumbnails, scene
  dump — `SceneCompositor.Overlay` is shared for all three). A feature wired into one path is silently
  absent from the output; that bug (flash + social bar "preview-only forever", 2026-09-26) is why the
  rule exists.
- **`VideoFrame.Epoch` is not optional.** A producer that recycles buffers MUST bump `Epoch` per
  frame, or `FramePump`'s buffer-identity paste cache and the full-render signature serve stale
  pixels. `BrandFlashPresenter` reuses one 8MB master buffer and stamps `Epoch` per read for exactly
  this reason.
- **`StripPool` owns four dedicated `BelowNormal` background threads** for strip-parallel
  compositing; strips below 64 rows run inline. Never demote threads inside `Parallel.For` — it runs
  on thread-pool threads and starves the scheduler (that mistake once broke audio-mix timing).
- **Read render numbers in Release only.** Debug pixel paths are several× slower; a Release 1080p
  composite measures ~1.0–1.1ms against the 16.67ms budget, and only Release takes are evidence.

### Output — ffmpeg, encoder, recording & audio — moved

The bundled `ffmpeg.exe` decision, the encoder subprocess (`FfmpegEncoder`/`IEncoderProcess`), local
recording, and the whole WASAPI capture → filter → mix → named-pipe audio chain live in
**[`docs/ai/output.md`](docs/ai/output.md)** — read it before touching ffmpeg args, the encoder
seams, REC/ON-AIR, or the audio mix.

Invariants that hold even if you never open that file:

- **ffmpeg is BUNDLED, never downloaded or fetched at runtime.** `IFfmpegLocator` checks the bundle
  first, PATH second, and throws loudly if neither exists; `GuardBundledFfmpeg` fails the build when
  the folder is missing. `TheLocatorCarriesNoNetworkCapability` prevents the old download design
  (Store policy 10.2.2, and it 404'd in production on 2026-09-01) from coming back.
- **Args stay pure** (`FfmpegArgs.Build`) and **never use `-re`** — the pump is the pacer; a second
  clock on the rawvideo demux fights it. Keyframes `-g fps×4` with a closed GOP, `-bf 0`, yuv420p.
- **Encoder choice is probed, never libx264** (GPL): NVENC → QSV → AMF → OpenH264. Never add a static
  or GPL codec path — see the licensing guardrails in `docs/ai/product.md`.
- **REC and ON-AIR are radio-exclusive by creator ruling.** The dual-output machinery stays generic,
  but the UI can never arm both: one ffmpeg, one output, "we're not them".
- **A modal's falsy result must never fall through into its side effect.** Cancel on the recording
  save dialog DISCARDS the temp file explicitly (the bug that kept recordings under the default name).
- **Torn-down sessions must leave no lit pill:** every pump death / go-live prep failure runs the full
  `StopStream` teardown, and `AudioMixer.StopLive` is idempotent for rollbacks that never started.
- **Audio stop order:** `StopStream` closes the audio pipe (`StopLive` → audio EOF) **BEFORE** stopping
  the frame pump (video EOF); the reverse order stalls on pipe backpressure.
- **Audio is honest by measurement, not by ear:** meters show true dBFS (`ToDisplay`), the mix runs
  `SpeechLeveler` + `MasterLimiter` (−1 dBFS ceiling), and gains are `Func` seams read live each tick
  (`AudioGainProvider`) — a decorative volume control is a bug. Judge loudness with the bundled
  ffmpeg (`ebur128`), never the in-app meter alone.

### The live frame pipeline — moved

`FramePump` and the whole path from frame producer to encoder (pacing, pool/paste caches, slice
history, teardown, health stats) live in
**[`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md)**.

Invariants that hold even if you never open that file:

- **rawvideo is stamped by ARRIVAL, never by a timestamp.** ffmpeg assigns pts from frame order at
  the declared fps, so supply rate = output speed. The pump duplicates the previous frame on a missed
  deadline (correct duration) and **never resets `nextTick` to wall-now** — rebasing turns a hitch
  into a silent time-lapse (take-2 lesson, 2026-09-01).
- **Submit never blocks.** The encoder queue is bounded and a drain task owns the stdin write +
  flush; on overflow the **newest frame is dropped and counted** (`DroppedFrames`) — overload costs
  content, never a frozen or smeared session.
- **Stop order:** `StopAsync` stops the encoder first — flushing the pending queue to EOF so the file
  finalizes with no accepted frame lost — **before** awaiting the pump loop; the reverse order
  deadlocked on pipe backpressure.
- **Judge output in Release, by the burned `_outputIndex`** — the 6-digit counter burned into each
  frame is what lets a recorded file be audited for drops after the fact; stats alone are not proof.


### Current limitations / TODOs

- `Helpers/OAuthCredentials.cs` contains the real ClientId/ClientSecret. Auth is complete and the session **persists via Windows DPAPI** (`Helpers/TokenStore.cs` → `%APPDATA%\ytLlive\ytLlive.auth`, CurrentUser scope), reloaded best-effort at startup with a proactive refresh of a near-expiry access token. Sign-in/Change Account lives **inside the Start Stream dialog** (two-state flow — no separate Connect button). Sign-out is **explicit only**, and that means **Logout — NOT Change Account**: `StopStream()` does NOT clear the session — TASK 18's 2026-08-29 reversal ("stopping a recording leaves the creator signed in") — the older "graceful End signs out" line here was stale and is corrected against the code (2026-09-01). `YouTubeAuthService` takes an optional `HttpClient` + `sessionChanged` + `openBrowser` callback (test seams + save hook; services are still constructed in `MainViewModel`). **Change Account must never sign out first** (fixed 2026-09-27): it used to `await SignOutYouTubeAsync(); await SignInAsync();`, and since `SignOutYouTubeAsync` → `TokenStore.Clear()` **deletes** `ytLlive.auth`, cancelling in the Google chooser left the creator logged out of a working account. It is now just `await SignInAsync()` — success overwrites the token via the `sessionChanged` hook, cancel/failure leaves the session untouched. The OAuth flow also **releases its loopback callback port on retry**: abandoning the browser leaves the attempt holding `localhost:8765` for its 5-minute timeout, so a new `AuthenticateAsync` cancels and awaits the abandoned attempt before binding, serialised by a `SemaphoreSlim` (`_authGate`) — without that gate, two overlapping calls both see a free port and race to bind it (the exact failure the takeover prevents)
- **Google Cloud project ownership (recorded 2026-09-27, do not re-hunt):** OAuth ClientId is `198374416003-s8tm3rb5csh91quvlg6mnec6srmrr6md.apps.googleusercontent.com`, so the **Cloud project number is `198374416003`**. That project is owned by the **`llamachile.shop@gmail.com`** ("gramps") Google account. ⛔ **This is NOT the same account as the YouTube channel a creator signs into the app with** — console access and channel identity are separate logins, and assuming they are the same was the trap that stalled the search. Direct link that resolves it in one hop: `https://console.cloud.google.com/apis/credentials?project=198374416003`. If the picker shows no match, it defaults to the **"Recent"** tab — switch to **"All"** before concluding the project is missing.
- Scene/source/asset layout + the social bar persist (SQLite, schema v8); the OAuth session persists (DPAPI); the paid-unlock state persists (LayoutStore Settings table — `LicenseKey`/`LicenseValidatedAt`/`IsPremium`, 14-day offline grace)
- `YouTubeStreamService` manages the **variable reusable stream** (shipped 2026-08-16): `GetOrCreateReusableStreamAsync` lists `liveStreams?mine=true` and reuses the existing `cdn.isReusable` stream, creating it only on first use (`resolution=variable`, `frameRate=variable`); the stream is cached via `LayoutStore` (`SaveReusableStream`/`LoadReusableStream`, Settings table) and bound at broadcast insert (`boundStreamId`). Health (shipped 2026-08-16): `GetStreamHealthAsync(streamId)` polls `liveStreams?part=status` for `healthStatus` + `configurationIssues[]` → `StreamHealth`; the real API nests them under `status.healthStatus = {status, lastUpdateTimeSeconds, configurationIssues[]}` — the parse accepts that object shape and the legacy flat-string shape (fix 2026-09-22: the object shape crashed every poll with `requires an element of type 'String'`); banner decision in pure `StreamHealthReporter`
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **the output compositor (TASK 4 ship step 1) is SHIPPED**, **the FFmpeg locator (TASK 4 ship step 2) is SHIPPED**, **the encoder + RTMP push (TASK 4 ship step 3) is SHIPPED**, **WASAPI audio capture (TASK 4 ship step 4) is SHIPPED**, **frame-pipeline wiring (TASK 4 ship step 5) is SHIPPED**, **health stats (TASK 4 ship step 6) is SHIPPED**, **one-click go-live + private-only enforcement (TASK 4 ship step 7) is SHIPPED** — TASK 4 (RTMP Ingest) is fully done; full plan in `TASKS.md`
- `StreamConfig` defaults (`TargetBitrate=6000`, `Resolution="1920x1080"`) are stale — the live dropdown drives `StreamHealth.CurrentBitrate`/`FPS` instead
- **v1 task queue (2026-08-19):** tasks 19-23 added for v1 feature completeness; TASK 3.18 (chat box) shipped; TASK 16 (infinity display) shipped; TASK 10 (Polar billing) shipped — steps 1-7 (Velopack bootstrap, PolarLicenseService, license entry UI, LayoutStore entitlement, BrandFlash toggle, startup re-validation, PremiumUrl wired); Velopack update URL pending:
  - TASK 10: Polar billing — license key entry, offline entitlement, brand flash gating, Velopack auto-updates
  - TASK 19/23 merged: Control Surface UX — director's control room with an explicit EDIT toggle, scene strip above the preview, readouts in the top-left cell, and preview-rail drawers; current behavior is in `docs/ai/ui-surface.md`
  - TASK 20: Hotkeys (global keyboard shortcuts) — the single biggest UX gap
  - Broadcast metadata pull-out + launch geometry — shipped 2026-08-24 ("Text" tab on the Live screen; see `docs/ai/ui-surface.md`)
  - TASK 21: Media source (video file playback) — starting soon videos, BRB loops
  - TASK 22: Audio sync offset — lip-sync correction for USB mics/capture cards
  - These were identified through competitive analysis and are table-stakes for any streaming software in 2025-2026. Full details in `TASKS.md`.