b37b8a30f9
Measure take ty-1824 on the slice-16 build: the downscale fix worked (conv ~47ms, ring allocs 0) but desktop band was still 88% frozen at 6.8 updates/s. Telemetry isolated the real wall — CreateCopyFromSurfaceAsync readback ~45ms of each conversion, serialized one-in-flight => ~17/s capture cap. Docs fact: pool-sized surfaces CLIP, not scale (Microsoft Learn), so readback stays native; the lever is concurrency. - MaxConcurrentConversions=3 with pool 2->5 buffers (in-flight frames fit) - new MonotonicGate (Interlocked compare-exchange): stale OLDER completions are dropped, never overwrite a newer LatestFrame (mirror of 1742 tear) - FrameRingBuffer.Rent/ConsumeAllocations now lock; downscale row scratch is per-conversion locals - Good Dog test PublishGate_TryPublish_OnlyStrictlyNewerWins; 296/296 green, 0 warnings; docs cited Microsoft screen-capture page + libyuv fixed-point. Local only, no push.
1421 lines
140 KiB
Markdown
1421 lines
140 KiB
Markdown
# ytLlive — AI Guide
|
||
|
||
> Memory map entry point. Conventions live in [`schema.md`](schema.md); task
|
||
> status and YouTube API research in [`TASKS.md`](TASKS.md); brand/marketing
|
||
> palette and launch strategy in [`MARCOM.md`](MARCOM.md) (gitignored — read
|
||
> it before any branding or marketing work); directory maps in each folder's
|
||
> `index.md`. Reading order: this file → `TASKS.md` → `MARCOM.md` → `<dir>/index.md` → source.
|
||
|
||
## Response style
|
||
|
||
No default "Plans & Pitfalls" / planning boilerplate. Respond directly and
|
||
concisely: **do the queued work, then report what changed and what's next.**
|
||
Skip feature pitch, step-by-step implementation plans, pros/cons tables, and
|
||
"potential pitfalls" sections unless the user explicitly asks for a plan first.
|
||
A short diff-style summary beats a proposal document every time.
|
||
|
||
## Derivative work (spin guard mandated in AGENTS.md)
|
||
|
||
Nothing we build is novel — streaming/overlay/WebView2 problems were solved by
|
||
OBS, CEV, and WPF ecosystems long before us. Two consequences:
|
||
1. **Proactive:** new features start with a quick external scan, not head-first
|
||
design (citation goes in the commit message).
|
||
2. **Objectively triggered spin guard:** a second failed fix for the same symptom
|
||
means STOP theorizing and research the established answer externally; there is
|
||
no unique bug in this repo that the wider ecosystem hasn't hit.
|
||
This project's hard lesson: the web-source bounding saga (2026-08) burned 9
|
||
commits rediscovering that a browser source is a *fixed canvas* — OBS keeps it at
|
||
a stable page size, crops/hugs content via the box, and clips at the edge. The
|
||
apparent "gap" that ended the saga was a widget's own CSS glow effect, not a bug.
|
||
|
||
**Derived-solution rule (2026-08-29, driven by the image-shrink incident):** when
|
||
you work out any one-off, reusable solution (recipe / workaround / how-to), write
|
||
it into `MyMistakes.md` → **Recipes registry that same session**, and grep it
|
||
before ever re-deriving. A solution recorded once ends the loop; an un-recorded
|
||
one guarantees the user hears it derived again (see `AGENTS.md` → Working rules → 🔬).
|
||
|
||
## LAN infrastructure (192.168.50.x)
|
||
|
||
All LAN credentials live in `CREDENTIALS.md` (gitignored). Server IPs and roles:
|
||
|
||
| Host | IP | Role |
|
||
|------|----|------|
|
||
| corsair | .132 | Gaming/streaming primary — RTX 5070 Ti (this machine) |
|
||
| llamavault | .86 | ODROID-HC4, 12TB RAID5, media storage (NFS). SSH: `ssh mshallop@192.168.50.86` |
|
||
| pivault | .158 | RPi5 8GB, 10.83TB RAID5 NAS |
|
||
| jarvis | .210 | AMD Ryzen 5 5600X, RX 6600 XT — AI inference, SearXNG, Docker registry, Immich |
|
||
| ultron | .108 | AMD Ryzen 7 7840HS — AI orchestrator, llama-server, Qdrant |
|
||
| gordito | .144 | Home Assistant, Alarmo, Zigbee |
|
||
| officerfriendly | .26 | Pi-hole DNS |
|
||
| octopi | .197 | OctoPrint (Prusa MK4) |
|
||
| BigBlinkyRouter | .1 | ASUS ROG GT-BE98 Pro, WiFi 7 |
|
||
|
||
Web infra (DO droplet): `llamacasty.com` + `llamachile.tube` at `143.244.176.131`.
|
||
|
||
### No-Fluff Mode (on demand)
|
||
|
||
Invoke with "no-fluff mode" (or similar) when you want ruthless review instead
|
||
of reassurance. In that mode:
|
||
|
||
- Strip all polite pleasantries, emojis, transitions, and conversational padding.
|
||
- Treat the user's input as a draft to be methodically deconstructed or
|
||
strengthened — argue, correct, and sharpen rather than agree.
|
||
- Give unvarnished truth, not reassurance.
|
||
|
||
This is an occasional, explicitly-invoked mode — **never the default**. The
|
||
default response style above stays in effect unless invoked.
|
||
|
||
## Brand
|
||
|
||
Brand palette and official assets live in `MARCOM.md` (gitignored — read it for hex values,
|
||
asset file names, and marketing copy). The brand red `#e94560` is the primary accent used
|
||
throughout the UI.
|
||
|
||
### Product name vs repo/assembly branding (2026-09-14)
|
||
|
||
The product is **llamacasty** (llamacasty.com, llamachile.tube, `llgit.llamachile.tube`). The
|
||
repo path, csproj `AssemblyName`/`RootNamespace`, DB/log paths (`%APPDATA%\ytLlive\...`),
|
||
and most internal namings are the legacy **ytLive/ytLlive** — a re-brand that never renamed
|
||
the internals. Treat the two as separate: external/user-facing language says "llamacasty",
|
||
code/assembly/repo names stay `ytLive`. If a full internal re-brand is ever done, this note
|
||
and the UI locator (Premium/wordmark) are the checklists; it is NOT a goal pre-1.0.
|
||
|
||
### AI transparency
|
||
|
||
This project is built through AI-assisted pair-programming. All code is generated
|
||
under the supervision of an experienced developer with decades of coding experience.
|
||
Architecture decisions, product direction, and quality gates are human-owned;
|
||
AI accelerates implementation. This is a testimonial to successful human-AI
|
||
collaboration, not autonomous code generation.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
# From WSL, ALWAYS use the Windows dotnet host — never Linux `dotnet` for this project:
|
||
"/mnt/c/Program Files/dotnet/dotnet.exe" build "C:\Users\gramp\Documents\Code\projects\ytLive\ytLive.csproj"
|
||
"/mnt/c/Program Files/dotnet/dotnet.exe" run
|
||
```
|
||
|
||
**Pre-commit gate (write it down — I forgot once):** ALWAYS run
|
||
`./scripts/verify.sh "<file1>" "<file2>" ...` (declared scope) before committing. It forces a
|
||
**clean** build (0 warnings), the full test suite (passes only the 2 known pre-existing `[FAIL]`s), and
|
||
the scope check in one shot. An **incremental** build can report `0 Warning(s)` while skipping
|
||
recompiles — that's how `CS8601` slipped past on 2026-08-29. Never claim 0 warnings from anything but
|
||
`verify.sh`'s clean build.
|
||
|
||
`EnableWindowsTargeting=true` in `ytLive.csproj` lets a cold restore work from WSL, but a Linux
|
||
`dotnet run`/`build` re-downloads 100M+ of `windowsdesktop.app.*` packs into the Linux NuGet cache
|
||
(which lacks them) over the slow 9p `/mnt/c` bridge — twice, because the WPF `_wpftmp` generated
|
||
project triggers a second restore (203s observed). The Windows cache has the SQLite packages and the
|
||
packs resolve from `C:\Program Files\dotnet\packs`, so the Windows host never re-downloads.
|
||
Never use `--no-restore` right after an interrupted restore — the stale `project.assets.json`
|
||
produces misleading `NETSDK1064` "package not found" errors. Running requires Windows anyway.
|
||
|
||
## Tests
|
||
|
||
xUnit in `ytLive.Tests` (net8.0-windows10.0.19041.0, matches the app TFM). Run on Windows —
|
||
WSL can't run net8.0-windows tests:
|
||
|
||
```bash
|
||
dotnet.exe vstest "C:\...\ytLive.Tests\bin\Debug\net8.0-windows10.0.19041.0\ytLive.Tests.dll"
|
||
```
|
||
|
||
Good Dog Rule: ONE integration test per change (no feature branches pre-1.0 — work lands on
|
||
`main`, see AGENTS.md). Current: TokenStore DPAPI
|
||
roundtrip/corrupt/missing/clear, OAuth exchange/refresh/ClearSession, CameraManager refcount +
|
||
frame pump + failure handling (fakes for the WinRT seams), real-`MainWindow` round-clip
|
||
interaction test, LayoutStore delete roundtrip, LayoutStore pre-round-rect-dims roundtrip,
|
||
LayoutStore background roundtrip, LayoutStore HasBackground roundtrip + v5→v6 non-Live backfill,
|
||
ScreenCaptureManager refcount + shared-bitmap + coalescing
|
||
(fake `IScreenCaptureSource` + a real background-STA `Dispatcher`), BackgroundTests, SceneCatalogTests,
|
||
the TASK 25 dirty-layout heal (BackgroundHealIntegrationTests), WebcamSafeguardTests,
|
||
SceneCompositorTests (full-scene composite + vertical tier), StretchMathTests, FfmpegLocatorTests,
|
||
FfmpegEncoderTests, FramePumpTests, the TASK 8/11/12 audio chain (AudioPipelineTests + MasterLimiter),
|
||
the Socials fediverse-heal roundtrip, AboutHubTests, NotificationAreaIntegrationTests (TASK 24),
|
||
GlobalHotkeyTests + HotkeyConfigTests (TASK 20), WebcamMenuGateTests (TASK 26), ChatLayerGateTests
|
||
(TASK 27), BroadcastPullOutTests (TASK 29), DefaultRecordFolder fallback (TASK 30), WebView2ManagerTests
|
||
(TASK 17), RecordingFileTests + OnAirSignTests (TASK 18), SessionTeardownTests +
|
||
WebcamOutputKeyTests + GameMeterHonestyTests (2026-09-01 recording-verification fixes) — **ZERO
|
||
known failures as of 2026-09-01. `AudioPipelineTests` 25/25 green in 26ms —
|
||
the "known failing" `Mix_HonorsProviderGains…` and the class's notorious STANDALONE HANG shared
|
||
one root cause: TASK 22's `_delayedMix` (nullable, never initialized) was dereferenced
|
||
(`delayed.Length` on null) every live-mix tick — a swallowed NRE starved the pipe the tests
|
||
read and flooded startup.log. Test was right, code drifted from the map's own contract
|
||
(loopbackGain = GameAudioVolume — now honored; the "unity because loopback scales with the
|
||
endpoint" assumption was disproven by the creator's 20%-volume meter observation). The former
|
||
RoundClip "known failure" (stale-test layers: namescoped `FindName` + `VisualTreeHelper.HitTest`
|
||
where `UIElement.InputHitTest` models input — see MyMistakes) was fixed the same day.
|
||
Per-class runs through the Windows dotnet.exe host execute the RealApp/MainWindow suites fine;
|
||
only the FULL-suite run still hangs (WASAPI teardown, pre-existing) — and the whole-suite
|
||
"247 total" era count is stale; trust per-class results.**
|
||
|
||
Reward-event capture (monetization awareness, see the Monetization section) will add its integration
|
||
tests here when it ships: one real chat-poll payload containing all seven reward event types →
|
||
assert the persisted canonical `RewardEvent` rows (type + amount/currency/tier/memberLevel/participant
|
||
ids) round-trip into SQLite.
|
||
|
||
### Real-MainWindow tests MUST be hermetic (DB pollution bug)
|
||
|
||
The integration test boots a real `MainWindow` → `MainViewModel` → real `LayoutStore`
|
||
(`%APPDATA%\ytLlive\ytLlive.db`). `Shutdown()` on close **saves the layout** (full rewrite:
|
||
DELETE all scenes/sources, re-insert), so any source a test adds would be persisted over the
|
||
user's real ones — this happened and wiped the real webcam source (DeviceId replaced by the
|
||
test's fake `test-camera`). Rule: a test that constructs `MainWindow` MUST first set
|
||
`MainViewModel.LayoutPathOverride` to a temp DB path and reset it (plus
|
||
`SqliteConnection.ClearAllPools()` + delete) in `finally`. The seam is
|
||
`internal static string? LayoutPathOverride` (line ~529 in `MainViewModel.cs`),
|
||
`ytLive.csproj` has `InternalsVisibleTo("ytLive.Tests")`.
|
||
|
||
The layout DB is a **full rewrite per save** (delete all, re-insert from memory), so
|
||
save/load round trips are exact: an element removed in the UI (`RemoveElement` →
|
||
`scene.Elements.Remove` → `OnElementsChanged` → debounced `ScheduleSave`, plus `Shutdown` on
|
||
close) does **not** come back after reload (`LayoutStorePersistenceTests` guards this).
|
||
|
||
## Architecture
|
||
|
||
C# / WPF (.NET 8) following MVVM:
|
||
|
||
| Path | Role |
|
||
|------|------|
|
||
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
|
||
| `ViewModels/` | MainViewModel — `public partial class`, one file per functional area (Scenes, Background, Webcam, Audio, Trax, Socials, Streaming, Chat, Overlays, Account, License, Recording — split complete, see `ViewModels/index.md`); **Chat.cs is a thin delegating facade over `Services/ChatOverlayLayer.cs` (Commit G, first true decomposition)**; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
|
||
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** |
|
||
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, **BuildStamp (wordmark release counter `#N`, +1 per commit, generated by `GenerateBuildStamp` in ytLive.csproj via git rev-list; the per-build GUID now logs to startup.log only — 2026-09-04)**, visibility converters |
|
||
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
|
||
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview + live controls below), left (scenes/sources), right (chat), bottom (gear + stream stats + resolution) |
|
||
|
||
### Key patterns
|
||
|
||
- `ViewModelBase.SetProperty<T>()` for property change notifications
|
||
- `RelayCommand` for all button actions; commands gate on state (e.g. Start only when Offline). **Typed `CommandParameter`s — no stringly-typed command tokens:** menu items that pick a source type pass the enum value itself (`CommandParameter="{x:Static models:SourceType.DisplayCapture}"`), so a typo breaks the build instead of silently adding an Image; `AddSource` still falls back to `Enum.TryParse<SourceType>(..., true)` for safety. The webcam item is its own `AddWebcamCommand` (it greys out via `CanAddWebcamToActiveScene` and isn't a `SourceType` — webcams are `WebcamSceneConfig`, not `Source` rows)
|
||
- **Audio is KISS by rule** — the whole of audio is *one knob*: **desktop/game audio is automatic** (WASAPI loopback from the default output at unity, zero UI — "it just is"); the **mic is the creator's only audio control** — sound meter + mute button + volume slider (`MicVolume`, defaults to 0.8) all sit together on the second line of the preview bottom row, BELOW the Socials+TRAX row, CENTERED beneath the preview panel. The TRAX button + Desktop Audio meter sit on the first line of that same row. Meter: 288px, muted slate track (`#3a3b52`) with ruler graduations and muted yellow/red zone tints at 60%/80%; fill = green → yellow → red via `MeterFillWidth`/`MeterBrush`; the meter is a **READ-ONLY realtime level display** — it shows the live input level scaled by the volume (raising the volume moves ambient noise up the bar), NOT the volume setting: the fill is `Math.Min(1, AudioLevelMeter.ToDisplay(AudioLevel) * MicVolume)` — `ToDisplay` maps the raw linear RMS onto a −60..0 dBFS display scale, because real speech sits around −40..−20 dBFS (0.01..0.1 linear) which would leave a flat scale dead (`AudioLevel` is fed by the audio mixer once capture lands, 0 with no input) and 0 while muted. While the volume slider is being dragged the bar previews the slider position (`SetVolumeAdjusting`, from `PreviewMouseLeftButtonDown/Up` + `LostMouseCapture` handlers) so the creator sees where they're setting it; on release it returns to the live level — with no input it bounces back to 0, exactly as it does today. Clicking the meter does nothing; **clicking the MIC label opens the mic picker** (`OpenMicPickerCommand`), and the picked voice source name (`MicSourceName`) is shown left-justified INSIDE the meter bar (FontSize 10, ellipsized to the bar) — the fill runs at 75% opacity so the text and the ruler markings stay visible through it. Mute (`ToggleMicMuteCommand`/`MicMuted`) is a plain clickable speaker icon (`MicSpeaker_MouseLeftButtonUp` code-behind handler — not a Button, `Stretch="Uniform"` so the glyph is never clipped) that swaps to a red do-not-symbol (slashed speaker) when muted. **The slider and the speaker can never disagree:** `MicMuted` is read-only, derived from `MicVolume == 0` — sliding the volume off flips the speaker to muted (storing the prior level in `_volumeBeforeMute`), sliding it up from 0 clears the mute indicator (and the stored level); the speaker button just runs the volume to 0 or restores it (default 0.8 if unknown). Muting zeroes the meter; **unmuting flashes the meter to the restored position for ~300ms** (`BeginVolumeFlash`/`EndVolumeFlash` on a DispatcherTimer, cancelled if the slider is grabbed) before it returns to the live level. Line 2 of the footer holds everything else: stream stats (bitrate/fps/dropped/duration/health) on the left, quality dropdown + gear on the right. The slider is a slim dimensional style in `Themes/Controls.xaml` (gradient track, beveled green fill on a 5px pill, gloss-sphere thumb with drop shadow — deliberately NOT flat). No device pickers (never show device names — no "install a device you didn't know existed"), no filter stacks, no monitoring, no routing — OBS's confusion (dynamic mixer, unintuitive names, four required filters) is deliberately absent. A production-ready mic chain (high-pass → noise gate → compressor) will be applied invisibly in the mixer, unconfigurable. Capture runs only while live (privacy indicator stays off otherwise). Capture pipeline = `IAudioSource` seam + NAudio `WasapiCapture`/`WasapiLoopbackCapture` + `AudioMixer` (pending — the UI is in place now). **Mic mute icon (2026-08-13):** a second 16px clickable glyph — a microphone, red + slash when muted — sits **between the meter and the speaker** (both mutes adjacent, spacing between the icons) and reuses the same `MicSpeaker_MouseLeftButtonUp` → `ToggleMicMuteCommand` handler. **REC / ON-AIR pills + signs (TASK 18, 2026-08-29):** the top-center area is now two **sliding pill toggles** (intent) next to two **status signs** (reality). REC pill (`RecordPillOn`) is the local-recording intent — works signed-out; ON-AIR pill (`OnAirPillOn`) is the streaming intent — greyed/disabled until `IsConnected`. The **REC sign** (`RecDotBrush`/`RecTextBrush`/`RecDotOpacity`) is dark-gray + dim offline and **green-tinged, pulsing only while actually recording** (`IsRecording`), not merely pill-on. The **ON-AIR sign** (`OnAirBrush` = `#555` offline, `#22c55e` live) is green while actually streaming (`IsLive`); the `PRIVATE` badge (`IsLivePrivate`) still shows when the broadcast is private. Both sign labels share the `StatusSignText` style (`Themes/Controls.xaml`) so REC and ON-AIR can't drift apart — status color lives on the dot only, never the label. The elapsed timer shows while `IsLive || IsRecording`. The connected account's avatar/name shows in the top bar next to the primary button (`AccountAvatarUrl`/`AccountDisplayName` via `SyncConnectedAccount`), so the creator always sees WHICH account will go live.
|
||
- ViewModels are constructed in XAML (`<vm:MainViewModel/>` as DataContext)
|
||
- **True decomposition beats partial-shuffling (2026-08-31, Commit G):** `MainViewModel` is a ~4100-line god-object split into 17 partials — but partials are a *myth of decomposition*: every partial shares the same class, same `SetProperty` state, same collaborators via `Scenes`/`StagedScene`/`IsLive`/`Source.VideoImageSource`, so an AI fetching one partial still reconstructs the whole class. The only thing that actually improves retrieval is a boundary where a feature **owns its own state + collaborators**. `Services/ChatOverlayLayer.cs` is the model (extracted from `MainViewModel.Chat.cs` 194→44): the layer owns the message buffer, `ChatBoxRenderer`, fade/mock timers, preview renders, and the live `RenderFrame`; the VM keeps only the binding surface (`ChatMessages` delegates to `_chatLayer.Messages` so XAML + LeftPanel `CollectionChanged` hold; the two computed gate props stay on the VM because their `OnPropertyChanged` is raised from VM setters). **Diagnostic — glue vs. component:** before extracting, count `OnPropertyChanged`/`SetProperty` touches (glue) and bound-property reads per partial. Zero/low-glue + a cohesive state blob (renderer/timers/buffer) = extract (chat). High glue over an already-extracted service (Audio, Webcam, Background, Scenes — `AudioMixer`/`CameraManager`/`ScreenCaptureManager`/`ChatBoxRenderer` already exist) = leave as VM glue; forcing it adds coupling, doesn't remove it. Don't manufacture seams to hit a line count — the line count is a guideline for context, not a design goal.
|
||
- **SceneGraph component + baked-crust compositor (TASK 31):** `Services/SceneGraph.cs` owns the scene collection (`Scenes` — the ViewModel's `Scenes` property delegates to it) and the element mutation surface (`AddElement`/`InsertElement`/`RemoveElement`/`MoveElement`, each invalidating the bake cache) plus queries that were scattered LINQ (`GetBackground`/`GetWebcam`/`GetChatBoxes`/`GetSplitPoint`/`IsStatic`). Elements expose `ElementKind Kind` (`Static` = images/background art, `Dynamic` = webcam/live capture/chat/web). The **split point** is the index of the first dynamic element; `SceneGraph.GetBakedBase` bakes/caches all static layers below it (keyed by scene id + static element identities), and `SceneCompositor.BakeStaticBase`/`CompositeLayers`/the `Render(.., staticBase, split)` overload composite the dynamic/above-split layers per frame. `FramePump.RenderScene` uses the optimized path when a `SceneGraph` is wired in (falls back to full render without one). Invariant: dynamic-only pixel changes never invalidate; static layout/opacity/visibility/asset changes do (via `InvalidateBake` from the VM's element-property and background-heal paths). **Defensive deviation from the TASK 31 spec:** `ChatOverlayLayer` keeps taking `IEnumerable<Scene>` instead of depending on `GetChatBoxes()` — it is deliberately decoupled from the graph (its doc comment says "without owning the scene graph"); and the static background helpers (`EnsureBackground`/`NormalizeBackgrounds`) stay on the ViewModel because `BackgroundTests.cs` unit-tests `MainViewModel.EnsureBackground` directly. Queries + invalidation moved; helpers stayed.
|
||
- Services are currently instantiated in MainViewModel's constructor — no DI container yet
|
||
- Layout persists to SQLite (`Microsoft.Data.Sqlite`); scenes/sources/asset bytes stored in the DB, asset identity is a SHA-256 content hash (1:M reuse, no file paths — assets are always available). Loaded sources always derive `IsBackground` from `Type` (OR'd with the persisted column, so legacy DisplayCapture backdrops keep their flag) — pre-derivation rows with `IsBackground=0` heal on load
|
||
- **Five-scene catalog (`Models/SceneCatalog.cs`):** the product is exactly Starting/Live/BRB/Chat/Ending — work with less, never more (the escape hatch for "more" is OBS). Scenes are matched **by name** (`SceneCatalog.Is`, case-insensitive trim). Empty DBs seed all five; the scenes-header "+" (`ShowAddScene`/`MissingScenes` on `MainViewModel`) only appears while ≥1 canonical scene is missing and its menu lists only the missing ones, re-adding them by name (`AddSceneCommand`). Renaming a canonical scene makes it missing again; `AddScene` rejects non-canonical names.
|
||
- Theming: all custom styles live in `Themes/Controls.xaml`, merged in `App.xaml` — never duplicate styles per-window (dialog duplicates were consolidated into this dictionary)
|
||
- Resolution tiers: 1080p60@8 (default) → 1080p30@8 → 720p60@6 → 720p30@6 → **Vertical 1080p60@8 (9:16, 1080×1920)**. The composition master frame is **always 1920×1080** — a tier is an output rect + target resolution over that master, so source geometry is never rewritten (no rounding drift). 16:9 tiers use the full frame; the vertical tier uses a centered **607×1080** window and the preview dims the cropped side strips at 55% black with an accent outline (semi-crop — the cut area stays visible). A resolution badge in the preview corner shows the active tier. Bottom bar: gear icon far left, stream stats (bitrate/FPS/dropped/duration) centered under preview, resolution dropdown far right. A **tooltip** explains finding upload bandwidth — an in-app speed test was deliberately dropped (unreliable). The future encoder crops the master to the rect and scales to the tier's Width×Height
|
||
- Crash diagnosis: `AppLog` writes startup checkpoints to `%APPDATA%\ytLlive\startup.log`; `App.xaml.cs` logs `DispatcherUnhandledException`/`AppDomain.UnhandledException`. When WPF won't run from WSL, this log is how you find the failure (it caught the `MenuItemRole.Separator` XAML crash and the ComboBox SelectionBoxItem bug)
|
||
|
||
### ⚠️ Known issue: screens/layers settings audit — RETIRED INTO THE GOLD PASS (2026-08-22, dispositioned 2026-09-01)
|
||
|
||
The AI hallucinated through multiple commits that night on background/scene property placement, repeatedly misreading the spec and introducing bugs. Specifically: moved properties to the wrong superclass level, used `new` instead of `override` breaking WPF bindings, removed working code, and added UI elements that weren't spec'd. The audit was never run since. The creator ruled (2026-09-01): it is a **going-gold requirement**, not session debt — the fine-tooth-comb pass (every screen's Background layer properties, pill toggle persistence, context menu visibility, "Add Layer"/"(+)" menu items) is a named checklist item under **TASK 36 (gold pass)** and this landmine is closed here.
|
||
|
||
### Current limitations / TODOs
|
||
|
||
- `Helpers/OAuthCredentials.cs` contains the real ClientId/ClientSecret. Auth is complete and the session **persists via Windows DPAPI** (`Helpers/TokenStore.cs` → `%APPDATA%\ytLlive\ytLlive.auth`, CurrentUser scope), reloaded best-effort at startup with a proactive refresh of a near-expiry access token. Sign-in/Change Account lives **inside the Start Stream dialog** (two-state flow — no separate Connect button). Sign-out is **explicit only** (Logout / Change Account → `SignOutYouTubeAsync`): `StopStream()` does NOT clear the session — TASK 18's 2026-08-29 reversal ("stopping a recording leaves the creator signed in") — the older "graceful End signs out" line here was stale and is corrected against the code (2026-09-01). `YouTubeAuthService` takes an optional `HttpClient` + `sessionChanged` callback (test seam + save hook; services are still constructed in `MainViewModel`)
|
||
- Scene/source/asset layout + the social bar persist (SQLite, schema v8); the OAuth session persists (DPAPI); the paid-unlock state persists (LayoutStore Settings table — `LicenseKey`/`LicenseValidatedAt`/`IsPremium`, 14-day offline grace)
|
||
- `YouTubeStreamService` manages the **variable reusable stream** (shipped 2026-08-16): `GetOrCreateReusableStreamAsync` lists `liveStreams?mine=true` and reuses the existing `cdn.isReusable` stream, creating it only on first use (`resolution=variable`, `frameRate=variable`); the stream is cached via `LayoutStore` (`SaveReusableStream`/`LoadReusableStream`, Settings table) and bound at broadcast insert (`boundStreamId`). Health (shipped 2026-08-16): `GetStreamHealthAsync(streamId)` polls `liveStreams?part=status` for `healthStatus` + `configurationIssues[]` → `StreamHealth`; banner decision in pure `StreamHealthReporter`
|
||
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **the output compositor (TASK 4 ship step 1) is SHIPPED**, **the FFmpeg locator (TASK 4 ship step 2) is SHIPPED**, **the encoder + RTMP push (TASK 4 ship step 3) is SHIPPED**, **WASAPI audio capture (TASK 4 ship step 4) is SHIPPED**, **frame-pipeline wiring (TASK 4 ship step 5) is SHIPPED**, **health stats (TASK 4 ship step 6) is SHIPPED**, **one-click go-live + private-only enforcement (TASK 4 ship step 7) is SHIPPED** — TASK 4 (RTMP Ingest) is fully done; full plan in `TASKS.md`
|
||
- `StreamConfig` defaults (`TargetBitrate=6000`, `Resolution="1920x1080"`) are stale — the live dropdown drives `StreamHealth.CurrentBitrate`/`FPS` instead
|
||
- **v1 task queue (2026-08-19):** tasks 19-23 added for v1 feature completeness; TASK 3.18 (chat box) shipped; TASK 16 (infinity display) shipped; TASK 10 (Polar billing) shipped — steps 1-7 (Velopack bootstrap, PolarLicenseService, license entry UI, LayoutStore entitlement, BrandFlash toggle, startup re-validation, PremiumUrl wired); Velopack update URL pending:
|
||
- TASK 10: Polar billing — license key entry, offline entitlement, brand flash gating, Velopack auto-updates
|
||
- TASK 19/23 merged: Control Surface UX — director's control room, transitions (Cut/Fade/Move), thumbnails above central monitor, edit mode offline only, right panel eliminated
|
||
- TASK 20: Hotkeys (global keyboard shortcuts) — the single biggest UX gap
|
||
- Broadcast metadata pull-out + launch geometry — shipped 2026-08-24 ("Text" tab on the Live screen, see its section below)
|
||
- TASK 21: Media source (video file playback) — starting soon videos, BRB loops
|
||
- TASK 22: Audio sync offset — lip-sync correction for USB mics/capture cards
|
||
- These were identified through competitive analysis and are table-stakes for any streaming software in 2025-2026. Full details in `TASKS.md`.
|
||
|
||
### Control Surface UX (TASK 19/23 — shipped 2026-08-24, verified by code survey)
|
||
|
||
The app is a **director's control surface**, not an editor. This replaces the OBS-style scene list with a TV-control-room metaphor.
|
||
|
||
**Layout:** Two-column. Left panel = one panel, two states. Center = thumbnails above central monitor.
|
||
|
||
- **Left panel:** When offline (edit mode) → LAYERS + PROPERTIES (same as current, minus SCREENS). When live/recording → YouTube Live Chat (entire panel replaced, never coexists with layers/properties). Mode is implicit (derived from `StreamStatus`), no toggle button.
|
||
- **Thumbnails:** Five scene previews above the central monitor, 16:9, drag-reorderable. Click = stage (offline) or transition (live). Active scene gets brand-red glow border.
|
||
- **Central monitor:** Shows staged scene (offline, editable in edit mode) or live output (live, read-only). During transitions: compositor-blended animation.
|
||
- **Right panel:** Eliminated. Chat moved to left panel (live mode).
|
||
- **Edit gating:** `IsEditMode` = `StreamStatus == Offline && !IsRecording`. When live/recording: all drag/resize, context menus, right-click property adjustments disabled. Gun safety — no accidental edits mid-stream.
|
||
|
||
**Scene reference split:** `ActiveScene` → `StagedScene` (central monitor target, editable offline) + `LiveScene` (encoder output). Go Live sets `LiveScene = StagedScene`. Thumbnail click sets `StagedScene` (offline) or fires transition (live).
|
||
|
||
**Transitions:** `TransitionService` (new file) — Cut/Fade/Move with configurable duration (default 300ms). `SceneCompositor.RenderBlend()` blends two scenes' frames. `FramePump` feeds `LiveScene` to encoder; during transitions, feeds blended frames. Go Live always = Cut (clean start).
|
||
|
||
**Thumbnails:** `Scene.Snapshot` property. `SceneCompositor.Render()` at 320×180 → `WriteableBitmap`. Only staged scene has live backdrop render; other 4 = static snapshots.
|
||
|
||
**Thumbnails:** `Scene.Snapshot` property. `RefreshSnapshotsAsync` fills minis from each scene's background art (`LoadBackgroundImage`); **minis never render live captures** — while the Live screen is staged its mini shows the green placeholder, unstaged it shows `live-background.jpg` (real-time surface = center monitor only; user decision 2026-08-23).
|
||
|
||
### Global hotkeys (TASK 20 — step 1 shipped 2026-08-24)
|
||
|
||
- `Services/GlobalHotkeys.cs`: `HotkeyId` enum (9 actions), `IHotkeyRegistrar` seam, `Win32HotkeyRegistrar` (`RegisterHotKey`, no modifiers), `GlobalHotkeyManager`. Fixed defaults: F1-F5 canonical scenes, F6 start/end stream, F7 mic mute, F8 desktop-audio mute, F9 TRAX play/pause.
|
||
- `MainWindow.OnSourceInitialized` creates the manager and forwards `Activated` → `MainViewModel.HandleHotkey`; window `Closed` detaches (unregister all).
|
||
- `MainViewModel.HandleHotkey` maps ids to `TransitionToScene`/start-end commands (honoring `CanExecute`)/mute toggles/`TraxLeftClick`.
|
||
- Tests: `RegistrarOverride` static seam mirrors `LayoutPathOverride`. **Landmine:** forcing a WPF window's handle with `WindowInteropHelper.Handle` does NOT raise `SourceInitialized` and returns Zero pre-show — use `EnsureHandle()` in tests; it both creates the HWND and fires the event.
|
||
|
||
### Broadcast metadata pull-out + launch geometry (2026-08-24)
|
||
|
||
**The "Text" tab** (always visible, 2026-08-24 revision): a white 30px tab with rotated YouTube-red "Text" sits on the right edge of the preview, vertically centered — **shown regardless of login/stream state** (creator decision; original IsLive gating removed same day). Click **toggles** the drawer (`ToggleDrawerCommand`); any left-click outside the form also collapses it (`Window_PreviewMouseLeftButtonDown` checks `TextPullOutHost` ancestry), as does the ✕. The drawer is a 320px dark panel over the preview's right edge (200ms CubicEase on `Width`, `BroadcastForm.IsDrawerOpen` drives it). The Update button still greys until `CurrentBroadcastId` exists.
|
||
|
||
**Fields — the always-editable snippet/status set** (⚠️ creator premise inverted: contentDetails fields like latency/DVR/embed are editable ONLY in created/ready, NOT while live; they are deliberately absent):
|
||
- Title / Description / Tags (comma-separated string → `snippet.tags[]`) / Visibility ComboBox (private/unlisted/public) / Made-for-Kids CheckBox — all live-editable pre- and post-launch
|
||
- Scheduled Start is read-only ("set at Go Live")
|
||
|
||
**Plumbing:** `Models/BroadcastMetadata` POCO; `Services/LiveBroadcastFormViewModel` (owns field state, persists every edit to Settings keys `Broadcast.*` immediately via `Func<LayoutStore>` seam — survives layout re-pointing; owns drawer state; `UpdateBroadcastCommand` gated by `CurrentBroadcastId != null && !IsUpdating`). `YouTubeStreamService.UpdateBroadcast` PUTs `liveBroadcasts?part=snippet,status` and **must echo scheduledStartTime — update replaces the whole snippet part**, omitting it clears the schedule. Go Live prefills the dialog from the form (`BroadcastForm.Title/Description`) and calls `CaptureGoLive(title, description, now)` on confirm. `MainViewModel.CurrentBroadcastId` exposes the private `_currentBroadcastId`; notified on create/end.
|
||
|
||
**The old Default Stream Title/Description** (un-persisted, App Settings overlay) are DELETED — the form owns those fields end-to-end; Go Live prefill routes through it.
|
||
|
||
**Launch geometry:** default **1920×1040**, MinWidth **1366**, MinHeight **768**, `WindowStartupLocation=Manual`. Window size/position persist on close via `RestoreBounds` (+ WindowState Normal/Maximized; Minimized treated as Normal) into Settings keys `Window.*`; restored in the MainWindow ctor BEFORE show, clamped to the minimums and the primary `SystemParameters.WorkArea` (a position saved on a since-disconnected secondary monitor falls back on-screen). Multi-monitor restore is not supported yet.
|
||
|
||
Test seam note: `YouTubeAuthService.SetSession()` is public — tests inject a channel with future `TokenExpiry` so `EnsureToken()` passes offline; pair with an `HttpClient(new StubHandler())`.
|
||
|
||
### Screen backdrop capture (TASK 3 ship task #1) — model reworked by TASK 25
|
||
|
||
**The Background is a locked per-screen layer (TASK 25): exactly one per canonical
|
||
scene, always named "Background", pinned at index 0, never addable/removable/reorderable,
|
||
absent from the (+) menu. Flavor is decided by scene: Live = `DisplayCapture` (the real-time
|
||
desktop/game surface with Show Desktop / monitor switch / fullscreen-game detect, falling back
|
||
to `live-background.jpg`); Starting/BRB/Chat/Ending = static `Background` art layers.**
|
||
`MainViewModel.NormalizeBackgrounds(scenes)` (internal static) heals any loaded layout to this
|
||
model on every load: keeps the correctly-flavored row (or converts a survivor in place — note
|
||
`Source.Type`'s setter derives `IsBackground`, so conversions must re-assert the flag), drops
|
||
duplicates/stale rows, seeds missing ones via `CreateBackground(name)`, renames, moves to index 0;
|
||
non-canonical scenes get `HasBackground=false` and lose any backgrounds. `HealBackgrounds()`
|
||
(instance) runs Normalize + stamps default art (`StampDefaultBackgroundAsset` reads
|
||
`Assets/{scene}-background.jpg` pack resources through `AddAsset`; custom Browse art wins).
|
||
This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Background`,
|
||
`SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) and `EnforceBackgroundPolicy`.
|
||
|
||
- **Model (schema v6):** `Source.IsBackground` (persisted) marks the one Background per scene; `Source.CaptureKey`
|
||
(persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The Live row is
|
||
a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackground(scene)`,
|
||
internal static — also used by the `StagedScene` setter so a scene switch can never expose a
|
||
missing Background), fixed at X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
|
||
the element template sets `IsHitTestVisible=false` for
|
||
backgrounds; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackground`
|
||
(persisted) is true for all five canonical scenes** (see `SceneCatalog.HasBackground`). Capture controls —
|
||
"Change Capture…"/"Refresh Capture"/"Capture Display"/"Show Desktop" — exist only while the **Live** scene is
|
||
staged (`CanChangeBackground` = staged scene is Live; the preview CanvasGrid menu binds it directly, the layer-row
|
||
menu MultiBindings it with the row's own `IsBackground` via `Helpers/AllTrueToVisibilityConverter`). Non-Live
|
||
screens keep the properties panel pill ("Use default background") + Browse for custom art.
|
||
- **Settings hygiene:** `LayoutStore.Save` purges orphaned `BackgroundUseDefault_{id}` /
|
||
`BackgroundPath_{id}` Settings keys whose id is no longer a live Source row (TASK 25 heal drops duplicates; their keys must not accumulate).
|
||
- **Layering invariant (regression lesson, 2026-08-23):** `ActiveBackgroundImage` renders *above*
|
||
`BackgroundImage`, so it is fed by **static-art rows only** (`Type == Background`). The capture-flavored
|
||
Live row must never populate it or it permanently covers the desktop/game capture; Live's canvas shows
|
||
`DisplaySource` (capture, falling back to its AssetId art). Flavor-blind `IsBackground` lookups are for
|
||
persistence/heal/minis — not this overlay.
|
||
- **Detection (event-driven):** `Win32FullScreenDetector` hooks `SetWinEventHook(EVENT_SYSTEM_FOREGROUND)`
|
||
— fires on the UI thread whenever the foreground window changes. The callback runs the same
|
||
`GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
|
||
`GetMonitorInfo` check; a window is full-screen when its frame covers all four monitor edges; own
|
||
process is excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
|
||
`ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
|
||
`IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
|
||
friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
|
||
submenu + `FullscreenMonitorChanged` event + `StartWatching()`/`StopWatching()`. At launch
|
||
`ReacquireScreenCaptures` heals backgrounds (`HealBackgrounds()`), then keys the Live background to the
|
||
full-screen game's
|
||
monitor; when **no game is detected**, `ResolveAutoCaptureKey()` returns `null` — stale `CaptureKey`
|
||
values are cleared, no capture session is started, and `Source.DisplaySource` falls back to
|
||
`_imageSource` (the static asset on the backdrop element). `StartWatching()` is called at the end of
|
||
`ReacquireScreenCaptures`. `OnFullscreenMonitorChanged` auto-redesignates when a fullscreen game appears
|
||
on a different monitor; null (no fullscreen / own window) is ignored — the existing capture keeps
|
||
running so loading screens / transitions don't break the stream. `RefreshBackdropAutoCapture` (bound to
|
||
"Refresh Capture" context menu + `Activated` handler) reads the current foreground directly as a manual
|
||
fallback. Deduplication: `_lastReportedMonitor` prevents redundant redesignates from rapid foreground
|
||
changes.
|
||
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
|
||
`Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
|
||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
|
||
`WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
|
||
`IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
|
||
flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
|
||
1920×1080 master are downscaled to the master (`DownscaleBgra`, integer 8.8 fixed-point bilinear —
|
||
slice 16: the same two-stage math as `SceneCompositor.Bilinear`; the previous double-per-pixel
|
||
version was ~30-45ms quiet / ~150ms under 240Hz-HDR load and froze the desktop layer ~90% of a take).
|
||
Conversions run as up to MaxConcurrentConversions (3) overlapping readbacks
|
||
(slice 17: the OS readback, not the downscale, is the ~47ms wall — see Slice 17) with a
|
||
monotonic LatestFrame publish gate (`MonotonicGate`: a slow OLDER completion can never
|
||
overwrite a newer frame), and are spaced by a 10ms floor
|
||
(`MinConvertInterval`): the monitor delivers at the **240Hz DWM cadence** (~4.2ms), far too fast for
|
||
the ~60/s the pump can use, so the extra arrivals are dropped (`skip busy`/`skip cadence` telemetry).
|
||
Hand-out buffers come from a **reuse-distance ring** (`FrameRingBuffer`, depth 8, redLine 4): a
|
||
buffer is only rewritten ≥4 rents after its last hand-out, else a fresh one is allocated — a frame a
|
||
consumer still holds (`session.LatestFrame` survives conversions; the dispatcher preview copy lags)
|
||
can never be read-while-overwritten (the 1742 tear). Rolling telemetry (frames/s, conv avg/max ms,
|
||
drops, ring allocs) is logged to `startup.log` every 2s while converting. Per-frame conversion
|
||
failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected content delivers black
|
||
frames (OS limitation, documented). Frame pool pauses while the app is minimized — capture keeps
|
||
running, the pool just stops delivering.
|
||
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
|
||
`WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
|
||
events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
|
||
a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
|
||
`IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
|
||
auto-detection.
|
||
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
|
||
so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
|
||
feature array) → QI `IDXGIDevice` → the WinRT interop export
|
||
`CreateDirect3D11DeviceFromDXGIDevice` → `MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
|
||
process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
|
||
no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
|
||
WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
|
||
`IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
|
||
reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
|
||
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
|
||
task); window capture (`window:<hwnd>`) and multi-monitor live re-targeting beyond the auto-detected
|
||
game are behind the picker; picker-based captures don't survive reload.
|
||
- **Focus-loss capture lag (known OS limit, NOT an in-app throttle — deferred):** when the app window loses
|
||
focus the preview visibly slows (mouse-movement lag). Nothing in the repo checks focus to slow capture —
|
||
the only focus hooks (`FullscreenMonitorChanged` event + `RefreshBackdropAutoCapture`) re-target the backdrop, never
|
||
pace it. The lag is inherited: `Windows.Graphics.Capture` is content-driven and DWM/OS-throttled while the
|
||
app is in the background (worse under a full-screen game on 24H2/26100), GPU readback
|
||
(`CreateCopyFromSurfaceAsync`) contends with the foreground game, the source's one-in-flight `_framePending`
|
||
gate drops frames during stalls, and the manager's `DispatcherPriority.Render` copies only run as fast as
|
||
WPF presents the window. Slice 16 shrank the per-conversion stall (fast downscale + 10ms cadence floor)
|
||
but the OS-level delivery throttle under focus loss remains. Recorded 2026-08-13; no mitigation attempted
|
||
yet (deferred by user decision).
|
||
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
|
||
to the encoder task.
|
||
- **Preview watermark:** the "Preview" placeholder hides while a background capture renders —
|
||
`ShowPreviewPlaceholder` also checks `BackgroundImage` (raised on change), so a scene with
|
||
live capture shows the feed instead of the "nothing here" label.
|
||
|
||
### Webcam capture (TASK 3 milestone 1)
|
||
|
||
- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
|
||
BGRA8) + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces. Tests inject fakes;
|
||
screen capture and background removal later feed the same seam.
|
||
- **CPU-first:** `MediaCaptureInitializationSettings { MemoryPreference = Cpu, StreamingCaptureMode = Video,
|
||
SharingMode = SharedReadOnly }`, frames pulled via
|
||
`CreateFrameReaderAsync(colorSource, MediaEncodingSubtypes.Bgra8)` — the pipeline does any format
|
||
conversion, so every `FrameArrived` yields a ready BGRA8 `SoftwareBitmap` (bytes read via
|
||
`WindowsRuntimeMarshal.TryGetDataUnsafe`, not marshalled copies).
|
||
- **Source pick, not first hit:** the frame reader is bound to the first source that is `VideoPreview`
|
||
(preferred) or `VideoRecord`, not blindly the first preview source. If a camera exposes neither, the
|
||
failure names the device and the stream types it *does* expose. `SharedReadOnly` lets the capture
|
||
coexist with other apps that share the camera.
|
||
- **Known failure: NVIDIA Broadcast** — it opens the physical webcam exclusively, so `InitializeAsync`
|
||
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
|
||
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
|
||
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
|
||
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
|
||
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
|
||
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
|
||
- **One webcam, many scenes (schema v3):** a singleton `Webcam` row holds the identity
|
||
(`Id`/`DeviceId`/`Name`); each scene gets its own `WebcamSceneConfig` (position/size/clip/mirror/
|
||
border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at most once, the webcam
|
||
(`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager` refcounts capture
|
||
sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps it; the last
|
||
release stops + disposes). **"Add Webcam" ALWAYS opens the Windows camera picker** — the creator is
|
||
never silently handed the previous camera (which used to happen after deleting one scene's webcam
|
||
while another scene still used it; that identity survived, so re-adding bypassed the choice).
|
||
Picking a different camera than the current app-wide one swaps it everywhere via
|
||
`SwapWebcamIdentityAsync` (the same path "Change Webcam…" uses), keeping the singleton honest;
|
||
picking the same one just places the config. The Add Webcam menu item greys out whenever a webcam
|
||
exists **anywhere** — one camera identity app-wide (`CanAddWebcam` = `StagedScene != null && _webcam == null`,
|
||
TASK 26; the old per-staged-scene gate let a second picker run from a scene that lacked the config).
|
||
**Removing the last webcam config anywhere clears the identity** (`_webcam = null`, raising
|
||
`CanAddWebcam`/`CanChangeWebcam`), which also drops "Change Webcam…" and re-enables Add.
|
||
The **YouTube Chat layer follows the same one-per-layout rule** (TASK 27):
|
||
`CanAddYouTubeChat` greys out the (+) item while any scene carries a ChatBox source (raised on
|
||
staging + elements change; `AddSource` refuses a second as defense in depth). Chat layers created
|
||
before the "YouTube Chat" default still carried the legacy name "Chat" (commit `65641d8` era) —
|
||
LoadLayout heals exactly that un-renamed default to "YouTube Chat"; creator renames are untouched.
|
||
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
|
||
snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
|
||
restores them when switching back — otherwise the Round resize lock (square) would leave a square
|
||
behind. The rect dims are **persisted** (`WebcamSceneConfig.RectWidth`/`RectHeight`, nullable), so a
|
||
reloaded Round webcam still restores its pre-Round aspect instead of staying square. A one-time
|
||
`HealLegacySquareRect` (load only) widens a pre-v4 `Traditional` config that ended up square to 16:9
|
||
(keeps height; Round and explicit rect dims are untouched).
|
||
- **Device swap / layout reload:** `ChangeWebcamAsync` (picker) and `ReacquireWebcam` (after load)
|
||
release the old device with `ReleaseAllAsync` — a forced full drop that zeroes the refcount and
|
||
stops the source regardless of how many scenes held it (the per-config count isn't known once the
|
||
scenes are replaced) — then `AcquireAsync` the new device once per config.
|
||
- **Shared bitmap, coalesced updates:** one `WriteableBitmap` per active camera, created on the UI thread
|
||
at the device's frame size (first frame), forwarded to every `WebcamSceneConfig.VideoImageSource` via
|
||
`PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
|
||
(at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
|
||
device never drowns the render thread.
|
||
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
|
||
frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
|
||
exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
|
||
webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
|
||
current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
|
||
`AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
|
||
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
|
||
(`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
|
||
0–1, `BorderWidth` 0–20, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
|
||
Electricity|Sparkles`). Rendered in the preview DataTemplate; toggled from the element's right-click
|
||
context menu (webcam menu: Change Webcam…, Border Effect submenu — all 9 items enabled, values persist,
|
||
rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
|
||
this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
|
||
preview canvas has its own Show Webcam entry.
|
||
- The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
|
||
(`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
|
||
renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
|
||
the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
|
||
geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
|
||
frame on the render thread, which is what made the live webcam crawl while round. The Round border is
|
||
a centered `Ellipse` at `Width/Height = RoundBorderSize`.
|
||
- Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
|
||
- **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
|
||
max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
|
||
**by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
|
||
may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
|
||
(`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
|
||
`RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
|
||
- **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
|
||
the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected
|
||
the element — making the corner handle ungrabbable. The element Grid carries
|
||
`Background="Transparent"` (whole rect draggable; the empty canvas Grid uses the same trick for
|
||
right-click Show Webcam) and the `SelectionOverlay` (dashed border + corner dot) is
|
||
`IsHitTestVisible="False"` so it never intercepts the click. Two things make the webcam menu work:
|
||
(1) the `ContextMenu` pins its own `DataContext` to `PlacementTarget.DataContext` — a `ContextMenu`
|
||
isn't in the visual tree, so without it the Click-handler `DataContext:` patterns (and the
|
||
IsChecked/slider bindings) silently fail; (2) `Themes/Controls.xaml` ships a full dark `MenuItem`
|
||
template — `PART_Popup` (submenu popups), a popup `ItemsPresenter` (the Border Opacity/Thickness
|
||
sliders live in Items, so they render in a hover flyout), a `✓` checkmark column, and a `›` arrow
|
||
driven by `HasItems`. An earlier bare `Border + Header` template dropped all three: submenus never
|
||
opened, sliders never rendered, checkmarks never showed — the menu looked dead even though the
|
||
Click handlers were fine.
|
||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||
to the encoder task.
|
||
- **Background removal = milestone 2** — ONNX Runtime + DirectML (CPU fallback), MediaPipe Selfie
|
||
Segmentation (Apache-2.0), wired into the same `VideoFrame` seam. Not part of milestone 1.
|
||
|
||
### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)
|
||
|
||
The encoder needs the master 1920×1080 frame **without** the preview's editing chrome (SelectionOverlay,
|
||
DimRects, output-rect outline, badge, placeholder). WPF's `RenderTargetBitmap` is software-rendered and
|
||
captures the visual tree *including* chrome, so the preview can't be captured — the output is a **second,
|
||
parallel software compositor** over the `VideoFrame` (BGRA8) seam, and the XAML preview
|
||
(`MainWindow.xaml` CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two
|
||
renderers must agree: geometry, `UniformToFill` cover-crop, round clip, mirror, border, z-order. Preview
|
||
stays XAML (editing view); the compositor is the output view.
|
||
|
||
- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H,
|
||
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the
|
||
vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920.
|
||
- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame
|
||
anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers
|
||
pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition;
|
||
if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one **behind
|
||
the same seam** — the CPU master buffer stays the contract.
|
||
- **Branding flash is composited by the output path too** (it's on the live output, per Monetization),
|
||
passed in as a pre-rendered `VideoFrame?` — the compositor core stays pure byte-math, no WPF. Likely a
|
||
bundled asset rather than runtime text rendering (deterministic, no font/layout risk).
|
||
- **Frame sources are injected** via a `Func<SceneElement, VideoFrame?>` resolver
|
||
(`SceneCompositor.Render(scene, frameFor, flashFrame, options)`) — the caller maps each element to
|
||
its frame (webcam → `DeviceId`, image → `AssetId` via `StaticPixelCache`, backdrop → `CaptureKey`),
|
||
so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that
|
||
resolver in the encoder step, not the compositor step. The master buffer (the compositor's return
|
||
value) is the seam a future D3D11 compositor would honor identically.
|
||
|
||
### FFmpeg locator (TASK 4 ship step 2 — shipped 2026-08-10, plan in TASKS.md)
|
||
|
||
The encoder's one external dependency is `ffmpeg.exe`; it's never shipped in the repo. `IFfmpegLocator`
|
||
resolves an absolute path on demand: **PATH probe first** (the user's own install wins — their choice,
|
||
their responsibility), then the cache (`%APPDATA%\ytLlive\tools\ffmpeg.exe`), then a **pinned** BtbN
|
||
LGPL-**shared** win64 zip (~75 MB) from which `ffmpeg.exe` **and the `libav*.dll` family** are extracted
|
||
(staged temp-write + move so a crash never corrupts the cache; Windows resolves the DLLs from the exe's
|
||
own directory). BtbN LGPL-shared (not gyan.dev, not static): it drops GPL-only libx264/x265 while keeping
|
||
NVENC/QSV/AMF + libopenh264 + native AAC, and dynamic linking means LGPL compliance is "license text +
|
||
source offer" with no static-relink (§6) material — see the Licensing guardrails below. The pin is a
|
||
dated autobuild tag (immutable); BtbN retention keeps the last 14 daily + each month-end for 2 years, so
|
||
a cold cache can outlive the pin → the seam throws a clear, logged error (recoverable; the pin is one
|
||
const). Constructor-injected search dirs / tools dir / downloader (`Func<string, CancellationToken,
|
||
Task<byte[]>>`) keep it hermetic: tests fake the network with a real in-memory zip. Constructed in the
|
||
encoder step (not yet — this PR ships the seam + impl + tests only).
|
||
|
||
### Live encoder + RTMP push (TASK 4 ship step 3 — shipped 2026-08-12, plan in TASKS.md)
|
||
|
||
The encoder is a **thin orchestrator over `ffmpeg.exe`** — no H.264/AAC code in the app. It spawns the
|
||
subprocess (path from `IFfmpegLocator`), feeds raw BGRA master frames into stdin, and parses `-stats`
|
||
stderr lines into `StreamHealth` (bitrate/FPS/duration, dropped-from-frame-count). `FfmpegEncoder`
|
||
(`IFfmpegEncoder` seam) holds: `StartAsync` (locate → probe `-encoders` → spawn → stderr loop →
|
||
drain loop), `SubmitFrameAsync` (slice 10: bounded-queue ENQUEUE — never a pipe write; the drain task
|
||
owns stdin writes), `StopAsync` (flush queue → stdin EOF → ffmpeg
|
||
finalizes + exits by itself; a 10s watchdog kills it), `Dispose` (force-kill + wait), and the
|
||
`HealthUpdated`/`ProcessFailed` events. **Pattern:** the encoder never touches `Process` — it drives the
|
||
`IEncoderProcess` seam (`FfmpegEncoderProcess` wraps the real `Process`, redirected stdin/stdout/stderr
|
||
+ exit control); a `Func<IEncoderProcess>` factory + the locator are constructor-injected, so the
|
||
integration test fakes the whole subprocess (probe + encoder) with a Channel-backed `TextReader` whose
|
||
`Complete()` is EOF (`null`), never a `ChannelClosedException`.
|
||
|
||
**Decisions (locked):** args are pure (`FfmpegArgs.Build`, no string building in the encoder):
|
||
`-f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS -i pipe:0` (**NO `-re`** — the FramePump
|
||
is the pacer since slice 9, 2026-09-10; `-re` added a second, fighting clock on the rawvideo demux)
|
||
+ a **real audio input — the
|
||
mixer writes IEEE-float stereo to a Windows named pipe** (`-f f32le -ar 48000 -ac 2
|
||
-i \\.\pipe\ytllive_audio`, name via `EncoderOptions.AudioPipeName`; replaced the old `-f lavfi -i
|
||
anullsrc` silence in the TASK 8 audio milestone) + explicit `-map 0:v -map 1:a` + `-c:v <enc> -b:v K
|
||
-maxrate K -bufsize 2K` + **`-g fps×4 -keyint_min fps×4 -sc_threshold 0 -bf 0 -pix_fmt yuv420p`**
|
||
(≤4s keyframes, closed GOP, H.264 compliance) + `-c:a aac -ar 48000 -ac 2 -f flv <rtmpUrl>`.
|
||
**Record output (TASK 18):** `FfmpegArgs.Build` emits one self-contained block per output, each its own
|
||
`-map 0:v -map 1:a` + codec tags (`AddVideoTags` helper). Stream block = `-f flv <rtmpUrl>`; record block =
|
||
`-f mp4 <RecordPath>`. `EncoderOptions.StreamEnabled`/`RecordEnabled`/`RecordPath` gate each block, so
|
||
the engine runs record-only (no RTMP) or stream-only — **stream+record simultaneously is OUT by creator
|
||
ruling (2026-09-01): the VOD is already the copy, and dual-encoding drags mid-range chassis and degrades
|
||
BOTH outputs ("we're not them"). The old "cheap on NVENC" claim was an unverified assumption; the
|
||
UI constraint (pills = radios) lands as a small change, the dual-block machinery stays generic.**
|
||
`FfmpegEncoder.StartAsync` throws unless at least one output is enabled.
|
||
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
|
||
NVENC → QSV → AMF → OpenH264 fallback, **never libx264** (GPL; see Licensing). `EncoderOptions.VideoEncoder`
|
||
forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate from the quality tier and
|
||
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
|
||
driven by the `FramePump` below.
|
||
|
||
### Local recording (TASK 18 — shipped 2026-08-29, VM + top-bar UX; plan in TASKS.md)
|
||
|
||
Recording is **independent from streaming** and needs no YouTube sign-in. Two pill toggles in the top bar
|
||
declare intent: **REC pill** (local file, works signed-out) and **ON-AIR pill** (streaming; greyed/disabled
|
||
until `IsConnected`). The pill is intent; `IsRecording`/`IsLive` are reality — the REC status dot only turns
|
||
green when a session is actually recording, the ON-AIR dot when actually live.
|
||
|
||
- **State model (`MainViewModel`):** pills `RecordPillOn`/`OnAirPillOn` (ON-AIR setter no-ops if
|
||
`!CanToggleOnAir`), `ShowPrimaryStartButton` = **always the idle face** (`IsOffline && !IsRecording`,
|
||
2026-09-01 — the old connected/record-only gate blanked the bar after stopping signed-out),
|
||
`ShowEndStreamButton`, `CanStartSession`, `AccountStatusLightToolTip`. Sign-in is a **context-menu
|
||
item on Start** ("Sign in to YouTube" → `SignInCommand`, visible while disconnected) — the standalone
|
||
Sign In button is gone (TASK 30's single-button rule, completed). `StartSession()` routes: ON-AIR on
|
||
→ GoLive dialog then stream (± record); REC on or **nothing armed** → `BeginRecordOnly()` (unarmed
|
||
Start lights the REC pill and records — no dead-end no-ops; local recording needs no account).
|
||
`IsEditMode` also requires `!IsRecording` (lock scrubbing while recording).
|
||
**Top bar order (creator spec 2026-09-01):** `[sign light] REC [pill] [sign light] ON-AIR [pill]`
|
||
— each reality lamp sits before its own intent switch. **Stop ends everything:** `StopStream`
|
||
clears both pills (see "Stop ends everything; failures roll back" below).
|
||
- **Filenames** (pure `Services/RecordingFile`): auto-name `ty-<yyyyMMdd>-<HHmm start>-0000.mp4` at start;
|
||
**rename-on-stop** to `ty-…-<hh2mm2 actual length>.mp4` (`FinalizeRecordingAsync` after the pump stops &
|
||
the file closes), numeric `-2`/`-3` suffix on collision (`UniquePath`). Length comes from `_liveElapsed`,
|
||
which the session timer walks while `IsLive || IsRecording`.
|
||
- **Folder:** default `%APPDATA%\ytLlive\recordings\`, user-overridable via `ChooseRecordFolderCommand`
|
||
(`OpenFolderDialog`), persisted through `LayoutStore.Load/SaveRecordFolder` (`RecordFolder` key).
|
||
- **Explicit sign-out only:** `StopStream` no longer clears the session/token. Sign out via Logout /
|
||
Change Account. Stopping a recording leaves the creator signed in.
|
||
- **Stop ends everything; failures roll back (2026-09-01):** `StopStream` also clears both intent
|
||
pills (`RecordPillOn`/`OnAirPillOn`) — a lit pill with no session behind it is a lie. Every
|
||
frame-pump death and every go-live prep failure now runs the full `StopStream` teardown instead of
|
||
leaving a limbo (`StreamStatus.Error` with `IsRecording=true` over a dead encoder — the first-launch
|
||
zombie recording). Toast copy distinguishes "Recording stopped" from "stream pipeline stopped";
|
||
`AudioMixer.StopLive` is idempotent-safe for rollbacks that never reached StartLive. Seams:
|
||
`OnFramePumpFailed` + `IsRecording` setter made internal (test-only, InternalsVisibleTo).
|
||
Test: `SessionTeardownTests`.
|
||
- `EncoderOptions.StreamEnabled/RecordEnabled/RecordPath` gate outputs; record+stream is one ffmpeg with two
|
||
output blocks. Rest of the engine (FramePump/encoder) is unchanged — `BuildEncoderOptions` now always
|
||
returns an options (with flags from the pills + `_activeRecordPath`), so record-only reaches the pump.
|
||
|
||
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12; game audio bar 2026-08-13; **TASK 8 audio milestone: real stream audio + filters + duck + TRAX, shipped 2026-08-14**, plan in TASKS.md)
|
||
|
||
**Capture runs for the app's lifetime and is KISS by rule**: desktop/game audio is automatic (WASAPI
|
||
loopback), the mic is the creator's only audio control (meter/mute/volume already shipped). The whole
|
||
layer sits behind an **`IAudioSource` seam** (`Services/Audio/`: `Start`/`Stop`/`Started`/`SampleReady`/
|
||
`Failed`, IDisposable) so the app never touches NAudio directly and tests inject hermetic fakes (no
|
||
devices, no timers).
|
||
|
||
- **Sources (NAudio 2.2.1 — `NAudio.Wasapi` + `NAudio.WinMM` (for `WaveOutEvent`), MIT — item 9 in
|
||
`THIRD-PARTY-NOTICES.txt`):** `WasapiLoopbackAudioSource` = `WasapiLoopbackCapture` on the default
|
||
render device; `WasapiMicAudioSource` = `WasapiCapture` with the device resolved by `FriendlyName`
|
||
matching `MicSourceName` (the app only persists the **DisplayName**), falling back to the default
|
||
capture endpoint. Both run at the **device's own mix format** — NAudio 2.2.1 exposes no overridable
|
||
`GetDefaultMixFormat` on `WasapiCapture`, so there is no forced 48 kHz path; the mixer's
|
||
`TinyResampler` normalizes any rate/channel layout to 48 kHz stereo (this replaced the earlier
|
||
"force IEEE-float 48k" plan — the resampler IS the design). Mic device resolution re-reads the name
|
||
provider `Func<string?>` at each `Start`, so a mic picked mid-session takes effect **immediately**
|
||
(the mixer restarts the mic on pick). Both sources raise `Started` once their capture loop actually
|
||
begins — the mixer turns that into `MicConnected`.
|
||
- **`AudioMixer`** owns both sources; **`StartMicCaptureAsync` starts the mixer once at startup** and
|
||
`Shutdown` disposes it — NOT go-live — so the meters preview live. Mic samples: downmix to mono →
|
||
`TinyResampler` → **`VoiceFilterChain` per sample** (bass `LowShelfFilter` 120 Hz +4 dB → treble
|
||
`HighShelfFilter` 8 kHz +3 dB → `NoiseGate` (threshold 0.005, hysteresis 0.5, attack 0.5, release
|
||
0.0005) → `Compressor` (threshold 0.5, ratio 4:1); all pure TDF2 DSP) → ring buffer → the
|
||
**post-filter** `AudioLevelMeter` (RMS, 0.2 exponential smoothing) → `MicLevelChanged` →
|
||
marshalled to the UI thread → `AudioLevel`. Loopback samples: resample → ring buffer → second meter
|
||
→ `LoopbackLevelChanged` → the game bar's `GameAudioLevel`. The raw linear level maps to the meter's
|
||
display scale by `AudioLevelMeter.ToDisplay` (−60..0 dBFS → 0..1 with **+10 dB input amplification**).
|
||
**Mic connection state surfaces as events:** `MicConnected` (source `Started`), `MicFailed` (source
|
||
`Failed`), and `RestartMic()` re-resolves + restarts just the mic (loopback keeps running). Failures
|
||
log via `AppLog`; a mic failure zeroes the meter, a loopback failure never kills the mic. Meter `Push`
|
||
is unconditional (a `?.` on the event would skip the meter update when nothing is subscribed yet).
|
||
- **Go-live audio (TASK 8):** `BeginGoLive` → `StartLive(pipeName)` (idempotent) creates the named pipe
|
||
server (`NamedPipeAudioWriter`, behind the `IAudioPipeWriter` seam) and a live loop task; every
|
||
`mixInterval` (default 10 ms) `FillAndMix` reads a chunk from each ring buffer (silence-fills
|
||
underruns), computes the post-filter mic RMS → `AutoDucker` (threshold 0.02, duck ×0.25 / −12 dB,
|
||
attack 0.05, release 0.005, always on), applies the **honest gains** via `Func<double>` seams
|
||
(`micGain = MicMuted ? 0 : MicVolume`; `loopbackGain = GameMuted ? 0 : GameAudioVolume`, × duck),
|
||
mixes mono mic + stereo loopback into interleaved stereo, runs the
|
||
**`MasterLimiter`** (a **−1 dBFS ceiling** so the honest gains can never sum past 0 dBFS and clip the
|
||
AAC encode — instant attack per frame, smoothed release toward unity), and writes the floats to the pipe. **The gains are wired through `AudioGainProvider`**
|
||
(`Services/Audio/AudioGainProvider.cs`) — the VM hands its `MicGain`/`LoopbackGain` Funcs to the
|
||
mixer at construction, read live each tick, so the volume sliders and mute buttons are stream-honest.
|
||
(Before this wiring the seams defaulted to unity and the controls were decorative.) `StopStream` →
|
||
`StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE** stopping the frame pump (video EOF) —
|
||
the reverse order stalls on pipe backpressure. `FfmpegEncoder` itself is untouched; the VM owns the
|
||
pipe lifecycle.
|
||
- **Audio sync offset (TASK 22):** after the limiter, `FillAndMix` routes the whole interleaved stereo
|
||
mix through a **`AudioSyncDelay`** (`Services/Audio/AudioSyncDelay.cs`) — a pure delay line whose
|
||
offset comes from a `Func<int> syncOffsetMs` seam (the VM's global `AudioSyncOffsetMs`, **−500..+500
|
||
ms**, persisted as `Audio.SyncOffsetMs`). This is OBS's documented fix for lip-sync: **positive**
|
||
delays audio when it runs ahead of video; **negative** advances audio when it runs behind by eating
|
||
the first `|N|` ms of the live stream head (see `AudioMixer.StartLive`; the delay line stays
|
||
clamped 0..500 internally — negative bypasses it entirely). The advance is armed once at `StartLive`
|
||
(eating the head mid-stream is impossible); positive is live-reactive (re-read per tick). A slider
|
||
on the mic bar (`"AUDIO SYNC"`, locked while live/recording via `IsEditMode`) + a status dot
|
||
(`IntToSyncBrushConverter`) surface it. Changing the delay flushes the line (a live change clicks
|
||
rather than smears).
|
||
- **TRAX — free background music (TASK 8):** `MusicPlayer` = NAudio `MediaFoundationReader`
|
||
(mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** bed (no slider) → `WaveOutEvent` on
|
||
the default device, **looping on any clean natural end** (`PlaybackStopped` with `e.Exception == null`
|
||
→ rewind + replay; the earlier `Position >= Length` check was unreliable for `MediaFoundationReader`
|
||
and let tracks play once then stop). The desktop/game bar's mute toggles `LocalGain` (0 or 1;
|
||
`UpdateTraxLocalGain` runs on every `GameMuted` change and on TRAX load), so muting kills TRAX
|
||
in the headphones — matching the stream, where it rides the loopback channel. The volume slider
|
||
drives system volume (`PushSystemVolume`) — which scales the HEADPHONES; the loopback capture is
|
||
pre-endpoint-volume (disproven-then-fixed 2026-09-01), so the STREAM follows the knob via
|
||
`AudioGainProvider.LoopbackGain = GameMuted ? 0 : GameAudioVolume`, and the game meter multiplies
|
||
its display by the same. One knob, three honest paths. No third mixer input. The footer
|
||
**TRAX** button lives **inside the game audio bar** (in the preview overlay), **left of the
|
||
"Desktop Audio" label** (the creator's pick — it rides the desktop channel, so the music control
|
||
sits with the desktop audio controls): status dot (**red** no track / **yellow** loaded stopped /
|
||
**green** playing) + "TRAX"; **left-click** toggles play/pause (opens the in-app picker via `OpenFileDialog` when no track is loaded); **right-click** always opens
|
||
the picker (`TraxButton_PreviewMouseRightButtonUp`, `e.Handled = true`, code-behind pattern); tooltip
|
||
shows the loaded/playing track name or "No track — right-click to choose background music". The picked
|
||
track persists via **schema v9** single-row `Music` (`TrackPath`/`IsEnabled`) in `LayoutStore`. Known
|
||
wrinkle (out of scope, future feature): YouTube mutes VODs carrying copyrighted music — "music on
|
||
live, off VOD" is tabled.
|
||
- **Mic status dot (`Models/MicStatus.cs`)** on the preview-bottom MIC button: green = `MicConnected`,
|
||
yellow = `MicFailed` (in use/unplugged), red = no mic device at startup (the mixer is never started,
|
||
so loopback and the game bar can't run either — no capture devices at all). **The picked mic persists:
|
||
`LayoutStore`'s `Settings` key/value table stores `MicSourceName` (saved on pick via
|
||
`SaveMicSourceName`), and the VM restores it at construction BEFORE the mixer's first `Start` — so a
|
||
restart reconnects the same already-vetted device (green) or reports it missing (yellow), instead of
|
||
silently falling back to the default endpoint.**
|
||
- **Game audio bar** (desktop/game, **always visible** — TASK 4's `IGameAudioDetector` show/hide gating was **removed 2026-08-15**: the bar used to appear only while a full-screen game with sound was up, which kept hiding the creator's desktop meter; the detector stack is gone, desktop audio is just automatic WASAPI loopback): overlaid **at the bottom of the preview window** (bottom-center, dark translucent
|
||
chip, a mirror of the mic bar: meter + mute + volume slider). It's monitoring UI **in the preview
|
||
grid** — the meter is display-only (fill = `ToDisplay(GameAudioLevel) × GameAudioVolume` — the ×volume
|
||
term was missing until 2026-09-01: loopback capture does NOT shrink with the endpoint, so an
|
||
unscaled meter left it pegged at 20% volume; `GameMeterHonestyTests` guards it), but its **volume
|
||
slider drives system volume** (`PushSystemVolume` — live headphones follow the knob; the capture
|
||
side is scaled in the mix, see TRAX note above) **and mute kills TRAX** (`MusicPlayer.LocalGain` = 0). Relabelled
|
||
**"Desktop Audio"** (TASK 8) since TRAX rides the same channel.
|
||
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
|
||
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
|
||
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
|
||
- Build **0 warnings**; **197 passing** (DSP/ring-buffer/ducker/resampler/pipe + mixer/hysteresis/
|
||
game-detector/meter-scale unit tests + the TASK 8 integration test
|
||
`AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe` + the polish-batch integration
|
||
test `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` proving the provider gains reach the
|
||
pipe and mute silences the loopback).
|
||
|
||
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, health stats 2026-08-13, plan in TASKS.md)
|
||
|
||
The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active
|
||
scene each tick, resolves every element to its latest frame, composites it into the tier's output frame,
|
||
and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns
|
||
pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps)
|
||
ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs
|
||
`FramePump stats: n/target frames per 5s, avg render Xms (resolve R), avg submit Yms` so a slow stage names itself — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped
|
||
into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup
|
||
("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are
|
||
readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected
|
||
seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
|
||
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, and an injectable pacing delay
|
||
(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers.
|
||
|
||
- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log
|
||
+ surface via the `Failed` event. `BuildEncoderOptions` now always returns options (TASK 18) with
|
||
`StreamEnabled`/`RecordEnabled`/`RecordPath` baked from the pills + `_activeRecordPath`; the pump skips the
|
||
encoder only when `options == null`, i.e. "no output configured". `MainViewModel._rtmpUrlProvider` is that
|
||
seam — a `Func<string?>` that now yields the reusable stream's ingest URL (TASK 9, shipped 2026-08-16):
|
||
loaded from the `LayoutStore` cache at startup and set fresh by `PrepareAndStartLiveAsync` before the pump
|
||
starts. **The pump reads the URL once at startup**, which is why go-live ensures the stream BEFORE
|
||
`StartAsync`.
|
||
- **Deadline pacing + row-blit render (take-3 starvation fix, 2026-09-03):** take 3 recorded 30s into a
|
||
2.1s file and the 5s stats line named the cause in one number — `17/300 frames per 5s, avg render
|
||
258.1ms, avg submit 1.5ms`. Two defects, both fixed: (1) the pump slept the FULL interval after each
|
||
render, so period = render + interval — OBS's `video_thread` (libobs/media-io/video-io.c) pattern
|
||
replaces it: absolute `nextTick += intervalTicks` deadline, sleep only the remainder, and on overrun
|
||
skip the wait. (NOTE, slice 9: the original "skip the missed ticks (rebase)" half of this fix was
|
||
wrong — see the slice 9 correction below.) (2) the compositor did
|
||
per-pixel float sampling + `Math.Round` blending over all 2.07M master pixels (backdrop) and scanned
|
||
the whole destination per overlay (a 64px social-bar strip cost 2M iterations). `SceneCompositor` now
|
||
follows the libyuv pattern (BSD-3, chromium.googlesource.com/libyuv/libyuv — cited per the
|
||
derivative-work rule): 1:1 aligned blits take a row-walk fast path (bilinear at scale 1 + integer
|
||
offset is the identity) with per-pixel alpha branch and integer fixed-point blend; `BlitOverlay`
|
||
clips to the intersection rect; a full-cover live backdrop skips the opaque-black pre-fill. The
|
||
general per-pixel path handled scaled/round/mirrored elements. Test:
|
||
`Pump_Paces_To_The_Deadline_Compensating_Render_Cost` records the requested wait
|
||
via the pacing seam (a seam fake must genuinely await — a synchronous completed task runs the whole
|
||
pump loop on `StartAsync`'s continuation and hangs the run; MyMistakes).
|
||
- **Slice 2 of the render fix (2026-09-04, take 4):** the pacing held (file no longer truncated at
|
||
loop level) but render stayed at 58.9ms — the per-pixel row walk was still 2M managed iterations and
|
||
every tick allocated a fresh 8.3MB master buffer. Three changes: (1) **`VideoFrame.IsOpaque`** — a
|
||
producer-contract flag (only the screen-capture and webcam paths set it; DWM/MediaCapture fill alpha
|
||
255 by contract); a full-canvas, aligned, opacity-1 blit of an opaque frame is now ONE
|
||
`Buffer.BlockCopy` (~1.5ms) instead of the loop, and the black pre-fill is skipped when the backdrop
|
||
covers. (2) **integer fixed-point bilinear** in the general `BlitContent` path (row-hoisted invariants,
|
||
no divisions, no `Math.Round`) — within ±1 of the float reference, inside the ±2 test tolerance.
|
||
(3) **scratch pool in the pump** — `AcquireScratch`/`ReleaseScratch` recycle the master buffer
|
||
(max 4, keyed by length, owned-by-reference so bake-cache/social-bar/static-art arrays can never be
|
||
captured); release happens strictly AFTER `SubmitFrameAsync` returns (the write to stdin copies),
|
||
and the free-list Contains guard makes the transition Cut path safe (`BlendFrame` returns the
|
||
to-frame itself, aliasing the scratch). The per-tick `fromScene` render in the transition branch was
|
||
**dead weight** (BlendFrame uses `TransitionService.FromFrame` captured at `Start`, never the pump's) —
|
||
removed, and the pump's `fromSceneProvider` seam + `MainViewModel` call site went with it. Tests:
|
||
`Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels` (the ONE: alternating backdrop colors
|
||
pin every frame's content, repeated backing-array identity proves the pool recycles),
|
||
`Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch` (memcpy branch + scratch
|
||
sentinel). `FakeEncoder.SubmitFrameAsync` now snapshots bytes like the real stdin write — holding the
|
||
reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`; the
|
||
vertical tier's final 1080×1920 `BilinearScale` still allocates fresh per frame (same GC lesson when
|
||
someone streams vertical — recorded as a follow-up, not silently "done").
|
||
- **Slice 3 — chat raster-on-change (2026-09-04, take 5):** take 5 measured `138/300, avg render
|
||
25.5ms` — the per-tick blits were cheap now, but `ResolveOutputFrame` → `RenderChatBox` ran a FULL
|
||
WPF raster (`ChatBoxRenderer`: FormattedText + `RenderTargetBitmap` + CopyPixels + channel swap)
|
||
EVERY tick whenever the message buffer was non-empty — and the buffer survives between sessions,
|
||
so even a signed-out record-only take paid it. Established answer, same as OBS text sources:
|
||
**raster on change, blit the cache every tick.** `ChatOverlayLayer` now keeps a content version
|
||
(`Messages.CollectionChanged` → `_contentVersion++`) plus a config key (box size + all Chat*
|
||
props); `RenderFrame` returns the cached `VideoFrame` by identity until either changes (the
|
||
compositor only ever reads a cached frame). Test: `ChatOverlayLayerCacheTests` (RealApp, real
|
||
renderer — `Same()` for unchanged inputs, `NotSame()` on message/config change, null on empty).
|
||
Accepted cost pending take 6: one slow tick (~15-25ms) per arriving message; if chat-burst frame
|
||
loss shows up, the next slice moves the re-render off-tick (debounced, dispatcher-side).
|
||
- **Slice 5 — paste cache for non-opaque layers (2026-09-04, take 7/8 data):** the render/resolve
|
||
split (added in the stamp commit) finally named the last thing honestly: `resolve ≈ 0` but
|
||
`render 26-27ms` (124-135/300, ≈2.2x) — the chat fix HAD worked; the remaining cost was the
|
||
compositor re-rasterizing EVERY layer per tick even when its pixels never changed (this creator's
|
||
Live scene: chat 159k + web widget 271k + image 95k + webcam 156k px ≈ 680k samples @ ~38ns each).
|
||
OBS's actual shape: sources cache their surface, the compositor pastes. `SceneCompositor` now
|
||
routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space,
|
||
TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round,
|
||
mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at
|
||
paste). **Correction (2026-09-12, the web-widget "black box"):** the raster builds on a
|
||
TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it
|
||
premultiplied the color into RGB and forced `alpha=255`. A translucent widget pixel then pasted
|
||
as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct.
|
||
`BlitContentRaw` now takes `transparentDst` (the raster call passes `true` and writes straight
|
||
color + straight alpha; the paste rows do the source-over). Master paths are bit-identical.
|
||
Guard: `PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk`. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale
|
||
content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device
|
||
frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path.
|
||
Position/opacity drags are now near-free (no resample — paste params, not cache keys).
|
||
ONE integration test: `PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates`
|
||
(raster-vs-paste byte equality incl. round-clip margins + new-array propagation); the whole
|
||
existing pixel suite guards sampler semantics. Take 9 must show `≈300/300, avg render ≤ ~8ms`;
|
||
if it lands there, the recording saga closes. Follow-ups unchanged: vertical-tier scale alloc,
|
||
debounced chat re-render under message bursts.
|
||
- **Slice 6 — the sleep quantum WAS the ceiling (take 9, 2026-09-04):** period measured ~37ms while
|
||
work (render+submit) was ~25 — the missing ~12ms is `Task.Delay` rounding EVERY request up to the
|
||
Windows clock tick (documented ~15.6ms default; learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.delay).
|
||
A frame finishing 3ms early requests a 3ms wait and sleeps a full 15.6 — capping the producer at
|
||
~27fps NO MATTER how fast the compositor ran. This is why takes 7→9 showed zero playback change
|
||
despite real render wins: the sleep floor dominated everything above it. Established fix (game-loop
|
||
canon — stackoverflow.com/questions/5441464; timeBeginPeriod — learn.microsoft.com/en-us/windows/win32/api/timeapi/nf-timeapi-timebeginperiod):
|
||
`timeBeginPeriod(1)` for the pump's lifetime (paired `timeEndPeriod` in the finally), sleep only the
|
||
BULK of the remainder (request minus 2ms), `Thread.SpinWait` the last ~2ms across the deadline;
|
||
blown deadlines still rebase. (SUPERSEDED in part: slice 9 deleted the rebase — deadlines only
|
||
advance; slice 10 replaced the burst re-write with one fresh composite per iteration.) Stats gained
|
||
`avg wait Xms` so render+submit+wait must ≈ period —
|
||
the accounting is closed, no stage can hide again. Same slice: the webcam's `IsOpaque` paste-cache
|
||
bypass was removed (it re-sampled ~156k px every tick even between identical device frames; the
|
||
cached paste beats the sampler on hits and costs the same on misses). Take 10 verdict: `≈300/300`
|
||
honest fps — if short, the wait/render split names the remaining term with no ambiguity left.
|
||
- **Slice 7 — the loop was ON the UI thread the whole time (take-8 numbers, 2026-09-04):** with
|
||
the wait finally measured, take 8 (59a02a5b) showed the impossible pair — `render 22ms, wait 10ms`
|
||
against a 16.7ms deadline: a blown deadline rebases with NO wait. The wait was the producer
|
||
**queued behind the live preview on the dispatcher**: `PumpAsync` starts from a UI command handler
|
||
and its `await` continuations inherit the UI `SynchronizationContext`, so the "WPF-free" compositor
|
||
actually rendered on the UI thread and ran only when WPF let it. OBS runs its media loops on
|
||
dedicated threads for exactly this reason. Fix: `_pumpTask = Task.Run(() => PumpAsync(...))`
|
||
(no sync context inside → continuations stay on the pool). Side effects handled, not ignored:
|
||
`StaticPixelCache.Get` now locks (pool miss-decodes race UI callers); `ChatOverlayLayer.RenderFrame`
|
||
checks its cache off-thread but MARSHALS the rare raster miss to the dispatcher (DrawingVisual/
|
||
RenderTargetBitmap are UI-thread objects) and re-validates there; pump events already marshalled.
|
||
Also: `GCLatencyMode.SustainedLowLatency` for the pump's life, stats gained `worst render Xms`
|
||
(spike visibility — bimodal averages hid them), webcam now routes through the paste cache (bypass
|
||
re-sampled 156k px even between identical device frames). Tests: `Pump_Produces_OffTheStartingContext`
|
||
(the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) +
|
||
70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); `worst render` names any
|
||
remaining raster-miss spikes; render+submit+wait still == period — the accounting holds.
|
||
- **Slice 8 — capture buffer ring + paste-cache Epoch (take 11, 2026-09-04):** take 11 validated
|
||
the architecture — typical frames now land `work ~10ms + wait ~6.8ms = 16.7`, exactly the deadline
|
||
(212/300). The whole remaining gap is PERIODIC 35-65ms worst-render spikes that worsened across
|
||
the take (189→147 frames/5s) — the signature of gen2 GC pauses, now provable via `gen2 +N` in
|
||
every stats window. Biggest churn was structural: `ScreenCaptureFrameSource` minted a fresh
|
||
~8.3MB `byte[]` per DWM frame (~500MB/s LOH). It now rotates a shared-frame ring (4-deep at slice 8,
|
||
deepened to 8-deep after the "flash" finding — consumer holds must never outlive depth × source
|
||
period), size-matched per slot, with reused downscale row scratch. Recycled
|
||
arrays would poison the paste cache (it keys on array IDENTITY), so `VideoFrame.Epoch` —
|
||
monotonic per producer frame, 0 for fresh-array producers — joins the key. Regression test
|
||
`PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` fails on the old key by
|
||
construction. The camera producer carried the same fresh-array churn (110-220MB/s at 30-60fps):
|
||
it now rotates its own 8-deep ring + Epoch (same identity rule), and the WebView2 capture
|
||
reuses a canvas scratch + 8-deep output ring + a reused WriteableBitmap instead of minting two
|
||
fresh arrays + a new bitmap per 10Hz tick. Next suspect if gen2 stays hot: the WPF preview load
|
||
itself driving gen2 — recorded as follow-up, untouched this slice.
|
||
- **Slice 9 — the "rebase" WAS the recording time-lapse (2026-09-10, the recording playback-timing
|
||
take):** slice 6's "blown deadlines still rebase" was the recording-timing bug all along. In a real
|
||
take the pump emitted `215-219/300 per 5s` (~43fps) while `render+wait == period` still looked closed —
|
||
the rebase ERASED every missed slot (deadline = wall-now again), so the missed ticks never showed in
|
||
the stats. rawvideo has no per-frame timestamps: ffmpeg muxes by frame count at `-framerate 60`, so a
|
||
43fps reality was authored into a 60fps container and **every recording played ~1.4x fast** (takes
|
||
confirmed with a WSL ticker visible in the recording: file duration 11.44s vs ~15.5s wall). Two
|
||
defects, both fixed the OBS way (`libobs/media-io/video-io.c` — the video thread NEVER resets its
|
||
deadline; every interval tick outputs ONE frame, and a late render means repeated content — judder —
|
||
never a skipped timestamp):
|
||
(1) **count-based CFR emission:** `while (GetTimestamp() >= nextTick) { SubmitFrame(frame); nextTick += intervalTicks; statFrames++; }`
|
||
submits exactly one frame per crossed slot, re-writing the current composite when the render overruns
|
||
(duplicated footage = correct duration, not a time-lapse), and `nextTick` is NEVER reset to wall-now;
|
||
(2) **`-re` removed from `FfmpegArgs`** — it was a second, fighting pacer on the rawvideo demux
|
||
(its "Resumed reading … after a lag" grew 0.79s→4.82s across the take, leaving the pump behind its own
|
||
honest-stats count). One fix covers the game background AND the webcam — both flow through this one
|
||
pump. Muxed duration is now frame-count ÷ fps == wall time by construction. Test: the pump suite stays
|
||
green unchanged — the pacing test's `<interval` wait assertion still holds because the frame emits at
|
||
the next slot boundary, not immediately. (Unrelated pre-existing failure found the same day:
|
||
`Composite_FullScene_MasterPixels` pixel (1380,700) cyan-vs-magenta — reproduces with this fix stashed,
|
||
untouched by it, recorded as a follow-up.)
|
||
- **Slice 10 — bounded encoder queue + drop policy + burned-in frame counter (2026-09-10, the
|
||
playback-pacing take):** slice 9 made the DURATION right but the CREATOR still read the recorded
|
||
ticker as "1...23...4...56..." — variable pacing. The aggregates (301/300, uniform PTS, 15.6s wall vs
|
||
15.74s file) could not see it, and the cause was finally measured in `FfmpegEncoder.SubmitFrameAsync`:
|
||
it BLOCKED on `WriteAsync(8.3MB) + FlushAsync` whenever ffmpeg lagged the pipe, and slice 9's
|
||
burst `while` loop then re-wrote that SAME composite for every slot that ticked past — frozen
|
||
content runs (the "smeared ticker"). Fixed the OBS way (the encoder queue in `libobs/obs-encoder.c`:
|
||
the encoder's thread NEVER couples back into the video thread; overload = dropped content, never a
|
||
frozen video thread):
|
||
(1) **`FfmpegEncoder.SubmitFrameAsync` is now an ENQUEUE** into a bounded `Channel<byte[]>` (cap 120 ≈
|
||
2s at 60fps) owned by a dedicated drain task with the stdin write + flush; the caller NEVER blocks on
|
||
the pipe. Pixels are copied into an `ArrayPool` buffer before enqueue (the write now happens later on
|
||
the drain thread, so the pump's scratch reusable the moment submit returns — same contract as before).
|
||
(2) **drop-on-overflow (creator's choice: freshness over coverage):** when the queue is full the NEWEST
|
||
frame is dropped and counted (`IFfmpegEncoder.DroppedFrames`, `Interlocked`); the session never freezes
|
||
or smears — it drops. Stop flushes the whole queue then EOF (`Channel.TryComplete` → drain writes
|
||
leftovers → closes stdin → ffmpeg finalizes+exits), so no accepted frame is ever lost at stop.
|
||
(3) **pump loops ONE submit per iteration** (replacing slice 9's burst): every due slot gets a FRESH
|
||
composite — no catch-up slot ever repeats frozen content; missed slots vanish as a count-based gap.
|
||
The deadline counter is still never reset to wall-now (slice 9 ruling).
|
||
(4) **burned-in frame counter (the new judge):** `_outputIndex` is burned into a 6-digit dot-matrix
|
||
strip (white box + black 5×7 glyphs, bottom-right corner) of EVERY composite before submit. The WSL
|
||
ticker is demoted because its own timers smear under Windows host load; decoding the recording and
|
||
reading the strip is clock-independent: the number advances +1 per frame and jumps by exactly the
|
||
counted drops (queue overflow OR skipped catch-up slots). Stats gained `worst submit Xms`, `dropped N`,
|
||
`stalls K`; a stall logger names any iteration > 2× interval with its render/submit split — with the
|
||
queue, submit is ~1ms, so a stall means render/resolve. The strip sits above the social bar (bar is
|
||
composited, then burned over) — a debug judge, tiny at 1080p. Test:
|
||
`Backpressure_QueueOverflow_DropsFrames_AndNeverBlocks` (the ONE: a 40ms-per-frame sink fake makes
|
||
every enqueue fill the queue; submit must return instantly, drops are counted, stop flushes exactly
|
||
submitted − dropped bytes). Full suite 289/290 (the pre-existing compositor pixel failure unchanged).
|
||
Audio untouched (follow-up). WSL ticker-under-load reliability check still to run (informational — the
|
||
burned counter is the judge).
|
||
- **Slice 11 — web-layer animation ran at ~1/6 speed; capture cadence 10Hz→~30Hz + de-throttle
|
||
(2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2
|
||
capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6×
|
||
under its native rate (repeated-footage slow-mo). Two stacked throttles:
|
||
> Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling
|
||
> design; frame-driven composition capture replaced them — nothing here is current architecture.
|
||
(1) **the 10Hz cap** (unconditional, code-proven) and
|
||
(2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s
|
||
per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused
|
||
or covered — the page is off-screen at (-5000,-5000), so it is a hidden page the moment the host
|
||
window loses occlusion.
|
||
Fixed:
|
||
(1) **`CaptureScheduler` (new, `Services/CaptureScheduler.cs`)** replaces the per-session
|
||
`DispatcherTimer`: a dispatcher timer that DROPS ticks while a capture is in flight (latest-wins,
|
||
never queues) — the in-flight drop is what made raising the cadence safe (concurrent full-HD PNG
|
||
`CapturePreviewAsync` calls would stack ~10-30ms encodes and publish stale-after-fresh). Interval is
|
||
owner-configurable: recording 33ms (~30Hz), idle 200ms. Unit-tested without a WebView2 runtime
|
||
(the ONE test: `CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes` — overlapping
|
||
ticks dropped, capture resumes when idle, driven by a TCS so it is deterministic).
|
||
(2) **de-throttle browser args**: the shared `CoreWebView2Environment` is created with
|
||
`--disable-backgrounding-occluded-windows --disable-renderer-backgrounding
|
||
--disable-features=CalculateNativeWinOcclusion` (the Electron/Streamlabs-class embedder answer for
|
||
occluded-window animation throttling) BEFORE `EnsureCoreWebView2Async`.
|
||
(3) **cadence hook**: `MainViewModel` sets `SetCaptureInterval(33)` on record/stream start and
|
||
`(200)` on stop (`MainViewModel.Streaming.Operations.cs`).
|
||
(4) **capture-cost telemetry**: first 30 captures per session log elapsed ms to startup.log —
|
||
PNG-encode+decode cost decides whether ~30Hz stays or drops to ~20Hz; the FramePump drops frames
|
||
(never time-lapses, slice 10) if UI-thread GC churn starves it, so the cost is observable.
|
||
Full suite 290/291 passing, the sole failure the pre-existing compositor pixel test. The web-overlay
|
||
transparency verification (the first-capture diagnostic bound to about:blank) and the audio-silence
|
||
item remain open push-gate items — both untouched by this slice.
|
||
- **Slice 12 (audio diagnostics, 2026-09-10):** the recorded audio is full-length silent AAC
|
||
(−91dB, 1124 frames / 23.95s in the latest take) — the named-pipe delivered ~9.1MB of zeros to
|
||
ffmpeg for the entire recording, so the loop ran and connected, but both WASAPI capture sources
|
||
delivered nothing. Sources started fine (`Audio: using system default mic...` logged), no failure
|
||
callbacks fired, and the mixer's existing integration test (`Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`)
|
||
proves the loop→pipe math is sound. The fault is capture-side: either the default render endpoint
|
||
carried nothing (audio played on a non-default device — common), or both endpoints were held
|
||
exclusive, or genuinely nothing played. Added permanent per-5s live-loop telemetry to startup.log
|
||
(`Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix`) and
|
||
`NamedPipeAudioWriter.DroppedWrites` — names the exact stage on the next take without new
|
||
code. `AudioMixer.FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)` for the telemetry
|
||
accumulation. Full suite 290/291, same pre-existing sole failure.
|
||
- **Slice 13 (web transparency diagnostic re-point, 2026-09-10):** the "?" in MyMistakes'
|
||
"RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the
|
||
FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted
|
||
for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump
|
||
`%TEMP%\ytLive-web-<id>-w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%])
|
||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||
- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone
|
||
(2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG
|
||
poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real
|
||
cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling:
|
||
the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a
|
||
`CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's
|
||
`webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which
|
||
captures the root `surface_` visual).
|
||
- **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the
|
||
parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was
|
||
`InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted).
|
||
`controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a
|
||
root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor`
|
||
(created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`,
|
||
`ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`,
|
||
`DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`.
|
||
Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) →
|
||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor
|
||
blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame
|
||
`FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to
|
||
2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE
|
||
crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation.
|
||
Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS
|
||
throttling; the capture path itself no longer polls.
|
||
- **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors
|
||
`ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs`
|
||
(owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor
|
||
`(Dispatcher, Func<string, IScreenCaptureSource>?)` so the tests instantiate zero WinRT; public
|
||
ctor `(Dispatcher)`).
|
||
- **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO
|
||
`DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` +
|
||
`FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct
|
||
`DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`,
|
||
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then
|
||
`new Compositor()` — all once on the WPF UI thread (the app dispatches Render there).
|
||
- **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN
|
||
`ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`;
|
||
`CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous
|
||
(`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`.
|
||
- **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three
|
||
`SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`.
|
||
`TransparentBackgroundScript` const is unchanged (a test pins it).
|
||
- **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4
|
||
control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to
|
||
`WebCaptureFrameSource.FindContentBounds`; the ONE integration test
|
||
`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with
|
||
`FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from
|
||
`ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` +
|
||
`IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares
|
||
the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range
|
||
spans rows wrongly).
|
||
- Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the
|
||
composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay
|
||
LOCAL (no push) until the user greenlights.
|
||
- **Slice 15 — one frame per deadline SLOT: duplicate-on-lag, never skip (2026-09-14, device takes
|
||
ty-1723/1726):** slice 10's freshness choice — a render overrun VANISHES the missed slots from the
|
||
file — authored ACCELERATED playback: ~35ms render cost vs the 16.6ms slot, one fresh frame per
|
||
35ms (log wall-1726: `FramePump stall: iteration 33ms (> 2× the 17ms interval): worst render
|
||
33-46ms`), and a 60fps container muxed on arrival → 1723: 697 frames = 11.62s video vs 11.84s
|
||
audio; 1726: 163 frames = 2.72s vs 2.93s. The video also ended 0.21–0.24s before the audio ("audio
|
||
speeds up, cuts off at the end"). OBS's answer is duplicate-on-lag: `libobs/media-io/video-io.c`
|
||
emits one frame per tick and a lagger DUPLICATES the newest frame, counted as "lagged frames due
|
||
to rendering lag/stalls" (obs-output.c) — a wall-time hole never exists (docs.obsproject.com/
|
||
backend-design: "If the video frame queue is full, it will duplicate the last frame"). The pump's
|
||
submit is now a bounded catch-up: `while (now >= nextTick) { submit; nextTick += intervalTicks; }`
|
||
over a `deadlineNow` captured once per iteration — the LATEST composite, fresh on the first missed
|
||
slot, repeated (OBS's duplication) for the rest. Duration == wall under any render load, at the
|
||
cost of a short judder during a stall. The burst is safe because `Channel.TryWrite`
|
||
never blocks (slice 10's queue — the take-9 smear was the BLOCKING pipe-write re-copying a stale buffer
|
||
during a long freeze; here each emit is nanoseconds). The burned `_outputIndex` (slice 10 judge)
|
||
moved INSIDE the submit loop so every emitted slot carries its own +1 (and the old unconditional
|
||
pre-gate bump no longer gaps the sequence on non-submitting fast-render iterations). **Good Dog
|
||
test** `Pump_Overrun_Renders_EmitsEverySlot_NotSkipped`: 60fps, resolver sleeps 35ms, asserts ≥0.65
|
||
of the wall slots are emitted (the skip-pump wrote ~1/35ms ≈ 200 in 7s; the slot pump ~415). Full
|
||
suite **294/294 green, 0 warnings**. The +0.6s audio-late clap reading on 1726 was confounded by
|
||
the 1.1x acceleration — re-measure on device; if a real residual remains it is the audio pipeline.
|
||
No push (web/A/V work stays local).
|
||
- **Slice 16 — the desktop capture conversion was the bottleneck: fast downscale +
|
||
cadence throttle + reuse-distance ring (2026-09-14, device take ty-1742):** slice 15
|
||
fixed pacing but the 1742 desktop layer was still jerky/frozen with a horizontal
|
||
tear. Decoded to raw frames and audited: the **desktop band was frozen 21s of 23.35s
|
||
(90%)** — ~6.1 content updates/s, freeze runs up to 2.28-2.78s, dup-run max 90
|
||
frames (1.5s). The render stat (`worst render 33-36ms`) was real but MOOT: the
|
||
capture CONVERSION was the wall. The monitor delivers at the **240Hz DWM cadence**
|
||
(~4.2ms); with one-in-flight conversions and each 2560×1440→1920×1080
|
||
`DownscaleBgra` at ~150ms under load, `LatestFrame` updated a handful of times/s —
|
||
the desktop feed inside a 60fps file read ~90% frozen. (Webcam + audio were their
|
||
own paths — fine, matching the report.) Three changes, all in
|
||
`Services/ScreenCaptureFrameSource.cs` (+ `Services/FrameRingBuffer.cs`):
|
||
(1) `DownscaleBgra` is now **integer 8.8 fixed-point, "shift only at the end"** —
|
||
the exact two-stage math of `SceneCompositor.Bilinear` (MyMistakes take-4 rule),
|
||
dropping double-per-pixel to a row-walk of integer ops (the capture downscale had
|
||
stayed the naive float twin of the 258ms disaster); (2) a **10ms conversion floor**
|
||
(`MinConvertInterval`) so the 240Hz tail stops queuing ~150ms of serialized
|
||
conversion per second — the open edge sits just above the ~60/s the pump can use;
|
||
(3) the ring is a **reuse-distance pool** (`FrameRingBuffer`, depth 8, redLine 4):
|
||
a buffer is only rewritten ≥4 rents after its last hand-out else a fresh allocation,
|
||
replacing the blind round-robin — since `ScreenCaptureManager` keeps
|
||
`session.LatestFrame` across conversions and the dispatcher preview copy lags,
|
||
"who released the buffer" needs a consumer API that doesn't exist; a reuse
|
||
DISTANCE needs no cooperation (the 1742 new-top/old-bottom tear is that
|
||
read-under-write, structural now). Telemetry added: a startup.log line every 2s
|
||
(`frames/s, conv avg/max ms, skip busy/cadence, ring allocs`) so the device take
|
||
can be judged numerically. **Good Dog test**
|
||
`Ring_NoLap_ReusesOnlyAfterRedLineRents` (depth 3/redLine 4 exercises the red-line
|
||
skip; the 8/4 config settles at 8 buffers and never grows). Full suite **295/295
|
||
green, 0 warnings**. C4 (compositor Epoch-cached downscale / blit-on-change) was
|
||
DEFERRED: the slice-15 approved plan assumed a native-res relocation; the
|
||
measurement recast it — relocating a ~30ms float downscale to the render thread
|
||
just moves the same cost into the slot budget. Re-measure on device; if render
|
||
still >16.6ms slots after capture feeds ≤60 real updates/s, add C4. No push.
|
||
- **Slice 17 — the OS readback was the real wall: overlapping conversions + monotonic
|
||
publish + deeper pool (2026-09-14, device take ty-1824):** slice 16's downscale fix
|
||
landed but the desktop was still ~90% frozen on the 1824 take (band 6.8/s updates, max
|
||
freeze 4.85s). The new 2s telemetry was decisive: `conv avg 46-50ms max ~61ms` with
|
||
`skip busy 37-83` — the **GPU→CPU readback (`CreateCopyFromSurfaceAsync`), not
|
||
`DownscaleBgra`**, is the ~47ms wall (240Hz HDR compositing + encoder + 2 capture
|
||
devices share the GPU); at one-in-flight that caps the desktop feed at ~17-20
|
||
updates/s, which is the file's whole-frame ~7 content-moments/s. Two facts reshaped
|
||
the fix: **(a)** shrinking the pool size does NOT scale the desktop (Microsoft docs:
|
||
"If content is larger than the frame, the contents are **clipped**") — readback stays
|
||
at native 2560×1440; **(b)** delivery is healthy (60-100 arrivals/s), so the lever is
|
||
conversion throughput, not the pool size. Changes in `Services/ScreenCaptureFrameSource.cs`:
|
||
conversions overlap up to **MaxConcurrentConversions = 3** (pool deepened to 5 buffers
|
||
so in-flight frames fit), each completion publishes ONLY if its Epoch is strictly
|
||
newer than the last published (`MonotonicGate` — a slow older completion must never
|
||
overwrite a newer LatestFrame), Epoch increments via `Interlocked`, and the
|
||
DownscaleBgra row-scratch became per-conversion locals (concurrent callers). The
|
||
render side (33-42ms → ~30 unique composites/s) is the NEXT cap after capture speeds
|
||
up — that's the C4 slice, queued right after this re-measure. **Good Dog test**
|
||
`PublishGate_TryPublish_OnlyStrictlyNewerWins`. Full suite **296/296 green, 0
|
||
warnings**. NOT YET DEVICE-VERIFIED; target: telemetry frames/s jumps ≥ ~30-40 and the
|
||
band audit drops below ~50% frozen. No push.
|
||
- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending
|
||
queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits;
|
||
an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was
|
||
a submit stuck on pipe backpressure; the drain thread owns that wait now, and stopping the encoder
|
||
first keeps the pump's own (non-blocking) submits from racing the completed queue as a spurious drop.
|
||
`ProcessFailed` self-stops the pump. `Failed` while live flips
|
||
`StreamStatus.Error` (minimal).
|
||
- **Health stats (ship step 6, shipped 2026-08-13):** `HealthUpdated` is bound to the bottom bar —
|
||
`MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the encoder's stderr loop raises on
|
||
a background thread) and copies into `CurrentHealth` (the bottom bar's existing `CurrentHealth.*`
|
||
bindings). `ResetHealth(status)` zeroes dropped/duration on go-live and on End so stats never linger
|
||
from a previous session; bitrate/FPS stay on the tier's targets (`ApplyStreamQuality`). The bar shows
|
||
real encoder values since TASK 9 (2026-08-16) wired the reusable stream URL into `_rtmpUrlProvider`.
|
||
- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` →
|
||
`CameraManager.GetLatestFrame(DeviceKeyForWebcam(WebcamId))` — **device-keyed** (configs carry the
|
||
identity GUID; sessions key on the Windows device id — the old direct-GUID lookup returned null
|
||
forever and the webcam was structurally absent from output; take-2 fix 2026-09-01,
|
||
`WebcamOutputKeyTests`), `Source { IsLiveCapture, CaptureKey }` →
|
||
`ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/
|
||
background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*`
|
||
doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`,
|
||
ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one.
|
||
- **Media source decoder (TASK 21, Increment B — 2026-08-31):** `MediaVideoSource` (`Services/MediaVideoSource.cs`)
|
||
decodes a local file to `VideoFrame`s by spawning ffmpeg with
|
||
`-f rawvideo -pix_fmt bgra -vf scale=WxH -an` and draining the raw BGRA stdout pipe through the pure
|
||
`RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raising `FrameAvailable` per frame. Implements
|
||
`IMediaFrameSource` (`Services/IMediaFrameSource.cs`: `Key`, `FrameAvailable`, `Completed`, `StartAsync`/`StopAsync`).
|
||
The subprocess sits behind `IDecodeProcess`/`FfmpegDecodeProcess` (binary-stdout mirror of the encoder's
|
||
`IEncoderProcess`), and the path comes from `IFfmpegLocator` — codec-agnostic, uses the ffmpeg already shipped,
|
||
testable with a fake process. Decode sessions are owned app-wide by `MediaVideoSourceManager`
|
||
(`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func<string, IMediaFrameSource?>` factory
|
||
seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`,
|
||
coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as
|
||
the pipe produces them. Wired into the live pipeline (slice 1 step 4): `ResolveOutputFrame` reads
|
||
`_mediaManager.GetLatestFrame(MediaPath)` for a `MediaSource`, and `Source.DisplaySource`
|
||
(`Models/Source.cs`) routes `VideoImageSource` for `MediaSource` so the preview/canvas show decoded video.
|
||
Slice 2a shipped the probe seam: `FfmpegFrameRateParser` (pure, `avg_frame_rate=` then
|
||
`r_frame_rate=` rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe`
|
||
from the located ffmpeg dir, reuses the `IDecodeProcess` seam for the ffprobe subprocess text; null if
|
||
ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive
|
||
(conditional — old caches without it just get no pacing). Slice 2b wired pacing: `MediaVideoSource`
|
||
takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` (default `Task.Delay`)
|
||
seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; no probe/unknown → no
|
||
pacing (ffmpeg's own pipe backpressure already throttles the decode). Slice 3 added loop control:
|
||
`IMediaFrameSource.Looping` (bool); `MediaVideoSource` takes a `Func<IDecodeProcess>` process factory (a single
|
||
`Process` can't be re-`Start()`ed, so each loop pass creates a fresh decoder) and wraps the decode in a
|
||
`do…while (Looping)` — restart on natural EOF instead of raising `Completed`. Production wiring (MainViewModel
|
||
media factory): passes `() => new FfmpegDecodeProcess()` as the factory, `FfmpegFrameRateProbe(new FfmpegLocator(),
|
||
() => new FfmpegDecodeProcess())` as the probe.
|
||
Still open: session acquisition on add/remove (UI picker) + wiring `Source.MediaIsLooping` into
|
||
`IMediaFrameSource.Looping` (needs a manager-level per-path loop provider, comes with the picker slice).
|
||
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
||
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
||
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
||
(WPF glue — WPF glue here is fine because the strip is rendered once per config change on the UI thread,
|
||
the resulting immutable frame is then composited pure-CPU by `SceneCompositor`), and `SceneCompositor.Render`
|
||
blits it **last — above the branding flash** at `socialBarTop` (0 = top, `SourceRectHeight − barHeight` =
|
||
bottom) in master space. `MainViewModel` owns the frame (`_socialBarFrame`, rebuilt by `RenderSocialBarFrame`
|
||
on load/save/`NotifySocialsChanged`).
|
||
- Known consideration: the pump reads the active scene on a background thread while the UI can still edit
|
||
it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than
|
||
crashing. The background thread + video pipeline is the new reality since this step.
|
||
|
||
### Social bar (TASK 6 — shipped 2026-08-12, plan in TASKS.md; bar bug-fix branch 2026-08-13)
|
||
|
||
A **global bar layer** (never a Source, no sources-list row) that sits over the bottom or top of the
|
||
output and carries the creator's social links — content-sized, centered, GREEN glow when ON, position is a
|
||
**click-toggle top ⇄ bottom** (default BOTTOM, persisted `SocialBarPosition`). `Models/Socials.cs`: `SocialService` enum
|
||
(YouTube/Twitch/X/Instagram/TikTok/Facebook/Discord/Kick/Threads/Bluesky/GitHub/LinkedIn/Pinterest/
|
||
Snapchat/Reddit/WhatsApp/Telegram/Link/Website/**Fediverse**), `SocialEntry` (Service/Handle/ProfileUrl/
|
||
`FediverseSoftware`), `SocialsConfig` (Entries + `BarPosition` + `BarEnabled`; `BarJustify` dropped,
|
||
column back-compat). Configured in the "Social Media Site Promotion" dialog (`SocialsDialog.xaml` +
|
||
`ViewModels/SocialsDialogViewModel`, WPF-free, injected `ISocialValidator` + sign-in/sign-out fakes):
|
||
ON/OFF bar switch (schema v8), 6 fixed slots — row 1 always YouTube (signed-in → channel handle;
|
||
signed-out → sign-in gate → OAuth; delete → confirm sign-out), row 2 free, rows 3–6 lock icons on
|
||
freemium. Service detection (`SocialServiceIcons.DetectService`): URL domain / fediverse `@user@domain` →
|
||
**Fediverse** / bare→Website. Validation (`Services/SocialValidator.cs`, `ISocialValidator` seam +
|
||
`HttpSocialValidator` default): async GET of the canonical profile URL; 200/redirect = valid, 404/failure =
|
||
rejected. Fediverse additionally does a **best-effort nodeinfo lookup** (`/.well-known/nodeinfo` →
|
||
`software.name`, stored in `SocialEntry.FediverseSoftware` / the `SocialEntry.Software` column,
|
||
column-presence migration, no version bump) so the entry shows the **instance's real logo**
|
||
(`LogoDataForFediverse`: mastodon/peertube/pixelfed/misskey/lemmy/pleroma/firefish, generic fediverse
|
||
honeycomb fallback — Simple Icons CC0 path data, initials badges gone); nodeinfo failure still validates
|
||
(glyph falls back). If the identity domain's nodeinfo is blocked (YunoHost SSO gates
|
||
`/.well-known/nodeinfo` behind the login page) but the bare root 302s to the real instance, the lookup
|
||
**follows the root redirect** and asks the resolved host. Cancel is a hard stop: `LookupAsync` takes a
|
||
`CancellationToken` (dialog VM owns a CTS; Cancel/X/Save abort in-flight lookups, canceled continuations
|
||
never touch slot state).
|
||
|
||
**Bar bug-fix branch (2026-08-13) — three changes:**
|
||
|
||
1. **Positioning is a click-toggle (KISS)** — the original drag set a local `Canvas.SetTop` value that
|
||
permanently overrides the `{Binding SocialBarTop}` (a binding can never win over a local value), so the
|
||
bar stayed wherever it was dropped. The first drag-snap fix (`SocialBarSnap.Decide` + `ClearValue` on
|
||
release) still failed for shaky hands — jitter around the ±6px deadzone snapped the bar up but wouldn't
|
||
let it come back down. **Superseded by the user's click-toggle:** clicking the bar flips it top ⇄ bottom
|
||
(`MainViewModel.ToggleSocialBarPosition`), the bar rides the binding alone, and `SocialBarSnap` is gone.
|
||
2. **Fediverse software self-heal** — the DB row `@gramps@llamachile.tube` had `Software = NULL` because
|
||
nodeinfo was only ever asked of the identity domain (a landing page; the real instance is
|
||
`mastodon.llamachile.tube`). `HttpSocialValidator.ResolveFediverseSoftwareAsync` now **probes
|
||
well-known subdomains** (`mastodon.` → `social.` → … `FediverseSubdomainCandidates`) when the identity
|
||
domain and its redirect both come up empty, under a ~15s linked-CTS budget. `MainViewModel` runs the
|
||
static `HealFediverseSoftwareAsync` off the UI thread on layout load, applies matches via the dispatcher,
|
||
and saves; `SocialEntry.FediverseSoftware` is settable and raises `LogoData`, so the icon updates in place.
|
||
3. **The bar renders on the live output** — `Compositor/SocialBarRenderer.cs` rasterizes the entries into a
|
||
transparent straight-alpha BGRA strip (1920-wide, 40px content + 24px glow pad, the green glow baked in,
|
||
Pbgra32→straight-alpha unpremultiply) and the compositor blits it above the flash (see "Live frame
|
||
pipeline").
|
||
|
||
### Toast notifications (TASK 24 — shipped 2026-08-23, plan in TASKS.md)
|
||
|
||
Non-blocking user-facing messages replace blocking MessageBoxes. **Seam:** `INotificationService.Show(title,
|
||
message, AppNotificationSeverity)` (`Services/INotificationService.cs`) + `Info/Success/Warning/Error`
|
||
extension methods; `MainViewModel` news up the real `NotificationService` at its field initializer
|
||
(no DI container — the VM is built from XAML). **Library:** `Notification.Wpf` 11.0.0 (Platonenkov fork,
|
||
MIT — notice #10 in `THIRD-PARTY-NOTICES.txt`). Facts that cost a hunt:
|
||
- The `NotificationArea` XAML control registers itself with the manager on `Loaded`; routing matches
|
||
the area's `Name` against the request's `AreaName` — our host is `x:Name="ToastArea"` in MainWindow's
|
||
root grid (last child = topmost z-order, `Grid.RowSpan=4`, bottom-right above the footer via
|
||
`Margin="0,0,12,96"`, outside the preview Viewbox). An unknown AreaName silently drops the toast.
|
||
- `NeverExpires()` sets `ExpirationTime = TimeSpan.MaxValue` (NOT null); `NotificationColor.ToHex()`
|
||
returns `#AARRGGBB` (alpha prefix).
|
||
- Lifetimes: Info/Success ~4s, Warning ~8s auto-dismiss, Error sticky until dismissed (the pure mapping
|
||
lives in `NotificationService.BuildRequest` + the `Cards` tint table: slate info `#3a3b52`, green
|
||
success `#1d5c38`, amber warning `#b8860b`, dark-red error `#8f1f1f` — matching the health-banner language).
|
||
- Calls marshal to the captured Dispatcher (`_dispatcher.Invoke`) — encoder/pump events fire off-thread.
|
||
|
||
Migrated MessageBoxes: both webcam-acquire warnings, image-read warning, sign-in unsuccessful/failed.
|
||
Promoted from log-only (each keeps its AppLog line): go-live prep failures ×3 (reusable stream,
|
||
broadcast insert, prep exception), frame-pump death while live, mic-missing at startup, premium lapse,
|
||
saved-session refresh failure (sign-out notice). Deliberately left log-only: offline license re-validation
|
||
skip (would spam every launch). Still modal by design: SocialsDialog's slot-delete YesNo confirm.
|
||
|
||
### Licensing — do not violate (GA = paid product; see `THIRD-PARTY-NOTICES.txt`)
|
||
|
||
This product is closed-source and paid. Every third-party component must stay inside the LGPL/BSD/MIT
|
||
guardrails below — written down so a future "quick fix" never reintroduces a GPL binary. **NEVER:**
|
||
|
||
- **Use a GPL FFmpeg build** — gyan.dev's builds are GPLv3 and ship libx264; BtbN's `gpl` variant is
|
||
GPL too. GPL in a distributed paid product is the #1 lawsuit risk. Only BtbN `lgpl` / `lgpl-shared`
|
||
builds are allowed.
|
||
- **Distribute the static lgpl build** — LGPLv2.1 §6 wants relinkable object files for static linking.
|
||
The **shared** (dynamic-DLL) build sidesteps that: compliance is "license text + source offer +
|
||
unmodified binaries". The pin is `lgpl-shared`; when the pin is refreshed, keep the shared variant.
|
||
- **Use BtbN's `nonfree` variant** — it adds fdk-aac (Fraunhofer code licensing). The native FFmpeg AAC
|
||
encoder is fine (no Fraunhofer code) but grants no AAC patent license — accepted low-risk posture for
|
||
RTMP→YouTube, since encoder vendors cover their implementations (Cisco OpenH264, NVIDIA NVENC, Intel
|
||
QSV, AMD AMF).
|
||
- **Link FFmpeg into the app** — it stays a separate subprocess fed frames over a pipe; that separation
|
||
keeps the app's own code out of LGPL reach.
|
||
- **Drop `THIRD-PARTY-NOTICES.txt`** from the shipped app or the About screen, or alter the FFmpeg
|
||
copyright/LGPL notices inside the downloaded binaries. Automating the download counts as distribution
|
||
— the obligations are not optional.
|
||
- **Pin to a moving target** — the `latest` BtbN release tag floats. Only immutable autobuild tags give
|
||
a reproducible source offer. Record the tag + variant beside the URL (TASKS.md) every time the pin moves.
|
||
- **Use non-CC0 icon art** — the social bar's bundled SVG logo path data comes from **Simple Icons**
|
||
(CC0 1.0, public domain — see `THIRD-PARTY-NOTICES.txt`). Replacing or adding logos must stay CC0 or
|
||
another public-domain source; a logo asset under a copyleft or attribution license would contaminate
|
||
the paid product. The About hub's logo is the creator's own art ("llama fortnite superman logo",
|
||
`Assets/llama-logo.png`) — owned by us, so it carries no third-party license either.
|
||
- **Forget the v1 license-texts gate** — `THIRD-PARTY-NOTICES.txt` links the canonical license texts; at
|
||
**v1 (GA)** the full texts of every license it names MUST ship alongside it (TASK 4 requirement 9 is the
|
||
release blocker). Queued early is wrong; the release pass owns it.
|
||
- **Break the in-app About requirement** — since TASK 11 the About overlay is the app's in-app creator
|
||
hub: the real logo, the creator links (YouTube channel, Mastodon, Buy me a coffee, and the greyed
|
||
"Unlock Premium" seam whose billing URL is tabled), and a **Licenses & legal** sub-panel that loads
|
||
`THIRD-PARTY-NOTICES.txt` from the executable directory and renders it in a scrolling panel —
|
||
**never the OS viewer, no Notepad, no browser tab**. The premium/coffee link constants live in
|
||
`MainViewModel` (`PremiumUrl`, `CoffeeUrl`, `ChannelUrl`, `MastodonUrl`); filling in the billing URL
|
||
lights up the premium button. Do not reintroduce an "open the notices in Notepad" button.
|
||
|
||
## Design Principle
|
||
|
||
> This software is so intuitive that even the most right-brained person can easily intuit and use it.
|
||
|
||
Apply this to every UI decision:
|
||
- One-click go-live with working defaults
|
||
- Prefilled YouTube defaults (RTMP URL, bitrate, resolution, latency)
|
||
- Visual/drag-and-drop scene building over property panels
|
||
- Every action produces a visible outcome — no dead ends
|
||
- **"We're not them" (2026-09-01):** assume the creator's hardware is mediocre, because it is.
|
||
Every feature spends the machine's budget once — never twice for the same result (record OR
|
||
stream, not both; one reusable stream; one webcam; two mixer inputs). If a feature only sings on
|
||
a high-end chassis, it doesn't ship — the OBS escape hatch is open by design.
|
||
|
||
## Monetization (design decision — the branding flash is the sword)
|
||
|
||
Full pricing, discount codes, and Polar product details in `MONETIZATION.md` (gitignored).
|
||
Brand palette and assets in `MARCOM.md` (gitignored). This section covers the in-app model.
|
||
|
||
Free forever: **all features unlocked for everyone** — no feature lock between free and paid.
|
||
The **only difference is watermarking**. This is deliberate: no creator will tolerate a watermark,
|
||
and as a good-guy developer, we give them complete access to every feature so no one can call us
|
||
crooked.
|
||
|
||
- **Free:** a periodic full-frame branding flash — "made with LlamaCasty!" rendered big and centered
|
||
at ~25% opacity for about one second (soft 250ms fade in/out), repeated every 300s, on the live
|
||
output (and on local recordings). Implemented as `BrandFlashLayer` in the preview compositor
|
||
(`MainWindow.xaml` CanvasGrid) + `BrandFlashTimer` in `MainViewModel` — cadence 300s, first flash
|
||
~5s after go-live, only while live or recording. An always-on watermark can be cropped or covered;
|
||
an intermittent full-frame flash can't be cropped and is impractical to edit around on a live feed.
|
||
**The flash is also the free tier's billboard** — every free stream advertises LlamaCasty to its
|
||
own viewers; the free tier is distribution, not compromise.
|
||
**Escalation model (2026-09-01, creator decision):** the cadence is *obnoxiously* self-promoting —
|
||
intervals shorten with use, starting at the 300s cadence and creeping toward a floor (the exact
|
||
curve is a build-time design knob). License activation still flips exactly one bit: `IsPremium` →
|
||
flash off. Nothing else changes between free and paid, ever.
|
||
**Pre-GA posture:** while the app is unreleased the flash renders **in the preview only** and is
|
||
never composited onto the live output or local recording — test VODs stay clean (same channel-
|
||
protection stance as the visibility lock), and creators can be shown what free looks like without
|
||
it ever touching a real broadcast. Flipping the flash live-on is a **TASK 36** unlock item.
|
||
- **Paid (annual subscription):** branding flash removed (flips `BrandFlashEnabled` off). That's it.
|
||
No feature gating. Alerts, social bar slots, voice filters, TRAX, recording — everything is free.
|
||
- **Pricing:** early-access founders rate **$49.99/yr** → **$99/yr list at GA** (v1). **Grandfathering:
|
||
early adopters keep $49.99/yr for as long as the subscription is maintained**; a lapse means renewal
|
||
at list. That's the whole policy — no escalation matrix (a realistic product lifetime is a few years;
|
||
keep the promise simple).
|
||
- **Billing:** **Polar (polar.sh)** — open-source MoR (Apache 2.0), handles payments, subscriptions,
|
||
license keys, and global tax compliance. Startup Program gives Scale plan free for 12 months.
|
||
Product: `d105dfa1-497e-423b-8cd4-e0ee2e3abbc0`. Checkout: `llamacasty.com` → Polar hosted page.
|
||
Org ID: `c05fb364-b967-4f6c-adf2-8a144e46d085` (org-scoped OAT — `organization_id` omitted from API calls).
|
||
Key prefix: `LCYT-`. Discounts: `LLAMAFOUNDER` (100% off, 50 uses), `LLAMA50` (50% off, 12 months). Details in `MONETIZATION.md`.
|
||
- **Support (creator's model, corrected 2026-09-01):** support = email + GitHub issues — the
|
||
once-planned in-app bug-reporter is **out of product** (TASKS.md → closed list). Most queries are
|
||
how-tos / feature requests / manual-skimmers. Maintenance cadence = "when I get around to it" with
|
||
emergency patches; not a 24/7 service promise. The only license chatter is the paid-user expiry
|
||
reminder (TASK 36 item 5); active subscribers see zero license UI.
|
||
|
||
**Monetization awareness (built-in, ungated, free — 2026-09-01; v1 SCOPE per the complete-v1
|
||
ruling — build order: capture → report → journey → Alerts):** the app is monetization-aware by
|
||
design, for **every** creator, free and ungated (the only subscriber swap remains the branding-flash
|
||
removal above). It has three layers, all free — this is the product's differentiator, not a paid tier:
|
||
|
||
- **Reward-event capture (the data foundation).** The live-chat poll (`YouTubeChatService.Poll`)
|
||
already receives every reward event over `liveChatMessages` but today only decodes two
|
||
(`superChatEvent`, `newSponsorEvent`); the other five — `superStickerEvent`, `membershipGiftingEvent`,
|
||
`giftMembershipReceivedEvent`, `memberMilestoneChatEvent`, `giftEvent` — are silently dropped. The
|
||
plan: decode all seven into a canonical `RewardEvent` and persist each to a SQLite table
|
||
(`RewardEvents`: broadcastId, type, timestamp, amountMicros, currency, tier, memberLevelName,
|
||
gifter/recipient channelIds). `superChatEvents.list` (30-day lookback) backfills prior broadcasts.
|
||
This single feed is the foundation that Alerts, the session report, and the journey tracker all read.
|
||
- **Session / broadcast report.** A session is created at Go Live; the poll + End roll that session's
|
||
reward totals (new members, Super Chat $, stickers $, gifts, milestones). This is the in-app
|
||
replacement for the emailed external "stream activity report" — the incumbents (Streamlabs et al.)
|
||
email near-useless post-stream summaries; we track the same data natively and maintain it.
|
||
- **Journey tracker.** On sign-in/startup, pull current `channels.list?mine=true&part=statistics`
|
||
(`subscriberCount`/`viewCount`/`videoCount` — the current OAuth scopes suffice). Compute position
|
||
toward each YPP tier threshold and project an ETA curve from the creator's own
|
||
per-stream/per-day velocity (subs + watch hours). **YPP thresholds are versioned, date-aware DATA,
|
||
never constants** — the Tier-2 bar doubles for new applicants on **2027-02-01** (long-form 4,000 →
|
||
8,000 qualified hrs / 365d; Shorts 10M → 20M / 90d), while the Tier-1 fan-funding bar (500 subs /
|
||
3,000 hrs / 3M Shorts) is unchanged. Hardcoding the pre-2027 numbers would ship a wrong ETA.
|
||
Analytics-API scopes (`yt-analytics.readonly`, additive `yt-analytics-monetary.readonly`) are an
|
||
additive capability behind a seam (`IChannelStatsProvider`), not a v1 blocker — current-scope data
|
||
is the v1 baseline; the Analytics scopes would require an OAuth re-consent, deferred.
|
||
|
||
**What was rejected:** always-on watermark (obscurable — replaced by the flash), hard stream-time
|
||
cutoffs (the worst dead end — a stream dying mid-broadcast reads as broken, and YouTube streams
|
||
routinely run 2-4 hours), soft-limit nagging, freemium feature tiers, and donation-only (relies on
|
||
the kindness of strangers). Resolution/quality ceilings stay rejected (fixed 2026-09-01, no longer
|
||
"deferred"): the free tier never loses quality — auto step-down (**TASK 33**) is a *protect the
|
||
stream* feature, never a monetization penalty.
|
||
|
||
## Auth gates Go Live, but not exploration
|
||
|
||
The app is fully usable without authentication: users can build scenes, add sources, compose
|
||
previews, and audition the software with zero commitment. But **going live requires authentication** —
|
||
it's the one capability gated behind YouTube sign-in. The sign-in should never pressure the user
|
||
("sign in (optional)", not a modal wall): the two-state top bar shows **Start Stream** (offline) /
|
||
**End Stream** (live), and the Start Stream dialog hosts the account — a saved session appears as
|
||
the default with "Change Account"; with none saved, a "Sign in to YouTube" button starts OAuth and
|
||
the Start button stays disabled until signed in.
|
||
|
||
## Account assumption (do not build an account setup flow)
|
||
|
||
Connecting uses Google OAuth ("Sign in with Google") to link an **existing** YouTube creator
|
||
account. ytLlive **never creates or sets up accounts** — that is YouTube's job. If the creator has no
|
||
YouTube channel, they go to YouTube first. This assumption is explicit and must never be silently
|
||
replaced by an in-app account-creation step. Zero state = the Start Stream dialog's "Sign in to
|
||
YouTube" button; going live is unreachable until an account is connected.
|
||
|
||
## YouTube Live API — design constraints (do not violate)
|
||
|
||
These are the hard facts behind every decision. Full list in `TASKS.md`.
|
||
|
||
- **One-click go-live** — never call `transition(live)`. Insert the broadcast with
|
||
`enableAutoStart=true`, `enableAutoStop=true`, `enableMonitorStream=false`,
|
||
`selfDeclaredMadeForKids=false`, `latencyPreference=low`. The encoder starting brings YouTube live.
|
||
`enableMonitorStream=false` is what lets us skip the testing stage.
|
||
- **Visibility picker (TASK 9 item 6) — DELIBERATE TEST-PHASE LOCK, not drift (2026-09-01).**
|
||
The "always Private" enforcement in `YouTubeStreamService.CreateBroadcast` stays while the creator
|
||
runs multi-month real-world testing: non-private test streams would clutter the channel with VODs
|
||
highlighting where the product breaks. `recordFromStart`/DVR stay ON — Private VODs are invisible
|
||
to subscribers and serve as post-mortem review tapes; bulk-delete pre-GA. The unlock is a
|
||
deliberate final-pass item in **TASK 36 (gold pass)**, wired with the dialog selection — never
|
||
opportunistic. The PRIVATE badge keeps showing when the stream is actually private.
|
||
- **Full broadcast form (TASK 9 item 7) — RESCOPED 2026-09-01** — the core editable fields ship (since 2026-08-24) as the always-visible **Text drawer**: title, description, tags, visibility, made-for-kids, live-editable; scheduling ships as **TASK 34** (drawer ☑ Scheduled + adoption at Start). The old **Advanced tab is permanently out of product** (the 10% margin — see TASKS.md → "Out of product"): latency locked `low`, DVR/record-from-start locked on, embed/projection/CC/region fixed at sane defaults, invisible — every exposed field is a support ticket. `categoryId` is removed from `CreateBroadcast` (not a `liveBroadcast` field, silently ignored). Monetization enablement (if ever needed) rides the reward-events chain via `liveBroadcasts.update`, not a form field. Go-live order (TASK 9, shipped 2026-08-16):
|
||
`BeginGoLive` → `PrepareAndStartLiveAsync` — ensure the reusable stream (`GetOrCreateReusableStreamAsync`,
|
||
cache it), create the broadcast bound to it (`CreateBroadcast(..., stream.Id)` → `boundStreamId`),
|
||
THEN start the pump (the URL must exist before `FramePump.StartAsync`, which reads it once). Failure →
|
||
`StreamStatus.Error`, never a crash; `StopStream` clears `_currentBroadcastId`.
|
||
- **Variable reusable stream** (shipped 2026-08-16) — `GetOrCreateReusableStreamAsync` lists
|
||
`liveStreams?mine=true` and reuses the existing `cdn.isReusable` stream, inserting once per channel
|
||
(`cdn.resolution=variable`, `cdn.frameRate=variable`, `isReusable=true`) only on first use; the
|
||
ingestion URL is cached via `LayoutStore` Settings (`SaveReusableStream`/`LoadReusableStream`) and
|
||
bound to each broadcast at insert (`boundStreamId`). Any quality tier works without recreating the
|
||
stream, and auto step-down rides the same property — we drop bitrate/resolution on the fly, zero
|
||
API calls — **but the deciding governor is not built**: the old present tense there was a map-lie,
|
||
corrected 2026-09-01; step-down is **TASK 33** (v1 scope).
|
||
- **Quality is greyed out while live** — resolution/frameRate/ingestionType are immutable after
|
||
stream creation; editing title/description/privacy is fine at any time.
|
||
- **Report-by-exception health (SHIPPED 2026-08-16, TASK 9 item 3)** — `YouTubeStreamService.GetStreamHealthAsync(streamId)`
|
||
polls `liveStreams.list?part=status`; render nothing on `good`/`ok`/`noData`, surface a banner only
|
||
on `configurationIssues[]` with `warning`/`error` severity. The decision is the pure
|
||
`Services/StreamHealthReporter.BannerFor` (null text = no banner; error beats warning). The VM polls
|
||
every **30s while live** (`DispatcherTimer` `_healthPollTimer`, first poll right after go-live) and
|
||
clears on End via `ResetHealth`; poll failures log only. UI = a full-width banner strip under the
|
||
top bar, `HealthIssueBanner` text + `HealthIssueBackground` (amber `#b8860b` warning / dark red
|
||
`#8f1f1f` error), hidden by `NotNullToVis`. The design's bottom-strip YouTube logo + green/red dot
|
||
(clickable → dialog) is still queued.
|
||
- **One dialog, three states** — not connected / connected-offline (all editable) / live
|
||
(title + description + visibility editable; quality + account greyed out). Both entry points
|
||
(Start Stream button + bottom strip) open it; prefilled from saved session profile.
|
||
- **End stream (SHIPPED 2026-09-01)** — stop encoder → `YouTubeStreamService.EndBroadcastAsync` POSTs `liveBroadcasts/transition?broadcastStatus=complete&id=…&part=status` (AFTER RTMP EOF, so no frames post-date the end — VOD finalizes immediately instead of ~1min of frozen "stream offline"), log-only on failure (`invalidTransition` when autoStop already fired — never throws, never toasts a finished session); `enableAutoStop` remains the safety net. The design had always called for this call; it was never built until the recording-verification session caught the gap. Record-only stops make ZERO API calls (guard: `wasLive && _currentBroadcastId != null`).
|
||
- **Encoder compliance** — keyframes ≤ 4s (gopSizeLong), closed GOP, H.264, AAC/MP3 @ 44.1/48kHz,
|
||
mono/stereo only. YouTube flags violations via health status.
|
||
- **Broadcast ID == Video ID** — one ID tracks status, health, and the auto-created VOD
|
||
(`recordFromStart` + `enableDvr` default true).
|