From 3bf053a3b10a3c7eaf03bedb3e52606dc9fce6f7 Mon Sep 17 00:00:00 2001 From: gramps Date: Mon, 31 Aug 2026 11:38:12 -0700 Subject: [PATCH] TASK 31: SceneGraph design + baked-crust compositor optimization Design captured in TASKS.md, pattern note in ai.md, handoff updated. --- HANDOFF.md | 3 ++ TASKS.md | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++ ai.md | 1 + 3 files changed, 132 insertions(+) diff --git a/HANDOFF.md b/HANDOFF.md index 8a74f77..16993f4 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -27,6 +27,9 @@ docs commit `cb54637` + **Phase 3 splits A–E** (tags `refactor-commit-A..E`) + Nothing code-in-flight — working tree clean. +**TASK 31 designed:** SceneGraph component + baked-crust compositor optimization. +Design captured in `TASKS.md`, pattern note in `ai.md`. Ready to implement. + **The directive (2026-08-31, user):** rewrite the project, breaking files into **functional components to compliment AI retrieval/processing** — NOT line-count chasing. Line count is a guideline for context management, not a design goal. diff --git a/TASKS.md b/TASKS.md index a9961ca..2c5e499 100644 --- a/TASKS.md +++ b/TASKS.md @@ -1221,6 +1221,134 @@ and a full layout restructure moving all live controls into the preview pane. --- +## TASK 31 — SceneGraph component + static/dynamic compositor optimization + +**Goal:** extract the scene collection, element inventory, and mutation surface into a standalone +`SceneGraph` component. Classify elements as static or dynamic. Enable the compositor to bake +static layers once and only recompose dynamic layers per frame. + +### Status: ☐ Not started + +### Design + +**Element classification:** + +Every `SceneElement` exposes `ElementKind Kind` — `Static` or `Dynamic`: + +| Type | Kind | Why | +|------|------|-----| +| `Source` (Background art) | Static | Pixels don't change at runtime | +| `Source` (Image) | Static | Pixels don't change at runtime | +| `Source` (DisplayCapture) | Dynamic | Live game/desktop feed | +| `Source` (ChatBox) | Dynamic | Live chat messages arrive continuously | +| `Source` (WebSource) | Dynamic | WebView content can change | +| `WebcamSceneConfig` | Dynamic | Camera feed, changes every frame | + +**Split point:** + +The **split point** is the index of the first dynamic element in a scene's z-ordered `Elements`. +Everything below it = baked base. Everything from it upward (including static layers above dynamic +elements) = composited per frame. If zero dynamic elements exist: entire scene is baked, +compositor skipped entirely. + +**BakedSceneCache:** + +``` +BakedSceneCache { + VideoFrame BaseFrame // composited static layers below split point + int Version // incremented on mutation + SceneElement[] BakedElements // for invalidation tracking +} +``` + +Invalidation triggers (re-bake): element added/removed/reordered, static element moved/resized/ +opacity changed, `IsVisible` toggled on a static element, background asset changed, scene switch. + +No invalidation when: only a dynamic element's pixels changed (webcam frame, chat message), +or a dynamic element moved/resized (per-frame compositing concern, not a base rebake). + +**Compositor integration:** + +``` +Render(scene, frameFor, options): + 1. Find split point (first Dynamic element index) + 2. If split == scene.Elements.Count: + → return cached base (or bake if stale) + 3. Otherwise: + → start from cached base (or bake if stale) + → composite elements[split..] on top using frameFor +``` + +**SceneGraph interface:** + +``` +SceneGraph { + // Collection + ObservableCollection Scenes + Scene? StagedScene + Scene? LiveScene + + // Mutation surface (single owner of element operations) + AddElement(scene, element) + RemoveElement(scene, element) + MoveElement(scene, element, newIndex) + + // Queries (replace scattered LINQ) + GetBackground(scene): Source? + GetWebcam(scene): WebcamSceneConfig? + GetChatBoxes(): IEnumerable + GetSplitPoint(scene): int + IsStatic(scene): bool + + // Events + SceneChanged + ElementAdded/Removed/Moved + SplitPointChanged +} +``` + +**What moves into SceneGraph:** + +| Current location | Moves to | +|-----------------|----------| +| `MainViewModel.Scenes.cs` — Scenes collection, StagedScene, scene switching | `SceneGraph` | +| `MainViewModel.Background.cs` — `NormalizeBackgrounds`, `EnsureBackground`, background queries | `SceneGraph` (background helpers) | +| `MainViewModel.Sources.cs` — `AddSource`, `RemoveElement`, element inventory | `SceneGraph` (mutation surface) | +| `MainViewModel.Webcam.cs` — webcam queries across scenes | `SceneGraph.GetWebcam(scene)` | +| `ChatOverlayLayer` — `scenes.SelectMany(...).OfType().Where(ChatBox)` | `SceneGraph.GetChatBoxes()` | + +**What stays on the ViewModel:** + +The binding surface — `StagedScene` setter still raises `OnPropertyChanged` for +`ShowEmptySceneHint`, `CanAddWebcam`, etc. But now it delegates to `SceneGraph` for actual +state queries. ViewModel becomes a thin binding facade over the SceneGraph (same pattern as +`MainViewModel.Chat.cs` over `ChatOverlayLayer`). + +The compositor resolver (`frameFor` callback) stays in the ViewModel/encoder layer — it bridges +WPF concepts (CameraManager, ScreenCaptureManager, ImageCache) into the pure `VideoFrame` seam. +SceneGraph doesn't know about capture sessions; it just knows element kinds. + +**Migration path:** + +1. Extract `SceneGraph` as a standalone class in `Services/` +2. Move `Scenes` collection + `StagedScene`/`LiveScene` + wiring +3. Move element mutation surface (add/remove/move) +4. Move background normalization/queries +5. Add `ElementKind` to `SceneElement` base +6. Add `GetSplitPoint` + `BakedSceneCache` +7. Update compositor to use split point +8. Wire ViewModel as thin facade +9. One integration test: baked static base + dynamic layer composite + +### Tests + +ONE integration test: `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly` +— a scene with static background + image + dynamic webcam; assert the baked base is cached +(renders once), dynamic layer composited on top, static element mutation invalidates the cache, +dynamic-only pixel change does not. + +--- + ## Backlog (future versions) 1. v1.1 — Stream Deck / Loupedeck integration (requires hotkey foundation from TASK 20) diff --git a/ai.md b/ai.md index 1a08d9c..075f7d3 100644 --- a/ai.md +++ b/ai.md @@ -167,6 +167,7 @@ C# / WPF (.NET 8) following MVVM: - **Audio is KISS by rule** — the whole of audio is *one knob*: **desktop/game audio is automatic** (WASAPI loopback from the default output at unity, zero UI — "it just is"); the **mic is the creator's only audio control** — sound meter + mute button + volume slider (`MicVolume`, defaults to 0.8) all sit together on the second line of the preview bottom row, BELOW the Socials+TRAX row, CENTERED beneath the preview panel. The TRAX button + Desktop Audio meter sit on the first line of that same row. Meter: 288px, muted slate track (`#3a3b52`) with ruler graduations and muted yellow/red zone tints at 60%/80%; fill = green → yellow → red via `MeterFillWidth`/`MeterBrush`; the meter is a **READ-ONLY realtime level display** — it shows the live input level scaled by the volume (raising the volume moves ambient noise up the bar), NOT the volume setting: the fill is `Math.Min(1, AudioLevelMeter.ToDisplay(AudioLevel) * MicVolume)` — `ToDisplay` maps the raw linear RMS onto a −60..0 dBFS display scale, because real speech sits around −40..−20 dBFS (0.01..0.1 linear) which would leave a flat scale dead (`AudioLevel` is fed by the audio mixer once capture lands, 0 with no input) and 0 while muted. While the volume slider is being dragged the bar previews the slider position (`SetVolumeAdjusting`, from `PreviewMouseLeftButtonDown/Up` + `LostMouseCapture` handlers) so the creator sees where they're setting it; on release it returns to the live level — with no input it bounces back to 0, exactly as it does today. Clicking the meter does nothing; **clicking the MIC label opens the mic picker** (`OpenMicPickerCommand`), and the picked voice source name (`MicSourceName`) is shown left-justified INSIDE the meter bar (FontSize 10, ellipsized to the bar) — the fill runs at 75% opacity so the text and the ruler markings stay visible through it. Mute (`ToggleMicMuteCommand`/`MicMuted`) is a plain clickable speaker icon (`MicSpeaker_MouseLeftButtonUp` code-behind handler — not a Button, `Stretch="Uniform"` so the glyph is never clipped) that swaps to a red do-not-symbol (slashed speaker) when muted. **The slider and the speaker can never disagree:** `MicMuted` is read-only, derived from `MicVolume == 0` — sliding the volume off flips the speaker to muted (storing the prior level in `_volumeBeforeMute`), sliding it up from 0 clears the mute indicator (and the stored level); the speaker button just runs the volume to 0 or restores it (default 0.8 if unknown). Muting zeroes the meter; **unmuting flashes the meter to the restored position for ~300ms** (`BeginVolumeFlash`/`EndVolumeFlash` on a DispatcherTimer, cancelled if the slider is grabbed) before it returns to the live level. Line 2 of the footer holds everything else: stream stats (bitrate/fps/dropped/duration/health) on the left, quality dropdown + gear on the right. The slider is a slim dimensional style in `Themes/Controls.xaml` (gradient track, beveled green fill on a 5px pill, gloss-sphere thumb with drop shadow — deliberately NOT flat). No device pickers (never show device names — no "install a device you didn't know existed"), no filter stacks, no monitoring, no routing — OBS's confusion (dynamic mixer, unintuitive names, four required filters) is deliberately absent. A production-ready mic chain (high-pass → noise gate → compressor) will be applied invisibly in the mixer, unconfigurable. Capture runs only while live (privacy indicator stays off otherwise). Capture pipeline = `IAudioSource` seam + NAudio `WasapiCapture`/`WasapiLoopbackCapture` + `AudioMixer` (pending — the UI is in place now). **Mic mute icon (2026-08-13):** a second 16px clickable glyph — a microphone, red + slash when muted — sits **between the meter and the speaker** (both mutes adjacent, spacing between the icons) and reuses the same `MicSpeaker_MouseLeftButtonUp` → `ToggleMicMuteCommand` handler. **REC / ON-AIR pills + signs (TASK 18, 2026-08-29):** the top-center area is now two **sliding pill toggles** (intent) next to two **status signs** (reality). REC pill (`RecordPillOn`) is the local-recording intent — works signed-out; ON-AIR pill (`OnAirPillOn`) is the streaming intent — greyed/disabled until `IsConnected`. The **REC sign** (`RecDotBrush`/`RecTextBrush`/`RecDotOpacity`) is dark-gray + dim offline and **green-tinged, pulsing only while actually recording** (`IsRecording`), not merely pill-on. The **ON-AIR sign** (`OnAirBrush` = `#555` offline, `#22c55e` live) is green while actually streaming (`IsLive`); the `PRIVATE` badge (`IsLivePrivate`) still shows when the broadcast is private. Both sign labels share the `StatusSignText` style (`Themes/Controls.xaml`) so REC and ON-AIR can't drift apart — status color lives on the dot only, never the label. The elapsed timer shows while `IsLive || IsRecording`. The connected account's avatar/name shows in the top bar next to the primary button (`AccountAvatarUrl`/`AccountDisplayName` via `SyncConnectedAccount`), so the creator always sees WHICH account will go live. - ViewModels are constructed in XAML (`` as DataContext) - **True decomposition beats partial-shuffling (2026-08-31, Commit G):** `MainViewModel` is a ~4100-line god-object split into 17 partials — but partials are a *myth of decomposition*: every partial shares the same class, same `SetProperty` state, same collaborators via `Scenes`/`StagedScene`/`IsLive`/`Source.VideoImageSource`, so an AI fetching one partial still reconstructs the whole class. The only thing that actually improves retrieval is a boundary where a feature **owns its own state + collaborators**. `Services/ChatOverlayLayer.cs` is the model (extracted from `MainViewModel.Chat.cs` 194→44): the layer owns the message buffer, `ChatBoxRenderer`, fade/mock timers, preview renders, and the live `RenderFrame`; the VM keeps only the binding surface (`ChatMessages` delegates to `_chatLayer.Messages` so XAML + LeftPanel `CollectionChanged` hold; the two computed gate props stay on the VM because their `OnPropertyChanged` is raised from VM setters). **Diagnostic — glue vs. component:** before extracting, count `OnPropertyChanged`/`SetProperty` touches (glue) and bound-property reads per partial. Zero/low-glue + a cohesive state blob (renderer/timers/buffer) = extract (chat). High glue over an already-extracted service (Audio, Webcam, Background, Scenes — `AudioMixer`/`CameraManager`/`ScreenCaptureManager`/`ChatBoxRenderer` already exist) = leave as VM glue; forcing it adds coupling, doesn't remove it. Don't manufacture seams to hit a line count — the line count is a guideline for context, not a design goal. +- **SceneGraph component + baked-crust compositor (TASK 31):** `SceneGraph` owns the scene collection, element inventory, and mutation surface. Elements are classified `Static` (images, background art) or `Dynamic` (webcam, live capture, chat). The compositor finds the split point (first dynamic element) and bakes all static layers below it into a cached `VideoFrame` — only dynamic layers and static layers above them are composited per frame. Scenes with zero dynamic elements are fully baked and the compositor is skipped. Pattern recorded in `TASKS.md`. - Services are currently instantiated in MainViewModel's constructor — no DI container yet - Layout persists to SQLite (`Microsoft.Data.Sqlite`); scenes/sources/asset bytes stored in the DB, asset identity is a SHA-256 content hash (1:M reuse, no file paths — assets are always available). Loaded sources always derive `IsBackground` from `Type` (OR'd with the persisted column, so legacy DisplayCapture backdrops keep their flag) — pre-derivation rows with `IsBackground=0` heal on load - **Five-scene catalog (`Models/SceneCatalog.cs`):** the product is exactly Starting/Live/BRB/Chat/Ending — work with less, never more (the escape hatch for "more" is OBS). Scenes are matched **by name** (`SceneCatalog.Is`, case-insensitive trim). Empty DBs seed all five; the scenes-header "+" (`ShowAddScene`/`MissingScenes` on `MainViewModel`) only appears while ≥1 canonical scene is missing and its menu lists only the missing ones, re-adding them by name (`AddSceneCommand`). Renaming a canonical scene makes it missing again; `AddScene` rejects non-canonical names.