LlamaCasty/TASKS/task-31-scenegraph-optimization.md

TASK 31 — SceneGraph component + static/dynamic compositor optimization

Catalog: TASKS.md — status and requirements live here.

Goal: extract the scene collection, element inventory, and mutation surface into a standalone SceneGraph component. Classify elements as static or dynamic. Enable the compositor to bake static layers once and only recompose dynamic layers per frame.

Status: ✅ Done (2026-08-31)

What landed — core optimization + SceneGraph component, all in one commit:

• ElementKind (Static/Dynamic) on SceneElement base, overridden in Source and WebcamSceneConfig
• Services/SceneGraph.cs: owns Scenes collection (ViewModel's Scenes property delegates to it); mutation surface AddElement/InsertElement/RemoveElement/MoveElement (each invalidates the bake); queries GetBackground/GetWebcam/GetChatBoxes/GetSplitPoint/IsStatic; and the GetBakedBase bake-cache (keyed by scene id + static element identities below the split)
• SceneCompositor.BakeStaticBase/CompositeLayers + Render(.., staticBase, split) overload — split-aware, builds the cached base in source-rect space, composites dynamic/above-split per frame
• FramePump takes an optional SceneGraph and uses the optimized path when wired (falls back to full render)
• MainViewModel: routes element mutations through the graph; invalidates the bake on static layout/opacity/visibility/useDefaultBackground changes and after background heal / EnsureBackground
• Integration test SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly (audio-free, verifies bake-once + cache hit + dynamic-on-top pixels + static-mutation invalidation + dynamic-only non-invalidation)

Defensive deviations from this spec (decision 2026-08-31):

1. ChatOverlayLayer keeps taking IEnumerable<Scene> rather than SceneGraph.GetChatBoxes() — it is deliberately decoupled from the graph (// without owning the scene graph). Forcing the graph in would couple a WPF-bound layer to it and violate that documented seam.
2. Background static helpers (EnsureBackground/CreateBackground/NormalizeBackgrounds) stay on the ViewModel because BackgroundTests.cs unit-tests MainViewModel.EnsureBackground directly. Queries moved to SceneGraph; helpers + their invalidation wiring stayed on the VM.
3. The full ViewModel-as-facade extraction (moving the ~all-binding-surface off Scenes/StagedScene/ LiveScene) was deliberately NOT done in this unsupervised pass — it is the "touch 11 files across 4 layers" regression risk the handoff flagged. SceneGraph owns the collection + mutation surface now; the remaining VM call sites binding to those still work because Scenes delegates. Revisit after 1.0.

Verification: SceneCompositorTests 4, StretchMathTests 4, BackgroundTests 16, SceneCatalogTests 18, LayoutStorePersistenceTests 12, FramePumpTests 9, SceneGraphTests 1 all green. RealAppHost GUI/collection tests (SourceNaming, RoundClip, BackgroundHeal, ...) construct a real MainWindow — CORRECTED 2026-09-01: they DO run from WSL when invoked per-class through the Windows dotnet.exe vstest host (the old "cannot run headless" claim conflated them with the full-suite WASAPI hang). Two first-launch crashes this refactor shipped with were caught on the first real native launch and fixed same day: ctor-order SceneGraph NRE (null! field assigned after first ctor use — now field-initialized) and window-scope EyeButton/EyeIconStyle consumed via StaticResource from the extracted LeftPanel (UserControl namescopes can't see window resources — styles moved to Themes/Controls.xaml, the app-scope rule honored). The RoundClip "known failure" was then root-caused to stale test code (window.FindName across namescopes + VisualTreeHelper.HitTest where UIElement.InputHitTest models input) — test green 2026-09-01, the sole remaining known failure is the audio one.

Design

Element classification:

Every SceneElement exposes ElementKind Kind — Static or Dynamic:

Type	Kind	Why
Source (Background art)	Static	Pixels don't change at runtime
Source (Image)	Static	Pixels don't change at runtime
Source (DisplayCapture)	Dynamic	Live game/desktop feed
Source (ChatBox)	Dynamic	Live chat messages arrive continuously
Source (WebSource)	Dynamic	WebView content can change
WebcamSceneConfig	Dynamic	Camera feed, changes every frame

Split point:

The split point is the index of the first dynamic element in a scene's z-ordered Elements. Everything below it = baked base. Everything from it upward (including static layers above dynamic elements) = composited per frame. If zero dynamic elements exist: entire scene is baked, compositor skipped entirely.

BakedSceneCache:

BakedSceneCache {
    VideoFrame BaseFrame           // composited static layers below split point
    int Version                    // incremented on mutation
    SceneElement[] BakedElements   // for invalidation tracking
}

Invalidation triggers (re-bake): element added/removed/reordered, static element moved/resized/ opacity changed, IsVisible toggled on a static element, background asset changed, scene switch.

No invalidation when: only a dynamic element's pixels changed (webcam frame, chat message), or a dynamic element moved/resized (per-frame compositing concern, not a base rebake).

Compositor integration:

Render(scene, frameFor, options):
    1. Find split point (first Dynamic element index)
    2. If split == scene.Elements.Count:
        → return cached base (or bake if stale)
    3. Otherwise:
        → start from cached base (or bake if stale)
        → composite elements[split..] on top using frameFor

SceneGraph interface:

SceneGraph {
    // Collection
    ObservableCollection<Scene> Scenes
    Scene? StagedScene
    Scene? LiveScene

    // Mutation surface (single owner of element operations)
    AddElement(scene, element)
    RemoveElement(scene, element)
    MoveElement(scene, element, newIndex)

    // Queries (replace scattered LINQ)
    GetBackground(scene): Source?
    GetWebcam(scene): WebcamSceneConfig?
    GetChatBoxes(): IEnumerable<Source>
    GetSplitPoint(scene): int
    IsStatic(scene): bool

    // Events
    SceneChanged
    ElementAdded/Removed/Moved
    SplitPointChanged
}

What moves into SceneGraph:

Current location	Moves to
MainViewModel.Scenes.cs — Scenes collection, StagedScene, scene switching	SceneGraph
MainViewModel.Background.cs — NormalizeBackgrounds, EnsureBackground, background queries	SceneGraph (background helpers)
MainViewModel.Sources.cs — AddSource, RemoveElement, element inventory	SceneGraph (mutation surface)
MainViewModel.Webcam.cs — webcam queries across scenes	SceneGraph.GetWebcam(scene)
ChatOverlayLayer — scenes.SelectMany(...).OfType<Source>().Where(ChatBox)	SceneGraph.GetChatBoxes()

What stays on the ViewModel:

The binding surface — StagedScene setter still raises OnPropertyChanged for ShowEmptySceneHint, CanAddWebcam, etc. But now it delegates to SceneGraph for actual state queries. ViewModel becomes a thin binding facade over the SceneGraph (same pattern as MainViewModel.Chat.cs over ChatOverlayLayer).

The compositor resolver (frameFor callback) stays in the ViewModel/encoder layer — it bridges WPF concepts (CameraManager, ScreenCaptureManager, ImageCache) into the pure VideoFrame seam. SceneGraph doesn't know about capture sessions; it just knows element kinds.

Migration path:

1. Extract SceneGraph as a standalone class in Services/
2. Move Scenes collection + StagedScene/LiveScene + wiring
3. Move element mutation surface (add/remove/move)
4. Move background normalization/queries
5. Add ElementKind to SceneElement base
6. Add GetSplitPoint + BakedSceneCache
7. Update compositor to use split point
8. Wire ViewModel as thin facade
9. One integration test: baked static base + dynamic layer composite

Tests

ONE integration test: SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly — a scene with static background + image + dynamic webcam; assert the baked base is cached (renders once), dynamic layer composited on top, static element mutation invalidates the cache, dynamic-only pixel change does not.