docs: restructure TASKS.md into a catalog — one file per task in TASKS/
TASKS.md is now the index (status table, open items, research pointer). 33 files: 32 task files + 1 research facts file. The full take-saga narrative and all design decisions are preserved verbatim; the catalog makes the queue readable without opening every task body. Schema and AGENTS.md updated to reflect the new layout.
This commit is contained in:
@@ -0,0 +1,170 @@
|
||||
# TASK 31 — SceneGraph component + static/dynamic compositor optimization
|
||||
|
||||
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
||||
|
||||
|
||||
**Goal:** extract the scene collection, element inventory, and mutation surface into a standalone
|
||||
`SceneGraph` component. Classify elements as static or dynamic. Enable the compositor to bake
|
||||
static layers once and only recompose dynamic layers per frame.
|
||||
|
||||
### Status: ✅ Done (2026-08-31)
|
||||
|
||||
What landed — core optimization + SceneGraph component, all in one commit:
|
||||
|
||||
- `ElementKind` (Static/Dynamic) on `SceneElement` base, overridden in `Source` and `WebcamSceneConfig`
|
||||
- `Services/SceneGraph.cs`: owns `Scenes` collection (ViewModel's `Scenes` property delegates to it);
|
||||
mutation surface `AddElement`/`InsertElement`/`RemoveElement`/`MoveElement` (each invalidates the bake);
|
||||
queries `GetBackground`/`GetWebcam`/`GetChatBoxes`/`GetSplitPoint`/`IsStatic`; and the
|
||||
`GetBakedBase` bake-cache (keyed by scene id + static element identities below the split)
|
||||
- `SceneCompositor.BakeStaticBase`/`CompositeLayers` + `Render(.., staticBase, split)` overload —
|
||||
split-aware, builds the cached base in source-rect space, composites dynamic/above-split per frame
|
||||
- `FramePump` takes an optional `SceneGraph` and uses the optimized path when wired (falls back to full render)
|
||||
- `MainViewModel`: routes element mutations through the graph; invalidates the bake on static
|
||||
layout/opacity/visibility/useDefaultBackground changes and after background heal / EnsureBackground
|
||||
- Integration test `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly` (audio-free,
|
||||
verifies bake-once + cache hit + dynamic-on-top pixels + static-mutation invalidation +
|
||||
dynamic-only non-invalidation)
|
||||
|
||||
**Defensive deviations from this spec (decision 2026-08-31):**
|
||||
1. `ChatOverlayLayer` keeps taking `IEnumerable<Scene>` rather than `SceneGraph.GetChatBoxes()` — it is
|
||||
deliberately decoupled from the graph (`// without owning the scene graph`). Forcing the graph in would
|
||||
couple a WPF-bound layer to it and violate that documented seam.
|
||||
2. Background static helpers (`EnsureBackground`/`CreateBackground`/`NormalizeBackgrounds`) stay on the
|
||||
ViewModel because `BackgroundTests.cs` unit-tests `MainViewModel.EnsureBackground` directly. Queries
|
||||
moved to SceneGraph; helpers + their invalidation wiring stayed on the VM.
|
||||
3. The full ViewModel-as-facade extraction (moving the ~all-binding-surface off Scenes/StagedScene/
|
||||
LiveScene) was deliberately NOT done in this unsupervised pass — it is the "touch 11 files across 4
|
||||
layers" regression risk the handoff flagged. SceneGraph owns the collection + mutation surface now;
|
||||
the remaining VM call sites binding to those still work because `Scenes` delegates. Revisit after 1.0.
|
||||
|
||||
**Verification:** SceneCompositorTests 4, StretchMathTests 4, BackgroundTests 16, SceneCatalogTests 18,
|
||||
LayoutStorePersistenceTests 12, FramePumpTests 9, SceneGraphTests 1 all green. RealAppHost GUI/collection
|
||||
tests (SourceNaming, RoundClip, BackgroundHeal, ...) construct a real `MainWindow` — **CORRECTED
|
||||
2026-09-01:** they DO run from WSL when invoked per-class through the Windows `dotnet.exe` vstest host
|
||||
(the old "cannot run headless" claim conflated them with the full-suite WASAPI hang). Two first-launch
|
||||
crashes this refactor shipped with were caught on the first real native launch and fixed same day:
|
||||
ctor-order `SceneGraph` NRE (`null!` field assigned after first ctor use — now field-initialized) and
|
||||
window-scope `EyeButton`/`EyeIconStyle` consumed via `StaticResource` from the extracted `LeftPanel`
|
||||
(UserControl namescopes can't see window resources — styles moved to `Themes/Controls.xaml`, the
|
||||
app-scope rule honored). The RoundClip "known failure" was then root-caused to stale test code
|
||||
(`window.FindName` across namescopes + `VisualTreeHelper.HitTest` where `UIElement.InputHitTest`
|
||||
models input) — test green 2026-09-01, the sole remaining known failure is the audio one.
|
||||
|
||||
### Design
|
||||
|
||||
**Element classification:**
|
||||
|
||||
Every `SceneElement` exposes `ElementKind Kind` — `Static` or `Dynamic`:
|
||||
|
||||
| Type | Kind | Why |
|
||||
|------|------|-----|
|
||||
| `Source` (Background art) | Static | Pixels don't change at runtime |
|
||||
| `Source` (Image) | Static | Pixels don't change at runtime |
|
||||
| `Source` (DisplayCapture) | Dynamic | Live game/desktop feed |
|
||||
| `Source` (ChatBox) | Dynamic | Live chat messages arrive continuously |
|
||||
| `Source` (WebSource) | Dynamic | WebView content can change |
|
||||
| `WebcamSceneConfig` | Dynamic | Camera feed, changes every frame |
|
||||
|
||||
**Split point:**
|
||||
|
||||
The **split point** is the index of the first dynamic element in a scene's z-ordered `Elements`.
|
||||
Everything below it = baked base. Everything from it upward (including static layers above dynamic
|
||||
elements) = composited per frame. If zero dynamic elements exist: entire scene is baked,
|
||||
compositor skipped entirely.
|
||||
|
||||
**BakedSceneCache:**
|
||||
|
||||
```
|
||||
BakedSceneCache {
|
||||
VideoFrame BaseFrame // composited static layers below split point
|
||||
int Version // incremented on mutation
|
||||
SceneElement[] BakedElements // for invalidation tracking
|
||||
}
|
||||
```
|
||||
|
||||
Invalidation triggers (re-bake): element added/removed/reordered, static element moved/resized/
|
||||
opacity changed, `IsVisible` toggled on a static element, background asset changed, scene switch.
|
||||
|
||||
No invalidation when: only a dynamic element's pixels changed (webcam frame, chat message),
|
||||
or a dynamic element moved/resized (per-frame compositing concern, not a base rebake).
|
||||
|
||||
**Compositor integration:**
|
||||
|
||||
```
|
||||
Render(scene, frameFor, options):
|
||||
1. Find split point (first Dynamic element index)
|
||||
2. If split == scene.Elements.Count:
|
||||
→ return cached base (or bake if stale)
|
||||
3. Otherwise:
|
||||
→ start from cached base (or bake if stale)
|
||||
→ composite elements[split..] on top using frameFor
|
||||
```
|
||||
|
||||
**SceneGraph interface:**
|
||||
|
||||
```
|
||||
SceneGraph {
|
||||
// Collection
|
||||
ObservableCollection<Scene> Scenes
|
||||
Scene? StagedScene
|
||||
Scene? LiveScene
|
||||
|
||||
// Mutation surface (single owner of element operations)
|
||||
AddElement(scene, element)
|
||||
RemoveElement(scene, element)
|
||||
MoveElement(scene, element, newIndex)
|
||||
|
||||
// Queries (replace scattered LINQ)
|
||||
GetBackground(scene): Source?
|
||||
GetWebcam(scene): WebcamSceneConfig?
|
||||
GetChatBoxes(): IEnumerable<Source>
|
||||
GetSplitPoint(scene): int
|
||||
IsStatic(scene): bool
|
||||
|
||||
// Events
|
||||
SceneChanged
|
||||
ElementAdded/Removed/Moved
|
||||
SplitPointChanged
|
||||
}
|
||||
```
|
||||
|
||||
**What moves into SceneGraph:**
|
||||
|
||||
| Current location | Moves to |
|
||||
|-----------------|----------|
|
||||
| `MainViewModel.Scenes.cs` — Scenes collection, StagedScene, scene switching | `SceneGraph` |
|
||||
| `MainViewModel.Background.cs` — `NormalizeBackgrounds`, `EnsureBackground`, background queries | `SceneGraph` (background helpers) |
|
||||
| `MainViewModel.Sources.cs` — `AddSource`, `RemoveElement`, element inventory | `SceneGraph` (mutation surface) |
|
||||
| `MainViewModel.Webcam.cs` — webcam queries across scenes | `SceneGraph.GetWebcam(scene)` |
|
||||
| `ChatOverlayLayer` — `scenes.SelectMany(...).OfType<Source>().Where(ChatBox)` | `SceneGraph.GetChatBoxes()` |
|
||||
|
||||
**What stays on the ViewModel:**
|
||||
|
||||
The binding surface — `StagedScene` setter still raises `OnPropertyChanged` for
|
||||
`ShowEmptySceneHint`, `CanAddWebcam`, etc. But now it delegates to `SceneGraph` for actual
|
||||
state queries. ViewModel becomes a thin binding facade over the SceneGraph (same pattern as
|
||||
`MainViewModel.Chat.cs` over `ChatOverlayLayer`).
|
||||
|
||||
The compositor resolver (`frameFor` callback) stays in the ViewModel/encoder layer — it bridges
|
||||
WPF concepts (CameraManager, ScreenCaptureManager, ImageCache) into the pure `VideoFrame` seam.
|
||||
SceneGraph doesn't know about capture sessions; it just knows element kinds.
|
||||
|
||||
**Migration path:**
|
||||
|
||||
1. Extract `SceneGraph` as a standalone class in `Services/`
|
||||
2. Move `Scenes` collection + `StagedScene`/`LiveScene` + wiring
|
||||
3. Move element mutation surface (add/remove/move)
|
||||
4. Move background normalization/queries
|
||||
5. Add `ElementKind` to `SceneElement` base
|
||||
6. Add `GetSplitPoint` + `BakedSceneCache`
|
||||
7. Update compositor to use split point
|
||||
8. Wire ViewModel as thin facade
|
||||
9. One integration test: baked static base + dynamic layer composite
|
||||
|
||||
### Tests
|
||||
|
||||
ONE integration test: `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly`
|
||||
— a scene with static background + image + dynamic webcam; assert the baked base is cached
|
||||
(renders once), dynamic layer composited on top, static element mutation invalidates the cache,
|
||||
dynamic-only pixel change does not.
|
||||
|
||||
Reference in New Issue
Block a user