feat: web sources frame-captured via composition (CoreWebView2CompositionController → Windows.Graphics.Capture); PNG poll + CaptureScheduler deleted
The ~30Hz CapturePreviewAsync PNG poll capped real cadence at ~20Hz (35–165ms full-HD encode+decode), so a 60fps widget still juddered at ~1/6 speed. Replaced polling with frame-driven capture of the composition controller's root visual — the mechanism WebView2CompositionControl and Flutter's webview_windows use (graphics_context.cc captures the root surface_ visual via CreateGraphicsCaptureItemFromVisual; reference: github.com/microsoft/Windows.UI.Composition.WinUI / flutter-internal webview_windows). Frames now arrive at the renderer's own pace; capture memory is epoch'd ring reuse + one crop-sized shared WriteableBitmap. New Services/WebCaptureFrameSource.cs owns GraphicsCaptureItem + free- threaded Direct3D11CaptureFramePool + session (Straight alpha readback, per-frame FindContentBounds → CropBounds). WebView2Manager reworked around per-session composition controllers + one UI-thread Compositor created via the CoreMessaging CreateDispatcherQueueController P/Invoke (the 19041 projection lacks CreateOnCurrentThread); internal seam ctor (Dispatcher, Func<string,IScreenCaptureSource>?) for hermetic tests. CaptureScheduler.cs deleted; the three SetCaptureInterval cadence hooks removed; InitWebView2() moved from MainWindow ctor to Loaded (a parent HWND must exist for the composition controller); the hidden WebViewHostPanel overlay deleted. TransparentBackgroundScript unchanged. Tests: WebView2ManagerTests reworked — 4 control-size + scheduler tests dropped, FindContentBounds tests moved to WebCaptureFrameSource, ONE integration test (Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds) drives the seam with a FakeWebSource + background-STA DispatcherPump. Suite 293/293, 0 warnings. NOTE: composition path NOT yet verified on a device — the take is the next step. Web work committed locally only (no push per standing rule). Docs same-commit: ai.md Slice 14 + supersede marker on Slice 11, HANDOFF, MyMistakes (CoreMessaging DQ + namespace-landmine recipe), TASK 17, Controls/ViewModels/Services indexes.
This commit is contained in:
+63
-55
@@ -1,14 +1,10 @@
|
||||
# HANDOFF — 2026-09-14 (signed audio sync SHIPPED + PUSHED: −500..+500, negative advances by eating the stream head; slider locked while live/recording)
|
||||
# HANDOFF — 2026-09-14 (web capture REDESIGNED: composition-capture, PNG polling + CaptureScheduler deleted — NOT yet verified on device, NOT pushed)
|
||||
|
||||
## Branch / Commit State
|
||||
|
||||
`main` HEAD = **`b22d08e`** (signed audio-sync, committed AND **pushed** to origin/main with the
|
||||
user's greenlight — 29 commits pushed in one go). Working tree **clean**.
|
||||
|
||||
DB now: **`Audio.SyncOffsetMs = 0`** (confirmed via sqlite3 — the creator slid the sync to 0
|
||||
and the write finally stuck; sound is correct at 0). The 0.54s residual in the 1128 take ≈ 300ms
|
||||
injected offset (leftover DB value) + ~240ms natural (partly webcam clap-quantization ±40–85ms,
|
||||
partly measurement).
|
||||
`main` HEAD = **`b22d08e`** (signed audio-sync, already pushed). Working tree **DIRTY** with the
|
||||
web-frame-capture redesign (below). Commits for it land **LOCALLY only** — the web work was never
|
||||
pushed; the user greenlights pushes at checkpoints.
|
||||
|
||||
## ⚠️ Branding (2026-09-14, creator-corrected): product = **llamacasty**, internals = ytLive
|
||||
|
||||
@@ -17,72 +13,84 @@ The product is **llamacasty**; the repo path, csproj `AssemblyName`/`RootNamespa
|
||||
language must say "llamacasty"; code/assembly/repo names stay ytLive. Full detail in
|
||||
`ai.md` → Brand → "Product name vs repo/assembly branding".
|
||||
|
||||
## ✅ SHIPPED (this dirty tree) — signed audio sync, −500..+500
|
||||
## 💥 The web-capture take history (why this redesign exists)
|
||||
|
||||
**What:** the audio-sync control is now a SIGNED offset. Positive = delay the mix (audio runs
|
||||
AHEAD of video — existing `AudioSyncDelay` behavior, unchanged and live-reactive). Negative =
|
||||
**advance** the audio (audio runs BEHIND video): OBS's "eat the head of the buffer" fix —
|
||||
the mixer drops the first |N| ms of the written stream at the pipe, re-anchoring the audio
|
||||
stream so every event lands |N| ms EARLIER relative to video.
|
||||
The recording was 60fps but web widgets ran at ~1/6 speed: capture was a `CapturePreviewAsync` PNG
|
||||
poll; every full-HD encode+decode cost **35–165ms** so the session budget capped real cadence at
|
||||
~20Hz — a 60fps widget still juddered. Slice 11's `CaptureScheduler` (30Hz) was the wrong lever:
|
||||
the fix is frame-driven, not faster polling.
|
||||
|
||||
**Mechanism (AudioMixer):** `StartLive` arms `_advanceSamplesRemaining = |N| ms → samples` at
|
||||
go-live (a negative offset can only eat the HEAD of the stream; it is armed once, not
|
||||
live-reactive). `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` samples off each write
|
||||
head while the budget lasts — the pipe writer accepts a partial chunk via `AsMemory(writeFrom)`.
|
||||
Positive path untouched (delay line still re-reads the Func every tick).
|
||||
## 🔬 IN FLIGHT (this dirty tree, NOT committed, NOT pushed) — composition capture
|
||||
|
||||
**UI/plumbing:**
|
||||
- `MainViewModel.Audio.cs` — clamp `Math.Clamp(value, -500, 500)`, doc updated.
|
||||
- `LayoutStore.Settings.cs` — `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs` clamp −500..500.
|
||||
- `PreviewPane.xaml` — label **SYNC → "AUDIO SYNC"**, `Minimum="-500"`, tooltip explains both
|
||||
directions (calibrate with a clap: clap late → negative; early → positive), and
|
||||
**`IsEnabled="{Binding IsEditMode}"`** — the slider locks during live AND recording (gun
|
||||
safety, same property `IsRecording`/`StreamStatus` already raise PropertyChanged for).
|
||||
- `AudioSyncDelay` unchanged (still clamps negative→0 internally; header doc updated to point
|
||||
at the mixer for the advance side).
|
||||
**What:** web sources now render through `CoreWebView2CompositionController` into
|
||||
`Windows.Graphics.Capture` (the Flutter `webview_windows` / `WebView2CompositionControl`
|
||||
mechanism) — frame-driven at the renderer's pace instead of polling PNGs.
|
||||
|
||||
**Regression test (the ONE integration test for this change):**
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead` in `AudioPipelineTests.cs` —
|
||||
−40 ms advance (8-tick budget at 5ms interval), emits 6×0.9 fresh right after StartLive (≤
|
||||
budget, so the head MUST be eaten) then a long 0.2 bed; asserts the wire max ≈ 0.2 (< 0.3).
|
||||
Fails loudly if −N no longer drops the head (0.9 leaks).
|
||||
**Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(appHwnd)` →
|
||||
`RootVisualTarget` = a `RelativeSizeAdjustment=1,1` child under a 1920×1080 root `ContainerVisual`
|
||||
on one `Compositor` (CoreMessaging DQ P/Invoke recipe — see ai.md Slice 14 / MyMistakes) →
|
||||
`GraphicsCaptureItem.CreateFromVisual(root)` → free-threaded `Direct3D11CaptureFramePool` (2
|
||||
buffers) → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (`BitmapAlphaMode.Straight`) →
|
||||
per-frame `FindContentBounds` bbox → `CropBounds` → epoch'd ring → dispatcher-coalesced crop copy
|
||||
into ONE shared `WriteableBitmap`.
|
||||
|
||||
**User directives (this session):**
|
||||
- Signed −500..+500 with positive=delay / negative=advance, relabel "AUDIO SYNC", default 0.
|
||||
- **Lock the sync control when live or recording** (done — `IsEditMode`).
|
||||
- **NOTE ONLY, no fix:** the live recording is completely different from the "recording
|
||||
results" shown in Chat view (recorded in `bugs.md` — do not rediscover as a surprise).
|
||||
- **Before 1.0:** write a detailed USER-DOC tutorial on the audio-sync feature (see TASKS.md
|
||||
note; add it to the gold-pass/1.0 checklist). `docs/` currently holds only the README image.
|
||||
**Files:** NEW `Services/WebCaptureFrameSource.cs`; reworked `Services/WebView2Manager.cs` (internal
|
||||
seam ctor `(Dispatcher, Func<string, IScreenCaptureSource>?)` for hermetic tests); DELETED
|
||||
`Services/CaptureScheduler.cs`; `MainViewModel.Web.cs` (`InitWebView2()` no-arg),
|
||||
`MainViewModel.Streaming.Operations.cs` (three `SetCaptureInterval` hooks gone — verified zero
|
||||
remaining), `MainWindow.xaml(.cs)` (`InitWebView2()` moved ctor→**Loaded**; `WebViewHostPanel`
|
||||
overlay deleted). `TransparentBackgroundScript` unchanged.
|
||||
|
||||
## Take verification so far (ring-backlog fix)
|
||||
**Tests:** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 control-size tests +
|
||||
`CaptureScheduler_Drops…`; `FindContentBounds` tests moved to `WebCaptureFrameSource`; the ONE
|
||||
integration test (`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds`) drives the
|
||||
seam with `FakeWebSource` + background-STA `DispatcherPump`: crop-sized preview published once →
|
||||
`CropBounds` + `IsOpaque=false` → back-to-back frames coalesce to latest. `RoundClipInteractionTests`
|
||||
comment updated (WebViewHostPanel gone). Suite **293/293 green**, app build **0 warnings**. Docs
|
||||
slant (ai.md Slice 14, HANDOFF, MyMistakes, TASK 17, Controls/ViewModels/Services indexes) updated in
|
||||
the same working set.
|
||||
|
||||
Take `ty-20260914-1128-0000-2.mp4` (1920×1080@60fps, 356 frames / 5.93s): audio clap RMS peak
|
||||
3.260s; video diff-frame 163 → 2.717s; **offset ≈ +0.54s** — collapsed from +2.11/2.22s; the
|
||||
remaining ~300ms was the baked-in `SyncOffsetMs=300` (now 0). Both streams `start_time=0` — not
|
||||
an `avoid_negative_ts` artifact.
|
||||
## ⚠️ Open items on this change (before it is PUSHABLE)
|
||||
|
||||
- **Device verification:** the composition path has never run against a real widget. Verify ~60fps
|
||||
web animation in a take (also: the OBS de-throttle flags stay; hidden-page JS throttling is fixed
|
||||
by them, capture pacing is now renderer-driven).
|
||||
- **No push yet** — web work is commit-local until the user says push.
|
||||
|
||||
## ✅ SHIPPED + PUSHED — signed audio sync, −500..+500 (`b22d08e`)
|
||||
|
||||
Positive = delay the mix (audio AHEAD — the `AudioSyncDelay` line, live-reactive). Negative =
|
||||
**advance** (audio BEHIND): OBS-style "eat the stream head" — `AudioMixer.StartLive` arms
|
||||
`_advanceSamplesRemaining = |N| ms`, `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` off each
|
||||
write head while it lasts. Slider relabeled **AUDIO SYNC**, `−500..500`, locked while
|
||||
live/recording (`IsEditMode`). One regression test:
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead`.
|
||||
|
||||
Take `ty-20260914-1128-0000-2.mp4`: **offset ≈ +0.54s** (collapsed from +2.11/2.22s; residual was
|
||||
the baked-in `SyncOffsetMs=300`, now 0 — confirmed via sqlite3). Both streams `start_time=0`.
|
||||
|
||||
## Open threads
|
||||
|
||||
- **Verify signed sync on device:** re-run the clap take with a NEGATIVE offset to confirm the
|
||||
advance direction end-to-end (the regression test proves the mixer; a take proves the file).
|
||||
- **Audio-silence verification** — fixed code (`724af14`) confirmed; creator heard real audio.
|
||||
- **Verify signed sync negative direction on device** (clap take with a negative offset).
|
||||
- **Audio-silence verification** — fixed (`724af14`); creator heard real audio.
|
||||
- **Webcam MJPG missing / ~10–14Hz**, layer SortOrder, truncation-with-dynamic-scenes — queued.
|
||||
- **Sync control tutorial in user docs — REQUIRED before 1.0** (creator directive).
|
||||
- **Sync control user-doc tutorial — REQUIRED before 1.0** (creator directive; TASK 22 file).
|
||||
- **Device verify the composition web capture (this change).**
|
||||
|
||||
## Landmines
|
||||
|
||||
- testhost shares startup.log — filter by time.
|
||||
- `cmd.exe /c "taskkill /F /IM ytLive.exe"` (WSL double-slashes mangle) before rebuilds.
|
||||
- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings.
|
||||
- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings —
|
||||
only `./scripts/verify.sh "<files>"`'s clean build counts.
|
||||
- ffmpeg/ffprobe: `/mnt/c/Program Files/Krita (x64)/bin/` with Windows paths.
|
||||
- `MyMistakes.md` has the **audio/video sync measurement recipe** (claps + cross-correlation)
|
||||
— grep it before re-deriving.
|
||||
- `MyMistakes.md` has the **audio/video sync measurement recipe** AND the **composition-capture
|
||||
CoreMessaging DQ recipe (2026-09-14)** — grep before re-deriving.
|
||||
- sqlite3 lives at `/home/gramps/android-sdk/platform-tools/sqlite3` (WSL) for the DB at
|
||||
`/mnt/c/Users/gramp/AppData/Roaming/ytLlive/ytLlive.db`.
|
||||
|
||||
## Next step
|
||||
|
||||
Open queue question: user-doc tutorial for the audio-sync feature (creator directive, REQUIRED
|
||||
before 1.0 — recorded in `TASKS/task-22-audio-sync-offset.md`). Also asked 2026-09-14 whether
|
||||
other root md files need the TASKS-style split — answered: no (see below for the criterion).
|
||||
From this dirty state: run `./scripts/verify.sh "<all changed files>"`, review the diff, commit
|
||||
locally (web work — NO push, per standing directive). Then a device take to verify ~60fps web
|
||||
animation, then present for push review.
|
||||
Reference in New Issue
Block a user