fix: FramePump paces one frame per deadline slot — duplicate-on-lag, never skip (slice 15)
ty-1723/1726 device takes showed accelerated playback + audio tail cut-off:
a render overrun (~35ms vs the 16.6ms slot) SKIPPED the missed slots (slice 10's
freshness choice), so a 60fps-authoring pump wrote one frame per 35ms into a
60fps container — 1723: 697 frames/11.62s vs 11.84s audio; 1726: 163/2.72s vs
2.93s, video ending 0.21-0.24s early.
OBS never leaves a wall-time hole: the video thread emits one frame per tick and
a lagging producer DUPLICATES the newest frame ("lagged frames due to rendering
lag/stalls" — obs-output.c; "If the video frame queue is full, it will duplicate
the last frame" — docs.obsproject.com/backend-design). The pump's submit is now a
bounded catch-up over the missed slots (while now >= nextTick), fresh on the first,
repeated after — duration == wall, judder not fast-forward. Safe because Channel.
TryWrite never blocks (the take-9 smear was the blocking pipe-write; each emit is
nanoseconds). Burned frame index moved inside the loop: every emitted slot carries
its own +1 (also fixes the old unconditional pre-gate bump that gapped the judge
sequence on non-submitting fast-render iterations).
Good Dog test: Pump_Overrun_Renders_EmitsEverySlot_NotSkipped (60fps, 35ms render
cost, asserts >=0.65 of the wall slots emitted). 294/294 green, 0 warnings.
No push — web/A/V work is commit-local until greenlight.
This commit is contained in:
+60
-65
@@ -1,96 +1,91 @@
|
||||
# HANDOFF — 2026-09-14 (web capture REDESIGNED: composition-capture, PNG polling + CaptureScheduler deleted — NOT yet verified on device, NOT pushed)
|
||||
# HANDOFF — 2026-09-14 (composition capture DEVICE-VERIFIED ✓; slice-15 pacing fix committed locally — A/V re-measure next)
|
||||
|
||||
## Branch / Commit State
|
||||
|
||||
`main` HEAD = **`c01206f`** (web-frame-capture redesign — committed LOCALLY, **NOT pushed**; web work
|
||||
commit-locally-until-greenlight). Before it: `b22d08e` (signed audio-sync, already on origin/main).
|
||||
`main` HEAD = **slice-15 commit** (FramePump duplicate-on-lag pacing — committed LOCALLY, **NOT
|
||||
pushed**; web/A/V work stays commit-local until greenlight). Before it: `c01206f` (composition
|
||||
capture redesign, also local/unpushed), before that `b22d08e` (signed audio-sync, pushed).
|
||||
Working tree **clean**.
|
||||
|
||||
## ⚠️ Branding (2026-09-14, creator-corrected): product = **llamacasty**, internals = ytLive
|
||||
|
||||
The product is **llamacasty**; the repo path, csproj `AssemblyName`/`RootNamespace`, DB/log paths
|
||||
The product is **llamacasty**; repo path, csproj `AssemblyName`/`RootNamespace`, DB/log paths
|
||||
(`%APPDATA%\ytLlive\...`), and most code names are the legacy **ytLive/ytLlive**. User-facing
|
||||
language must say "llamacasty"; code/assembly/repo names stay ytLive. Full detail in
|
||||
`ai.md` → Brand → "Product name vs repo/assembly branding".
|
||||
language says "llamacasty"; code/assembly/repo names stay ytLive. See `ai.md` → Brand.
|
||||
|
||||
## 💥 The web-capture take history (why this redesign exists)
|
||||
## ✅ DEVICE-VERIFIED — composition capture looks good; then the A/V defect surfaced
|
||||
|
||||
The recording was 60fps but web widgets ran at ~1/6 speed: capture was a `CapturePreviewAsync` PNG
|
||||
poll; every full-HD encode+decode cost **35–165ms** so the session budget capped real cadence at
|
||||
~20Hz — a 60fps widget still juddered. Slice 11's `CaptureScheduler` (30Hz) was the wrong lever:
|
||||
the fix is frame-driven, not faster polling.
|
||||
The creator ran real takes off the `c01206f` build and reports the video looks **very good** — the
|
||||
Windows.Graphics.Capture web path is confirmed on device (open item closed). BUT the same takes
|
||||
showed the A/V defect we have seen before: **both desktop and webcam playback are accelerated, the
|
||||
audio "gets speedier", then cuts off at the end.**
|
||||
|
||||
## 🔬 Committed locally as `c01206f` — composition capture (NOT pushed)
|
||||
## 🔬 Sliced and measured (evidence)
|
||||
|
||||
**What:** web sources now render through `CoreWebView2CompositionController` into
|
||||
`Windows.Graphics.Capture` (the Flutter `webview_windows` / `WebView2CompositionControl`
|
||||
mechanism) — frame-driven at the renderer's pace instead of polling PNGs.
|
||||
With the A/V recipe (`MyMistakes.md` → "Measuring audio-video A/V sync" + `/tmp/opencode/avsync.py`):
|
||||
|
||||
**Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(appHwnd)` →
|
||||
`RootVisualTarget` = a `RelativeSizeAdjustment=1,1` child under a 1920×1080 root `ContainerVisual`
|
||||
on one `Compositor` (CoreMessaging DQ P/Invoke recipe — see ai.md Slice 14 / MyMistakes) →
|
||||
`GraphicsCaptureItem.CreateFromVisual(root)` → free-threaded `Direct3D11CaptureFramePool` (2
|
||||
buffers) → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (`BitmapAlphaMode.Straight`) →
|
||||
per-frame `FindContentBounds` bbox → `CropBounds` → epoch'd ring → dispatcher-coalesced crop copy
|
||||
into ONE shared `WriteableBitmap`.
|
||||
- `ty-20260914-1723-0000-2.mp4`: 697 video frames @60fps = **11.62s** vs audio **11.84s**.
|
||||
- `ty-20260914-1726-0000-2.mp4` (clap take): 163 frames = **2.72s** vs **2.93s** audio. Video ends
|
||||
0.21–0.24s before audio → the cut-off. Clap: audio env peak 1.655s vs video motion 0.75–1.03s;
|
||||
cross-correlation lag **+0.667s** (audio late).
|
||||
- `startup.log`: `FramePump stall: iteration 33-46ms (> 2× the 17ms interval): worst render 33ms …
|
||||
dropped 0`.
|
||||
|
||||
**Files:** NEW `Services/WebCaptureFrameSource.cs`; reworked `Services/WebView2Manager.cs` (internal
|
||||
seam ctor `(Dispatcher, Func<string, IScreenCaptureSource>?)` for hermetic tests); DELETED
|
||||
`Services/CaptureScheduler.cs`; `MainViewModel.Web.cs` (`InitWebView2()` no-arg),
|
||||
`MainViewModel.Streaming.Operations.cs` (three `SetCaptureInterval` hooks gone — verified zero
|
||||
remaining), `MainWindow.xaml(.cs)` (`InitWebView2()` moved ctor→**Loaded**; `WebViewHostPanel`
|
||||
overlay deleted). `TransparentBackgroundScript` unchanged.
|
||||
**Root cause (slice 15):** slice 10's freshness choice skipped the overrun's missed slots — one
|
||||
fresh frame per ~35ms stall authored into a 60fps container = **accelerated playback**. OBS's
|
||||
answer is duplicate-on-lag (`libobs/video-io.c`, docs.obsproject.com/backend-design): fill each
|
||||
missed slot by repeating the newest frame — duration == wall, judder never fast-forward.
|
||||
|
||||
**Tests:** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 control-size tests +
|
||||
`CaptureScheduler_Drops…`; `FindContentBounds` tests moved to `WebCaptureFrameSource`; the ONE
|
||||
integration test (`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds`) drives the
|
||||
seam with `FakeWebSource` + background-STA `DispatcherPump`: crop-sized preview published once →
|
||||
`CropBounds` + `IsOpaque=false` → back-to-back frames coalesce to latest. `RoundClipInteractionTests`
|
||||
comment updated (WebViewHostPanel gone). Suite **293/293 green**, app build **0 warnings**. Docs
|
||||
slant (ai.md Slice 14, HANDOFF, MyMistakes, TASK 17, Controls/ViewModels/Services indexes) updated in
|
||||
the same working set.
|
||||
## 🔬 Committed locally — slice 15: one frame per deadline slot (duplicate-on-lag)
|
||||
|
||||
## ⚠️ Open items on this change (before it is PUSHABLE)
|
||||
**What:** `Services/Encoder/FramePump.cs` submit is now a bounded catch-up —
|
||||
`while (now >= nextTick) { submit latest composite; nextTick += intervalTicks; }` clamped to a
|
||||
`deadlineNow` captured once per iteration. First missed slot gets the fresh composite, the rest get
|
||||
repeats of it (OBS duplication) — recording duration == wall time under any render load, no
|
||||
acceleration, no audio tail cut. The burned `_outputIndex` moved inside the submit loop (every
|
||||
emitted slot gets its own +1; also fixes the old unconditional bump that gapped the judge sequence
|
||||
on non-submitting fast-render iterations).
|
||||
|
||||
- **Device verification:** the composition path has never run against a real widget. Verify ~60fps
|
||||
web animation in a take (also: the OBS de-throttle flags stay; hidden-page JS throttling is fixed
|
||||
by them, capture pacing is now renderer-driven).
|
||||
- **No push yet** — web work is commit-local until the user says push.
|
||||
**Good Dog test:** `ytLive.Tests/FramePumpTests.cs` → `Pump_Overrun_Renders_EmitsEverySlot_NotSkipped`
|
||||
(60fps, 35ms render cost, asserts ≥0.65 of the wall slots emitted — the old skip-pump wrote ~1/35ms).
|
||||
**294/294 green, app build 0 warnings.** Files: `Services/Encoder/FramePump.cs`,
|
||||
`ytLive.Tests/FramePumpTests.cs`. Docs in same commit: ai.md Slice 15, MyMistakes (supersedes the
|
||||
slice-10 "no burst re-write" clause + latest numbers), this HANDOFF.
|
||||
|
||||
## ✅ SHIPPED + PUSHED — signed audio sync, −500..+500 (`b22d08e`)
|
||||
## ⚠️ Open items (before PUSHABLE)
|
||||
|
||||
Positive = delay the mix (audio AHEAD — the `AudioSyncDelay` line, live-reactive). Negative =
|
||||
**advance** (audio BEHIND): OBS-style "eat the stream head" — `AudioMixer.StartLive` arms
|
||||
`_advanceSamplesRemaining = |N| ms`, `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` off each
|
||||
write head while it lasts. Slider relabeled **AUDIO SYNC**, `−500..500`, locked while
|
||||
live/recording (`IsEditMode`). One regression test:
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead`.
|
||||
- **Device re-verify:** a take on the slice-15 build must play at REAL time (no acceleration, no
|
||||
cut-off audio tail). Confirm via ffprobe: video stream duration ≈ audio ≈ container.
|
||||
- **Re-measure the true A/V offset** with a clap take now that video pacing is honest
|
||||
(`/tmp/opencode/avsync.py`). The +0.6s reading on 1726 was confounded by the 1.1x
|
||||
acceleration. If a real residual remains after the pacing fix, the backlog is the AUDIO pipeline
|
||||
(mixer ring / advance), not video.
|
||||
- **No push yet** — commit-locally-until-greenlight for web/A/V work.
|
||||
|
||||
Take `ty-20260914-1128-0000-2.mp4`: **offset ≈ +0.54s** (collapsed from +2.11/2.22s; residual was
|
||||
the baked-in `SyncOffsetMs=300`, now 0 — confirmed via sqlite3). Both streams `start_time=0`.
|
||||
## Open threads (carried)
|
||||
|
||||
## Open threads
|
||||
|
||||
- **Verify signed sync negative direction on device** (clap take with a negative offset).
|
||||
- **Audio-silence verification** — fixed (`724af14`); creator heard real audio.
|
||||
- **Webcam MJPG missing / ~10–14Hz**, layer SortOrder, truncation-with-dynamic-scenes — queued.
|
||||
- **Sync control user-doc tutorial — REQUIRED before 1.0** (creator directive; TASK 22 file).
|
||||
- **Device verify the composition web capture (this change).**
|
||||
- Audio-silence verification — fixed (`724af14`); creator heard real audio.
|
||||
- Webcam MJPG missing / ~10–14Hz, layer SortOrder, truncation-with-dynamic-scenes — queued.
|
||||
- Sync control user-doc tutorial — REQUIRED before 1.0 (creator directive; TASK 22).
|
||||
- Signed A/V sync: verify the negative (advance) direction on device.
|
||||
|
||||
## Landmines
|
||||
|
||||
- testhost shares startup.log — filter by time.
|
||||
- `cmd.exe /c "taskkill /F /IM ytLive.exe"` (WSL double-slashes mangle) before rebuilds.
|
||||
- `cmd.exe /c "taskkill /F /IM ytLive.exe"` (WSL double-slashes mangle) before rebuilds — a live
|
||||
app process locks `ytLive.exe` and the apphost copy fails (MSB3021, seen today).
|
||||
- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings —
|
||||
only `./scripts/verify.sh "<files>"`'s clean build counts.
|
||||
only `./scripts/verify.sh "<files>"`'s clean build counts. Building `ytLive.csproj` alone does
|
||||
NOT rebuild `ytLive.Tests.dll` — run the Tests csproj before `vstest`.
|
||||
- ffmpeg/ffprobe: `/mnt/c/Program Files/Krita (x64)/bin/` with Windows paths.
|
||||
- `MyMistakes.md` has the **audio/video sync measurement recipe** AND the **composition-capture
|
||||
CoreMessaging DQ recipe (2026-09-14)** — grep before re-deriving.
|
||||
- sqlite3 lives at `/home/gramps/android-sdk/platform-tools/sqlite3` (WSL) for the DB at
|
||||
- `MyMistakes.md` has the **A/V sync measurement recipe**, the **deadline-pacing** lessons
|
||||
(item 1 + slice 15), and the **CoreMessaging DQ recipe** — grep before re-deriving.
|
||||
- sqlite3 at `/home/gramps/android-sdk/platform-tools/sqlite3` for
|
||||
`/mnt/c/Users/gramp/AppData/Roaming/ytLlive/ytLlive.db`.
|
||||
|
||||
## Next step
|
||||
|
||||
From this dirty state: run `./scripts/verify.sh "<all changed files>"`, review the diff, commit
|
||||
locally (web work — NO push, per standing directive). Then a device take to verify ~60fps web
|
||||
animation, then present for push review.
|
||||
Have the creator record a clap take on the slice-15 build → ffprobe durations (video == audio ==
|
||||
wall) + `/tmp/opencode/avsync.py` for the honest offset. If durations match, the acceleration
|
||||
defect is closed; then decide push with the user, and attack any true audio-latency residual as its
|
||||
own work unit.
|
||||
Reference in New Issue
Block a user