diff --git a/HANDOFF.md b/HANDOFF.md index 4c7dde0..9c150d2 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -10,6 +10,46 @@ Unpushed: `5ad0106` broadcast diagnostic → `74da99c` + `0fb2f9c` privacy/Test `8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement → `b2f2197` handoff → `UNIT 0` frame-cadence measurement. +## 🚨 ANSWERED 2026-10-05 — the judder is the COMPOSITOR, not the camera. Unit 1 is the wrong next move. + +Take `ty-20261005-1201-556-0001.mp4` (75.3s, 1920×1080, 60fps) + `startup.log`, read together: + +| Counter | Value | Reading | +|---|---|---| +| `Camera cadence … arrivals` | **30.0/s, 0 gaps ≥50ms**, avg 33.4ms | the camera is **perfect**. Not the culprit. | +| `Camera cadence … camera` (device stamps) | 30.0/s, identical | the USB link and driver are clean too | +| `FramePump … from N renders` | **34 renders per 5s = 6.8/s**, avg render **141ms**, worst 294ms | one full composite takes longer than four camera frames | +| `webcam …` on the pump's line | 13.0/s, 34 gaps ≥50ms | only 13 of the 30 delivered frames ever reached a composite | +| the recording itself | **6.8 distinct webcam images/s, 88.7% duplicates, ZERO arrival hitches** | the judder is *only* us repeating a frame — nothing arrives late | + +**⇒ The camera delivers all 30 of its frames, evenly. The compositor can only draw ~7 composites a +second, so the 60fps container is 89% repeats of a face held for 140ms.** That is exactly what a +lipreader experiences as "cutting out". + +**⛔ Do NOT do Unit 1 (720p / exclusive lock) as the judder fix — it is the wrong lever.** 720p would +triple the bytes the compositor stretches per render on a path that is already 4.4× over its budget. +The exclusive lock is still worth having **for sharpness** (the C920 is pinned at 640×480 YUY2 and +the Chat scene upscales that ~2.1×), but it must be scheduled as a quality win, not a smoothness win, +and it should land **after** the compositor work — otherwise it makes the judder measurably worse. + +**The number that needs explaining:** the 10:01 take hit **25.6** distinct images/s; this one hit +**6.8**. Same scene shape both times (`full-render split=0 elements=6 dynamic=5`), same camera, same +640×480. Composites went **29–34ms at 10:03 → 143–155ms at 12:01**, a 4.5× regression between +sessions with nothing in the log explaining it. `resolve` is 0.1ms, so the cost is pure pixel work, +not frame lookup. **That regression is the next thing to find, and the log cannot say what changed** — +no scene dump is written at stream start. ⛔ Do not guess at it. + +**Cheapest next step, and it is a measurement not a fix:** log the per-element composite breakdown +(size + ms for each of the 6 elements) plus the scene contents at pump start. One run names which +element eats 140ms and whether the 10:03→12:01 regression was a scene change or something else. +Everything else in the compositor is guesswork until that exists. + +**Also worth knowing (a counter subtlety, so nobody re-derives it):** the pump's `webcam` line reads +13/s while the file shows 6.8/s. Not a discrepancy — the counter stamps a frame when the RESOLVER +first sees it, and one composite takes 141ms, so ~4 camera frames arrive during it: the signature +pass records one epoch and the actual draw uses a later one. **The file measurement is the truth; +the pump counter overcounts by design** (it is a pipeline-health signal, not a frame-delivery metric). + ## 🚧 IN FLIGHT 2026-10-05 — UNIT 0 shipped: a dropped webcam frame is finally measurable **The measurement the map never had.** Take `ty-20261005-1001-821-0001.mp4` (85.6s, 1920×1080, diff --git a/TASKS.md b/TASKS.md index 4a5c4e2..2537a38 100644 --- a/TASKS.md +++ b/TASKS.md @@ -160,13 +160,22 @@ second encoder path needing a "redirect". Record+simulcast is one ffmpeg with tw webcam judders, ~1 hitch in 5, invisible in the output file). **UNIT 0 shipped** (counters + `Helpers/FrameCadence.cs`; see `ai.md` → "Frame cadence"). **Needs a creator recording before the next unit.** Remaining units, all agreed 2026-10-05, one Good Dog test each: - - **Unit 1:** `MediaCaptureSharingMode.ExclusiveControl` + MJPG ≥30fps up to 1280×720 via the - existing `SetMediaStreamPropertiesAsync` path (refused only because `SharedReadOnly`); same-camera - `SharedReadOnly` retry. **The creator reversed the ban on taking an exclusive camera lock.** - API probe already done — see HANDOFF.md, incl. the `SetFormatAsync` dead end. - - **Unit 2:** blinking amber **"SD" badge**, preview only, never in the recording/stream, when a - better format was available and not obtained. Top-right corner, or the 1:30 rim point when the - clip is round. + - **⛔ UNIT 1 IS NOT NEXT — re-scoped 2026-10-05.** The first real reading exonerated the camera: + it delivers 30.0/s with zero hitches; the compositor manages **6.8 composites/s at ~141ms each**, + so the file is **88.7% duplicate frames** and that is the judder. 720p triples the bytes stretched + per render on an already-over-budget path. Keep `ExclusiveControl` + 720p as a **sharpness** win + and schedule it AFTER the compositor work. + - **⏭ ACTUAL NEXT: per-element composite timing + a scene dump at pump start.** One run names which + of the 6 elements eats 140ms and explains the 10:03→12:01 regression (29–34ms → 143–155ms, same + scene shape). Measurement only — no fix until it names the element. + - **Unit 2 (later, unchanged):** blinking amber **"SD" badge**, preview only, never in the + recording/stream, when a better format was available and not obtained. Top-right corner, or the + 1:30 rim point when the clip is round. + - **Deferred to after Unit 1:** `MediaCaptureSharingMode.ExclusiveControl` + MJPG ≥30fps up to + 1280×720 via the existing `SetMediaStreamPropertiesAsync` path (refused only because + `SharedReadOnly`); same-camera `SharedReadOnly` retry. **The creator reversed the ban on taking an + exclusive camera lock.** API probe already done — see HANDOFF.md, incl. the `SetFormatAsync` + dead end. - ⛔ **No fallback-camera feature, ever** — *"that's bypassing the user's explicit choice."* - ⛔ **No background removal** — re-affirmed out 2026-10-05. - **TASK 40 — App Settings round** — plan saved 2026-09-22 (`TASKS/task-40-app-settings-round.md`), diff --git a/ai.md b/ai.md index eeffc7e..d79fcf0 100644 --- a/ai.md +++ b/ai.md @@ -648,10 +648,32 @@ happens per submit). Regression test: compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below `arrivals` = the compositor missed frames the reader had already delivered. -**Still unmeasured (next unit):** the camera is stuck at the C920 default 640×480 YUY2 @30 because -all 185 `SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is -on the table and the compositor's headroom at ~3× the bytes is unknown. Measure first, then change -the acquisition. +**✅ WHAT THE FIRST REAL READING SAID (2026-10-05) — the camera was never the problem.** +`ty-20261005-1201-556-0001.mp4` + its log: the C920 delivers **30.0/s with ZERO gaps ≥50ms**, and the +device's own stamps are identical, so the camera, USB link, and driver are all clean. The pump did +**34 composites per 5s (6.8/s) at ~141ms each** (worst 294ms), and the file holds **6.8 distinct +webcam images/s at 88.7% duplicates with no arrival hitches at all**. The judder is entirely us +holding one face for 140ms and padding the rest of a 60fps container. `resolve` is 0.1ms, so it is +pure pixel work in the composite, not frame lookup. Two consequences, both load-bearing: + +- **720p is a sharpness win, NOT a smoothness win** — it triples the bytes stretched per render on a + path already 4.4× over budget. Schedule the exclusive lock AFTER the compositor work or it makes + the judder measurably worse. +- **The 10:01 take hit 25.6 images/s and the 12:01 take 6.8, same scene shape + (`full-render split=0 elements=6 dynamic=5`), same 640×480 camera.** Composites went 29–34ms → + 143–155ms between sessions with nothing logged explaining it. **There is no per-element composite + breakdown and no scene dump at pump start**, so the log cannot name the culprit — that gap is the + next thing to close, and guessing at it is explicitly out of bounds. + +**Counter subtlety, so it is not re-derived:** the pump's `webcam` line read 13/s while the file held +6.8/s. Not a bug — the counter stamps a frame when the resolver FIRST sees it, and one composite takes +141ms, so ~4 camera frames arrive during it (signature pass stamps one epoch, the actual draw uses a +later one). The file is the truth; the pump counter is a pipeline-health signal and overcounts by +design. + +**Still unmeasured:** the camera is stuck at the C920 default 640×480 YUY2 @30 because all 185 +`SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is available +— but see above, it is a quality win only. ### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)