UNIT 0: make a dropped webcam frame measurable, not just inferable
The creator is DHH and lipreads their own recordings, so a webcam that judders is an accessibility defect, not a polish item. Take ty-20261005-1001-821-0001.mp4 holds 25.6 distinct webcam images/s against the camera's 30fps with 281 gaps of 50-100ms - roughly one hitch in five - in a file that muxes flawlessly. Nothing in the app could see that: rawvideo carries no per-frame timestamps (ffmpeg stamps by ARRIVAL at the declared 60fps), and the stats line divided every average by the number of frames SUBMITTED, which deadline padding always fills to the full target. So the pump reported "300/300 frames, avg render 12.4ms" while producing 60 distinct composites. Helpers/FrameCadence.cs is one pure counter fed from three points: MediaCaptureFrameSource logs the device's own SystemRelativeTime stamps against our arrival times (TryAcquireLatestFrame keeps only the LATEST frame, so falling behind is silent by design), and FramePump logs the distinct webcam Epoch values that reached a composite. Read together they name the culprit - device/USB, our reader, or our compositor - instead of guessing. Render/resolve/wait now divide by the real render count, and "frames per 5s from R renders" is the duplicate-on-lag ratio stated honestly. An out-of-order timestamp counts as a frame but produces no gap and does not become the next interval's reference, so one bad stamp cannot invent a multi-second hitch twice. Also corrects a stale ai.md line (the "webcam renders at ~18fps" figure came from a different session and counted every newly rendered layer) and re-records the background-removal ruling as permanently out. Records the agreed-but-unbuilt Units 1-2, the no-fallback-camera ruling, and the SetFormatAsync dead end found by probing the 19041 projection. Suite 402/402, 0 warnings. The one red, LayerReorderPersistenceTests' RealMouseDrag, fails identically on a stashed clean tree - it needs an interactive desktop session this shell lacks.
This commit is contained in:
+94
-20
@@ -1,13 +1,79 @@
|
||||
# HANDOFF — current state
|
||||
|
||||
**Branch:** `main` (pre-1.0, no feature branches — creator ruling 2026-08-24).
|
||||
**Pushed 2026-10-04:** `origin/main` = `048cab7`. **NOT PUSHED since — local `main` is 5 commits
|
||||
ahead and the tree is clean.** The stale-push-state failure `MyMistakes.md` warns about has now bitten
|
||||
**Pushed 2026-10-04:** `origin/main` = `048cab7`. **NOT PUSHED since — local `main` is ahead and the
|
||||
tree is clean.** The stale-push-state failure `MyMistakes.md` warns about has now bitten
|
||||
twice, so the rule stands: **check `git rev-list --count origin/main..main` before trusting any line
|
||||
here about push state.**
|
||||
|
||||
Unpushed: `5ad0106` broadcast diagnostic → `74da99c` + `0fb2f9c` privacy/Test rulings →
|
||||
`8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement.
|
||||
`8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement →
|
||||
`b2f2197` handoff → `UNIT 0` frame-cadence measurement.
|
||||
|
||||
## 🚧 IN FLIGHT 2026-10-05 — UNIT 0 shipped: a dropped webcam frame is finally measurable
|
||||
|
||||
**The measurement the map never had.** Take `ty-20261005-1001-821-0001.mp4` (85.6s, 1920×1080,
|
||||
H.264 NVENC, 5124 frames @60) holds **25.6 distinct webcam images/s** in its lower-right region
|
||||
against the camera's 30fps: 1,153 normal 33ms intervals, **281 gaps of 50–100ms**, 96 bunched 17ms
|
||||
gaps. One hitch in five — the judder a DHH creator lipreads against, invisible in a file that muxes
|
||||
flawlessly. Measured with `signalstats`/YDIF over `crop=480:270:1400:780`; the bundled ffmpeg has no
|
||||
`mpdecimate`. ⛔ The old "webcam renders at ~18fps" claim was wrong (different session, counted all
|
||||
newly rendered layers) — corrected in `ai.md`.
|
||||
|
||||
**Shipped:** `Helpers/FrameCadence.cs` (pure counter: frames, gaps, hitches ≥50ms, longest/avg), fed
|
||||
from three points — `MediaCaptureFrameSource` logs `Camera cadence WxH: arrivals … | camera …`
|
||||
(our delivery vs the device's own `SystemRelativeTime`), and `FramePump` logs distinct webcam
|
||||
`Epoch`s that reached a composite. Full detail in `ai.md` → "Frame cadence".
|
||||
|
||||
**Also fixed, and it was a real measurement bug:** every pump average was divided by the number of
|
||||
frames **submitted**, which deadline padding always fills to the full 60fps target — so a pump fed
|
||||
one webcam frame per 200ms logged "300/300 frames, avg render 12.4ms". Render/resolve/wait now divide
|
||||
by `statRenders` and the line reads `300/300 frames per 5s from 60 renders`. That pair IS the
|
||||
duplicate-on-lag ratio. Test: `FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_…`.
|
||||
|
||||
**⏭ NEXT STEP — needs the creator, one short recording (60s is plenty):** record with the webcam
|
||||
visible and paste nothing — I read `%APPDATA%\ytLlive\startup.log` for the `Camera cadence` and
|
||||
`FramePump stats` lines. Reading them together names the culprit: camera steady + arrivals ragged =
|
||||
we drop it (reader/compositor); ragged in both = device/USB/driver; pump `webcam` below `arrivals`
|
||||
= the compositor missed frames the reader already had. **Then Unit 1** (exclusive lock + 720p, see
|
||||
below) is judged against those numbers.
|
||||
|
||||
**Agreed but NOT yet built (decided 2026-10-05, so a later session does not re-derive it):**
|
||||
- **Unit 1 — exclusive camera lock + 720p on the camera the creator picked.** Open with
|
||||
`MediaCaptureSharingMode.ExclusiveControl`; if init is refused, reopen the **SAME camera** in
|
||||
`SharedReadOnly`. Target MJPG ≥30fps up to 1280×720, ranked by frame rate then size (this also fixes
|
||||
the existing candidate-ordering bug). **The creator REVERSED the ban on taking an exclusive camera
|
||||
lock** — that ban was never written into the map, which is why the reversal must be recorded this
|
||||
time. Accepted trade-off: an app needing camera *control* that opens it after us will fail.
|
||||
- ✅ **API probe DONE 2026-10-05 against the actual 19041.57 projection** (a throwaway
|
||||
compile-probe file, since deleted — do not re-derive, and note the build cost of finding out):
|
||||
`MediaCaptureSharingMode.ExclusiveControl` **is** projected, so the old comment in
|
||||
`MediaCaptureFrameSource.cs` ("Exclusive is not projected by the 19041 SDK") is wrong and must be
|
||||
corrected. `Exclusive` and `SharedReadOnly` are the **only two** `MediaCaptureSharingMode` values.
|
||||
`MediaFrameSource.SupportedFormats` and `CurrentFormat` exist.
|
||||
⛔ **`MediaFrameSource.SetFormatAsync(MediaFrameFormat)` is a DEAD END here** — it takes a
|
||||
`MediaFrameFormat`, and that type has **no public constructor and no static factory** in the
|
||||
projection (it is only obtainable from a live frame's `VideoMediaFrame.Format`). The two-argument
|
||||
`(subtype, properties)` overload in Microsoft's docs does not exist here. So the format change must
|
||||
go through the `VideoDeviceController.SetMediaStreamPropertiesAsync` path the code ALREADY has —
|
||||
it is only refused because `SharingMode` is `SharedReadOnly`. Under an exclusive lock the existing
|
||||
negotiation should simply start working, which is a much smaller change than planned.
|
||||
- **Remaining unknown (needs a real device, not research):** whether `SetMediaStreamPropertiesAsync`
|
||||
actually succeeds under `ExclusiveControl` on this C920. The `MediaFrameSource.SupportedFormats`
|
||||
route is the fallback probe if it refuses — enumerate, and log what the device offers before
|
||||
assuming anything.
|
||||
- **Unit 2 — blinking "SD" badge in the preview only**, when a better format was available and we did
|
||||
not get it. Top-right corner of the webcam, or the 1:30 point ON THE RIM for a round clip (a round
|
||||
webcam has no corner); outside the mirror transform. ~1Hz opacity blink (under the 3/sec
|
||||
photosensitivity limit), tooltip on hover only, **must never appear in the recording or stream**
|
||||
(the compositor draws from camera frames, not the WPF tree). Style lives in `Themes/Controls.xaml`
|
||||
— a DataTrigger storyboard cannot be stopped inside a DataTemplate (`PreviewPane.xaml:168-171`).
|
||||
- ⛔ **NO fallback-camera setting, ever** — creator ruling: *"no fallback - that's bypassing the
|
||||
user's explicit choice."* Picking a vcam is a user action in the camera picker. Corollary, also
|
||||
recorded in `ai.md`: **Windows Studio Effects is NOT a vcam** — it runs inside the physical camera's
|
||||
driver and never appears as a selectable device. Only Broadcast / XSplit VCam install one.
|
||||
- ⛔ **Background removal: permanently out.** Creator confirmed the existing note: *"no - not doing
|
||||
it. the note is correct and accurate."* `ai.md` said "milestone 2", which was stale — now corrected.
|
||||
|
||||
## 🚨 FIXED 2026-10-04 — broadcasts never went live because they were never BOUND
|
||||
|
||||
@@ -281,11 +347,12 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS
|
||||
|
||||
## Landmines
|
||||
|
||||
- **Frame pump jitter, unquantified (2026-10-04).** The same test stream logged **86 stalls
|
||||
per 5s** (`FramePump stall: iteration 40–50ms` against a 17ms budget; 142 such lines in an
|
||||
8-second window) while `FramePump stats` still read 301/300 frames, 0 dropped. Nominal 60fps
|
||||
with zero drops *and* repeated 3× overruns is a contradiction worth one measurement, not a
|
||||
guess. Do not "fix" it from the log alone.
|
||||
- **Frame pump jitter — counters now honest, cause NOT yet named (2026-10-05).** The same test
|
||||
stream logged **86 stalls per 5s** (`FramePump stall: iteration 40–50ms` against a 17ms budget)
|
||||
while `FramePump stats` read 301/300 frames, 0 dropped. UNIT 0 removed the reason those two
|
||||
numbers could disagree (the averages were diluted by padded submits) and added the counters that
|
||||
separate reader from compositor — **but no one has read a post-UNIT-0 log yet.** A creator
|
||||
recording is required first. Do not "fix" the pump before that reading.
|
||||
- **Stale fact inside `TASKS.md` (left alone, flagged here per the Scope Lock).** The "1.0 gates"
|
||||
section asserts "**TASK 36 is now shipped**", but the catalog row 36 reads **☐ Queued** and
|
||||
`task-36-gold-pass.md` still shows items 1–6 unchecked. The line most likely means *item 2*
|
||||
@@ -293,9 +360,10 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS
|
||||
flagging rather than fixing because it is not this unit's job.
|
||||
- **A stale `ytLive.exe` (PID 2544) locked `bin/.../ytLive.exe`** and broke `dotnet run` with
|
||||
MSB3027. Killed. **If the build fails to copy `ytLive.exe`, check for a running instance first.**
|
||||
- **`LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` is flaky by
|
||||
environment** (needs an interactive desktop session; 2 fail / 1 pass in isolation). **Run the
|
||||
suite with ytLive CLOSED.** Not a regression.
|
||||
- **`LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` needs an
|
||||
interactive desktop session.** As of 2026-10-05 it fails deterministically here (`ImgC` vs
|
||||
`ImgA`), verified identical on a clean tree. **Run the suite with ytLive CLOSED** and expect this
|
||||
one to fail in this shell. Not a regression — don't diagnose it from a WSL-shell failure.
|
||||
- **`RealAppHost.RunAsync` exists for a reason** — a frame-pump test MUST `await` inside it. A
|
||||
blocking wait occupies the one shared STA thread and **hangs the whole suite with no output.**
|
||||
Always background a `vstest` run and poll the log; a foreground piped `vstest` returns nothing
|
||||
@@ -308,11 +376,17 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS
|
||||
|
||||
## Test state
|
||||
|
||||
**395/395, 0 warnings, re-verified 2026-10-04 on a clean test-project build** (381 + 3 diagnostic
|
||||
`FirstItem` cases + 1 window fact + 3 placement theory cases + 1 bind test, less 1 removed).
|
||||
Build the **test** project, not just the app project, or VSTest runs a stale DLL.
|
||||
**402/402, 0 warnings, verified 2026-10-05** (395 + 6 `FrameCadenceTests` + 1 pump-stats integration
|
||||
test). Build the **test** project, not just the app project, or VSTest runs a stale DLL.
|
||||
**Launch exe for the creator's test run: `bin/Debug/net8.0-windows10.0.19041.0/ytLive.exe`.**
|
||||
|
||||
**Confirmed environmental flake, NOT a regression (re-checked 2026-10-05, and it now fails
|
||||
deterministically rather than 2-of-3):** `LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_
|
||||
PersistsTheReorder` fails with `Expected "ImgC", Actual "ImgA"` at `RealAppHost.Run` line 68. **It
|
||||
fails identically on a stashed clean tree**, so it has nothing to do with UNIT 0 — it needs an
|
||||
interactive desktop session this shell does not have (the mouse drag never lands). Do not chase it;
|
||||
it is listed under Landmines as the known interactive-session test.
|
||||
|
||||
**⚠️ New flake, do not chase it as a regression:** on the *first* full-suite run,
|
||||
`AudioPipelineTests.MasterLimiter_CapsTheLiveMix_OnThePipe` failed with `Expected 0.891,
|
||||
Actual 0` — a **zero**, not a slightly-off value, which is the signature of the pipe not being
|
||||
@@ -335,12 +409,12 @@ whole-suite scheduling artifact of the same family as the `RealAppHost` STA land
|
||||
1. **~PUSH, then the creator's 60-second Test** — see the FIXED box above. Packaging is NOT next;
|
||||
the creator's ruling 2026-10-04: still in active development, so ship-time work waits and queued
|
||||
features come first.
|
||||
2. **Frame-pump jitter (queued, unquantified):** ~86 stalls per 5 s in one log window, 142 in an
|
||||
8-second window, while stats reported 301/300 frames and **zero** drops. Measure before touching
|
||||
anything — the counter and the missing-frame count currently disagree, and "fixing" the pump
|
||||
without knowing which number is wrong risks making it worse. **Creator to-do, not yet queued:
|
||||
long-run resource/consumption/recycling benchmarking** (how long the app stays healthy under full
|
||||
burden) — record only, do not start it.
|
||||
2. **Webcam cadence → Unit 1 (exclusive lock + 720p).** UNIT 0 is shipped and the counters are in
|
||||
`startup.log`; the reading needs one 60-second creator recording (details + the exact
|
||||
read-the-log-together table in the IN FLIGHT box above). Then Unit 1: exclusive camera lock +
|
||||
720p, and Unit 2: the blinking SD badge. **Creator to-do, not yet queued: long-run
|
||||
resource/consumption/recycling benchmarking** (how long the app stays healthy under full burden) —
|
||||
record only, do not start it.
|
||||
2. **The packaging project + `Package.appxmanifest`** (TASK 48 items 2–3) — the next code unit.
|
||||
Full trust, `webcam`/`microphone`, English only; Velopack deleted. Add the
|
||||
`DefaultRecordFolder()` comment in the same unit, and the WebView2 framework dependency +
|
||||
|
||||
Reference in New Issue
Block a user