gramps/LlamaCasty
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

UNIT 0: make a dropped webcam frame measurable, not just inferable

Browse Source 
The creator is DHH and lipreads their own recordings, so a webcam that
judders is an accessibility defect, not a polish item. Take
ty-20261005-1001-821-0001.mp4 holds 25.6 distinct webcam images/s against
the camera's 30fps with 281 gaps of 50-100ms - roughly one hitch in five -
in a file that muxes flawlessly. Nothing in the app could see that: rawvideo
carries no per-frame timestamps (ffmpeg stamps by ARRIVAL at the declared
60fps), and the stats line divided every average by the number of frames
SUBMITTED, which deadline padding always fills to the full target. So the
pump reported "300/300 frames, avg render 12.4ms" while producing 60
distinct composites.

Helpers/FrameCadence.cs is one pure counter fed from three points:
MediaCaptureFrameSource logs the device's own SystemRelativeTime stamps
against our arrival times (TryAcquireLatestFrame keeps only the LATEST
frame, so falling behind is silent by design), and FramePump logs the
distinct webcam Epoch values that reached a composite. Read together they
name the culprit - device/USB, our reader, or our compositor - instead of
guessing. Render/resolve/wait now divide by the real render count, and
"frames per 5s from R renders" is the duplicate-on-lag ratio stated
honestly.

An out-of-order timestamp counts as a frame but produces no gap and does not
become the next interval's reference, so one bad stamp cannot invent a
multi-second hitch twice.

Also corrects a stale ai.md line (the "webcam renders at ~18fps" figure
came from a different session and counted every newly rendered layer) and
re-records the background-removal ruling as permanently out. Records the
agreed-but-unbuilt Units 1-2, the no-fallback-camera ruling, and the
SetFormatAsync dead end found by probing the 19041 projection.

Suite 402/402, 0 warnings. The one red, LayerReorderPersistenceTests'
RealMouseDrag, fails identically on a stashed clean tree - it needs an
interactive desktop session this shell lacks.
This commit is contained in:
gramps
2026-10-05 11:10:22 -07:00
parent b2f219739d
commit 55ca1029a3
9 changed files with 532 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Services/Encoder/FramePump.cs
+33 -5
View File
@@ -338,6 +338,27 @@ public sealed class FramePump : IDisposable
int stalls = 0;
long lastCacheHits = 0, lastCacheRenders = 0;
var statsNext = DateTime.UtcNow + TimeSpan.FromSeconds(5);
// Real-render accounting (UNIT 0, 2026-10-05): every average above was
// divided by statFrames — the number of frames SUBMITTED, which deadline
// padding always fills to the full target even when the renderer only
// produced a fifth of them. A 60fps container therefore reported "300/300
// frames, avg render 12ms" while the source actually fed it 60 distinct
// images. Render cost is now divided by the render count, and the render
// count is reported beside the submit count — the two numbers side by side
// ARE the duplicate-on-lag ratio, which is the only honest reading of how
// much of the stream is a repeat.
int statRenders = 0;
// Webcam cadence (UNIT 0, 2026-10-05): the pump sees each element's resolved
// frame, and VideoFrame.Epoch is the webcam ring's monotonic generation — so a
// NEW epoch means the webcam actually delivered something new between two of
// our ticks. This is the counterpart to MediaCaptureFrameSource's arrivals
// line: it measures what reached the OUTPUT, not what the camera produced. If
// the camera reports 30/s and this reports 25/s, the loss is between them.
var webcamCadence = new Helpers.FrameCadence();
var seenWebcamEpochs = new HashSet<long>();
void ReportStats()
{
if (DateTime.UtcNow < statsNext) return;
@@ -360,19 +381,23 @@ public sealed class FramePump : IDisposable
}
_log?.Invoke(statFrames == 0
? "FramePump stats: NO frames produced in 5s (loop stalled?)" + (detail.Length > 0 ? " | " + detail : "")
: $"FramePump stats: {statFrames}/{target:F0} frames per 5s, " +
$"avg render {renderTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms " +
$"(resolve {resolveTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}), " +
: $"FramePump stats: {statFrames}/{target:F0} frames per 5s from {statRenders} renders, " +
$"avg render {renderTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / Math.Max(1, statRenders):F1}ms " +
$"(resolve {resolveTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / Math.Max(1, statRenders):F1}), " +
$"avg submit {submitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms, "
+ $"avg wait {waitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms, "
+ $"avg wait {waitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / Math.Max(1, statRenders):F1}ms, "
+ $"worst render {worstRender / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, "
+ $"worst submit {worstSubmit / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, "
+ $"dropped {dropped}, stalls {stalls}" + cacheDetail + (detail.Length > 0 ? " | " + detail : ""));
+ $"dropped {dropped}, stalls {stalls}{cacheDetail}, {webcamCadence.Summarize().Describe("webcam")}"
+ (detail.Length > 0 ? " | " + detail : ""));
worstRender = 0;
worstSubmit = 0;
renderTicks = submitTicks = resolveTicks = waitTicks = 0;
statFrames = 0;
statRenders = 0;
stalls = 0;
webcamCadence.Reset();
seenWebcamEpochs.Clear();
statsNext = DateTime.UtcNow + TimeSpan.FromSeconds(5);
}
@@ -387,6 +412,8 @@ public sealed class FramePump : IDisposable
var frame = _frameResolver(element);
resolveSw.Stop();
resolveTicks += resolveSw.ElapsedTicks;
if (element.IsWebcam && frame != null && seenWebcamEpochs.Add(frame.Epoch))
webcamCadence.Add(System.Diagnostics.Stopwatch.GetTimestamp());
return frame;
}
@@ -444,6 +471,7 @@ public sealed class FramePump : IDisposable
renderSw.Stop();
renderTicks += renderSw.ElapsedTicks;
if (renderSw.ElapsedTicks > worstRender) worstRender = renderSw.ElapsedTicks;
statRenders++;
IFfmpegEncoder? encoder;
lock (_gate) encoder = _encoder;
Services/MediaCaptureFrameSource.cs
+51 -1
View File
@@ -50,7 +50,9 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
// Fallback ladder: prefer the camera's VideoPreview stream, but if the
// reader refuses to start there (NoVideoFrameAvailable), retry against
// its VideoRecord stream — some devices only deliver through it.
// (MediaCaptureSharingMode.Exclusive is not projected by the 19041 SDK.)
// (The exclusive-lock upgrade is a separate, later unit — creator
// ruling 2026-10-05; see HANDOFF.md. It is NOT blocked by an SDK gap:
// `MediaCaptureSharingMode.ExclusiveControl` IS projected in 19041.)
if (!await TryStartAsync(preferVideoRecord: false))
{
if (!await TryStartAsync(preferVideoRecord: true))
@@ -246,6 +248,7 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
_frameReader = reader;
_lastError = null;
_isFailed = false;
StartCadenceWindow();
return true;
}
@@ -287,6 +290,30 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
}
}
private const int CadenceWindowSeconds = 5;
private void StartCadenceWindow()
{
_arrivalCadence.Reset();
_cameraCadence.Reset();
_cadenceNextReportTicks = System.Diagnostics.Stopwatch.GetTimestamp()
+ System.Diagnostics.Stopwatch.Frequency * CadenceWindowSeconds;
}
/// <summary>Logs one cadence window and opens the next. The size rides along
/// because the whole question started as "the webcam is soft": a future format
/// change must be readable in the SAME line that proves the frame rate.</summary>
private void ReportCadence(int width, int height)
{
_cadenceNextReportTicks = System.Diagnostics.Stopwatch.GetTimestamp()
+ System.Diagnostics.Stopwatch.Frequency * CadenceWindowSeconds;
var arrivals = _arrivalCadence.Summarize();
var camera = _cameraCadence.Summarize();
_arrivalCadence.Reset();
_cameraCadence.Reset();
AppLog.Write($"Camera cadence {width}x{height}: {arrivals.Describe("arrivals")} | {camera.Describe("camera")}");
}
private void UnsubscribeCaptureEvents(MediaCapture capture)
{
capture.Failed -= OnCaptureFailed;
@@ -349,12 +376,35 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
return fresh;
}
// Arrival vs production cadence (UNIT 0, 2026-10-05): the recorded file cannot
// tell a dropped camera frame from a padded repeat — rawvideo stamps frames by
// ARRIVAL at the declared 60fps, so a webcam delivering 25 of its 30 frames still
// muxes a full-length file. Two counters, one reader, settle it: the camera's OWN
// SystemRelativeTime stamps (device/USB/driver cadence) against the moment the
// reader handed each frame to us (our cadence, including coalescing drops — the
// reader deliberately keeps only the LATEST frame, so falling behind is silent).
// Steady camera + ragged arrivals = we are behind. Ragged in both = the device.
private readonly FrameCadence _arrivalCadence = new();
private readonly FrameCadence _cameraCadence = new();
private long _cadenceNextReportTicks;
private void OnFrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
var arrivedTicks = System.Diagnostics.Stopwatch.GetTimestamp();
using var frame = sender.TryAcquireLatestFrame();
var videoFrame = frame?.VideoMediaFrame?.SoftwareBitmap;
if (videoFrame == null) return;
// SystemRelativeTime is QPC-domain, the same clock Stopwatch reads here, so the
// camera's own frame spacing is measurable without a second conversion. It is
// NULLABLE in the projection (null = the device supplied no stamp), so the
// production side of the comparison is then simply missing from the line —
// arrivals alone still measure what reached us.
_arrivalCadence.Add(arrivedTicks);
if (frame!.SystemRelativeTime is { } cameraStamp)
_cameraCadence.Add((long)(cameraStamp.TotalSeconds * System.Diagnostics.Stopwatch.Frequency));
if (arrivedTicks >= _cadenceNextReportTicks) ReportCadence(videoFrame.PixelWidth, videoFrame.PixelHeight);
var bitmap = videoFrame.BitmapPixelFormat == BitmapPixelFormat.Bgra8
? videoFrame
: SoftwareBitmap.Convert(videoFrame, BitmapPixelFormat.Bgra8);