Mouth-motion against speech is a statistical proxy and it fails on ordinary talking clips: the
per-window estimates swing by 700 ms and the honest answer is "unmeasurable". A clap is not a
proxy — it has one unmistakable audio spike and one unmistakable visual spike, a fixed distance
apart, with no periodicity to lock onto wrongly.
So when the correlation is weak, the tool now detects sharp audio transients (99th percentile,
one event per 0.6 s), finds the biggest visual change within ±400 ms of each, and reports the
median of the cluster within 60 ms of it — at least three pairs or it still says unmeasurable.
The cluster matters: an arm entering the frame moves more than the clap itself and lands early
(two pairs on the validation clip came out at +248 and +283 ms against -285).
Validated on ty-20261112-1405-621-0000.mp4: the correlation path said +98 ms and the clap path
+116 ms (three tight pairs at +117, +115, +107), so the applied 350 ms should come down to about
240. Two methods, same answer, no listening required.
On ty-20261112-1401-952-0000-2.mp4 the per-window estimates ranged from +400 to -370 ms and
only one window cleared 0.35, yet the tool printed "-370 ms, adjust by +370" from an overall
correlation of 0.08. A number that wrong is worse than no number: it invites another build.
Now: windows above 0.35 must also agree within 150 ms, and anything weaker reports
"unmeasurable" with the remedy — three claps with the hands in the webcam frame, two seconds
apart, which gives a sharp visual and audio transient instead of a mouth-motion proxy.
Also reverted an onset-based envelope tried on the same clip: it dropped the best window's
correlation from 0.47 to 0.08, so the plain level envelope stays.
What the run did establish: lengths match (15.555 s video against 15.601 s audio, 0.3%), and
the log confirms "mic delay 350 ms" was applied — the drift fix and the delay wiring both hold.
Three numbers, from the recording itself: each stream's length, the drift implied by a length
mismatch (a clock problem), and the residual offset between mouth motion and speech (a constant
problem). Per-3s window table with correlations, so a weak reading announces itself instead of
reporting a confident wrong number, and a median over only the strongly-correlated windows — a
whole-clip search on periodic speech locks onto the wrong beat and returns nonsense (it said
-400 ms at the search limit on the clip where the windows said -185).
Validated on ty-20261112-1228-457-0000.mp4: lengths 13.271/13.305 s (match, so no drift) and
-185 ms (sound early), which agrees with the independent visual read of the same clip.
The creator is deaf, so "does it sound right" cannot be the loop. This is.
Needs numpy; uses the bundled ffmpeg.exe.
gramps