Finding the same moment by sound, not timestamps.

1. The problem

The best shot of a show is often not the one the artist paid for. It is thirty seconds filmed by someone in the fourth row who will post it once and never think about it again. An artist who wants that footage faces two separate problems: getting permission to use it, and making sense of it once they have it.

Timestamps cannot do it. Consumer device clocks drift on the order of tens of milliseconds per minute [13], capture time is frequently stripped or rewritten in the path from camera roll to social platform, and the granularity that survives is coarser than the thing being measured. Two clips stamped the same minute may be ninety seconds apart; two clips stamped differently may overlap exactly.

Audio is the way through. Everyone in the venue recorded the same sound source. If a clip’s audio can be located inside a reference recording of the same show, that clip has a position on the show’s timeline, and comparing positions becomes arithmetic. The shift from timestamp metadata to acoustic self-localisation is the whole idea, and it is not new: it underpins the founding work on organising crowd-sourced concert video [9][10].

2. System architecture

CrowdClip is a NestJS API over PostgreSQL, with media in secure storage and the signal processing isolated in a Python child process. The pipeline saves its progress at each stage, so interrupted work can resume instead of starting over.

Trust boundaries

• CrowdClip follows Meta’s platform rules.
• CrowdClip gives the fan a secure upload link that expires. It does not store the working link in a database.

3. Alignment method

CrowdClip follows Concert Stitch [1], which addresses this exact problem: organising and synchronising crowd-sourced recordings. Concert Stitch in turn builds on the Haitsma–Kalker fingerprint [2].

3.1 Fingerprinting

Audio is downmixed to mono and resampled to 5 kHz, then framed and reduced to 32 bits per frame. The parameters are taken directly from the paper:

Each bit is the sign of the energy difference taken along the frequency axis and then along the time axis, which is what makes the representation survive a phone microphone in a loud room: it encodes how energy changes, not how much of it there is, so overall level, EQ, and compression largely fall out.

A cool implementation detail is that the bits are stored as ±1 floats rather than packed binary. This means an FFT cross-correlation of two fingerprint sequences directly counts matching bits. Normalised similarity is then just 1 − the bit error rate, and the entire offset search is a single transform rather than a sliding Hamming-distance loop.

3.2 Departure one: affine scale search, not DTW

The paper’s main pipeline runs subsequence dynamic time warping over the fingerprint distance matrix and trains an SVM to pick the true overlap path, using cross-correlation only for sample-accurate refinement. CrowdClip implements neither the DTW stage nor the classifier, and the reason is the source material.

DTW is there to absorb playback speed that varies within a recording, specifically the analog wow-and-flutter of the tape-sourced Grateful Dead collection evaluated in the paper. Smartphone recordings do not do that. Their dominant distortion is a small constant clock-rate error, which is a single scalar, not a warping path. So the analyser brute-forces time-scale factors from 0.92 to 1.08 in steps of 0.001, resampling the query fingerprint sequence at each scale and cross-correlating against the reference. The best (scale, offset) pair wins. It is a cheaper model that matches the actual physics of the input.

3.3 Departure two: confidence statistics, not a fixed threshold

The paper’s cross-correlation baseline accepts a match below a fixed 0.35 bit error rate. Its own authors flag this threshold as fragile on unseen data. Rather than inherit a magic number, the analyser reports the shape of the correlation peak and lets the caller decide:

Acceptance is then a policy rather than a constant. Defaults live in the process contract: similarity ≥ 0.59, peak_margin ≥ 0.02, and a time scale inside 0.95–1.05. A caller can tighten or relax them per catalogue without touching the DSP. Gating on margin as well as absolute similarity is the standard false-match suppressor in the fingerprinting literature [4], and the implausible-scale rejection catches sped-up re-uploads that would otherwise align at nonsense offsets.

3.4 Grouping

Once clips carry offsets, same-moment detection is deliberately boring. Two clips are the same moment when they matched the same reference track in the same catalogue and their reference offsets fall within a five-second window. Everything else, including a different catalogue, a different track, an unmatched analysis, an offset outside the window, returns an explicit reason rather than a bare false, which is what makes threshold tuning tractable when a group looks wrong.

4. Evaluation

Two evaluations exist. The first was a feasibility spike: clips cut from one audience recording of a 1977 show, placed against a different recording of the same performance. It is the figure below, playable with the audio actually used.

The second is the one the thresholds come from. Using a 2019 concert from the Live Music Archive, the matrix source served as the reference and a separate audience source supplied the queries: eight fifteen-second clips, plus eight clips taken from a different song in the same show as deliberate false candidates.

All eight true matches resolved to the same constant offset within a 145 ms spread, which is the result that actually matters: consistency across independent clips is evidence the offsets are real, and it needs no ground-truth annotation to verify. Note how much wider the separation is on margin than on similarity: 0.066 versus 0.002, two orders of magnitude, against a similarity gap of only about 0.09. The margin statistic is doing most of the discriminative work.

Degradation behaviour

Fans singing along are one source of degradation, especially when someone close to the phone is louder than the PA. I simulated that by mixing a detuned, 150 ms-late copy of the song over the clip. Similarity falls to 0.64 when the singer is as loud as the music, and 0.59 when the singer is 6 dB louder. Both still recover the exact offset. This costs slightly more similarity than uncorrelated noise at equal loudness because correlated interference flips fingerprint bits in a biased direction rather than a random one.

Under ±3% clock drift plus 10 dB SNR noise, the correct offset is still recovered while raw similarity falls below the 0.59 floor, but peak_z_score still separates cleanly (5–6 for true matches against roughly 4.7 for unrelated audio). If real uploads prove noisier than archive audience recordings, the correct response is to gate on z-score, not to lower the similarity floor.

5. Use cases

• Cleared recap edits. The immediate one. Fan footage of the show, permission recorded per clip, original full-quality files rather than re-compressed reposts.
• Multi-angle stitching. Same moment, four vantage points, already frame-aligned. The cut becomes an editorial choice instead of a synchronisation chore.
• Moment ranking. Overlap density is itself a popularity signal. The parts of the show the most people chose to film are the parts worth cutting to [9].
• Audio repair. With clips aligned, the best-sounding source in a group can supply the audio bed for the best-looking one.
• Setlist reconstruction. Matched reference tracks plus offsets produce a timestamped record of what was played and when, derived from the crowd rather than entered by hand.

6. References

1. V. Subramanian and A. Lerch. Concert Stitch: Organization and Synchronization of Crowd-Sourced Recordings. ISMIR, Paris, 2018. The method CrowdClip implements.
2. J. Haitsma and T. Kalker. A Highly Robust Audio Fingerprinting System. ISMIR, 2002. The fingerprint itself.
3. A. Wang. An Industrial-Strength Audio Search Algorithm. ISMIR, 2003. The Shazam algorithm; assumes the same signal, so it does not cross from studio to live.
4. J. Six and M. Leman. Panako: A Scalable Acoustic Fingerprinting System Handling Time-Scale and Pitch Modification. ISMIR, 2014. Closest open-source analogue to this output tuple.
5. R. Sonnleitner and G. Widmer. Quad-based Audio Fingerprinting Robust to Time and Frequency Scaling. DAFx, 2014.
6. S. Chang et al. Neural Audio Fingerprint for High-Specific Audio Retrieval Based on Contrastive Learning. ICASSP, 2021. The escape hatch if peak-based matching fails on crowd noise.
7. Z. Rafii, B. Coover and J. Han. An Audio Fingerprinting System for Live Version Identification. ICASSP, 2014.
8. T. Tsai, T. Prätzlich and M. Müller. Known-Artist Live Song Identification Using Audio Hashprints. ISMIR, 2016. The reference for the studio-catalogue regime, where constraining search to a known artist is the highest-leverage move.
9. L. Kennedy and M. Naaman. Less Talk, More Rock: Automated Organization of Community-Contributed Collections of Concert Videos. WWW, 2009. The founding paper for this problem.
10. C. Cotton and D. Ellis. Audio Fingerprinting to Identify Multiple Videos of an Event. ICASSP, 2010. On matching degraded recordings against each other rather than against a clean master.
11. S. Bano and A. Cavallaro. Discovery and Organization of Multi-Camera User-Generated Videos of the Same Event. Information Sciences, 2015. The closest published system to this whole flow.
12. J. Liang et al. Temporal Localization of Audio Events for Conflict Monitoring in Social Media. AAAI, 2017. Pairwise sync followed by a global consistency solve.
13. M. Guggenberger, M. Lux and L. Böszörmenyi. A Synchronization Ground Truth for the Jiku Mobile Video Dataset. MMM, 2015. Ground truth for this exact task, plus the companion device clock-drift study.