You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

PedzSTAR-AVD-WavLM (v1.0-paper)

PedzSTAR Lab β€” Adult Voice Detector, WavLM Base Plus fine-tune.

β–Ά In plain words: this model listens to a child's clip and, for every 1/50 of a second (20 ms), says how likely it is that a child, an adult, or a non-voice noise is sounding. Its highest adult score anywhere in the clip ("WavLM adult max") was the main evidence used to decide that a clip is child-only (below 0.2) or contains an adult (0.8 or higher, together with an active examiner microphone or a strong warning sign).

What is in this repository

Path Contents
checkpoints/fold0 … fold4 One model per held-out child fold (the models behind the paper's numbers)
checkpoints/all_data Trained on all folds: use this one for new recordings
each checkpoint model.safetensors (WavLM + head, 95.3 M weights, 381 MB), config.json (WavLM config), avd_config.json
avd_wavlm.py Standalone loader + scorer (frame probabilities and clip summaries)

Converted checkpoints reproduce the study's saved scores exactly (max absolute difference 0.0 on 40 clips per model; GPU, bfloat16 as in the study).

How to use

from avd_wavlm import AVDWavLM
m = AVDWavLM.from_pretrained("checkpoints/all_data")   # GPU if available
s = m.score_file("clip.flac")                           # mono, resampled to 16 kHz if needed
print(s["wl_adult_max"], s["frames"].shape)            # frames: (n_frames, 3) = child, adult, noise

CLI: python avd_wavlm.py --model checkpoints/all_data a.flac b.flac > scores.csv

Training data (described; no recordings are included)

All training audio comes from the PedzSTAR GFTA-3 recordings (secured-access research data; sites ATL, BLT, DLS, ISN, LAX, ORL, STL). Children's clips were cut from ~12.7-minute sessions recorded with a room microphone (Zoom H6 XY); the examiner also wore a lavalier microphone.

Ingredient Amount
Child-only base clips (from the verified first-stage child-only list) 1,031
Real examiner speech segments, cut from the full sessions between child clips 6,372 (0.61 h, 116 sessions)
Extra adult speech: LibriSpeech dev-clean (40 speakers, CC BY 4.0), room-simulated 1,768 segments
Room noises (taps, bumps) and room tone from the sessions 7,628 segments
Human-labelled clips (two listeners) 300 (85 with an adult: 81 two-speaker, 4 adult-only)
Human-verified child-only clips (blind listening check) 150

Practice mixtures were built from a child base clip plus one of: nothing; a noise; another clip of the same child; adult speech cut off at the end, cut off at the start, a quiet background murmur (βˆ’32 to βˆ’15 dB), overlapping speech, or a separate adult turn. Added adult speech had to be β‰₯ 8 dB above the session's noise floor. Children were split into 5 site-balanced folds (177 children); each fold model never saw its held-out children, their examiners' segments or their noises.

Training procedure

Setting Value
Base microsoft/wavlm-base-plus (94,381,936 weights; pre-trained on 94k h of unlabelled speech)
Trained 90,181,488 WavLM weights (convolutional front-end frozen; layer-drop 0) + new head 920,336 weights
Head softmax-weighted mix of all 13 hidden layers β†’ bidirectional LSTM (128 per direction) β†’ 3 sigmoid outputs per 20 ms frame
Batch each step: 24 freshly generated mixtures + 8 real clips (labelled or verified), ≀ 4 s
Steps 8,000 per model (β‰ˆ192,000 unique mixtures seen)
Loss per-frame binary cross-entropy (weights child 1, adult 3, noise 1) + 0.5 Γ— clip-level adult loss (log-sum-exp over frames, sharpness 3; real clips weighted Γ— 2)
Optimiser AdamW, weight decay 0.01; learning rate 2e-5 (WavLM) and 1e-3 (head); 300-step warm-up then linear decay to 5 %; gradient clipping 5.0
Precision, hardware bfloat16 autocast; 1 Γ— NVIDIA H100; β‰ˆ8.3 min and 6.6 GB per model
Software torch 2.11.0+cu128, transformers 5.23.0.dev0

Evaluation (held-out children; 300 labelled clips, 85 with an adult)

Measure Result
AUC, WavLM adult max (out-of-fold) 0.938 (95 % range 0.90–0.97)
At adult max β‰₯ 0.5 71 of 85 adult clips caught; 24 of 215 child-only flagged; 11 of 150 verified-clean flagged
At adult max β‰₯ 0.2 (the rescue limit) 76 of 85 caught; 41 of 215 child-only flagged; 16 of 150 verified-clean flagged
Child-only clips kept while catching all 85 adult clips 33 of 215 (thresholds set on the same labels)
Adult-confirmed rule: adult max β‰₯ 0.8 and either the examiner microphone is active (β‰₯ 3 strong frames in a session where it works normally) or one of three strong warning signs fires (examiner-style phrase; Nemotron hears a second voiced speaker; Nemotron says 2+ people or an adult) right on 61 of 63 labelled clips (97 %)

Adult clips caught at threshold 0.5, by kind:

Kind Clips WavLM Nemotron Either
At the same time 33 31 30 32
Both kinds in one clip 33 28 31 33
One after the other 15 10 13 13
Adult only (no child) 4 2 3 3

WavLM is strongest when the adult talks at the same time as the child, and weaker for adult-only clips and adults speaking in a separate turn. Its practice mixtures had few such cases. In the study the fine-tuned Nemotron covered that gap. Use in the study: 2,454 of 4,401 files were listed as child-only. Two blind listening checks found no adult in 300 random clips, so at most 2.5 % of the list contains an adult (95 % confidence).

Limitations

  • Only 300 labelled clips; one task (GFTA-3), one room-microphone setup, children from 7 US sites.
  • Practice mixtures had no adult-only clips and few adult-led ones.
  • Frame timing ("at the same time" vs "one after the other") matched listeners in 49 of 58 vs 3 of 13 labelled cases.

Privacy, access and responsible use

  • This model was trained on secured-access recordings of young children and their examiners. Fine-tuned models can, in rare cases, reveal traces of their training data. Keep this repository private/gated; share only with people covered by the PedzSTAR data use agreement, and make it public only after PI/IRB/data-governance approval.
  • Do not try to recover training audio or transcripts from the weights.
  • No audio, transcripts, participant IDs or file names are stored in this repository.

Licences

  • WavLM Base Plus is distributed by Microsoft (UniSpeech repository) under Creative Commons Attribution-ShareAlike 3.0 Unported, so this adaptation is shared under the same licence (LICENSE_WavLM_UniSpeech_CC-BY-SA-3.0.txt). Base model: Chen et al. (2022), WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing.
  • LibriSpeech segments used in training: CC BY 4.0 (Panayotov et al., 2015).

Citation

PedzSTAR Lab (2026). PedzSTAR-AVD: adult-voice detection for child speech recordings (manuscript in preparation). Please cite the manuscript once published, plus the base model.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for BharathYedla/PedzSTAR-AVD-WavLM

Finetuned
(43)
this model