Research Assistant at National Institute of Education, Oct 2023 – Mar 2024
Finding who spoke when in classroom recordings
Speech processing on classroom recordings: a speaker diarization system, built with professors, to measure student participation.
Outcomes
- hours of classroom recordings
- 200+
- better than open-source baselines
- 30%
- after fine-tuning on transcripts I annotated
The project
At the National Institute of Education, I worked with professors studying how students take part in group discussions. They had more than 200 hours of classroom recordings and wanted to know how speaking patterns, interruptions and turn-taking related to learning outcomes.
First, someone had to work out who spoke when. That’s speaker diarization, and classroom audio makes it hard: voices overlap, students talk over each other and the microphones pick up the whole room.
What I built
I built an ASR and diarization pipeline on NVIDIA NeMo. I transcribed and labelled recordings myself, for evaluation and for fine-tuning, and the fine-tuned models beat the open-source baselines by 30%.
To clean up the audio, we used beamforming with a pair of microphones. A voice reaches the two microphones at slightly different times. Delay one signal by that gap and add the two together, and sound from the speaker’s direction gets louder while sound from other directions partly cancels out.
Looking back
Labelling the recordings myself was slow, but it gave us a test set that came from real classrooms, noise and all. The 30% is measured on that.