Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

The consequence of differing levels of agreement across raters is rarely studied. Subsequently, knowledge is limited on how number of raters affects the outcome. The present study aimed to examine the impact on pre-linguistic outcome classifications of 12-month-old infants when using four raters compared to three. Thirty experienced Speech and Language Therapists (SLTs) from five countries assessed 20 minute video recordings of four 12-month-old infants during a play session with a parent. One recording was assessed twice. A naturalistic listening method in real time was used. This involved: (1) assessing, each syllable as canonical or non-canonical, and (2) following the recording, assessing if the infant was babbling canonically and listing the syllables the infant produced with command. The impact that four raters had on outcome, compared to three, was explored by classifying the outcome based on all possible combinations of three raters and determining the frequency that the outcome assessment changed when a fourth assessor was added. Results revealed that adding a fourth rater had a minimal impact on canonical babbling ratio assessment. Presence/absence of canonical babbling and size of consonant inventory showed a negligible impact on three out of four recordings, whereas the size of syllable inventory and presence/absence of canonical babbling was minimally affected in one recording by adding a fourth rater. In conclusion, adding a forth rater in assessment of pre-linguistic utterances in 12-month-old infants with naturalistic assessment in real time does not affect outcome classifications considerably. Thus, using three raters, as opposed to four, is recommended.

Original publication




Journal article


Clinical linguistics & phonetics

Publication Date





138 - 153


Institute of Neuroscience and Physiology, Speech and Language Pathology Unit, Sahlgrenska Academy, University of Gothenburg , Gothenburg, Sweden.


Humans, Speech Disorders, Child Language, Parents, Phonetics, Video Recording, Child, Infant