A novel rater agreement methodology for language transcriptions: evidence from a nonhuman speaker
The ability to measure agreement between two independent observers is vital to any observational study. We use a unique situation, the calculation of inter-rater reliability for transcriptions of a parrot’s speech, to present a novel method of dealing with inter-rater reliability which we believe can be applied to situations in which speech from human subjects may be difficult to transcribe. Challenges encountered included (1) a sparse original agreement matrix which yielded an omnibus measure of inter-rater reliability, (2) “lopsided” $$2\times 2$$ 2 × 2 matrices (i.e. subsets) from the overall matrix and (3) categories used by the transcribers which could not be pre-determined. Our novel approach involved calculating reliability on two levels—that of the corpus and that of the above mentioned smaller subsets of data. Specifically, the technique included the “reverse engineering” of categories, the use of a “null” category when one rater observed a behavior and the other did not, and the use of Fisher’s Exact Test to calculate $$r$$ r -equivalent for the smaller paired subset comparisons. We hope this technique will be useful to those working in similar situations where speech may be difficult to transcribe, such as with small children. Copyright Springer Science+Business Media Dordrecht 2014
Volume (Year): 48 (2014)
Issue (Month): 4 (July)
|Contact details of provider:|| Web page: http://www.springer.com|
|Order Information:||Web: http://www.springer.com/economics/journal/11135|
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Roel Popping, 1983. "Traces of agreement: On the DOT-product as a coefficient of agreement," Quality & Quantity: International Journal of Methodology, Springer, vol. 17(1), pages 1-18, February.
- Roel Popping, 1984. "Traces of agreement: On some agreement indices for open-ended questions," Quality & Quantity: International Journal of Methodology, Springer, vol. 18(2), pages 147-158, February.
When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:48:y:2014:i:4:p:2329-2339. See general information about how to correct material in RePEc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla)or (Rebekah McClure)
If references are entirely missing, you can add them using this form.