Using Generalizability Theory to Examine Scoring Reliability and Variability of Judging Panels in Skating Competitions

Generalizability (G-) theory is increasingly being used by assessment professionals in a variety of assessment contexts, especially those involving performance assessments. This study expands the use of G-theory and its possible application to highly competitive sporting events that involve judging panels. Specifically, using G-theory, this study examined both the scoring variability and reliability of all four events of the 2004 World Figure Skating Championships. For each event, it examined the independent sources of score variation contributing to the variability of scores assigned by the raters (judges), and the score reliability (or generalizability coefficient). For the Ice Dancing event, an additional examination of the impact of treating "rater" and "rating" as different facets of G-theory studies on the score reliability was conducted. The results show that rater/rating was identified as a component of concern in all events. For the Singles events, in particular, the percentages of the variance for rater (judge) reached approximately 10 percent of the total variance, indicating that there were serious concerns about the scoring reliability. The generalizability coefficients for the events of Men's Singles and Ladies' Singles were both 0.93. However, the generalizability coefficients for Pairs and Ice Dancing were 0.99 and 0.98, respectively, indicating that the scores for these two events were extremely reliable. Using rater vs. rating in the Ice Dancing scores appeared to result in very little difference.