Investigating expert-rater agreement and inter/intra-rater reliability of two fundamental movement skills for the locomotor subscale of the FG-COMPASS

The Furtado-Gallagher Computerized Observational Movement Pattern Assessment System (FG-COMPASS) is an observational rating scale using sequential decisions to assess fundamental movement skills. The current version of the test has 3 locomotor and 5 manipulative skills. This study aimed to assess expert-rater agreement and inter/intra rater reliability of two new scales to be added to the locomotor subtest. This study was divided into two phases. In Phase I, 60 children between the ages of 5 and 10 were filmed performing the skills of galloping and vertical jumping. An expert in motor behavior classified the videotapes using the newly created rating scales. Next, 8 video clips were selected for training purposes and 24 video clips for testing purposes. In Phase II, 30 undergraduate and graduate students served as raters and underwent a training session prior the testing session. Participants were instructed not to classify the video clips based on the apparent age of the children as skill levels were distributed across all age levels. Further, to avoid guessing, participants were not told how many videos of each level they would be rating. Unlike the training sessions, participants did not receive feedback during testing sessions. Weighted kappa (Kw) and Intraclass Correlation Coefficient (ICC) were used to analyze data. Results suggested a “very good” agreement between expert and rater consensus for vertical jumping (Kw = .96) and galloping (Kw=.89) and an “excellent” agreement for vertical jump (ICC = .98) and galloping (ICC = .95). Inter-rater reliability resulted in “very good” agreement for vertical jumping (Kw=.92) and a “good” agreement for galloping (Kw=.78), and an “excellent” agreement for vertical jumping (ICC = .98) and galloping (ICC=.95). Intra-rater reliability resulted in a “very good” agreement for both vertical jumping (Kw=.96) and galloping (Kw=.85) and an “excellent” agreement for vertical jumping (ICC= .98) and galloping (ICC= .92).