Validating an assessment of scientific modeling competence and mapping junior high school students’ development status

Background: Modeling competence is an essential scientific practice. Nevertheless, different definitions of modeling competence exist, and most studies do not provide enough validity evidence for the assessment instruments. Purpose: This study applied the construct-modeling approach to design and validate an assessment of modeling competence and also to characterize students’ development status of modeling competence in junior high school. We propose an assessment framework integrating both the practice aspects of modeling and the epistemic considerations of the nature and purpose of models. Sample: Forty-four participants consisting of 7th, 8th, 9th and 10th graders and experts participated in a pilot study. Ninety-six 9th graders participated in a Rasch study. Design and methods: A pilot study was conducted to improve the quality of individual items and the rubrics, and to revise the framework. A Rasch testing study was then conducted to offer empirical validity evidence based on Item Response Theory. Finally, students’ modeling competence levels were mapped to their general science achievement level to understand their status. Results: Results of the pilot study showed that the difficulty of all items increased as students’ grades increased, except the model evaluation item and the exception led to revisions to our framework levels and the corresponding rubrics. Through model comparisons, the Rasch testing showed a further refinement to the framework levels and indicated that our designed assessment and revised framework had satisfactory psychometric properties. The estimated competence showed that ninth graders obtaining the proficient science achievement levels were more proficient in mastering model construction, evaluation and prediction, but not in model interpretation and model revision. Conclusions: The construct modeling approach is a useful approach to systematically guide the design and validation of an assessment in the modeling competence field. The designed assessment can be used to understand students’ development status of their modeling competence.