Development and Validation of a Multimedia-based Assessment of Scientific Inquiry Abilities

Che Yu Kuo, Hsin-Kai Wu, Tsung Hau Jen, Ying-Shao Hsu

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

The potential of computer-based assessments for capturing complex learning outcomes has been discussed; however, relatively little is understood about how to leverage such potential for summative and accountability purposes. The aim of this study is to develop and validate a multimedia-based assessment of scientific inquiry abilities (MASIA) to cover a more comprehensive construct of inquiry abilities and target secondary school students in different grades while this potential is leveraged. We implemented five steps derived from the construct modeling approach to design MASIA. During the implementation, multiple sources of evidence were collected in the steps of pilot testing and Rasch modeling to support the validity of MASIA. Particularly, through the participation of 1,066 8th and 11th graders, MASIA showed satisfactory psychometric properties to discriminate students with different levels of inquiry abilities in 101 items in 29 tasks when Rasch models were applied. Additionally, the Wright map indicated that MASIA offered accurate information about students’ inquiry abilities because of the comparability of the distributions of student abilities and item difficulties. The analysis results also suggested that MASIA offered precise measures of inquiry abilities when the components (questioning, experimenting, analyzing, and explaining) were regarded as a coherent construct. Finally, the increased mean difficulty thresholds of item responses along with three performance levels across all sub-abilities supported the alignment between our scoring rubrics and our inquiry framework. Together with other sources of validity in the pilot testing, the results offered evidence to support the validity of MASIA.

Original languageEnglish
Pages (from-to)2326-2357
Number of pages32
JournalInternational Journal of Science Education
Volume37
Issue number14
DOIs
Publication statusPublished - 2015 Sep 22

Fingerprint

multimedia
ability
student
psychometrics
evidence
secondary school
responsibility
participation

Keywords

  • Computer-based Assessment
  • Inquiry Ability
  • Item Response Theory
  • Multimedia-based Assessment
  • Validation

ASJC Scopus subject areas

  • Education

Cite this

Development and Validation of a Multimedia-based Assessment of Scientific Inquiry Abilities. / Kuo, Che Yu; Wu, Hsin-Kai; Jen, Tsung Hau; Hsu, Ying-Shao.

In: International Journal of Science Education, Vol. 37, No. 14, 22.09.2015, p. 2326-2357.

Research output: Contribution to journalArticle

@article{b4d5901827f64cdeac1545ad7303c1a9,
title = "Development and Validation of a Multimedia-based Assessment of Scientific Inquiry Abilities",
abstract = "The potential of computer-based assessments for capturing complex learning outcomes has been discussed; however, relatively little is understood about how to leverage such potential for summative and accountability purposes. The aim of this study is to develop and validate a multimedia-based assessment of scientific inquiry abilities (MASIA) to cover a more comprehensive construct of inquiry abilities and target secondary school students in different grades while this potential is leveraged. We implemented five steps derived from the construct modeling approach to design MASIA. During the implementation, multiple sources of evidence were collected in the steps of pilot testing and Rasch modeling to support the validity of MASIA. Particularly, through the participation of 1,066 8th and 11th graders, MASIA showed satisfactory psychometric properties to discriminate students with different levels of inquiry abilities in 101 items in 29 tasks when Rasch models were applied. Additionally, the Wright map indicated that MASIA offered accurate information about students’ inquiry abilities because of the comparability of the distributions of student abilities and item difficulties. The analysis results also suggested that MASIA offered precise measures of inquiry abilities when the components (questioning, experimenting, analyzing, and explaining) were regarded as a coherent construct. Finally, the increased mean difficulty thresholds of item responses along with three performance levels across all sub-abilities supported the alignment between our scoring rubrics and our inquiry framework. Together with other sources of validity in the pilot testing, the results offered evidence to support the validity of MASIA.",
keywords = "Computer-based Assessment, Inquiry Ability, Item Response Theory, Multimedia-based Assessment, Validation",
author = "Kuo, {Che Yu} and Hsin-Kai Wu and Jen, {Tsung Hau} and Ying-Shao Hsu",
year = "2015",
month = "9",
day = "22",
doi = "10.1080/09500693.2015.1078521",
language = "English",
volume = "37",
pages = "2326--2357",
journal = "International Journal of Science Education",
issn = "0950-0693",
publisher = "Taylor and Francis Ltd.",
number = "14",

}

TY - JOUR

T1 - Development and Validation of a Multimedia-based Assessment of Scientific Inquiry Abilities

AU - Kuo, Che Yu

AU - Wu, Hsin-Kai

AU - Jen, Tsung Hau

AU - Hsu, Ying-Shao

PY - 2015/9/22

Y1 - 2015/9/22

N2 - The potential of computer-based assessments for capturing complex learning outcomes has been discussed; however, relatively little is understood about how to leverage such potential for summative and accountability purposes. The aim of this study is to develop and validate a multimedia-based assessment of scientific inquiry abilities (MASIA) to cover a more comprehensive construct of inquiry abilities and target secondary school students in different grades while this potential is leveraged. We implemented five steps derived from the construct modeling approach to design MASIA. During the implementation, multiple sources of evidence were collected in the steps of pilot testing and Rasch modeling to support the validity of MASIA. Particularly, through the participation of 1,066 8th and 11th graders, MASIA showed satisfactory psychometric properties to discriminate students with different levels of inquiry abilities in 101 items in 29 tasks when Rasch models were applied. Additionally, the Wright map indicated that MASIA offered accurate information about students’ inquiry abilities because of the comparability of the distributions of student abilities and item difficulties. The analysis results also suggested that MASIA offered precise measures of inquiry abilities when the components (questioning, experimenting, analyzing, and explaining) were regarded as a coherent construct. Finally, the increased mean difficulty thresholds of item responses along with three performance levels across all sub-abilities supported the alignment between our scoring rubrics and our inquiry framework. Together with other sources of validity in the pilot testing, the results offered evidence to support the validity of MASIA.

AB - The potential of computer-based assessments for capturing complex learning outcomes has been discussed; however, relatively little is understood about how to leverage such potential for summative and accountability purposes. The aim of this study is to develop and validate a multimedia-based assessment of scientific inquiry abilities (MASIA) to cover a more comprehensive construct of inquiry abilities and target secondary school students in different grades while this potential is leveraged. We implemented five steps derived from the construct modeling approach to design MASIA. During the implementation, multiple sources of evidence were collected in the steps of pilot testing and Rasch modeling to support the validity of MASIA. Particularly, through the participation of 1,066 8th and 11th graders, MASIA showed satisfactory psychometric properties to discriminate students with different levels of inquiry abilities in 101 items in 29 tasks when Rasch models were applied. Additionally, the Wright map indicated that MASIA offered accurate information about students’ inquiry abilities because of the comparability of the distributions of student abilities and item difficulties. The analysis results also suggested that MASIA offered precise measures of inquiry abilities when the components (questioning, experimenting, analyzing, and explaining) were regarded as a coherent construct. Finally, the increased mean difficulty thresholds of item responses along with three performance levels across all sub-abilities supported the alignment between our scoring rubrics and our inquiry framework. Together with other sources of validity in the pilot testing, the results offered evidence to support the validity of MASIA.

KW - Computer-based Assessment

KW - Inquiry Ability

KW - Item Response Theory

KW - Multimedia-based Assessment

KW - Validation

UR - http://www.scopus.com/inward/record.url?scp=84942193020&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84942193020&partnerID=8YFLogxK

U2 - 10.1080/09500693.2015.1078521

DO - 10.1080/09500693.2015.1078521

M3 - Article

VL - 37

SP - 2326

EP - 2357

JO - International Journal of Science Education

JF - International Journal of Science Education

SN - 0950-0693

IS - 14

ER -