TY - JOUR
T1 - Toward an integrated model for designing assessment systems
T2 - An analysis of the current status of computer-based assessments in science
AU - Kuo, Che Yu
AU - Wu, Hsin Kai
N1 - Funding Information:
This study was based upon work supported by the National Science Council of Taiwan under NSC 100-2511-S-003-041-MY3 and NSC 101-2811-S-003-005 .
PY - 2013
Y1 - 2013
N2 - Drawing upon an integrated model proposed by Bennett and Bejar (1998), this review study examined how 66 computer-based science assessments (CBSAs) in basic science and medicine took advantage of advanced technologies. The model regarded a CBSA as an integrated system, included several assessment components (e.g., assessment purpose, measured construct, test and task design, examinee interface, and scoring procedure), and emphasized the interplay among these components. Accordingly, this study systematically analyzed the item presentations of interactive multimedia, the constructs measured, the response formats in formative and summative assessments, the scoring procedures, the adaptive test activities administrated based on the algorithms related to the item response theory (IRT) and other rules beyond IRT, and the strategies for the automatic provision of informative hints and feedbacks in the CBSAs. Our analysis revealed that although only 19 out of 66 assessments took advantage of dynamic and interactive media for item presentations, CBSAs with these media could measure integrated understanding of science phenomena and complex problem-solving skills. However, we also found that limitations in automated scoring may lead to infrequent use of the automated provision of hints and feedbacks with open-ended and extended responses. These findings suggest the interrelatedness of the assessment components, and thus we argue that designers should repeatedly consider the relationships between components of CBSAs to ensure the validity of the assessments. Finally, we also indicate the issues for future research in computer-based assessments.
AB - Drawing upon an integrated model proposed by Bennett and Bejar (1998), this review study examined how 66 computer-based science assessments (CBSAs) in basic science and medicine took advantage of advanced technologies. The model regarded a CBSA as an integrated system, included several assessment components (e.g., assessment purpose, measured construct, test and task design, examinee interface, and scoring procedure), and emphasized the interplay among these components. Accordingly, this study systematically analyzed the item presentations of interactive multimedia, the constructs measured, the response formats in formative and summative assessments, the scoring procedures, the adaptive test activities administrated based on the algorithms related to the item response theory (IRT) and other rules beyond IRT, and the strategies for the automatic provision of informative hints and feedbacks in the CBSAs. Our analysis revealed that although only 19 out of 66 assessments took advantage of dynamic and interactive media for item presentations, CBSAs with these media could measure integrated understanding of science phenomena and complex problem-solving skills. However, we also found that limitations in automated scoring may lead to infrequent use of the automated provision of hints and feedbacks with open-ended and extended responses. These findings suggest the interrelatedness of the assessment components, and thus we argue that designers should repeatedly consider the relationships between components of CBSAs to ensure the validity of the assessments. Finally, we also indicate the issues for future research in computer-based assessments.
KW - Architectures for educational technology system
KW - Post-secondary education
KW - Secondary education
KW - Simulations
UR - http://www.scopus.com/inward/record.url?scp=84879960065&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84879960065&partnerID=8YFLogxK
U2 - 10.1016/j.compedu.2013.06.002
DO - 10.1016/j.compedu.2013.06.002
M3 - Article
AN - SCOPUS:84879960065
SN - 0360-1315
VL - 68
SP - 388
EP - 403
JO - Computers and Education
JF - Computers and Education
ER -