A validity study for Yes/No Angoff standard setting method using cluster analysis

Fen Lan Tseng, Jia Min Chiou, Yao-Ting Sung

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique- Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.

Original languageEnglish
Title of host publication2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015
EditorsZhuo Tang, Jiayi Du, Shu Yin, Renfa Li, Ligang He
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages727-731
Number of pages5
ISBN (Electronic)9781467376822
DOIs
Publication statusPublished - 2016 Jan 13
Event12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015 - Zhangjiajie, China
Duration: 2015 Aug 152015 Aug 17

Publication series

Name2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015

Other

Other12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015
CountryChina
CityZhangjiajie
Period15/8/1515/8/17

Fingerprint

Cluster analysis
Cluster Analysis
Hierarchical Clustering
Expert Judgment
Expectation Maximization
Score Test
Gaussian distribution
Tool Support
Proportion
Clustering
Standards

Keywords

  • cluster analysis
  • hierarchical clustering
  • mixture of Gaussian models
  • standard setting

ASJC Scopus subject areas

  • Discrete Mathematics and Combinatorics
  • Modelling and Simulation
  • Computer Networks and Communications
  • Control and Systems Engineering

Cite this

Tseng, F. L., Chiou, J. M., & Sung, Y-T. (2016). A validity study for Yes/No Angoff standard setting method using cluster analysis. In Z. Tang, J. Du, S. Yin, R. Li, & L. He (Eds.), 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015 (pp. 727-731). [7382032] (2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FSKD.2015.7382032

A validity study for Yes/No Angoff standard setting method using cluster analysis. / Tseng, Fen Lan; Chiou, Jia Min; Sung, Yao-Ting.

2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015. ed. / Zhuo Tang; Jiayi Du; Shu Yin; Renfa Li; Ligang He. Institute of Electrical and Electronics Engineers Inc., 2016. p. 727-731 7382032 (2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Tseng, FL, Chiou, JM & Sung, Y-T 2016, A validity study for Yes/No Angoff standard setting method using cluster analysis. in Z Tang, J Du, S Yin, R Li & L He (eds), 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015., 7382032, 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015, Institute of Electrical and Electronics Engineers Inc., pp. 727-731, 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015, Zhangjiajie, China, 15/8/15. https://doi.org/10.1109/FSKD.2015.7382032
Tseng FL, Chiou JM, Sung Y-T. A validity study for Yes/No Angoff standard setting method using cluster analysis. In Tang Z, Du J, Yin S, Li R, He L, editors, 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015. Institute of Electrical and Electronics Engineers Inc. 2016. p. 727-731. 7382032. (2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015). https://doi.org/10.1109/FSKD.2015.7382032
Tseng, Fen Lan ; Chiou, Jia Min ; Sung, Yao-Ting. / A validity study for Yes/No Angoff standard setting method using cluster analysis. 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015. editor / Zhuo Tang ; Jiayi Du ; Shu Yin ; Renfa Li ; Ligang He. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 727-731 (2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015).
@inproceedings{b01d666f61d1412eb98a38bf5d2108b5,
title = "A validity study for Yes/No Angoff standard setting method using cluster analysis",
abstract = "Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique- Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.",
keywords = "cluster analysis, hierarchical clustering, mixture of Gaussian models, standard setting",
author = "Tseng, {Fen Lan} and Chiou, {Jia Min} and Yao-Ting Sung",
year = "2016",
month = "1",
day = "13",
doi = "10.1109/FSKD.2015.7382032",
language = "English",
series = "2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "727--731",
editor = "Zhuo Tang and Jiayi Du and Shu Yin and Renfa Li and Ligang He",
booktitle = "2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015",

}

TY - GEN

T1 - A validity study for Yes/No Angoff standard setting method using cluster analysis

AU - Tseng, Fen Lan

AU - Chiou, Jia Min

AU - Sung, Yao-Ting

PY - 2016/1/13

Y1 - 2016/1/13

N2 - Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique- Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.

AB - Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique- Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.

KW - cluster analysis

KW - hierarchical clustering

KW - mixture of Gaussian models

KW - standard setting

UR - http://www.scopus.com/inward/record.url?scp=84966642441&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84966642441&partnerID=8YFLogxK

U2 - 10.1109/FSKD.2015.7382032

DO - 10.1109/FSKD.2015.7382032

M3 - Conference contribution

AN - SCOPUS:84966642441

T3 - 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015

SP - 727

EP - 731

BT - 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015

A2 - Tang, Zhuo

A2 - Du, Jiayi

A2 - Yin, Shu

A2 - Li, Renfa

A2 - He, Ligang

PB - Institute of Electrical and Electronics Engineers Inc.

ER -