Using cluster analysis to validate the angoff standard setting method in mixed-format assessments

Fen Lan Tseng, Jia Min Chiou, Yao Ting Sung

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique-Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study concludes that cluster analysis appears useful for helping to set standards on educational tests. In addition, it suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.

Original languageEnglish
Title of host publicationICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery
EditorsLiang Zhao, Lipo Wang, Guoyong Cai, Kenli Li, Yong Liu, Guoqing Xiao
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2540-2546
Number of pages7
ISBN (Electronic)9781538621653
DOIs
Publication statusPublished - 2018 Jun 21
Event13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, ICNC-FSKD 2017 - Guilin, Guangxi, China
Duration: 2017 Jul 292017 Jul 31

Publication series

NameICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery

Other

Other13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, ICNC-FSKD 2017
CountryChina
CityGuilin, Guangxi
Period17/7/2917/7/31

Fingerprint

Cluster analysis
Cluster Analysis
Hierarchical Clustering
Expert Judgment
Expectation Maximization
Score Test
Gaussian distribution
Tool Support
Proportion
Standards
Clustering
Hierarchical clustering

Keywords

  • cluster analysis
  • hierarchical clustering
  • mixture of Gaussian model
  • standard setting

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems
  • Information Systems and Management
  • Logic
  • Modelling and Simulation
  • Statistics and Probability

Cite this

Tseng, F. L., Chiou, J. M., & Sung, Y. T. (2018). Using cluster analysis to validate the angoff standard setting method in mixed-format assessments. In L. Zhao, L. Wang, G. Cai, K. Li, Y. Liu, & G. Xiao (Eds.), ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (pp. 2540-2546). (ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FSKD.2017.8393175

Using cluster analysis to validate the angoff standard setting method in mixed-format assessments. / Tseng, Fen Lan; Chiou, Jia Min; Sung, Yao Ting.

ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery. ed. / Liang Zhao; Lipo Wang; Guoyong Cai; Kenli Li; Yong Liu; Guoqing Xiao. Institute of Electrical and Electronics Engineers Inc., 2018. p. 2540-2546 (ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Tseng, FL, Chiou, JM & Sung, YT 2018, Using cluster analysis to validate the angoff standard setting method in mixed-format assessments. in L Zhao, L Wang, G Cai, K Li, Y Liu & G Xiao (eds), ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery. ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, Institute of Electrical and Electronics Engineers Inc., pp. 2540-2546, 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, ICNC-FSKD 2017, Guilin, Guangxi, China, 17/7/29. https://doi.org/10.1109/FSKD.2017.8393175
Tseng FL, Chiou JM, Sung YT. Using cluster analysis to validate the angoff standard setting method in mixed-format assessments. In Zhao L, Wang L, Cai G, Li K, Liu Y, Xiao G, editors, ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery. Institute of Electrical and Electronics Engineers Inc. 2018. p. 2540-2546. (ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery). https://doi.org/10.1109/FSKD.2017.8393175
Tseng, Fen Lan ; Chiou, Jia Min ; Sung, Yao Ting. / Using cluster analysis to validate the angoff standard setting method in mixed-format assessments. ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery. editor / Liang Zhao ; Lipo Wang ; Guoyong Cai ; Kenli Li ; Yong Liu ; Guoqing Xiao. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 2540-2546 (ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery).
@inproceedings{74b0bd6c2c6842a9b25eacab70808831,
title = "Using cluster analysis to validate the angoff standard setting method in mixed-format assessments",
abstract = "Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique-Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study concludes that cluster analysis appears useful for helping to set standards on educational tests. In addition, it suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.",
keywords = "cluster analysis, hierarchical clustering, mixture of Gaussian model, standard setting",
author = "Tseng, {Fen Lan} and Chiou, {Jia Min} and Sung, {Yao Ting}",
year = "2018",
month = "6",
day = "21",
doi = "10.1109/FSKD.2017.8393175",
language = "English",
series = "ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "2540--2546",
editor = "Liang Zhao and Lipo Wang and Guoyong Cai and Kenli Li and Yong Liu and Guoqing Xiao",
booktitle = "ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery",

}

TY - GEN

T1 - Using cluster analysis to validate the angoff standard setting method in mixed-format assessments

AU - Tseng, Fen Lan

AU - Chiou, Jia Min

AU - Sung, Yao Ting

PY - 2018/6/21

Y1 - 2018/6/21

N2 - Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique-Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study concludes that cluster analysis appears useful for helping to set standards on educational tests. In addition, it suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.

AB - Test validity is a property of the interpretation assigned to test scores. To provide an objective validating evidence for a standard-referenced assessment is especially important. In this study we utilize a statistical technique, cluster analysis, to explore the validity of one of the expert judgement technique-Yes/No Angoff standard setting method. We first segregated each examinee ability cluster using the hierarchical clustering (HC). Assume that each ability cluster is a Gaussian distribution and that the distribution of each test subject data can be modeled by mixture of Gaussians (MoG), where the mean, variance and the proportion of each cluster were initialized by the HC results. Finally, the ability clustering was implemented by the expectation maximization (EM) method. The results from the traditional standard-setting procedure and cluster analysis were compared. The study concludes that cluster analysis appears useful for helping to set standards on educational tests. In addition, it suggested that cluster analysis could be applied as a support tool to provide validating information in the process of standard setting for high-stakes achievement tests.

KW - cluster analysis

KW - hierarchical clustering

KW - mixture of Gaussian model

KW - standard setting

UR - http://www.scopus.com/inward/record.url?scp=85050218821&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050218821&partnerID=8YFLogxK

U2 - 10.1109/FSKD.2017.8393175

DO - 10.1109/FSKD.2017.8393175

M3 - Conference contribution

AN - SCOPUS:85050218821

T3 - ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery

SP - 2540

EP - 2546

BT - ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery

A2 - Zhao, Liang

A2 - Wang, Lipo

A2 - Cai, Guoyong

A2 - Li, Kenli

A2 - Liu, Yong

A2 - Xiao, Guoqing

PB - Institute of Electrical and Electronics Engineers Inc.

ER -