The quality of experimental designs in mobile learning research

A systemic review and self-improvement tool

Yao-Ting Sung, Han Yueh Lee, Je Ming Yang, Kuo-En Chang

Research output: Contribution to journalReview article

Abstract

The intervention effects of mobile learning have become one of the most popular topics in education research. Despite scholars critiquing the quality of mobile learning research, there is an obvious lack of quantitative studies exploring the design quality of experimental studies of mobile learning as well as tools for self-checking and improving experimental design. Based on the newly developed CREED (Checklist for the Rigor of Education-Experiment Designs), which focuses on the internal, construct, and statistical-conclusion validities of research, this study investigated the experimental research designs of mobile-learning studies. By analyzing 342 experimental studies published in refereed journals from 2006 to 2016, this study found that 72% of the mobile-learning studies exhibited experimental designs with a low or medium-low level of rigor. The mobile-learning research suffered from methodological limitations such as a lack of baseline equivalence between the experimental and control groups, not examining the fulfillment of statistical assumptions for specific statistical methods, poor quality of outcome assessment tools, and insufficient statistical powers. This study concluded that while utilizing advanced technologies, mobile-learning research employs laggard methodologies compared with standards proposed in educational and psychological research. Recommendations for overcoming these limitations are provided for use in further research and practices.

Original languageEnglish
Article number100279
JournalEducational Research Review
Volume28
DOIs
Publication statusPublished - 2019 Nov 1

Fingerprint

learning
lack
statistical method
equivalence
research planning
education
experiment
methodology
Group

Keywords

  • Checklist for the Rigor of Education-Experiment Designs (CREED)
  • Experimental design
  • Mobile learning
  • Research quality
  • Study Design and Implementation Assessment Device (Study DIAD)
  • What Works Clearing House (WWC)

ASJC Scopus subject areas

  • Education

Cite this

The quality of experimental designs in mobile learning research : A systemic review and self-improvement tool. / Sung, Yao-Ting; Lee, Han Yueh; Yang, Je Ming; Chang, Kuo-En.

In: Educational Research Review, Vol. 28, 100279, 01.11.2019.

Research output: Contribution to journalReview article

@article{0632f8ca9d714ca699e55c439f70b1be,
title = "The quality of experimental designs in mobile learning research: A systemic review and self-improvement tool",
abstract = "The intervention effects of mobile learning have become one of the most popular topics in education research. Despite scholars critiquing the quality of mobile learning research, there is an obvious lack of quantitative studies exploring the design quality of experimental studies of mobile learning as well as tools for self-checking and improving experimental design. Based on the newly developed CREED (Checklist for the Rigor of Education-Experiment Designs), which focuses on the internal, construct, and statistical-conclusion validities of research, this study investigated the experimental research designs of mobile-learning studies. By analyzing 342 experimental studies published in refereed journals from 2006 to 2016, this study found that 72{\%} of the mobile-learning studies exhibited experimental designs with a low or medium-low level of rigor. The mobile-learning research suffered from methodological limitations such as a lack of baseline equivalence between the experimental and control groups, not examining the fulfillment of statistical assumptions for specific statistical methods, poor quality of outcome assessment tools, and insufficient statistical powers. This study concluded that while utilizing advanced technologies, mobile-learning research employs laggard methodologies compared with standards proposed in educational and psychological research. Recommendations for overcoming these limitations are provided for use in further research and practices.",
keywords = "Checklist for the Rigor of Education-Experiment Designs (CREED), Experimental design, Mobile learning, Research quality, Study Design and Implementation Assessment Device (Study DIAD), What Works Clearing House (WWC)",
author = "Yao-Ting Sung and Lee, {Han Yueh} and Yang, {Je Ming} and Kuo-En Chang",
year = "2019",
month = "11",
day = "1",
doi = "10.1016/j.edurev.2019.05.001",
language = "English",
volume = "28",
journal = "Educational Research Review",
issn = "1747-938X",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - The quality of experimental designs in mobile learning research

T2 - A systemic review and self-improvement tool

AU - Sung, Yao-Ting

AU - Lee, Han Yueh

AU - Yang, Je Ming

AU - Chang, Kuo-En

PY - 2019/11/1

Y1 - 2019/11/1

N2 - The intervention effects of mobile learning have become one of the most popular topics in education research. Despite scholars critiquing the quality of mobile learning research, there is an obvious lack of quantitative studies exploring the design quality of experimental studies of mobile learning as well as tools for self-checking and improving experimental design. Based on the newly developed CREED (Checklist for the Rigor of Education-Experiment Designs), which focuses on the internal, construct, and statistical-conclusion validities of research, this study investigated the experimental research designs of mobile-learning studies. By analyzing 342 experimental studies published in refereed journals from 2006 to 2016, this study found that 72% of the mobile-learning studies exhibited experimental designs with a low or medium-low level of rigor. The mobile-learning research suffered from methodological limitations such as a lack of baseline equivalence between the experimental and control groups, not examining the fulfillment of statistical assumptions for specific statistical methods, poor quality of outcome assessment tools, and insufficient statistical powers. This study concluded that while utilizing advanced technologies, mobile-learning research employs laggard methodologies compared with standards proposed in educational and psychological research. Recommendations for overcoming these limitations are provided for use in further research and practices.

AB - The intervention effects of mobile learning have become one of the most popular topics in education research. Despite scholars critiquing the quality of mobile learning research, there is an obvious lack of quantitative studies exploring the design quality of experimental studies of mobile learning as well as tools for self-checking and improving experimental design. Based on the newly developed CREED (Checklist for the Rigor of Education-Experiment Designs), which focuses on the internal, construct, and statistical-conclusion validities of research, this study investigated the experimental research designs of mobile-learning studies. By analyzing 342 experimental studies published in refereed journals from 2006 to 2016, this study found that 72% of the mobile-learning studies exhibited experimental designs with a low or medium-low level of rigor. The mobile-learning research suffered from methodological limitations such as a lack of baseline equivalence between the experimental and control groups, not examining the fulfillment of statistical assumptions for specific statistical methods, poor quality of outcome assessment tools, and insufficient statistical powers. This study concluded that while utilizing advanced technologies, mobile-learning research employs laggard methodologies compared with standards proposed in educational and psychological research. Recommendations for overcoming these limitations are provided for use in further research and practices.

KW - Checklist for the Rigor of Education-Experiment Designs (CREED)

KW - Experimental design

KW - Mobile learning

KW - Research quality

KW - Study Design and Implementation Assessment Device (Study DIAD)

KW - What Works Clearing House (WWC)

UR - http://www.scopus.com/inward/record.url?scp=85067177805&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067177805&partnerID=8YFLogxK

U2 - 10.1016/j.edurev.2019.05.001

DO - 10.1016/j.edurev.2019.05.001

M3 - Review article

VL - 28

JO - Educational Research Review

JF - Educational Research Review

SN - 1747-938X

M1 - 100279

ER -