Improving robotics competitions for real-world evaluation of AI

John Anderson, Hansjoerg (Jacky) Baltes, Kuo Yang Tu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)

Abstract

While embodied robotic applications have been a strong influence on moving artificial intelligence toward focussing on broad, robust solutions that operate in the real world, evaluating such systems remains difficult. Competition-based evaluation, using common challenge problems, is one of the major methods for comparing AI systems employing robotic embodiment. Competitions unfortunately tend to influence the creation of specific solutions that exploit particular rules rather than the broad and robust techniques that are hoped for, however, and physical embodiment in the real world also creates difficulties in control and repeatability. In this paper we discuss the positive and negative influences of competitions as a means of evaluating AI systems, and present recent work designed to improve such evaluations. We describe how improved control and repeatability can be achieved with mixed reality applications for challenge problems, and how competitions themselves can encourage breadth and robustness, using our rules for the FIRA HuroCup as an example.

Original languageEnglish
Title of host publicationExperimental Design for Real-World Systems - Papers from the AAAI Spring Symposium
Pages1-8
Number of pages8
Publication statusPublished - 2009 Nov 5
EventExperimental Design for Real-World Systems - Papers from the AAAI Spring Symposium - Stanford, CA, United States
Duration: 2009 Mar 232009 Mar 25

Publication series

NameAAAI Spring Symposium - Technical Report
VolumeSS-09-03

Other

OtherExperimental Design for Real-World Systems - Papers from the AAAI Spring Symposium
CountryUnited States
CityStanford, CA
Period09/3/2309/3/25

Fingerprint

Robotics
Artificial intelligence

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Anderson, J., Baltes, H. J., & Tu, K. Y. (2009). Improving robotics competitions for real-world evaluation of AI. In Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium (pp. 1-8). (AAAI Spring Symposium - Technical Report; Vol. SS-09-03).

Improving robotics competitions for real-world evaluation of AI. / Anderson, John; Baltes, Hansjoerg (Jacky); Tu, Kuo Yang.

Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium. 2009. p. 1-8 (AAAI Spring Symposium - Technical Report; Vol. SS-09-03).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Anderson, J, Baltes, HJ & Tu, KY 2009, Improving robotics competitions for real-world evaluation of AI. in Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium. AAAI Spring Symposium - Technical Report, vol. SS-09-03, pp. 1-8, Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium, Stanford, CA, United States, 09/3/23.
Anderson J, Baltes HJ, Tu KY. Improving robotics competitions for real-world evaluation of AI. In Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium. 2009. p. 1-8. (AAAI Spring Symposium - Technical Report).
Anderson, John ; Baltes, Hansjoerg (Jacky) ; Tu, Kuo Yang. / Improving robotics competitions for real-world evaluation of AI. Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium. 2009. pp. 1-8 (AAAI Spring Symposium - Technical Report).
@inproceedings{bc8f1808bb964f2a989151483e853d0a,
title = "Improving robotics competitions for real-world evaluation of AI",
abstract = "While embodied robotic applications have been a strong influence on moving artificial intelligence toward focussing on broad, robust solutions that operate in the real world, evaluating such systems remains difficult. Competition-based evaluation, using common challenge problems, is one of the major methods for comparing AI systems employing robotic embodiment. Competitions unfortunately tend to influence the creation of specific solutions that exploit particular rules rather than the broad and robust techniques that are hoped for, however, and physical embodiment in the real world also creates difficulties in control and repeatability. In this paper we discuss the positive and negative influences of competitions as a means of evaluating AI systems, and present recent work designed to improve such evaluations. We describe how improved control and repeatability can be achieved with mixed reality applications for challenge problems, and how competitions themselves can encourage breadth and robustness, using our rules for the FIRA HuroCup as an example.",
author = "John Anderson and Baltes, {Hansjoerg (Jacky)} and Tu, {Kuo Yang}",
year = "2009",
month = "11",
day = "5",
language = "English",
isbn = "9781577354109",
series = "AAAI Spring Symposium - Technical Report",
pages = "1--8",
booktitle = "Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium",

}

TY - GEN

T1 - Improving robotics competitions for real-world evaluation of AI

AU - Anderson, John

AU - Baltes, Hansjoerg (Jacky)

AU - Tu, Kuo Yang

PY - 2009/11/5

Y1 - 2009/11/5

N2 - While embodied robotic applications have been a strong influence on moving artificial intelligence toward focussing on broad, robust solutions that operate in the real world, evaluating such systems remains difficult. Competition-based evaluation, using common challenge problems, is one of the major methods for comparing AI systems employing robotic embodiment. Competitions unfortunately tend to influence the creation of specific solutions that exploit particular rules rather than the broad and robust techniques that are hoped for, however, and physical embodiment in the real world also creates difficulties in control and repeatability. In this paper we discuss the positive and negative influences of competitions as a means of evaluating AI systems, and present recent work designed to improve such evaluations. We describe how improved control and repeatability can be achieved with mixed reality applications for challenge problems, and how competitions themselves can encourage breadth and robustness, using our rules for the FIRA HuroCup as an example.

AB - While embodied robotic applications have been a strong influence on moving artificial intelligence toward focussing on broad, robust solutions that operate in the real world, evaluating such systems remains difficult. Competition-based evaluation, using common challenge problems, is one of the major methods for comparing AI systems employing robotic embodiment. Competitions unfortunately tend to influence the creation of specific solutions that exploit particular rules rather than the broad and robust techniques that are hoped for, however, and physical embodiment in the real world also creates difficulties in control and repeatability. In this paper we discuss the positive and negative influences of competitions as a means of evaluating AI systems, and present recent work designed to improve such evaluations. We describe how improved control and repeatability can be achieved with mixed reality applications for challenge problems, and how competitions themselves can encourage breadth and robustness, using our rules for the FIRA HuroCup as an example.

UR - http://www.scopus.com/inward/record.url?scp=70350554054&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70350554054&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:70350554054

SN - 9781577354109

T3 - AAAI Spring Symposium - Technical Report

SP - 1

EP - 8

BT - Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium

ER -