Improving robotics competitions for real-world evaluation of AI

John Anderson, Hansjoerg (Jacky) Baltes, Kuo Yang Tu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)

Abstract

While embodied robotic applications have been a strong influence on moving artificial intelligence toward focussing on broad, robust solutions that operate in the real world, evaluating such systems remains difficult. Competition-based evaluation, using common challenge problems, is one of the major methods for comparing AI systems employing robotic embodiment. Competitions unfortunately tend to influence the creation of specific solutions that exploit particular rules rather than the broad and robust techniques that are hoped for, however, and physical embodiment in the real world also creates difficulties in control and repeatability. In this paper we discuss the positive and negative influences of competitions as a means of evaluating AI systems, and present recent work designed to improve such evaluations. We describe how improved control and repeatability can be achieved with mixed reality applications for challenge problems, and how competitions themselves can encourage breadth and robustness, using our rules for the FIRA HuroCup as an example.

Original languageEnglish
Title of host publicationExperimental Design for Real-World Systems - Papers from the AAAI Spring Symposium
Pages1-8
Number of pages8
Publication statusPublished - 2009 Nov 5
EventExperimental Design for Real-World Systems - Papers from the AAAI Spring Symposium - Stanford, CA, United States
Duration: 2009 Mar 232009 Mar 25

Publication series

NameAAAI Spring Symposium - Technical Report
VolumeSS-09-03

Other

OtherExperimental Design for Real-World Systems - Papers from the AAAI Spring Symposium
CountryUnited States
CityStanford, CA
Period09/3/2309/3/25

ASJC Scopus subject areas

  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Improving robotics competitions for real-world evaluation of AI'. Together they form a unique fingerprint.

  • Cite this

    Anderson, J., Baltes, H. J., & Tu, K. Y. (2009). Improving robotics competitions for real-world evaluation of AI. In Experimental Design for Real-World Systems - Papers from the AAAI Spring Symposium (pp. 1-8). (AAAI Spring Symposium - Technical Report; Vol. SS-09-03).