The design of puzzle selection strategies for ESP-like GWAP systems

The games with a purpose (GWAP) genre is a type of human computation that outsources certain steps of the computational process to humans. Although most GWAP studies focus on the design and analysis of GWAP systems, a systematic and thorough evaluation of existing systems is lacking. We address the issue in this paper. Taking the ESP game as an example, we propose a metric, called system utility, for evaluating the performance of GWAP systems, and use analysis to study the properties of the ESP game. We argue that GWAP systems should be designed and played with strategies. To this end, based on our analysis, we implement an optimal puzzle selection strategy (OPSA) to improve GWAP systems. Using a comprehensive set of simulations, we show that the proposed OPSA approach can improve the system utility of the ESP game significantly. In addition, we implement a quasi ESP game, called ESP Lite, which embeds three puzzle selection algorithms transparently and records the complete game trace for evaluation and further research. During a one-month experiment, we have investigated the inner properties of the three strategies in real-world GWAP systems, and verified that the OPSA scheme achieves the best system utility for the ESP game. The results of this study demonstrate that GWAP systems are more efficient if they are designed and played with strategies.