Experimental Studies of Intelligence

Although my dissertation work included systematic experiments in 1979, I did not become deeply interested in the experimental evaluation of intelligent systems until I moved to UCI. Our research group in machine learning there became dissatisfied with the state of evaluation in the field, and this led to a search for something better.

When I became Executive Editor of the new journal Machine Learning in 1985, I began to encourage authors to include experimental evidence that their methods worked as claimed, but my ideas on this front were not yet well formed. This changed with the redefinition (and it was a redefinition) of learning as improvement on some performance task, following the leads of Ross Quinlan, Jeff Schlimmer, and Doug Fisher.

This realization led to clear ideas for the experimental evaluation of learning algorithms, many of them borrowed from the literature on cognitive psychology. The earliest general statement of these ideas appeared in my 1988 editorial in Machine Learning, which Dennis Kibler and I expanded into an invited workshop paper. Mark Drummond and I later extended this scheme to experimental studies of planning.

The current state of experimental research in machine learning incorporates some of these ideas but, alas, not others, and has become increasingly careful but increasingly narrow at the same time. For instance, it has come to rely mainly on comparative runs over data sets from the UCI repository (originally collected by David Aha), but ignores the need for synthetic data to test explicit hypotheses. I discuss this issue in my editorial for IEEE Expert and provide one example of such a study in a KDD paper.

Related Publications

Langley, P., & Messina, E. (2004). Experimental studies of integrated cognitive systems. Proceedings of the Performance Metrics for Intelligent Systems Workshop. Gaithersburg, MD.
Kalton, A., Langley, P., Wagstaff, K., & Yoo, J. (2001). Generalized clustering, supervised learning, and data assignment. Proceedings of the Seventh International Conference on Knowledge Discovery and Data Mining (pp. 299-304). San Francisco: ACM Press.
Langley, P. (2000). Crafting papers on machine learning. Proceedings of the Seventeenth International Conference on Machine Learning (pp. 1207-1211). Stanford, CA: Morgan Kaufmann.
Langley, P., & Fehling, M. (1998). The experimental study of adaptive user interfaces (Technical Report 98-3). Institute for the Study of Learning and Expertise, Palo Alto, CA.
Langley, P. (October, 1996). Relevance and insight in experimental studies. IEEE Expert, 11-12.
Langley, P., & Kibler, D. (1991). The experimental study of machine learning. Unpublished manuscript, AI Research Branch, NASA Ames Research Center, Moffett Field, CA.
Langley, P., & Drummond, M. (1990). Toward an experimental science of planning. Proceedings of the 1990 DARPA Workshop on Innovative Approaches to Planning, Scheduling, and Control (pp. 109-114). San Diego, CA: Morgan Kaufmann.
Kibler, D., & Langley, P. (1988). Machine learning as an experimental science. Proceedings of the Third European Working Session on Learning (pp. 81-92). Glasgow: Pittman.
Langley, P. (1988). Machine learning as an experimental science. Machine Learning, 3, 5-8.

For more information, send electronic mail to langley@isle.org