Human model evaluation in interactive supervised learning

Fiebrink, Rebecca; Cook, Perry R. and Trueman, Dan. 2011. 'Human model evaluation in interactive supervised learning'. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vancouver, Canada 7-12 May 2011. [Conference or Workshop Item]

[img]
Preview
Text
FiebrinkCookTrueman_CHI2011.pdf - Accepted Version

Download (336kB) | Preview

Abstract or Description

Model evaluation plays a special role in interactive machine learning (IML) systems in which users rely on their assessment of a model's performance in order to determine how to improve it. A better understanding of what model criteria are important to users can therefore inform the design of user interfaces for model evaluation as well as the choice and design of learning algorithms. We present work studying the evaluation practices of end users interactively building supervised learning systems for real-world gesture analysis problems. We examine users' model evaluation criteria, which span conventionally relevant criteria such as accuracy and cost, as well as novel criteria such as unexpectedness. We observed that users employed evaluation techniques---including cross-validation and direct, real-time evaluation---not only to make relevant judgments of algorithms' performance and interactively improve the trained models, but also to learn to provide more effective training data. Furthermore, we observed that evaluation taught users about what types of models were easy or possible to build, and users sometimes used this information to modify the learning problem definition or their plans for using the trained models in practice. We discuss the implications of these findings with regard to the role of generalization accuracy in IML, the design of new algorithms and interfaces, and the scope of potential benefits of incorporating human interaction in the design of supervised learning systems.

Item Type:

Conference or Workshop Item (Paper)

Identification Number (DOI):

https://doi.org/10.1145/1978942.1978965

Related URLs:

Departments, Centres and Research Units:

Computing
Computing > Embodied AudioVisual Interaction Group (EAVI)

Dates:

DateEvent
2011Published
1 December 2010Accepted

Event Location:

Vancouver, Canada

Date range:

7-12 May 2011

Item ID:

9426

Date Deposited:

04 Nov 2013 12:26

Last Modified:

29 Apr 2020 17:03

URI:

https://research.gold.ac.uk/id/eprint/9426

View statistics for this item...

Edit Record Edit Record (login required)