Using Informative Behavior to Increase Engagement in the TAMER Framework

Abstract

Noting the increasing integration of autonomous agents into our daily lives, we seek to increase such agents' usefulness by improving their interfaces for human interaction. In this paper, we address a relatively unexplored aspect of designing agents that learn from human training by investigating how the agent's non-task behavior can elicit human feedback of higher quality and quantity. Although many approaches have been developed to enable agents to learn with human assistance, we use the TAMER framework, which facilitates the training of agents by human-generated reward signals, i.e., judgements of the quality of the agent's actions, as the foundation for our investigation. Based on the TAMER framework, we consider two new training interfaces to increase human engagement in the training process and thereby improve the agent's task performance. One provides information on the agent's uncertainty, the other on its performance. We tested the two new interfaces and a control through a user study of 51 human subjects. Our results show that these interfaces can induce the trainers to train longer and give more feedback. The agent's performance, however, only increases in response to the addition of performance-oriented information, not by sharing uncertainty levels. Subsequent analysis of our results suggest that the organizational maxim about human behavior, ``you get what you measure''—i.e., sharing metrics with people causes them to focus on maximizing or minimizing those metrics while de-emphasizing other objectives— also applies to the training of agents.