Goals & Visions

Research themes

Projects

Courses

Events

People

Useful links

Back to top-page

The Stockholm HMI Center

Models for human interaction with mobile service robots

Collaborative project between HMI (Graduate School in Human-Machine Interaction) and CAS (Center for Autonomous Systems)

Henrik Christensen and Kerstin S. Eklundh, May 1997

The Centre for Autonomous Systems does research in autonomous systems such as mobile platforms. The research programme includes a thematic project on the 'Intelligent Service Agent', which is supposed to be mobile platform that can perform cleaning and transportation tasks in a domestic setting. In addition it may be used as a dextrous assistant to handicapped and elderly. It is considered unlikely that the ordinary person (and handicapped and elderly) will use well known paradigms like mouse/keyboard for instructing a service agent to perform specific tasks. There is thus a need for a study of how humans can interact with such robots in a flexible manner.

An initial proposal suggests use of gestures in combination with speech. Such a multi-modal interface will open up for research in the interrelationship between visual and spoken commands and the corresponding representations. It seems obvious that the coupling will generate a need for interrelated control of both modalities. A command like "Pick that (speficied by a gesture) up and up it in the kitchen" will require that the system identifies/localises the object "that". In contrast the command "pick up the cup and put it in the kitchen" will require localisation of the item "cup". These examples are rather simple, but illustrate an initial form for interaction.

On the output side, there are a range of issues pertaining to feedback about the robot's state and capacity. In a given situation, how can the robot display its communicative competence and current assumptions to the user?

If the concept above is generalised to human interaction with 'intelligent machines' then it will naturally include a rich set of different items in a domestic setting ranging for simple interaction with household appliances to intelligent service agents. The potential is here extremely large, and at the same time very little work has been done in this area.

Recently the National Science Foundation (NSF) and the Department of Energy (DOE) have defined that human-robot interaction is of strategic importance and noted that the research is only in its infancy. The need is motivated by the fact that robots gradually will move from well controlled production lines into daily life situations. To accommodate this it is important that features like safety and acceptability are studied, to ensure a robust dialogue/interaction that is both intuitive and flexible enough to be used by people without any special training. See http://www.sandia.gov/isrc/RIMCC_Whitepaper.html and http://rvsl.ece.neu.edu/pub/ras/GC.html for more details.

In this joint research project the issue of Human-Robot Interaction will be studied. The project will include:

1. A study of user acceptability requirements for human-robot interaction
2. Construction of a prototype system for gesture and speech based robot interaction
3. A study of more general dialogue systems for human-robot interaction where emphasis will be on the linguistic aspects of such dialogues. In particular, this will include a theoretical and empirical analysis of situated language use in the context of a "command and instruct" dialogue such as the one envisioned for the intelligent service agent.

The Centre for Autonomous Systems has about 8 PhD students working on the Intelligent Service Robot, of which one or two will be working on gesture interpretation and its integration with a speech interpretation system. CAS will not do any research related to speech interpretation or synthesis, these components will be based on research / commercial systems available from elsewhere. The HMI project will contribue to the joint project with a single PhD student.

For detailed specification of the joint project, an M.Sc. project will be initiated during the fall of 1997.

Kerstin S. Eklundh, HMI and IPLab will be responsible for the project. Henrik I Christensen will be the CAS contact for the project. Also, Lars Oestreicher and Zayera Khan, IPLab, will be working in this project.