SuperAssist: Personal Assistants for Distributed Supervision of Complex Task Environments

Geert de Haan (1), Mark A. Neerincx (1),(2) and Charles A.P.G. van der Mast (1)

(1) Unit of Man-Machine Interaction
Department of Mediamatics, Delft University of Technology, Mekelweg 4, 2628 CD,
Delft, the Netherlands,

(2) TNO Human Factors Research Institute
P.O. Box 23, 3769 ZG,
Soesterberg, the Netherlands


This paper describes the planned research in the SuperAssist project, introducing personal assistants in the care of diabetes patients among the patients themselves, medical specialists looking after their healthcare and technical specialist maintaining the health of the devices involved. The paper discusses the issues of trust and cooperation as the critical success factors within this multi-user multi-agent project and within the future of agent-based healthcare attempting to increase the self-help abilities of individual patients.


Human-computer interaction, personalization, supervision, ambient intelligence, health care, personal assistants, cooperative problem-solving.


TNO Human Factors, Delft University of Technology and Leiden University Medical Centre are developing guidelines, models and methods for joint user-"electronic assistant" supervision of critical equipment and information. The aim is to establish effective and efficient distributed supervision of networked information compilations and technical equipment, which is trustworthy for the user and takes place in a socially approved manner. Specific innovative project results are: communication and interaction model for these assistants; methods for joint human-computer supervision; improved test methods, tools and criteria for systematic assessment of user experience; "best practice" implementation-method and guidelines; and a "proof of concept" in the transmural health care domain (e.g. diabetes). For the medical application domain, the SuperAssist framework will reduce costs by improving the local, self-care capacity of people by efficient employment of remote, distributed expertise.

The project's business partners - Science & Technology, Philips Research, Pemstar and Sigmax PDA Solutions - bring in their technology and contribute to the development and validation of SuperAssist elements. The joint activities are included. in cognitive engineering cycles, in which the foundation, specification and demos of the SuperAssist concept are being refined and extended. The SuperAssist research takes place in the medical domain, but it aims at a generic solution for the distributed supervision of complex environments.


Ms. Brown is a vital 72 year old who, after loosing her husband several years ago, refound her balance and takes a keen interest in bicycling and reading. Two years ago she was diagnosed with type II diabetes for which a diet and medication were prescribed. Although she is sincere in following up the medical directions, sometimes her memory fails her and she simply forgets things like when she has last taken her pills or that she should only buy low glucose products. Like many of her friends she is enjoying life but every now and then coping with the restrictions of old-age takes the form of a struggle. Although she accepts having an incurable disease, she is definitely not going to live her life like a sick person.

Since ms. Brown is well overweight, she has been asked to participate in a therapeutic program whose purpose is to attempt to let participants lose some of that weight in a responsible yet playful manner. As a result, every day before breakfast and before bedtime Ms. Brown consults her personal diabetes assistant, provided to her as part of the program, to check her blood glucose levels and to help her remind whether she has taken her pills or not. She can also fill in a questionnaire regarding food intake, exercise, and stress in order to help her manage her food intake in relation to her activities. Since ms. Brown does not to be her illness she has kindly refused to carry the assistant all day as a portable dietary advisor. Because the diabetes assistant is not used as a portable and to address her deteriorating eyesight, the assistant is equipped with a extra-large extra-bright screen, a few large and clearly labelled buttons. In addition, the form and content of the dialogue have been adapted for ms. Brown according to her preferences and to the way she uses the diabetes assistant.

On three consecutive days, ms. Brown's blood glucose level has been slightly higher than normal and today it is rather high. An abnormal reading is not necessarily a cause for alarm. One possibility is that there is nothing wrong with ms. Brown but rather that the technique let her down. The reading of the blood glucose level is derived from the light passing through the finger that ms. Brown offered for testing, which is much more comfortable but also less reliable then blood sampling. Ms. Brown may call the district-nurse to do blood tests but she prefers not to, because he is busy enough with the "really sick people". Finally, the measurement might be genuine. It may be that ms. Brown has forgotten to take her night time pills. However, the display of the assistant clearly shows that she has confirmed taking her pills until this morning and besides, the compartments of her pill-box are empty. It may be -again- that she has accidentally taken a regular instead of a low sugar desert.

A little alarmed, ms. Brown at first doesn't know what to do but then she sees the help-button on the diabetes assistant and remembers from the instruction session that "if you would like to ask a question, first try the help-button". This she does and a friendly voice assures her that there is only a little worry but not an acute problem. Furthermore, the assistant suggests her to redo the measurement, this time using the little finger of her other hand. Using her other little finger, ms. Brown now learns that her blood glucose level is only slightly higher than normal and her assistant asks her to take her pills, including an extra TZD "you know, the big blue one" just to be on the safe side.


By this time, also behind the scenes some activity has taken place. Because of the measurement error, a maintenance module inside the assistant increased the problem count of the measurement module, reset it and performed a self evaluation test in order to exclude a range of technical problems. Also a temporary wireless network is set-up for a number of routine and problem reports. When the assistants' problem count exceeds a certain level of serious measurement or transmission problems, a technical expert is informed for further maintenance or for a replacement of the device. In addition, with serious problems the district nurse will be informed of the existence of a possibly faulty device, so that he can check out to take over or provide a replacement diabetes assistant. Because in this case, the measurements remained within the safety boundaries, there was no need to alarm the specialist diabetes team, although, as a matter of routine, both blood glucose readings and the measurement problem are fed into ms. Browns medical dossier. Finally, because there has been a change in ms. Brown's otherwise rock stable blood glucose level which deviated from normal over a number of days, a reminder is send to the district nurse. The personal patient-visiting assistant agent will remind the nurse the next time he goes on a periodical visit to ms. Brown. Even though the diabetes assistant is "allowed" within certain boundaries to change a patient's medication, only medical specialists (or in their place the district nurses) are allowed to make any enduring or substantial changes.


In the SuperAssist project we intent to follow a scenario-based approach [1] in that the project consists of a number of phases, each of which is approached iteratively, and guided by hypothetical and real scenario's of would-be situations [2]. In dealing with the novel situation of having multiple types of users communicate and cooperate with multiple types of agents, the domain is relatively new and, according to Erickson [3], a scenario-based approach is the preferred method. In addition to such standard HCI practices, it is a "sine qua non" that users should participate in the development of a personal service or consumer product that SuperAssist aims at [4].

Even though the project targets at a novel type of application in a new area of multi-agent multi-user interaction, it does not start from scratch. Apart from being concerned with software agents which assist human beings in the area of ubiquitous computing [5, 6], SuperAssist attempts to build further upon the PALS project, a Dutch project investigating the requirements to meet changing user needs and usage contexts with respect to web-services (see:. [7, 8, 9]). From this and other projects, two factors turned out to be of critical importance for the success of human-agent interaction: trust and cooperation. Trust in the sense that people need a basic level of trust in a system or personal assistant to "do business" with them, and cooperation in the sense that, like in human-human cooperation, participants rely on images, models or ideas about the communication and cooperation abilities of their discussion partners in order to set appropriate expectations.

The trust of a patient, a medical specialist or a nurse in their personal assistant is a prerequisite to delegate part of their responsibilities to the electronic device. To enable trust and delegation between people and electronic assistants, it is a first necessity that the form and content of the interaction dialogue are acceptable to the human user, including a possibly frightened patient or an excessively busy medical worker. Here, it may be possible to adapt the dialogue to the current context of use including aspects like fear and stress [9]. Apart from a humanly usable dialogue for solving problems, it is necessary that there is a shared view on the problem domain. This may be achieved by creating a representation of the problem space in which rules exist or may be derived to guide the behaviour of the assistants. Note that, even in a well-delimited problem domain as diabetes, this is a major undertaking due to influences from outside the system, such as the nurses' other responsibilities or the complex (side) effects between things like medication, stress and diet.

Cooperation for problem solving between people and their assistants and the cooperation between assistants among themselves requires some kind of model which at least describes 'what to expect from whom' in terms of questions, actions, etc. In (software) multi-agent architectures like e.g. FIPA (see: [10]) the question 'what to expect from whom' is specified in agent-communication models, either by way of static models, specified beforehand, or in on-request models, used in dynamically changing environments. FIPA is a proposal mainly aimed at enabling communication between software agents but for heterogeneous networks with human agents, it is similarly required that some model exists to specify what other participants may contribute and how to ask them; either directly or by means of some user-agent. In the scenario this is exemplified by ms. Brown's diabetes assistant asking her (on behalf of some supervision agent) to redo the measurement of her blood glucose. Also here, personalisation and adaptation come into play, for instance in deciding whether is appropriate to ask the patient to do something or to alarm the nurse. In addition to this it is probably even more important to utilise inter-agent communication to prevent the human participants from becoming overloaded with questions from their software counterparts.

The SuperAssist project attempts to set up an integrated healthcare service in the area of diabetes treatment, assisted by electronic devices and software agents. From this research we expect to be able to derive models, architectures, guidelines and general design knowledge to introduce human-agent systems that enable people to help themselves to a greater extend than they are able to do, now. As such, the SuperAssist case is intended as a test case for a range or healthcare and other services and not just as yet another smart solution for a particular problem. Only by introducing, testing and validating general tools, we will be able to help keep future healthcare costs within the boundaries of affordability and manageability - at the very least for individual patients like ms. Brown.

The SuperAssist project also employs diabetes as a vehicle to learn general lessons about how to introduce trustworthy and cooperative electronic or software assistants into the everyday life of real people. Until recently, human-agent interaction has been studied in settings where a single agent assisted a single user, as in supporting searching, information filtering, and critiquing systems. In the SuperAssist project, research and development are extended to multi-user multi-agent communication and cooperative problem solving. This novel research area does not only introduce new types of questions and new types of problems but it is also a starting point for the integration of separate intelligent applications into intelligent overall services.


1. Carroll, J.M. (1995). Scenario-based Design: Envisioning Work and Technology in System Development. New York, John Wiley.

2. Rizzo, A and Bacigalupo, M. (2004). Report on testing of full scale scenarios. Deliverable 3.2 of the CARE/Innovative Action - CREA Project. Available from: as CREA_UNISI_WP3_D3.2_v1.1.pdf

3. Erickson, T. (1995). Notes on Design Practice: Stories and Prototypes as Catalysts for Communication. Carroll, J.M. (ed.) Scenario-Based Design: Envisioning Work and Technology in System development, 37-58.

4. Kemp, J.A.M. and van Gelderen, T. (1996). Co-discovery Exploration: an informal method for the iterative design of consumer products. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A. and McClelland, I.L. (eds.) Usability Evaluation in Industry, pp. 139-146. Taylor and Francis.

5. Maes, P. (1998). Reflections on ... Agents that Reduce Work and Information Overload. In: Maybury, M.T. and Wahlster, W. (eds.), Readings in Intelligent User Interfaces, Morgan Kaufmann. pg. 536.

6. Weiser, M. (1991). The Computer for the 21st Century. Scientific American, 265(3), 94-104.

7. PALS, Personal Assistant for onLine Services. IOP/MMI project website; see

8. Herder, E. (2003). Utility-Based Evaluation of Adaptive Systems . Proceedings of the 2nd Workshop on Empirical Evaluation of Adaptive Systems at the 9th Intl. Conference on User Modeling - Johnstown PA, 25-30.

9. Neerincx, M.A. and Streefkerk, J.W. (2003). Interacting in Desktop and Mobile Context: Emotion, Trust and Task Performance. Proceedings of the first European Symposium on Ambient Intelligence (EUSAI), Eindhoven, The Netherlands. Springer-Verlag.

10. FIPA, the Foundation for Intelligent Physical Agents. See:

Position paper presented at the Eusai 2004 workshop on Wellbeing at Home. 8 Nov 2004, TU Eindhoven, The Netherlands.