Becky Hill & John Long 

Ergonomics & HCI Unit, University College London,

26 Bedford Way, London WC1H 0AP, England.

Tel: +44 (0)171 387 7050.

Fax: +44 (0)171 580 1100.

E-mail: b.hill@ucl.ac.uk.

ABSTRACT 

The emergency management combined response system is a planning and control system, set up to manage disasters. This paper presents a preliminary model of the planning and control of the combined response system and its domain. The model is based on a framework for modelling planning and control for multiple task work and on data from a training scenario. The application of the framework has extended its scope to the new domain of emergency management combined response. In addition, the preliminary model is used to identify tasks and behaviours of the combined response system which highlight issues of co-ordination.

Keywords 

Planning and control, multiple tasks, emergency management

INTRODUCTION 

This paper presents an informal analytic assessment of a model of the emergency management combined response (EMCR) worksystem with respect to the data from a training scenario. The model was derived by applying a framework for modelling planning and control in multiple task work (PCMT) to the data from an emergency management training scenario. This framework has been applied to develop models of three office administration planning and control domains – Medical Reception (Hill, Smith, Long and Whitefield 1995) -Secretarial Office Administration (Smith, Hill, Long and Whitefield 1992) and Solicitors Legal Work. The intention here was to extend the scope of the PCMT framework by using it to model a domain other than office administration. The first section of the paper characterises the domain to be modelled, identifying issues with the current design of the EMCR worksystem. The following section outlines the PCMT framework. Next, the data collection, training scenario, and example data are described. The model of PCMT- EMCR is then presented. Using the model to identify overlap between worksystem behaviours is exemplified. Last, issues related to the use of this framework for modelling the domain of EMCR are identified, along with plans for future work.

EMCR WORKSYSTEM 

The planning and control system, which is set up for emergency response to disaster, is that of the ‘combined’ response (EMCR). The EMCR worksystem has a command and control organisation with a three tier structure. Different agencies use this structure to organise their own planning behaviours, so that they co-ordinate effectively with one another. The three levels of command and control are operational, tactical and strategic. At each level the agencies have their own commander for co-ordinating the response. At the strategic level, these commanders make up a ‘senior co-ordinating group’. The operational response is carried out by each agency, concentrating on its specific roles (tasks) within its areas of responsibility. For example, the Fire Service fight fires. The tactical response determines the priority in allocating resources, for example additional fire engines. It also plans and co-ordinates the overall response, obtaining other resources as required. The strategic co-ordinating group formulates the overall policy within which the response to a major incident is made. This EMCR worksystem has been specified to ensure better co-ordination of the different agencies involved in the response. Co-ordination in this context is defined as the ‘harmonious integration of the expertise of all the agencies involved with the object of effectively and efficiently bringing the incident to a successful conclusion.'(Emergency planning college document 1995). The principal agencies involved are the Police, Fire Service and Ambulance Service. The primary objectives for all the emergency services are: ‘to save life, prevent escalation of the disaster, to relieve suffering, to facilitate investigation of the incident, safeguard the environment, protect property and restore normality’ (Home Office Publication 1994).

However, a succession of public enquiry documents analysing various disasters have identified co-ordination problems within EMCR. The Home Office Emergency Planning Research Group has also identified problems relating to co-ordination within EMCR (see acknowledgements).

Each emergency service has its own major incident plan. Each plan specifies a set of roles/tasks that need to be supported, when responding to a disaster, for example, evacuation, setting up a casualty clearing station etc. Some of the tasks, at the most generic level of description, are the same for the three emergency services, for example the Police and the Fire Service both support evacuation tasks. Most of the behaviours which support these tasks for the two services are different, but some are the same. Sometimes the behaviours overlap between the different services, when they are supporting different tasks. The Fire Service may be attempting to set up an inner cordon to contain a hazardous scene; the Ambulance Service may be

attempting to access the same scene to locate casualties. It is proposed here, that when behaviours overlap, co-ordination between the services may become a problem. (Behaviours do not necessarily have to be the same to overlap.)

In developing the present preliminary model of EMCR with respect to a training scenario, it is intended to identify such overlapping behaviours to support better the identification of the co-ordination problems raised by the system.

FRAMEWORK 

A framework was developed for modelling the planning and control of multiple task work (PCMT) in office administration. The PCMT framework makes a fundamental distinction between an interactive worksystem, comprising one or more users and computers/devices, and its domain of application, constituting the work carried out by the worksystem. This distinction allows for an expression of the effectiveness (performance) with which the work is carried out, conceptualised as a function of two factors: the quality of the task (i.e. whether the desired goals have been achieved), and resource costs (i.e. the resources required by the worksystem to accomplish the work) (Dowell and Long 1989).

In the framework, a domain of application (or work domain e.g. the disaster) is described in terms of objects, which may be abstract or physical. Objects are defined by their attributes, which have values. The attribute values of an object may be related to the attribute values of one or more other objects. An object at any time is determined by the values of its attributes. The worksystem performs work by changing the value of domain objects (i.e. by transforming their attribute values) to their desired values, as specified by the work goal.

The framework defines a number of worksystem structures for the planning and control of multiple task work. These structures are expressed at both abstract and physical levels of description. First, the framework describes the worksystem’s abstract cognitive structures. These structures comprise four processes (planning, controlling, perceiving and executing), and two representational structures (plans and knowledge-of-tasks). The four processes support the behaviours of planning, control, perception and execution respectively. A complete description for this set of structures, and the associated rationale, is to be found elsewhere (Smith, Hill, Long and Whitefield, 1992).

At the first (abstract) level of description, Plans are specifications of required transformations of domain objects and/or of required behaviours (to achieve goals). They may be partial (in the sense that they may specify only some of the behaviours or transformations). They may also be general (in the sense that some behaviours or transformations may be specified only generally, and not at a level which can be directly executed). Planning behaviours thus specify the required transformations and/or behaviours to support domain object transformations.

Perception and execution behaviours are, respectively, those whereby the worksystem acquires information about the domain objects and those whereby it carries out work which changes the value of the attributes for those objects as desired. Information about domain objects from perception behaviours is expressed in the knowledge-of-tasks representation. Control behaviours entail deciding which behaviour to carry out next, both within and between tasks.

The second level of description of planning and control structures is physical, wherein the framework describes the distribution of the abstract cognitive structures across the physically separate user and devices of particular worksystems.

This framework was chosen to model the combined response worksystem because:

(i) the EMCR worksystem is a planning and control system;

(ii) the work involves multiple tasks; and

(iii) there is a need to identify the tasks and behaviours of the worksystem in relationship to the work, to better identify co-ordination problems within the EMCR worksystem.

DATA COLLECTION 

Data were collected by means of a training scenario which took place at the Home Office Emergency Planning Training College. The trainees were members of the emergency services and local authority emergency planning officers. The exercise required the trainees to describe their response to the disaster scenario from initial response to restoration of normality. Data were collected by transcribing the descriptions given by the trainees, and through presentations given by the trainees about their response.

There are various phases in response to the disaster scenario, each of which can be clearly defined, and which involve different worksystem configurations and different worksystem behaviours. In this paper, only one worksystem configuration and its behaviours is modelled – that of the initial response phase to the scenario.

Scenario 

‘At 9.30am on a weekday during school term time, a tanker train en-route from a refinery to an airport fuel depot is derailed whilst passing over a railway bridge. The railway bridge carries the railway over the main access road to a town from east to west. The main access road bisects the town in a north/ south direction. There are market stalls set up on the roadway on either side of the bridge. At the time of the accident, a tourist’ bus carrying 45 foreign tourists is passing beneath the railway bridge. One of the tank cars is ruptured during the derailment and aviation fuel flows down the sides of the embankment onto the roadway. Flammable vapours from the fuel have been ignited by an open gas burner from a catering caravan. The explosion has created

severe structural damage in a 30 metre radius and moderate structural damage in 100 metre radius. At least 50 people have been killed including some of the foreign tourists, many people have received burns, and many people are trapped. A number have also been contaminated with aviation fuel. The leaking fuel has run down the road and is entering the canal and watercourses at the bottom of the incline. There is evidence to believe that vandalism may be responsible for the derailment.’

DESCRIPTION OF THE INITIAL COMBINED RESPONSE DATA 

A major incident was declared within 15 minutes of assessing the situation. Thus, all the emergency services initiated their major incident plans. Each service has a set of defined roles or tasks specified in their individual plans. For this scenario, examples of initial Police roles/tasks were:

setting up and manning an outer cordon around the site (a larger area than the scene).

preserving and managing the scene

logging all personnel entering the outer cordon

establishing routes and rendezvous points and marshalling areas for all the emergency services

Examples of the initial Fire Service roles/tasks were:

set up an inner cordon

control the fire

carry out immediate rescue

Stem the flow of the hazardous substance

Monitor the health and safety of all inside the inner cordon

Examples of the initial Ambulance Service roles/tasks were:

set up a casualty clearing point

set up a triage point

get medics to the scene

help with evacuation of bedridden people

For each of the tasks the personnel required were identified by, and are shown in, the model.

PCMT-EMCR MODEL 

The model describes the EMCR worksystem and its domain for the initial response scenario data. The initial response in this scenario had no strategic level of command. Thus, although the strategic level physical worksystem users and devices are represented in the model, they are included for completeness. No further description is offered. Also, the operational physical users and devices change rapidly over time. The model represents only those entities present for initial response. The local authority response has not been modelled.

EMCR Domain 

Based on the PCMT framework, the EMCR domain is expressed as those objects, whose transformation constitutes the work of the EMCR worksystem.

In the model, the domain is conceptualised as having a single disaster object comprising other abstract objects, such as lives, property, environment etc. The disaster object and its component sub-objects are abstract. The sub-objects of the domain are based on the identified primary objectives of EMCR (see earlier). The domain in addition contains physical objects.

The work carried out by the EMCR worksystem is thus the transformation of a ‘disaster object’s’ attribute values. Each task carried out by the EMCR worksystem transforms the attribute values of the disaster object. For example, one attribute is stability, which has values along a continuum (i.e. very unstable, slightly unstable, unstable, nearly stable, moderately stable, stable). The work of the EMCR worksystem is to transform this attribute to a desired level of stability. In order to transform this attribute, the attributes of the component sub-objects must be transformed. This attribute value changes by manipulating the values of the attributes of the other objects of the domain, for example, transforming the ‘lives object’ attribute survivor from not rescued to rescued. The transformation of the abstract object attributes result from the manipulation of the physical domain objects of the worksystem. Attributes may be affordant or dispositional. Affordant attributes are transformed by the worksystem; their transformation constitutes the work performed. Dispositional attributes are relevant to the work (they need to be perceived by the worksystem, but are not changed by the worksystem ). For example for the ‘lives object’ the attribute survivor has a value of mobility injured or mobility uninjured. This value needs to be perceived by the EMCR worksystem, so that the Ambulance Service knows there is a casualty to be transported. However, the worksystem does not change the injured state of the survivor. In Figure 1, all the starred attributes are dispositional.

Figure 1 shows the physical domain objects (but not their attributes) in the model. Included are all the objects identified from the scenario data. The objects include those associated with the example which follows.

EMCR Worksystem 

The model of the EMCR worksystem (see Figure 1) shows the cognitive structures of the PCMT framework. The cognitive structures in Figure 1 embody the planning and control behaviours described earlier. The framework comprises four process structures: planning; controlling; perceiving; and executing and two representation structures: plans and knowledge of tasks. For EMCR, these cognitive structures were identified as follows:

A perceiving process which acquires information about property, lives and other domain objects, such as their risk status. This information appears in the knowledge-of-tasks representation. A task in the EMCR will be different for the different agencies. A task for the Police could be setting up a casualty clearance point. A task for the Fire Service could be making sure all personnel are safe within the inner cordon.

An executing process which transforms domain object attribute values. For example, moving injured survivors

away from the scene transforms the ‘lives object’ attribute survivor from not rescued to rescued.

A controlling process which decides which of the other processes should be carried out next based on the plans and knowledge-of-tasks. For example the controlling process might direct the executing process to perform next the execution behaviours of evacuating people (based on the perceiving process that identified where people were, and the knowledge-of-tasks information that people in this situation are at risk), rather than directing the planning process to specify how to preserve the scene. (Thus, there is prioritisation of the tasks carried out by EMCR.)

A planning process, which constructs plans based on knowledge-of-tasks. For example using information from knowledge-of-tasks that the scene needs to be preserved for investigation, the planning process constructs a plan of the sequence of behaviours to carry out this task.

The Plans representation structure embodies the different types of plan used in the combined response.

A knowledge-of-tasks structure, which represents knowledge of relevant aspects of the work domain. For example, the type of hazardous substance.

The physical worksystem comprises the Police and their devices, the Fire Service and their devices, the Ambulance Service and their devices (plus other agencies and their devices which may be involved). The framework allows the construction of alternative models of the distribution of cognitive structures across the user and devices. Thus, it supports reasoning about allocation of function. In the case of the EMCR worksystem, planning and control are distributed across the different levels of the worksystem hierarchy. In the model, the distribution of the structures across the physical worksystem is not shown. However, these physical worksystem structures support its behaviours. In the example which follows, the worksystem behaviours are identified for a selection of data from the training scenario to illustrate this distribution across the worksystem.

RELATIONSHIP BETWEEN THE DOMAIN AND THE WORKSYSTEM 

This section describes an example of two tasks being carried out by the combined response system, in terms of the planning, control, perception and execution behaviours and the transformations these behaviours perform in the domain. This description helps to identify potential conflicts within the combined response system behaviours which may cause co-ordination problems. It also highlights some issues related to using the PCMT framework for modelling EMCR, which will be discussed in the final section.

The Fire Service operational commander carries out perception behaviours that update his knowledge-of-tasks with the information that there are structurally damaged buildings, fires and leaking hazardous fuels. He then carries out control behaviours that direct him to consult the major incident plan. According to the plan, the Fire Service is responsible for setting up an inner safety cordon when there are hazards and dangers at the scene. It also maintains the safety of the emergency service personnel within this cordon. Based on this plan, the operational commander then carries out control behaviours that direct him to consult the operational plan for setting up of a cordon. The operational plan gives guidance for cordon set up and regulations.

The operational commander then carries out control behaviours that direct him to carry out planning, based on the operational plan and the knowledge-of-tasks. The planning specifies how the inner cordon should be set up and what the regulations are for entering it. The operational commander then carries out control behaviours that direct him to carry out an execution behaviour of setting up the cordon. This execution behaviour is carried out by the operational personnel setting up the cordon and maintaining specified safety regulations. It is the manipulation of the physical objects that transform the abstract disaster scene objects attribute scene from uncontained to contained.

The operational commander then carries out control behaviours that direct him to inform the tactical incident officer of the inner cordon set up. The tactical incident officer thus carries out perception behaviours which update his knowledge-of-tasks about the inner cordon set up. The tactical incident officer then carries out control behaviours that direct him to consult his plan to assess the resources required for the set up. He then carries out planning behaviours to specify the resources required for this task.

At the same time, the operational Ambulance senior officer is carrying out perception behaviours that update his knowledge-of-tasks with the information that there are a number of casualties at the scene. He then carries out control behaviours that direct him to consult his major incident plan. According to the plan, casualties must be located and then either treated at the scene and/or transported to hospital. He then carries out control behaviours that direct him to carry out planning behaviours to specify in the operational plan what personnel are required and how to access the casualties. This plan then directs him to carry out control behaviours to direct the execution behaviours of accessing, treating and transporting casualties. These execution behaviours are carried out by the Ambulance operational personnel. It is the manipulation of the physical objects which transform the abstract lives object attribute survivor from not transported to transported.

However, the Ambulance operational senior officer has not carried out perception behaviours that update his knowledge of tasks that the scene is now contained. Therefore, when the Ambulance personnel attempt to carry out their execution behaviours, they do not fulfill the proper safety requirements which would allow them to enter the inner cordon. Therefore, the execution behaviours of transporting casualties cannot be carried out. The primary objective of EMCR is to save life. Here, there is an overlap of behaviours which is

hindering this objective. Also, if the Ambulance Service is not allowed to carry out its execution behaviour, in trying to increase the saving of life, the fire service must carry out rescue execution behaviours to move the casualties to the edge of the inner cordon. These rescue execution behaviours will decrease the resources available for carrying out the execution behaviours of controlling the hazard, thus decreasing the effectiveness of the response to the secondary objective of preventing escalation of disaster

To attempt to rectify this situation, the Ambulance senior officer informs the Ambulance incident officer of the inner cordon regulations. Thus, the Ambulance incident officer carries out perception behaviours that update his knowledge-of-tasks about the inner cordon regulations. He then carries out control behaviours that direct him to carry out planning behaviours to specify in the plan the required equipment for all resources directed to the scene. Until these regularised resources arrive, the Ambulance incident officer needs to communicate with the Fire Service incident officer to try and negotiate access for the operational personnel. The Fire Service incident officer should have informed the Ambulance incident officer of the regulations, to obviate the problem of co-ordination.

CONCLUSIONS 

This paper has described the application of the PCMT framework to modelling the domain of Emergency management combined response (EMCR). A preliminary model has been developed based on data from a training scenario. This final section will discuss issues related to using the PCMT framework for modelling this domain, and the use of the model for identifying overlapping behaviours that highlight co-ordination issues within EMCR.

The PCMT framework is for use in planning and control for multiple task domains. EMCR is such a domain, but there are differences between this domain and the other domains already modelled. First, EMCR has a changing worksystem. The PCMT representation does not currently represent a changing worksystem. Thus, the preliminary model only represents one phase of response. Second, EMCR is multiple task, but unlike the previous domains studied, there are not multiple objects at the highest level i.e. multiple disaster objects, rather there are multiple sub-objects, such as multiple lives objects. Third, EMCR is made up of multiple agents within a complex three tier command structure. The PCMT framework has so far only modelled domains with a single level of operation. Thus, interactions between the different horizontal layers and different vertical layers of the system are difficult to describe in terms of the present framework. The difficulty can be appreciated in the example, where the different commanders liase, at the same horizontal level, and information flows between vertical levels.

However, the model provides a description of the system and its domain that can be used to identify overlapping behaviours, albeit for a simple example and in a limited context. The allocation of the abstract structures across the physical worksystem can be inferred from the description of the physical behaviours, which can then be used to identify possible reconfigurations of the worksystem to support more effective co-ordination.

Future work will develop more complete models of EMCR in response to a disaster scenario and in so doing extend the PCMT framework to accommodate the issues described earlier.

ACKNOWLEDGEMENTS 

This work is part funded by the Home Office Emergency Planning Research group under the EPSRC CASE scheme.

Thanks are due to the Home Office Emergency Planning college for their infinite co-operation with the data collection

REFERENCES 

Dowell, J. and Long, J. (1989) Towards a conception for an engineering discipline of human factors. Ergonomics, 32, 1513-1536.

Hill, B., Long, J.B., Smith, W. and Whitefield, A.D. (1995) A model of medical reception – the planning and control of multiple task work, Applied Cognitive Psychology, 9, 81-114.

Smith, M.W., Hill, B., Long, J.B. and Whitefield, A.D. (1992) Modelling the Relationship Between Planning, Control, Perception and Execution Behaviours in Interactive Worksystems. In D.Diaper, M.Harrison and A.Monk (Eds) People and Computers VII; Proceedings of HCI ’92. Cambridge University Press.

Dealing with Disaster 2nd Edition (1994). Home Office Publication HMSO London.

Emergency planning college document (1995) Management levels in response to a major incident – The Principles of Command and Control

Figure 1 The PCMT-EMCRS Model

 

Grand Challenges in HCI: the Quest for

Theory-led Design

Alistair Sutcliffe

Introduction

Centre for HCI Design, School of Informatics, University of

Manchester, PC Box 88, Manchester, UK

Tel: +44 161200 3315

Email: ags@manchester.ac.uk

A grand challenge of theory-led design is proposed for HCI. The history

and state of the art in HCI theory and knowledge is reviewed, expanding

on Long & Dowell’s conception of HCI. A new approach to bridging from

theory to design practice is proposed that uses generic task models as a

means of locating theory-based design advice. Theory-based knowledge

is also transferred to design as bridging models and as critical cognitive

aspects which are applied to task models. Design of specific applications

uses theory-based knowledge via mappings to generic task models and by

the application of bridging models. The approach is illustrated by a case

study investigation of the design issues in notifier systems and explained

with a design scenario of hospital patient monitoring.

Keywords: theory, cognitive models, claims, bridging representations.

1 Introduction

Grand challenges have been much in vogue in the computer science community

[UKCRC 2005]; however, the computational emphasis in the grand challenges

competition has sidelined HCI to a lip-service role in Memories for Life challenge.

Fred Brooks [2003], however, was more enlightened and cites design of effective

user interfaces as one of three major challenges for the 21st century. The HCI

community should not be depressed by this outcome; instead, I argue it should

propose more ambitious and wider ranging grand challenges of its own, and in doing

so demonstrate that it has a broader ambition than computer science. In this paper I

will propose and expound the challenge of Theory-led Design.

The development and application of theory is a sign of intellectual respect in

most scientific or engineering disciplines, and HCI should be no exception. Long &

Dowell’s conception of HCI as craft, science and engineering urged systematic

development of principles, laws and guidelines derived from theory [Long & Dowell

1989; Dowell & Long 1998]. Barnard and colleagues in the AMODEUS project

represent the most concerted effort to translate theory into design practice over

several years. The cognitive theory of Interacting Cognitive Sub-Systems [Teasdale

& Barnard 1993], in combination with more formal system modelling was proposed

as a toolbox for designers [Barnard et al. 2000]. In restricted case studies, these

models could be demonstrated to ‘cohabit’, i.e. applications of each model analysed

different views on the same problem [Harrison & Barnard 1993]. However, a

key problem has been how to translate knowledge in psychological theory into a

form that is usable by designers. The task artefact theory has been proposed as

a means of developing HCI knowledge [Carroll & Rosson 1992], using claims to

express fragments for theory-based knowledge with contextual references to design

problems, artefacts and scenarios [Sutcliffe 2000; Sutcliffe & Carroll 1999]. While

claims continue to make modest progress [Carroll 2000; Sutcliffe et al. 2003], they

have not as yet become widespread as a bridging representation between theory

and design practice. The state of the art for HCI advice still remains as principles

[Sutcliffe 2003; Dix et al. 2003], or golden rules [Shneiderman & Plaisant 2004]

and the legion of, one suspects, under-used ISO guidelines [ISO 1997, 1998, 2000].

More recently HCI patterns have proliferated [Borchers 2001; Van Welie 2005],

often without any validation. While patterns can express knowledge in a more

contextualized manner, unless they are composed into a pattern language [Alexander

et al. 1977], they can present incompatible, inconsistent and fragmentary pieces of

HCI knowledge. The standardization process of developing an HCI pattern language

has been slow to emerge and one can question whether it would be necessary

since patterns duplicate much knowledge already present in better regulated sets of

guidelines.

Some may consider development of exemplar artefacts as a more productive

way to advance HCI. However, I believe that without theory, HCI will be doomed to

the invention of stimulating yet ephemeral artefacts [e.g. Ishii et al. 2001] without

generating generalized long-term knowledge. The thesis in this paper is to urge

the HCI community to adopt the Science of Design [NSF 2004] and thereby become

more than a sub-discipline of computer science. Instead, HCI can become the Science

of User-Centred Technological Design. In the remainder of this paper, I will review

the role of theory and HCI knowledge, and then propose an innovative approach

based on a new role for familiar task models. This will be followed by application of

the approach to investigate a particular area of attentive user interfaces, with a brief

discussion to conclude the paper.

2 Theories and HCI Knowledge

HCI is not short of cognitive theories to apply to design; see for example ACT-R,

[Anderson & Lebiere 1998], EPIC [Kieras & Meyer 1997] and LICAI [Kitajima

& Poison 1997]. However, theories of cognition have proved difficult to apply in

practice. They can be used to motivate experiments and demonstrate interesting and

useful design phenomena, for example Homoff’s work on optimizing menu layout

Figure 1: Long & Dowell’s conception of HCI, revisited.

for efficient search, and eyetracking studies that demonstrate that banner adverts are

not attended to or remembered but do impose subliminal cognitive cost [Burke et al.

2004]. This work uses EPIC as its theoretical lens. GOMS is the prime exemplar

of applicable theory [John & Kieras 1995], and continues to be the focus of many

studies although evidence of its effective use in industry is hard to find. A cutdown

version of LICAI has been successfully developed in combination with an

information searching algorithm (latent semantic indexing) in the CoLiDes systems

that can predict design flaws in website navigation and link cues given a model of

the user’s search goals [Blackmon et al. 2003].

The quest for bridging models from theory to design was pursued with some

vigour, first via design rationale, providing the glue to synthesize contributions from

different modelling approaches [Bellotti et al. 1995], and more convincingly by

developing Cognitive Task Models to analyse task sequences and diagnose potential

design problems indicated by reasoning with the ICS cognitive architecture [Barnard

et al. 2000]. Barnard, in his 1998 plenary address to this conference, proposed

* syndetic theory’ and a blending of several theoretical influences that could be

applied to design problems at different levels of granularity — micro and macro

theory. Unfortunately, application of this approach to scaled-up industrial design

problems has never been demonstrated.

Interest in theoretical approaches has continued in cut-down approaches such

as the Resource Theory that proposes an event modelling view incorporating

cognitive issues such as working memory and selective attention [Wright et al.

2000]. Interactors [Dix & Mancini 1998] provided an interesting approach for

encapsulating HCI knowledge in what are essentially ‘abstract interaction types’

(cf. ADTs) as specifications or code for generic interactive functions, e.g. multilevel

undo. More specialized approaches, such as Information Foraging Theory,

continue to motivate research in information retrieval [Pirolli & Card 1999], while

Activity Theory [Nardi 1996] has been much in vogue as an analytic framework

for ethnographic-style studies of CSCW and CMC systems. Although theory is

being used, its prime role seems to be an indirect influence on design via analytic

frameworks or general conceptual influences. For instance, it is difficult to extract

prescriptive design advice from the concepts of conflict, activity level, artefacts and

zone of proximal development in Activity Theory even though some have attempted

to derive heuristics from such opaque theory [Bertelsen & B0dker 2003]. The range

of applicable theory has been demonstrated by interest in theories of emotion [e.g.

Ortony et al. 1988], as an inspiration to frame design questions [Norman 2004], and

more directly as an embedded cognitive model to drive the emotional response that

embodies conversational agents [Cassell et al. 1999]. So theory is far from dead,

but the pathway for theory application in design research is fragmented, and more

critically to design practice, is unclear. The status quo is summarized in Figure 1.

The influence of theory-based research on design is tending to be recruited via the

scientific route in Long & Dowell’s framework, i.e. theory motivates experiments or

case studies, the results of which provide design guidelines.

So is there a consensus of what has been achieved so far? In the majority of

HCI text books, for example [Dix et al. 2003; Preece et al. 2002; Shneiderman &

Plaisant 2004], theory is present as the Model Human Processor [Card et al. 1983],

and as a bridging model in Norman’s [1986] action cycle. What is missing is the

meta-level guidance on how to exploit and develop theory in the design process: the

grand challenge of theory-led design.

We need to return to the debate on bridging from theory to design. Some

promising directions have emerged in cognitive dimensions [Green & Petre 1996]

which propose principles for the design and evaluation initially of notations but more

recently of interactive systems. Object system models [Connell et al. 2003] attempt

to apply cognitive dimensions within design contexts for particular problems, while

critical parameters focus attention on key cognitive requirements for interactive

systems [Newman et al. 2000]. The important direction, I believe, is to focus on

typical classes of problems which are common in HCI, understand the cognitive

and social implications of those problem classes and then apply the appropriate

theories to solving them. HCI needs to investigate tasks more deeply in order

to understand the psychological implications of certain classes of problem. Take

monitoring tasks as a specific example. Monitoring tasks are known to be error

prone and people find it difficult to concentrate over long periods of time. The types

of errors people make and their problem-solving strategies can be predicted using

several theories [Norman 1988; Rasmussen 1986; Hollnagel 1998]. Human error

theory predicts that interruptions and time pressures produce frequency and recency

gambling pathologies resulting in sub-optimal problem-solving strategies [Reason

1990].

Monitoring tasks involve a perceptual process of recognizing significant events,

cognitive processes of interpreting events, and domain knowledge to evaluate the

significance of events in context. Norman’s [1986] cycle of action can be reversed

to set these issues in an interaction context; i.e. once a significant event has

been recognized, interpreted, and evaluated one has to plan, specify and execute

the appropriate response. The critical cognitive aspects of monitoring tasks are

Figure 2: Process map of theory-led design.

selective attention and recognition of events, reasoning with domain knowledge to

interpret the events and then planning appropriate responses. To develop effective

support systems the designer has to understand the problem, isolate the critical

cognitive aspects, then use the appropriate theories or experimental evidence to find

appropriate solutions. This approach is summarized in Figure 2.

By linking theory to generic task models and bridging representations —

be they cognitive dimensions, claims or critical parameters — a library of HCI

knowledge can be built which enables effective reuse. Generic task models [Sutcliffe

2002] can play a key role by providing an access point from task analysis to reusable

knowledge. If specific tasks could be mapped to generalized tasks, and if the latter

are associated with theoretical, sound design advice, then designers will have advice

placed in the context of their current problem. They still have to specialize the advice

and reason about trade-offs but this is an advance on ploughing through libraries of

design guidelines or patterns. However, generic task models need to be associated

with ‘critical aspects’ to place theory in its application context. I have used the aspect

oriented programming analogy since it illustrates the problem. Theory provides

knowledge that is applicable in many parts of a design, i.e. cross-cutting concerns.

The problem is to define the theory treatment for the aspect and when to apply it at

‘point cuts’ in the design. Linking critical aspects to generic task models indicates

the point cuts, but defining the treatment is not so simple. This requires reasoning

about trade-offs using bridging models. There is no silver bullet for producing high

quality designs; however, theory-led design can make the bullet easier to aim. In the

following section I will put this approach to the test with an exploration of design for

notifier user interfaces, which support monitoring tasks.

3 Application of Theory-led Design

To illustrate how we should apply theory in HCI, I will investigate a current research

area as a ‘gedanken experiment’. Notifiers, attentional user interfaces or dual task

displays have attracted considerable attention over recent years [Horvitz et al. 2003;

McCrickard et al. 2003]. This research is motivated by a general user need, i.e. to

enhance support for multitasking and to monitor information such as stock markets

or the weather, while performing another primary task. The research problem is best

encapsulated by a scenario. Marilyn, a senior staff nurse, is in charge of a chronic

care ward and has to keep an eye on her patients during the night shift. The patients

are automatically given medication via drips for management of chronic and postoperative

conditions, e.g. pain relievers, anti-coagulants, etc. The patients’ heart and

respiratory rates and blood pressure are automatically monitored and displayed on

a screen in Marilyn’s office. She has to monitor patients’ condition and adjust the

rate of medication if the measured parameters depart from the expected treatment

plan. More urgent alerts must be triggered if the patients experience distress which

could be picked up by heart ECG sensors or an emergency call button which can be

activated by patients. If all is quiet Marilyn is expected to catch up on the day’s paper

work to complete duty rosters, treatment records, etc.

Several approaches have been taken to this problem and many systems have

been implemented which provide background monitoring displays as information

tickers [Maglio & Campbell 2000], ambient wallpaper [Stasko et al. 2004] and

radar metaphors to communicate the status of monitored information to the user

[McCrickard & Chewar 2003]. However, these designs are based on intuition and

give no guarantee of their effectiveness. Horvitz et al. [2003] have taken a more

principled approach by proposing a cost of interruption measure expressed as the cost

of diverting from the primary task compared with the benefit from the information

gained from the monitoring tasks. This enables reasoning about when to interrupt

using Bayesian networks with user defined valuation of the task activity, cost of

interruption and environmental measures. Chewar et al. [2004] have elaborated

this idea with a conceptual framework composed of three dimensions: Interruption,

Reaction and Comprehension, that draws attention to the cognitive parameters of the

problem, namely, when and how to alert the user to draw their attention away from

the primary task, how to communicate information relevant to the secondary task

and how to promote comprehension of such information. While these approaches

are better than craft-level design, they do not constitute a theory-based approach.

To examine the problem in a more principled manner we have to understand

the problem from a psychological viewpoint, and this is the first impact of theory:

the ability to conceptualize interaction problems. The first part of the problem is

alerting the users to relevant information for the secondary task. This poses the

system initiative question. Should the monitored information be accessible to the

user all the time, and hence possibly distracting from the primary task, or should it

only be made available when the system decides it is critical for the user’s interests?

The second issue is how to ensure the user has seen or heard the critical information

and responded to it. The third part of the problem is to support the user’s transition

from the primary task to the secondary task.

When to alert requires a model of the user’s task, the user and the context. This

is part of a user modelling grand challenge since acquiring a model of the user and

their context requires solving the difficult problem of inferring complex states from

Table 1: Cognitive task audit: Monitor and Evaluate tasks.

low-level data streams. I shall finesse this challenge and return to my main theme.

Task analysis is the conventional HCI approach to understanding the user’s domain

[Diaper 2002; Johnson 1992]. While a goal model and procedural sequences can

give us breakpoints where intervention should be more appropriate it doesn’t go far

enough. The cognitive demands of the primary task will influence when the user

interface should signal an alert. The role of theory is to provide a cognitive audit so

we can understand the critical aspects of a particular task and plan appropriate action.

An example of the cognitive task audit is illustrated in Table 1. In the hospital patient

monitoring scenario, working memory has to cope with keeping track of several

patients while selective attention will be distributed between the several patient

monitoring devices and the foreground tasks of updating drug registers and patient

records. Reasoning will be rule-based using the nurse’s training, but knowledge

intensive judgement [Rasmussen 1986] will have to be applied when the unexpected

happens.

A cognitive task audit requires application of general psychological knowledge

to the task in question. While this does utilize theoretical knowledge, it does so

only second hand via education. The designer has to know the theory in order to

apply it in a cognitive task analysis. However, if a set of generic task models could

be developed and annotated with cognitive aspects, then theoretical knowledge could

be reused by non-HCI expert designers. A taxonomy of generic tasks exists [Sutcliffe

2002]; however, the process of mapping from specific task to these generic models,

annotating them with cognitive properties and then applying such knowledge are

open research questions.

Three tasks involved in the notifier problem are illustrated in Figure 3, showing

the critical cognitive aspects. For recognition the salience and modality of the

stimulus is important. This is set against environmental obstacles such as complexity

of the environment and competing, similar stimuli. Precise design advice depends on

knowledge of perceptual psychology [e.g. Ware 2000] which can not be encapsulated

Figure 3: Monitor, Interpret and Evaluate generic tasks annotated with critical aspects and environment

 

Figure 4: Components in the Embedded Interaction Model for system initiative control.

in simple models, although design principles for choice of media and modalities

provide some guidance [Sutcliffe 2003]. These need to be combined knowledge

critical aspects such as selective attention to understand possible problems arising

from competing stimuli on the same modality, and the effect of distractions on the

monitoring task. In the Interpret and Evaluate tasks the critical aspects point to the

need to reduce working memory burden by supplying appropriate information for

understanding the event within a display appropriate to the user’s task and context.

Grand Challenges in HCl: the Quest for Theory-led Design 499

Attention Resource Theory [Wickens 2002] can provide an appropriate

framework for analysing the selective attention and when to intervene problem.

This provides the basis for specifying an embedded cognitive model which predicts

when to intervene, given a task model and monitored inputs of the user’s activity.

This embedded interaction model (see Figure 4) could be configured to other

notifier systems, thereby contributing another conduit by which theory can influence

design via reusable computational models that embed theory for solving interaction

problems.

Knowledge of the salience of perceptual stimuli and how the stimulus properties

affect attentional executive processing are necessary to design the alerting stimuli.

Such knowledge is only partially formulated even in cognitive theory such as EPIC

[Kieras & Meyer 1997] and experimental derived knowledge [Ware 2000], so the

designer will have to use several sources to understand how the salience of stimuli

interacts with users’ knowledge and motivation in perceptual mechanism. This

points to a need for new theoretical development. However, sufficient knowledge

already exists to suggest a design solution for the appropriate alerting stimulus and

modality. In our scenario, the alert will depend on the criticality of the information.

If the monitored parameters stray out of a normal range but into a non-critical zone

then a low-level stimulus will be used (e.g. colour change in the information display);

whereas if parameters stray into a critical zone then more salient alerts, such as voice

or sound coupled with visual highlighting, or animated displays, are required. Note

that this design adaptation will also be driven by the embedded interaction model.

Applying theory also helps understanding how to manage the transition from the

primary to a secondary task, although once again the task has a critical intermediary

role. Marilyn has to monitor several stimuli, interpret the measurements and then

decide on appropriate adjusting action. However, when an emergency occurs she

has to rapidly interpret the event, diagnose the problem and take remedial action.

In both cases her training will prepare her for the correct response. However,

the patient-monitoring task is close to a skill and requires little reasoning; in

contrast, responding to an emergency requires more conscious rule- and knowledgebased

reasoning (Rasmussen, 1986). Diagnosis places different cognitive demands

and hence requires information displays localizing the problem, while suggesting

possible causes and remedial actions

The storyboard design that might be prompted by the application of theory-led

design is illustrated in Figure 5.

The user’s foreground task occupies most of the screen, as windows with Word

and Excel that Marilyn uses for updating her records, writing reports and planning

rosters. The monitor is placed on continuous display on the side of the screen, since

change in the background monitored system is gradual so Marilyn can sample a

patient’s state on demand. A current and recent past display enables her to glance

at the display and see if the state of any patient has changed in the recent time

period. The display maps to rooms occupied by individual patients on the ward,

with colour coding used for change in parameters: from green within normal range

to red signalling a dangerous digression. Dangerous changes trigger an audio alarm

and highlight the patient’s location. The audio alarm uses canned speech messages

 

Figure 5: Storyboard design for the patient monitoring system.

which can be tailored to reinforce the message. Below the ‘room map’ display a time

line gives the sequence of visits to check on condition, changes to medication, and

reminders when periodic checks are due.

The above design decisions were motivated by several critical aspects. Use of a

continuous display monitor was motivated by the periodic nature of the monitoring

task and the cognitive aspect on reducing working memory burden. When to use

an explicit alert used the embedded model to calculate the trade-off between the

criticality of the event and the cost of interrupting the foreground task. Third was

how to alert the user to change; a variety of strategies were indicated linking the

strength of the stimulus to the criticality of the event. This used media design

principles as well as knowledge of language theory and selective attention. The final

design issue is how to make the transition between the foreground and background

tasks. System actions are tuned to the criticality of the monitoring task. For lowlevel

alerts the foreground task is automatically saved but no further action is taken

apart from displaying more information on the critical patients. For critical alarms

the work is saved, the Word and Excel windows are closed, and patient information

is displayed with appropriate treatment suggestions for the individual concerned.

In this case the system has made the transfer from supporting documentation to a

diagnosis task.

4 Conclusions

This exercise in theory-led design proposed a solution via three complementary

strands for applying theory to design: internalized knowledge of simplified

psychological theory, reuse of knowledge located via mapping generic tasks

Grand Challenges in HCl: the Quest for Theory-led Design 501

applicable to the problem, and design of reusable embedded interaction models. It

will always be difficult to separate the relative contribution to a design solution from

internalized and tacitly applied theory, reusable knowledge, external guidance and

advice. To an extent I argue for both as a necessary prerequisite for good design.

My quest is to provide different pathways by which knowledge from theory can

be applied to the design process. This is where I believe generic tasks and critical

aspects can play their role, first as a retrieval mechanism and secondly as a method

of structuring HCI to teach it in the first place.

It is germane to reflect whether this approach is really new. For example,

application of Ecological Interface Design [Vicente 1999] could have led to the

same metaphor domain representation of the patient monitor interface, although this

method would not have provided advice on cognitive aspects of media selection.

Claims and the task artefact cycle provide a similar access path to theory-based

knowledge, which I have proposed as a precursor to this approach [Sutcliffe 2000].

In an extended schema [Sutcliffe & Carroll 1999] claims have been linked to

generic task models and organized in taxonomies. Claims could therefore provide

design advice for specific critical aspects. The critical aspects are not dissimilar

to cognitive dimensions, and provide summaries of cognitive knowledge linked to

a particular design concern [Green & Petre 1996]. Generic task models, embedded

cognitive models and critical aspects supplement those views, but bring theory-based

knowledge closer to the designer by providing a more direct mapping to tasks and

design problems.

The example used in this paper has inevitably biased the focus towards cognitive

theory and single user systems. The impact of social theory and critical aspects for

collaborative systems has hardly been researched. For example are there critical

aspects of shared identity, mutual awareness, collective motivation, the glue of social

relationship as factors that influence the success of groups? So far we have but a

few heuristics to use, although a more well formed theory, such as Small Groups as

Complex Adaptive Systems [Arrow et al. 2000; Sutcliffe 2005] could provide more

insight into theory-led design for collaborative systems.

In conclusion, theory will continue to play a vital role in design, not only in

HCI but in the wider context of technology design, as computers become part of

everyday products. The quest to translate theory into usable knowledge for design

is still underway, and in this paper I have argued for a multi-path route via reuse

of knowledge in the head and theory embedded in computation models, as well as

localizing knowledge in a task context. This grand challenge has a wide-ranging

scope as a general theoretical approach to design not only of computing related and

computer embedded artefacts but to all designs that involve human interaction or

use. HCI could transform itself into the Science of Design by rising to this challenge.

References

Alexander, C, Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I. & Angel, S.

[1977], A Pattern Language: Towns, Buildings, Construction^ Oxford University Press.

Anderson, J. R. & Lebiere, C. [1998], Representing Cognitive Activity in Complex Tasks,

Lawrence Erlbaum Associates.

502 Alistair Sutcliffe

Arrow, H., McGrath, J. E. & Berdahl, J. L. [2000], Small Groups as Complex Systems:

Formation, Coordination, Development and Adaptation, Sage Publications.

Barnard, P., May, J., D, D. & Duce, D. [2000], Systems, Interactions and Macrotheory, ACM

Transactions on Computer-Human Interaction 7(2), 222-62.

Bellotti, v., Buckingham Shum, S., MacLean, A. & Hanmiond, N. [1995], Multidisciplinary

Modelling in HCI Design: In Theory and in Practice, in I. Katz, R. Mack, L. Marks, M. B.

Rosson & J. Nielsen (eds.). Proceedings of the SIGCHI Conference on Human Factors in

Computing Systems (CHr95), ACM Press, pp. 146-53.

Bertelsen, O. W. & B0dker, S. [2003], Activity Theory, in J. M. Carroll (ed.), HCI Models,

Theories and Frameworks: Toward a Multidisciplinary Science, Morgan-Kaufmann,

pp.291-324.

Blackmon, M. H., Kitajima, M. & Poison, P. G. [2003], Repairing Usability Problems

Identified by the Cognitive Walkthrough for the Web, in V. Bellotti, T Erickson, G. Cockton

& P. Korhonen (eds.), Proceedings of SIGCHI Conference on Human Factors in Computing

Systems (CHI’OS), CHI Utters 5(1), ACM Press, pp.497-504.

Borchers, J. [2001], A Pattern Approach to Interaction Design, John Wiley & Sons.

Brooks, F. P [2003], Three Grand Challenges for Half Century Old Computer Science,

Journal of the ACM 50(10), 25-6.

Burke, M., Gorman, N., Nilsen, E. & Homof, A. [2004], Banner Ads Hinder Visual Search

and Are Forgotten, in E. Dykstra-Erickson & M. Tscheligi (eds.), CHI’04 Extended Abstracts

of the Conference on Human Factors in Computing Systems, ACM Press, pp.1139-42.

Card, S. K., Moran, T. P & Newell, A. [1983], The Psychology of Human-Computer

Interaction, Lawrence Erlbaum Associates.

Carroll, J. M. [2000], Making Use: Scenario-based Design of Human-Computer

Interactions, MIT Press.

Carroll, J. M. & Rosson, M. B. [1992], Getting Around the Task-Artefact Framework: How

to Make Claims and Design by Scenario, ACM Transactions on Office Information Systems

10(2), 181-212.

Cassell, J., Bickmore, J., Billinghurst, M., Campbell, L., Chang, K., Vilhjalmsson, H. & Yan,

H. [1999], Embodiment in Conversational Interfaces: REA, in M. G. Williams & M. W.

Altom (eds.). Proceedings of the SIGCHI Conference on Human Factors in Computing

Systems: The CHI is the Limit (CHr99), ACM Press, pp.520-7.

Chewar, C. M., McCrickard, D. S. & Sutcliffe, A. G. [2004], Unpacking Critical Parameters

for Interface Design: Evaluating Notification Systems with the IRC, in D. Gruen & I. McAra-

McWilliams (eds.). Proceedings of the Symposium on Designing Interactive Systems:

Processes, Practices, Methods and Techniques (DIS’04), ACM Press, pp.279-88.

Connell, I., Green, T. R. G. & Blandford, A. [2003], Ontological Sketch Models:

Highlighting User-system Misfits, in E. O’Neill, P. Palanque & P. Johnson (eds.). People and

Computers XVII: Designing for Society (Proceedings ofHCVOS), Springer-Verlag, pp. 163-

78.

Grand Challenges in HCI: the Quest for Theory-led Design 503

Diaper, D. [2002], Tasks, Scenarios and Thought, Interacting with Computers 15(4), 629-38.

Dix, A., Finlay, J., Abowd, G. D. & Beale, R. [2003], Human-Computer Interaction, third

edition, Prentice-Hall.

Dix, A. & Mancini, R. [1998], Specifying History and Back Tracking Mechanisms, in

Formal Methods in Human-Computer Interaction, Formal Approaches to Computing and

Information Technology, Springer-Verlag, Chapter 1, pp. 1-23.

Dowell, J. & Long, J. B. [1998], Conception of the Cognitive Engineering Design Problem,

Ergonomics 41(2), 126-39.

Green, T. R. G. & Petre, M. [1996], Usability Analysis of Visual Progranmiing

Environments: A ‘Cognitive Dimensions’ Framework, Journal of Visual Languages and

Computing 7(2), 131-74.

Harrison, M. D. & Barnard, P. J. [1993], On Defining the Requirements for Interaction,

in S. Fickas & A. Finkelstein (eds.). Proceedings of the IEEE International Symposium on

Requirements Engineering (RE’93), IEEE Computer Society Press, pp.50-5.

Hollnagel, E. [1998], Cognitive Reliability and Error Analysis Method (CREAM), Elsevier.

Horvitz, E., Kadie, C. M., Paek, T. & Hovel, D. [2003], Models of Attention in Computing

and Conmiunications: From Principles to Application, Communications of the ACM

46(3). 52-9.

Ishii, H., Mazalek, A. & Lee, J. [2001], Bottles as a Minimal Interface to Access Digital

Information, in J. A. Jacko & A. Sears (eds.), CHI’Ol Extended Abstracts of the Conference

on Human Factors in Computing Systems, ACM Press, pp. 187-8.

ISO [1997], ISO 9241 International Standard. Ergonomic Requirements for Office Systems

with Visual Display Terminals (VDTs). International Organization for Standardization,

Geneve, Switzerland.

ISO [1998], ISO 14915 International Standard. Multimedia User Interface Design Software

Ergonomic Requirements — Part 1: Introduction and Framework. International Organization

for Standardization, Geneve, Switzerland.

ISO [2000], ISO 14915 International Standard. Multimedia User Interface Design Software

Ergonomic Requirements — Part 3: Media Combination and Selection. International

Organization for Standardization, Geneve, Switzerland.

John, B. E. & Kieras, R. E. [1995], The GOMS Family of User Interface Analysis

Techniques: Comparison and Contrast, ACM Transactions on Computer-Human Interaction

3(4), 320-51.

Johnson, P. [1992], Human-Computer Interaction: Psychology, Task Analysis and Software

Engineering, McGraw-Hill.

Kieras, D. E. & Meyer, D. E. [1997], An Overview of the EPIC Architecture for Cognition

and Performance with Application to Human-Computer Interaction, Human-Computer

Interaction 12(4), 391-438.

504 Alistair Sutcliffe

Kitajima, M. & Poison, P. G. [1997], A Comprehension-based Model of Exploration,

Human-Computer Interaction 12(4), 345-89.

Long, J. & Dowell, J. [1989], Conceptions of the Discipline of HCI: Craft, Applied

Science and Engineering, in A. Sutcliffe & L. Macaulay (eds.), People and Computers V

(Proceedings ofHCP89), Cambridge University Press, pp.9-34.

Maglio, P P & Campbell, C. S. [2000], Trade-offs in Displaying Peripheral Information, in

T. Turner & G. Szwillus (eds.), Proceedings of the SlGCHl Conference on Human Factors

in Computing Systems (CHI’OO), CHI Utters 2(1), ACM Press, pp.241-8.

McCrickard, D. S., Catrambone, R., Chewar, C. M. & Stasko, J. T. [2003], Establishing

Tradeoffs that Leverage Attention for Utility: Empirically Evaluating Information Display

in Notification Systems, International Journal of Human-Computer Studies 58(5), 547-82.

McCrickard, D. S. & Chewar, C. M. [2003], Attuning Notification Design to User Goals and

Attention Costs, Communications of the ACM 46(3), 67-72.

Nardi, B. A. (ed.) [1996], Context and Consciousness: Activity Theory and Human-

Computer Interaction^ MIT Press.

Newman, W. M., Taylor, A. S., Dance, C. R. & Taylor, S. A. [2000], Performance

Targets, Model and Innovation in Design of Interactive Systems, in D. Boyarski & W. A.

Kellogg (eds.). Proceedings of the Symposium on Designing Interactive Systems: Processes,

Practices, Methods and Techniques (DIS2000), ACM Press, pp.381-7.

Norman, D. A. [1986], Cognitive Engineering, in D. A. Norman & S. W. Draper (eds.). User

Centered System Design: New Perspectives on Human-Computer Interaction, Lawrence

Erlbaum Associates, pp.31-62.

Norman, D. A. [1988], The Psychology of Everyday Things, Basic Books.

Norman, D. A. [2004], Emotional Design: Why We Love (or Hate) Everyday Things, Basic

Books.

NSF [2004], Science of Design, http://www.nsf.gov/funding/pgm_summ.jsp?pims_id= 12766

(last accessed 2005-04-26). National Science Foundation: Directorate for Computer &

Information Science & Engineering.

Ortony, A., Clore, G. L. & Collins, A. [1988], The Cognitive Structure of Emotions,

Cambridge University Press.

Pirolli, P & Card, S. [1999], Information Foraging, Psychological Review 106(4), 643-75.

Preece, J., Rogers, Y. & Sharp, H. (eds.) [2002], Interaction Design: Beyond Human-

Computer Interaction, John Wiley & Sons.

Rasmussen, J. [1986], Information Processing and Human-Machine Interaction: An

Approach to Cognitive Engineering, North-Holland.

Reason, J. [1990], Human Error, Cambridge University Press.

Shneiderman, B. & Plaisant, C. [2004], Designing the User Interface: Strategies for Effective

Interaction, fourth edition, Addison-Wesley.

Grand Challenges in HCI: the Quest for Theory-led Design 505

Stasko, J., Miller, T, Pousman, Z., Plaue, C. & Ullah, O. [2004], Personalized Peripheral

Information Awareness through Information Art, in N. Davies, E. Mynatt & I. Siio (eds.),

UbiComp 2004: Ubiquitous Computing (Proceedings of the Sixth International Conference

on Ubiquitous Computing), Vol. 3205 of Lecture Notes in Computer Science, Springer,

pp. 18-35.

Sutcliffe, A. G. [2000], On the Effective Use and Reuse of HCI Knowledge, ACM

Transactions on Computer-Human Interaction 7(2), 197-221.

Sutcliffe, A. G. [2002], The Domain Theory: Patterns for Knowledge and Software Reuse,

Lawrence Erlbaum Associates.

Sutcliffe, A. G. [2003], Multimedia and Virtual Reality: Designing Multisensory User

Interfaces, Lawrence Erlbaum Associates.

Sutcliffe, A. G. [2005], Extending Small Group Theory for Analysing Complex Systems, in

C. W. Johnson (ed.). Proceedings of the Workshop on Complexity in Design and Engineering,

Department of Computer Science, University of Glasgow, pp. 139-48. GIST Report 2005-1.

Sutcliffe, A. G.. Fickas, S., Sohlberg, M. M. & Ehlhardt, L. A. [2003], Investigating the

Usability of Assistive User Interfaces, Interacting with Computers 15(4), 577-601.

Sutcliffe, A. G. & Carroll, J. M. [1999], Designing Claims for Reuse in Interactive Systems

Design, International Journal of Human-Computer Interaction 50(3), 213-42.

Teasdale, J. D. & Barnard, P J. [1993], Affect, Cognition and Change: Re-modelling

Depressive Thought, Lawrence Erlbaum Associates.

UKCRC [2005], Grand Challenges for Computing Research,

http://www.ukcrc.org.uk/grand_challenges/index.cfm (last accessed 2005-04-26). UK

Computing Research Conmiittee.

Van Welie, M. [2005], Patterns in Interaction Design: Web Design Patterns,

http://www.welie.com/pattems (last accessed 2005-04-26).

Vicente, K. J. [1999], Cognitive Work Analysis: Towards Safe, Productive and Healthy

Computer-based Work, Lawrence Erlbaum Associates.

Ware, C. [2000], Information Visualization: Perception for Design, Morgan-Kaufmann.

Wickens, C. [2002], Multiple Resources and Performance Prediction, Theoretical Issues in

Ergonomics Science 3(2), 150-77.

Wright, P C, Fields, R. E. & Harrison, M. D. [2000], Analysing Human-Computer

Interaction as Distributed Cognition: The Resources Model, Human-Computer Interaction

15(1), 1-41.

Introduction

 

Viewpoints

Mehrdad Sabetzadeh

Simula Research Laboratory

Oslo, Norway

mehrdad@simula.no

Anthony Finkelstein

University College London

London, UK

a.finkelstein@cs.ucl.ac.uk

Michael Goedicke

University of Duisburg-Essen, Campus Essen

Essen, Germany

michael.goedicke@s3.uni-due.de

November 8, 2009

Abstract

The construction of any sizable software system involves many

agents, each with their own perspective of the system being built.

Viewpoints provide a framework for guiding and managing development

in a multiple-perspective setting, where a variety of agents with

different areas of concern collaborate towards building a system. In

this article, we explain the main concepts and techniques underlying

viewpoint-based development and illustrate them using a number of

examples.

Keywords. Perspectives, Separation of Concerns, Integration, Inconsistency

Management, Merging, Parallel Composition, Weaving.

 

1 Introduction

Large-scale software development is necessarily a collaborative effort, involving

multiple agents (also called, participants or actors) with different

perspectives of the system they are trying to develop. Individual perspectives

are partial descriptions of the overall system, arising in response to the

specific responsibilities or roles assigned to the agents. These responsibilities

or roles may be organizationally-defined, be governed by the physical

distribution of agents across development sites, follow some pre-specified

structuring of the system at hand, or reflect the agents’ knowledge, experience,

and descriptive capabilities.

Inevitably, the perspectives of those involved in the development process

overlap, interact or compete with one another, giving rise to a need for

coordination. The relationships between the perspectives, however, may be

far from obvious because the agents may use different vocabularies and representation

notations to express their concerns. Further, since development

may be done concurrently, different perspectives may be at different stages

of elaboration and hence subject to different development strategies.

The problem of how to guide and manage development in this setting –

multiple agents, diverse knowledge and expertise, heterogeneous representation

notations, differing development strategies – is known as the “multiple

perspectives problem”. In this article, we introduce viewpoints – a conceptual

framework for addressing the multiple perspectives problem.

Intuitively, a viewpoint is an independent and locally-managed object

encapsulating a particular perspective. The main feature of viewpoint-based

development is its recognition of perspectives and the relationships between

them as first-class artifacts, so that perspectives can be directly defined by

users, and the relationships between them can be specified, modified, and

analyzed explicitly.

Viewpoints have often been studied as part of Requirements Engineering.

Although the multiple-perspective problem is particularly acute during

the requirements gathering stage where a diverse group of stakeholers with

different goals and needs are involved, the problem is not restricted to it. In

fact, viewpoints arise with increasing frequency in software design and implementation

as well, due to such factors as evolution, functional variability,

globalization, etc.

It is not our intention to exhaustively examine all ramifications of viewpoints

in software engineering. Our main goal is instead to describe the

key ideas that motivate the use of viewpoints and to outline the integration

activities that go hand in hand with them. To achieve our goal, we

are going to draw on a number of examples throughout the article. While

we have tried to ensure reasonable breadth in our choice of examples, we

acknowledge that these examples are not reflective of the full range of the

applications of viewpoints reported in the literature. General references for

further details about viewpoints and their broader set of applications are

provided later in the article (see Section 7).

We use software models as the primary context for our examples. This

enables us to build on the growing familiarity of the general software engineering

community with modelling languages such as the UML (Unified

Modeling Language, 2003), hence alleviating the need to provide extensive

background on the notations used.

The remainder of this article is structured as follows:

We begin in Section 2 with an example of a multiple-perspective setting.

In Section 3, we introduce viewpoints as a vehicle for capturing perspectives.

In Section 4, we discuss how different viewpoints can be related using explicit

relationships. We continue in Section 5 with reviewing the core design

principles underpinning viewpoints. In Section 6, we concentrate on the

most essential activity in viewpoint-based development, called integration,

and highlight the different facets of the problem. We conclude the article in

Section 7 with a summary and references for further reading.

 

2 An Example of Multiple Perspectives

To illustrate multiple perspectives, we use a simple example capturing some

basic aspects of a Hospital Information System (HIS). A HIS is a a specialized

kind of information system designed to manage the clinical and

administrative functions of a hospital.

An HIS has to meet the needs of a variety of stakeholders. This includes

people at the management levels of the hospital, medical staff, technicians,

administrators, and so on. These people clearly have different areas

of concern, and hence different perspectives. The perspectives are usually

expressed in different ways, and capturing them often necessitates the use

of different representation notations, chosen to be particularly appropriate

to the descriptions provided by each stakeholder.

 

Figure 1 shows a few representative perspectives in an HIS. The formal

specification of perspectives as structured models is seldom done directly by

the stakeholders, but rather by the requirements and design analysts. For

simplicity, we ignore this technicality and directly associate the models to

the stakeholders.

The perspectives in Figure 1 describe an HIS along different dimensions

and at different levels of abstraction. The hospital head focuses on the highlevel

objectives of the system and expresses her view as a goal model in

KAOS (van Lamsweerde, 2009) notation. In her model (P1 in the figure),

achieving patient satisfaction is declared as a top goal, and a decomposition

of this goal into more concrete and lower-level goals is provided.

The medical director uses a UML activity diagram (P2) to capture the

normal workflow for patient care at the hospital. The process begins with

diagnosis, after which a patient is classified as either an inpatient or an

outpatient. The patient then gets the proper care based on this classification

and is subsequently discharged.

The doctor and the nurse each contribute two perspectives, a UML class

diagram and a UML sequence diagram. The class diagrams (P3 and P5)

capture the concepts and associations relevant to each stakeholder, and the

sequence diagrams (P4 and P6) describe usage scenarios. The usage scenario

for the doctor (P4) concerns the routine visiting of a patient and updating

her medical record. The usage scenario given by the nurse (P6) captures

the daily check up conducted in a hospital ward, where a nurse evaluates

each patient individually, and notes her observations in the patient’s medical

chart.

Finally, the computer systems administrator, who is in charge of ensuring

the integrity and performance of the HIS, expresses a requirement that any

update made to the persistent data of the system should be logged in a file,

so that problems can be tracked in case of a system failure. This requirement

is modelled as a rewrite rule for sequence diagrams (P7). If the left hand side

of the rule (denote L) matches a fragment of a sequence diagram, then that

fragment is rewritten with the right hand side of the rule (denoted R). The

left hand side would be deemed a match if update*() (i.e. a method whose

name begins with the “update” prefix) is called on a persistent data object.

If a match is found, the sequence diagram in question will be modified as

prescribed by R (the right hand side). That is, SystemLog.append() is called

after the update to append an entry to the system log.

The models in our example relate to one another in many ways. Some

of these relationships are readily identifiable. For example, the objects in

P4 are instances of the classes declared in P3, and the messages in P4

are occurrences of the class methods in P3. The relationship between P5

and P6 is similar. Another obvious pair of relationships are P7,P4 and

P7,P6. The match between the left hand side of P7 and P4 is {Object1 !

:Doctor, Object2 ! :MedicalRecord, update*() ! update()}. The P7,P6 relationship

is defined similarly.

There are also relationships whose existence is almost certain, but whose

details cannot be established with full certainty because of terminological

and structural differences between the perspectives. For example, an anal-

ysis of P3 and P5 would hint at some possible overlaps between the two

perspectives: the Patient class in P3 is likely to be the same as that in P5;

MedicalRecord in P3 is likely to correspond to Chart in P5; and the aggregation

between Patient and MedicalRecord in P3 may be an alternative representation

of the assigned to link between Patient and Chart in P5. Unless these correspondences

are explicitly validated with the stakeholders, one can never be

entirely sure about their correctness. For example, it may turn out that

the doctor’s notion of Patient encompasses both inpatients and outpatients,

while the nurse may mean only inpatients when she refers to Patient in her

models. Similarly, MedicalRecord and Chart may be distinct entities.

There are several other relationships between the perspectives that are

less obvious:

• Since P3 and P5 most likely have overlaps, P4 and P6 will be related

as well, in the sense that they can access and modify shared objects.

For example, it is possible for a doctor and a nurse to both want to

update an individual patient’s chart simultaneously. Hence, P4 and

P6 may need to synchronize on accesses to shared objects.

• P2 implicitly characterizes the different states that a patient goes

through at the hospital. Hence P2 would be related to the design

of the Patient class in P3 and P5.

 

• Finally, at a higher level of abstraction, all P2–P7 contribute to the

satisfaction of the system’s main objectives, and are, directly or indirectly,

traceable to the goals expressed P1.

In Figure 2, we have shown a bird’s eye view of the perspectives in our

example along with the relationships that capture the known or hypothesized

interlocking constraints on the structure and behaviour of the perspectives.

The use of perspectives made it a lot easier to ensure that all the stakeholders

are properly represented, and much less likely to miss information that is

critical to the success of the system. At the same time, the flexibility to

have perspectives results in several inter-related models that need to be

maintained independently and may be inconsistent with one another.

Effective use of perspectives in system development requires a systematic

framework for management and integration of perspectives and resolving

their inconsistencies. Viewpoints, as we describe in the subsequent sections,

provide the foundation for such a framework.

3 Viewpoints

The basic building block for organizing and supporting multiple perspectives

is a viewpoint. A viewpoint can be thought of as a combination of the idea

of an agent, role, or knowledge source and the idea of a perspective. In

software terms, a viewpoint is a loosely coupled, locally managed object

which encapsulates a perspective about the system and domain, specified

in a particular representation notation. Each viewpoint is composed of the

following components, called slots:

• a domain delineating the part of the “world” the viewpoint is concerned

with;

• a representation style defining the notation used by the specification

slot (below);

• a specification expressing the perspective of interest, represented in

the viewpoint’s style

Additionally, a viewpoint may contain knowledge of the process of its

design. This knowledge is captured using the following slots:

• a work plan describing the process by which the specification can be

built;

• a work record giving an account of the history and current state of

development

As an example, let us consider the specification provided by the medical

director (P2) in the motivating example of the previous Section. In Figure 3,

we have shown the complete viewpoint for this specification. The representation

style of the viewpoint is UML activity diagrams, and the domain of

concern is patient care. The specification is the medical director’s current

knowledge about the procedure for processing patients at the hospital, expressed

as a UML activity diagram. The work plan explains how to build an

activity diagram and how, and in what circumstances, to check consistency

with the other viewpoints. The work record gives the current state of the

specification and an account of its development in terms of the work plan.

Medical Director

Admission

Diagnosis

Treatment

Ambulatory

Care

Discharge

[inpatient] [outpatient]

Patient Care

UML Activity Diagrams

how to build UML activity diagrams,

and how to check consistency with

other viewpoints …

 

 

4 Relationships Between Viewpoints

The key feature that distinguishes viewpoint-based development from conventional

development methods is that viewpoints are built and maintained

independently of one another. This means that viewpoints are not just

projections of a common underlying specification, but rather distinct and

separately-evolving artifacts with potentially different vocabularies and frames

of reference. As a result, one cannot merely rely on conventions (e.g., name

or identifier equivalence between elements of different viewpoints) to give

the desired relationships between the viewpoints. Instead, one needs to explicitly

specify these relationships. Doing so is a critical prerequisite for any

meaningful integration of the viewpoints, and for ensuring that the impact

of changes made to the viewpoints can be properly analyzed, scoped, and

propagated.

A relationship typically refers to an (explicit) mapping between the contents

or interfaces of the specification slots in different viewpoints. In a

broader sense, a relationship may also encompass the “organizational structure”

of the development team (i.e., the owners of the viewpoints). In this

article, we shall deal only with the former type of relationships, i.e., those

between specifications.

There is no general rule as to what constitutes an inter-viewpoint relationship.

This depends on a variety of factors, most importantly the semantics

of the viewpoints involved, the stage of development the viewpoints are

in, and the degree of detail one wishes to capture in a relationship.

In the remainder of this section, we illustrate some common relationships

between specifications expressed as models. There is no restriction on the

number of specifications an individual relationship can connect, but usually,

relationships are defined pairwise, i.e., they are between two specifications.

Our example relationships, shown in Figure 4, are all pairwise.

In Figure 4(a), we depict the relationship (denoted R) between the perspectives

contributed by the doctor (previously shown in Figure 1). The relationship

specifies how the objects in the sequence diagram (V2) correspond

to the classes in the class diagram (V1) through “instance of” mappings, and

how the interaction messages in V2 correspond to the methods in V1 through

“occurrence of” mappings. If the two diagrams are assumed to be views on

a monolithic model of the system, this relationship is derivable from the

underlying model and can hence be left implicit. But as we stated earlier,

such an assumption is typically not made in viewpoint-based approaches,

and as a result, the relationships are always specified explicitly.

In Figure 4(b), the specifications involved are in different abstraction

layers. V1 is the extends–implements fragment of the meta-model for Java class

diagrams, and V2 – an instance class diagram conforming to the meta-model.

The relationship between these specifications is a type function mapping

each element of V2 to an element of V1. The relationship respects the structure

of the specifications, in the sense that if it maps an edge e of V2 to

an edge u of V1, it will also map the endpoints of e to those of u. Such

preservation of structure ensures that the edges in the V2 are well-typed.

 

Our next example, in Figure 4(c), shows the relationship between two

perspectives on the data schema of the hospital’s payroll subsystem. Both

perspectives are described as entity-relationship diagrams. The vocabularies

used by the perspectives differ, but there are conceptual overlaps between

the perspectives. These overlaps are specified by a mapping that equates

the corresponding elements. The relationship in Figure 4(c) states that the

entity referred to as StaffMember in V1 is the same as Employee in V2. Further

the relationship declares the name attribute of these two entities (as well as

the edges linking name to the respective entities) as being equivalent. For

a problem of this size, the relationship may be created by hand, but larger

problems require automation, usually achieved through heuristic techniques

for finding concept matches. These techniques are almost always inexact,

i.e., they may yield matches that are incorrect, or they may miss correct

matches. Despite this, the techniques can be of significant assistance to a

human expert for exploring the relationships between different stakeholder

vocabularies.

The relationships we talked about so far were expressed as collections of

tuples of the form (e1, e2) where e1 and e2 are elements of different specifications.

While tuple-based techniques are very common and widely used,

they are not the only possible way for specifying inter-viewpoint relation-

ships. For example, a relationship may be expressed declaratively using

logical formulas. An example is given below:

In Figure 4(d), V1 and V2 are two alternative ways of expressing the

concept of a person along with their gender. V1 models gender as an attribute

of the Person class, whereas V2 models gender through sub-classing.

This structural discrepancy between the viewpoints is more conveniently

expressed using first-order logic formulas than tuples of mapped elements.

The formulas shown in the figure specify how Female and Male in V1 relate

to Person in V2. To avoid ambiguity, predicate names in the formulas

are subscripted with a number denoting the model of origin. For example,

Person1 refers to Person in V1.

As a final illustration, we consider, in Figure 4(e), the relationship between

viewpoints capturing different components of a system. In contrast

to all our previous examples, the relationship in Figure 4(e) is not between

the contents of the viewpoints involved, but rather between their interfaces

(viewpoint contents are not shown). Here, V1 denotes a user process that

accesses shared resources during its execution, and V2 is one such resource.

The relationship between V1 and V2 describes how the two components are

supposed to bind together, hence establishing a communication channel between

them. More specifically, the relationship requires that the acquire (resp.

release) action from V1 should synchronize with lock (resp. unlock) from V2.

5 Principles Underlying Viewpoints

The use of viewpoints in software engineering is motivated primarily by

the separation of concerns principle – the observation that, for a large and

complex system, it is more effective to build several partial specifications

that focus on individual concerns, rather than to attempt to construct a

single monolithic specification for the whole system.

Separation of concerns may be carried out along different dimensions.

For example, in the motivating problem described in Section 2, the areas of

concern were defined along the organizational roles of the involved participants

(hospital head, doctors, nurses, etc.). A finer-grained separation could

result in developing independent viewpoints for every relevant participant,

e.g., individual doctors and nurses.

The factors influencing the choice of dimensions along which viewpoints

are built are many. They include, among others, the nature of the problem

at hand, organizational considerations, the degree of involvement of the

stakeholders, component architecture of the system, the development teams’

dynamics, and policies for future maintenance of the system.

Separation of concerns naturally leads to the related principle of het-

erogeneity of representation. Viewpoints reinforce this principle by allowing

participants to choose the notation in which they want to describe their perspectives.

The choice of notation for each viewpoint is recorded explicitly

in the representation style slot of that viewpoint.

The next major principle underlying viewpoints is decentralization – the

idea that the knowledge within each viewpoint is maintained and manipulated

locally. Decentralization moves away from any notion of a monolithic

system specification that can be managed globally. Such flexibility becomes

particularly crucial for development teams that are spread across multiple

geographical sites. Decentralization makes it possible for the members of

these teams to function independently without having to constantly coordinate

their work with one another.

A consequence of having multiple viewpoints in a project is the inevitable

inconsistencies that arise between them. The approach taken by viewpoints

is that of living with inconsistency. In other words, there is no requirement

that a set of viewpoints should always be consistent. This stance towards

inconsistencies draws on two important observations: Firstly, immediate

resolution of inconsistencies can be highly intrusive, particularly in early

stages of development, e.g., the requirements elicitation phase, where ambiguities

and conflicts tend to occur too frequently. Having to address every

inconsistency as soon as it arises can adversely affect productivity.

Secondly, maintaining consistency at all times can lead to premature

commitment to decisions that have not yet been sufficiently analyzed. For

example, resolving an inconsistency identified at the requirements stage may

require information from the detailed design stage which might not have yet

even started. Hence, any attempt to fix the inconsistency early on may fail

to amend the problem, and can in fact even worsen it.

Obviously, living with inconsistency should not be viewed as suggesting

living with defects in the final system. What this principle does promote is

the flexibility to deal with each inconsistency at the right time.

6 Viewpoint Integration

Viewpoint integration is primarily concerned with ensuring that a set of

viewpoints fit together in a consistent way while retaining their original

structure. In this sense, viewpoint integration is often synonymous with inconsistency

management. Despite the general desire to maintain viewpoints

as independent objects, there are situations where a collection of viewpoints

need to be combined into a single larger specification, e.g., to gain a unified

perspective, to build a complete blueprint for the system, or to better explore

the interactions among viewpoints. To this end, viewpoint integration

may further encompass three other activities, called merging, parallel composition

and weaving. We briefly explain and exemplify each of these four

facets of integration.

6.1 Inconsistency Management

Consistency is usually defined using inter-viewpoint rules expressing the

desired constraints that must hold between viewpoints. As an example,

a consistency rule that one might want to express over the viewpoints in

Figure 4(a) is that each message in the sequence diagram has a corresponding

method in the class diagram. This is captured by the following rule:

! = ” msg, obj , cls · Target(msg, obj ) # R(obj , cls) =$

% mtd · MethodOf(mtd, cls) # R(msg, mtd).

In the above formula, R(x, y) holds if x and y are related by an interviewpoint

relationship (see the relationship R in Figure 4(a)); Target(x, y)

holds if message x is invoked on object y in the sequence diagram; and

MethodOf(x, y) holds if x is a method of class y in the class diagram. In this

example, the viewpoints satisfy !; but if we, say, remove the getRecord()

method from the Patient class in V1, the rule would no longer hold.

Inconsistency management is the process of handling the potential plethora

of consistency rules, an example of which was shown above, and dealing with

the viewpoints when the rules fail to hold. Inconsistency management involves

several steps:

• Consistency checking, focusing on monitoring, detecting, and reporting

of consistency rule violations.

• Inconsistency classification, focusing on identifying the kinds of inconsistencies

detected. Inconsistencies may be classified according to their

causes, or according to pre-defined kinds prescribed by developers.

• Inconsistency handling, focusing on acting in the presence of inconsistencies.

For example, when an inconsistency is detected, the appropriate

response may be to ignore, tolerate, resolve, or circumvent

it.

• Inconsistency measurement, focusing on calculating the impact of inconsistencies

on different aspects of development, and prioritizing the

inconsistencies according to the severity of their impact. The actions

taken to handle an inconsistency typically depend on the results of

inconsistency measurement.

 

 

 

6.2 Merging of Viewpoints

The main goal of merging is unification of overlaps between viewpoints.

More precisely, given a set of viewpoints, merge combines the viewpoints

together in a way that only one copy of the overlapping parts is included in

the result. This property is known as non-redundancy.

To illustrate merge, consider the two viewpoints in Figure 4(c) and the

relationship that specifies their overlaps. Figure 5 shows the merge of these

viewpoints (with respect to the relationship between them). This merge is

redundancy-free because it has only one copy of the common parts. This

ensures that database instances created over the merged schema will not

have data redundancies.

Non-redundancy is only the most basic requirement for merge. Viewpoint

merging is often subject to additional correctness criteria. The most

notable of these criteria are:

• Completeness: Merge should not lose information, i.e., it should represent

all the source viewpoints completely.

• Minimality: Merge should not introduce information that is not present

in or implied by the source viewpoints.

• Semantic Preservation: Merge should support the expression and preservation

of semantic properties. For example, if viewpoints are expressed

as state machines, one may want to preserve their temporal behaviours

to ensure that the merge properly captures the intended meaning of

the source viewpoints.

• Totality: Merge should be well-defined for any set of viewpoints,

whether or not they are consistent. This property is of particular

importance if one wants to tolerate inconsistency between the source

viewpoints.

These additional criteria are not universal to all model merging problems.

For example, completeness and minimality, in the strong sense described

above, may be undesirable if viewpoint merging involves conflict resolution

as well, in which case the final merge can potentially add or delete information.

Semantic preservation may be undesirable when one wants to induce a

design drift or perform an abstraction during merge. Such manipulations are

usually not semantics-preserving. And, totality may be undesirable when

the source viewpoints are expected to seamlessly fit together. In such a case,

the source viewpoints should be made consistent before they are merged.

6.3 Parallel Composition of Behavioural Viewpoints

Parallel composition refers to the process of assembling a set of behavioural

viewpoints that capture different components of a system, and verifying that

the result fulfills the properties of interest.

In contrast to merge, the inter-viewpoint relationships built for parallel

composition are between the interfaces of the viewpoints, and not between

their internal contents. As such, the line of sight of the relationships into

the viewpoints is limited to the interfaces that the viewpoints expose to the

outside world. A second difference between parallel composition and merge

is that, in parallel composition, multiple copies of the same viewpoint may

be participating to denote the fact that the system has multiple components

of the same type.

Figure 6 shows a simple component diagram, expressed in the visual

modelling notation of the Darwin architectural language (Magee et al.,

1995). The diagram describes the interconnections between three concur-

 

rent processes in the hospital system: C1, the process taken by a doctor to

add a prescription to a patient’s chart; C2, the process taken by a nurse to

update a patient’s chart after a checkup; and C3, the process representing a

patient’s chart as a resource. The semantics of the relationships between the

processes in Figure 6 is the same as that of the relationship R in Figure 4(e),

discussed earlier.

The behaviours of C1–C3 are shown in Figures 7(a)–(c), respectively.

The notation used to describe these behaviours is Labelled Transition Systems

(LTSs) (Milner, 1989). To make sure that patients’ charts are protected

against concurrent changes, accesses to C3 by C1 and C2 need to be mutually

exclusive.

The parallel composition of C1–C3 is shown in Figure 7(d). We have

used the LTS Analyzer (LTSA) tool (Magee & Kramer, 2006) for expressing

and computing the parallel composition. In Figure 7(e), we show the speci-

 

 

fication of the processes and their parallel composition in the input language

of the LTSA tool.

As seen from Figure 7(d), an update made by the doctor is preceded

by doctor.chart.lock (i.e., doctor’s request to acquire access to the chart), and

followed by doctor.chart.unlock (i.e., doctor’s request to release the chart). The

same is true for the nurse: her update is preceded by nurse.chart.lock and

followed by nurse.chart.unlock. By computing a parallel composition like this,

one can verify that the updates are mutually exclusive.

6.4 Weaving of Aspectual Viewpoints

Weaving is the predominant notion of integration in Aspect-Oriented Software

Development (AOSD) (Filman et al., 2004). AOSD is an attempt to

provide better functional decomposition in complex systems. This can in

part be achieved by means of encapsulating different concerns into distinct

system components. However, some concerns defy encapsulation as they

cut across many parts of the system. A classic example of a cross-cutting

concern is logging which affects all logged activities in a system. The perspective

of the Computer Systems Administrator (P7) in Figure 1 illustrated

this concern for sequence diagrams.

Given a set of cross-cutting concerns and a base system, weaving is

the process of incorporating the concerns into the base system. If no base

system is provided, weaving would refer to the process of incorporating the

given concerns into one another. Weaving operators may be implemented in

various ways depending on the nature of the base system and the concerns

involved, and whether weaving is performed statically (at compile time) or

dynamically (at runtime).

Aspect-oriented languages usually come equipped with built-in constructs

for defining the weaving operator. For example, aspect-oriented programming

languages provide pointcut constructs by which programmers specify

where and when additional code (i.e., an aspect) should be executed in place

of or in addition to an already-defined behaviour (i.e., the base program).

In aspect-oriented modelling, weaving is usually defined by patterns, to be

chosen either manually or automatically using pattern matching techniques.

Similar to other integration activities, weaving may result in undesirable

side effects. Thus, automated analysis techniques may be required to ensure

that the result of weaving satisfies the desired correctness properties.

As an example, Figure 8 shows the result of weaving P7 in Figure 1

into P4 and P6 in the same figure. For this problem, weaving is performed

statically, and the weaving operator can be most conveniently implemented

using graph rewriting.

7 Conclusion and Further Reading

In this article, we gave a brief introduction to viewpoints as a vehicle for

addressing the multiple perspectives problem. We discussed the basic con-

 

cepts involved in building software systems as a collection of inter-related

viewpoints, and explained the core integration activities associated with this

kind of development.

The principles upon which viewpoints are founded continue to be valuable

and relevant in many areas of software engineering today. In particular,

several major challenges faced in the areas of global software engineering

(Herbsleb, 2007), and model-based development (France & Rumpe, 2007)

are intimately related to the problems that have been studied under the

umbrella of viewpoints for the past two decades. The thread that ties these

areas to viewpoints is the pervasive need to manipulate systems made up

of many tangled artifacts, with potential inconsistencies in their design and

usage. This is indeed what viewpoints research has been aimed at all along.

In the light of the arguments presented in this article, viewpoint concepts

seem to be naturally applicable for expressing many properties and

structures in software development processes. Thus, the question arises why

no production-quality tools have been developed to date, incorporating all

the core viewpoint principles. We can see a number of tools with support

for multiple development views, e.g. UML-based tools; however, these tools

do not implement viewpoints in the sense we described here.

There are inherent properties to viewpoints that complicate their implementation.

In particular, implementing the various operators for matching

partial structures and creating and maintaining connections between

viewpoints is a challenging task. This is mainly due to the complexity of

the graph algorithms involved in the operators, and also to the potential

existence of structural and terminological mismatches between viewpoints,

which makes establishing precise inter-viewpoint relationships difficult.

In addition, it appears that the decentralized nature of viewpoints is

not fully compatible with the existing project management practices. Par-

ticularly, for project management purposes, one would often like to define

milestones and demand progress reports from the analysts and developers.

Distributing the work across multiple viewpoints means that a distributed

consensus has to be reached if something like a milestone is required, but arriving

at a consensus in a distributed setting is not always straight-forward.

Another property of distributed systems applies here as well: the difficulty

to build a global state. In the case of viewpoints, the notion of

an overarching document containing all specification information in a linear

form is lacking. Of course, the local view expressed by a viewpoint

and the connections to other viewpoints is a perfect basis for a hypertext

document, but this does not automatically ensure that the viewpoints fit together

properly and that they are consistent with one another. Developing

a more comprehensive and usable suite of tools based on viewpoints requires

first addressing the challenges discussed above. These challenges map out a

vision and an agenda for future research on viewpoints.

Numerous papers are available on viewpoints for further reading. Providing

links to all these papers is not possible here due to space limitations.

Below, we outline the expository and technical references used as the basis

for this article. These references together provide an extensive body of

bibliographic information on viewpoints.

Specification of viewpoints and their relationships. Several frameworks

for specifying and inter-relating viewpoints in requirements engineering

are surveyed in (Darke & Shanks, 1996). A more up to date bibliography

of viewpoints in requirements engineering is maintained by J. Leite at

http://www.requirementsviewpoints.blogspot.com/.

To read more about the applications of viewpoints in the broader context

of software engineering, one can refer to (Spanoudakis et al., 1996) where a

diverse agenda on viewpoints is provided.

The detailed anatomy of viewpoints and inter-viewpoint relationships,

as described in this article, is based on work on the ViewPoints framework

(Finkelstein et al., 1992; Nuseibeh et al., 1994; Nuseibeh et al., 2003). This

framework is largely attributed as having set forth the principal ideas of

viewpoint-based development as they exist today.

Inconsistency management. Several inconsistency management approaches

have been proposed, in general based on the success of the ViewPoints framework.

The main questions in this work centre on appropriate notations for

expressing consistency rules, and automated support for monitoring, diagnosis,

and resolution of inconsistencies. A direct successor to the ViewPoints

framework is xlinkit – a lightweight application service that provides rulebased

consistency checking and diagnostics generation for distributed web

content. This tool has been described in (Nentwich et al., 2003).

An overview of inconsistency management activities for viewpoints is

given in (Nuseibeh et al., 2000). Our discussion in Section 6.1 was based

on this reference. A detailed survey of existing inconsistency management

techniques is available in (Spanoudakis & Zisman, 2001).

Viewpoint merging. Several papers study merging of partial viewpoints

in specific domains including database schemata, requirements models, state

machines, and web ontologies. A survey of recent approaches to viewpoint

merging is available in (Sabetzadeh, 2008). The description in Section 6.2

was based on this reference.

Parallel composition. Parallel composition is a well-studied notion for

combining the behaviours of distributed system components, and has been

formally characterized for a variety of behavioural formalisms. Further detail

on analysis tools for concurrent distributed systems is available in (Magee

& Kramer, 2006).

Aspect weaving. Aspect-oriented concepts were originally coined for programming

languages, e.g. see AspectJ (Kiczales et al., 2001). Recently,

these concepts have been adapted to software models and applied to software

requirements and design viewpoints. The treatment of cross-cutting

viewpoints in Section 6.4 is similar to the approach of (Whittle et al., 2007).

For further references on aspect-oriented modelling and aspect weaving,

see (France et al., 2007; Morin et al., 2009).

Acknowledgments. The description of viewpoints in this article is based

on the ideas developed in the ViewPoints framework (Finkelstein et al.,

1992), to which many colleagues have contributed over the years. In particular,

we wish to acknowledge Bashar Nuseibeh and Jeff Kramer for their

significant contributions. We thank Steve Easterbrook, Shiva Nejati, and

Marsha Chechik for providing valuable comments on earlier versions of this

work. The first author acknowledges financial support from the Natural

Sciences and Engineering Research Council of Canada (NSERC), and is

grateful to University College London for hosting a visit during which this

article was shaped.

References

Darke, P., & Shanks, G. 1996. Stakeholder viewpoints in requirements

definition: A framework for understanding viewpoint development approaches.

Requirements Engineering, 1(2), 88–105.

Filman, R., Elrad, T., Clarke, S., & Aksit, M. 2004. Aspect-Oriented Soft-

ware Development. Reading, USA: Addison-Wesley.

Finkelstein, A., Kramer, J., Nuseibeh, B., Finkelstein, L., & Goedicke, M.

1992. Viewpoints: A Framework for Integrating Multiple Perspectives

in System Development. International Journal of Software Engineering

and Knowledge Engineering, 2(1), 31–58.

France, R., & Rumpe, B. 2007. Model-Driven Development of Complex

Software: A Research Roadmap. Pages 37–55 of: Future of Software

Engineering Track of the 29th International Conference on Software

Engineering.

France, R., Fleurey, F., Reddy, R., Baudry, B., & Ghosh, S. 2007. Providing

Support for Model Composition in Metamodels. Pages 253–266

of: Proceedings of the 11th IEEE International Enterprise Distributed

Object Computing Conference.

Herbsleb, J. 2007. Global Software Engineering: The Future of Socio-

Technical Coordination. Pages 188–198 of: Future of Software Engineering

Track of the 29th International Conference on Software Engineering.

Kiczales, G., Hilsdale, E., Hugunin, J., Kersten, M., Palm, J., & Griswold,

W. 2001. An Overview of AspectJ. Pages 327–353 of: Proceedings of

the 15th European Conference on Object-Oriented Programming.

Magee, J., & Kramer, J. 2006. Concurrency: State models and Java Programming:

2nd Edition. Wiley.

Magee, J., Dulay, N., Eisenbach, S., & Kramer, J. 1995. Specifying Distributed

Software Architectures. Pages 137–153 of: Proceedings of the

5th European Software Engineering Conference.

Milner, R. 1989. Communication and Concurrency. New York, USA:

Prentice-Hall.

Morin, B., Barais, O., Nain, G., & J´ez´equel, J. 2009. Taming Dynamically

Adaptive Systems using Models and Aspects. Pages 122–132 of: Proceedings

of the 31st International Conference on Software Engineering.

Nentwich, C., Emmerich, W., Finkelstein, A., & Ellmer, E. 2003. Flexible

Consistency Checking. ACM Transactions on Software Engineering and

Methodology, 12(1), 28–63.

Nuseibeh, B., Kramer, J., & Finkelstein, A. 1994. A Framework for Expressing

the Relationships Between Multiple Views in Requirements Specification.

IEEE Transactions on Software Engineering, 20(10), 760–773.

Nuseibeh, B., Easterbrook, S., & Russo, A. 2000. Leveraging Inconsistency

in Software Development. IEEE Computer, 33(4), 24–29.

Nuseibeh, B., Kramer, J., & Finkelstein, A. 2003. ViewPoints: Meaningful

Relationships Are Difficult! Pages 676–683 of: Proceedings of the 25th

International Conference on Software Engineering.

Sabetzadeh, M. 2008. Merging and Consistency Checking of Distributed

Models. Ph.D. thesis, University of Toronto.

Spanoudakis, G., & Zisman, A. 2001. Handbook of Software Engineering and

Knowledge Engineering. World scientific. Chap. Inconsistency management

in software engineering: Survey and open research issues, pages

329–380.

Spanoudakis, G., Finkelstein, A., & Emmerich, W. (eds). 1996. Viewpoints

96: International Workshop on Multiple Perspectives in Software Development.

Unified Modeling Language. 2003. The Unified Modeling Language.

http://www.uml.org/.

van Lamsweerde, A. 2009. Requirements Engineering: From System Goals

to UML Models to Software Specifications. Wiley.

Whittle, J., Moreira, A., Ara´ujo, J., Jayaraman, P., Elkhodary, A., & Rabbi,

R. 2007. An Expressive Aspect Composition Language for UML State

Diagrams. Pages 514–528 of: Proceedings of the 10th International

Conference on Model Driven Engineering Languages and Systems.

 

Introduction to Phil Barnard

 

Introduction to the paper

 

 

Bridging between Basic Theories and the

Artifacts of Human-Computer Interaction

Phil Barnard

In: Carroll, J.M. (Ed.). Designing Interaction: psychology at the human-computer interface.

New York: Cambridge University Press, Chapter 7, 103-127. This is not an exact copy of

paper as it appeared but a DTP lookalike with very slight differences in pagination.

Psychological ideas on a particular set of topics go through something very much

like a product life cycle. An idea or vision is initiated, developed, and

communicated. It may then be exploited, to a greater or lesser extent, within the

research community. During the process of exploitation, the ideas are likely to

be the subject of critical evaluation, modification, or extension. With

developments in basic psychology, the success or penetration of the scientific

product can be evaluated academically by the twin criteria of citation counts and

endurance. As the process of exploitation matures, the idea or vision stimulates

little new research either because its resources are effectively exhausted or

because other ideas or visions that incorporate little from earlier conceptual

frameworks have taken over. At the end of their life cycle, most ideas are

destined to become fossilized under the pressure of successive layers of journals

opened only out of the behavioral equivalent of paleontological interest.

In applied domains, research ideas are initiated, developed, communicated,

and exploited in a similar manner within the research community. Yet, by the

very nature of the enterprise, citation counts and endurance are of largely

academic interest unless ideas or knowledge can effectively be transferred from

research to development communities and then have a very real practical impact

on the final attributes of a successful product.

If we take the past 20-odd years as representing the first life cycle of research

in human-computer interaction, the field started out with few empirical facts and

virtually no applicable theory. During this period a substantial body of work

was motivated by the vision of an applied science based upon firm theoretical

foundations. As the area was developed, there can be little doubt, on the twin

academic criteria of endurance and citation, that some theoretical concepts have

been successfully exploited within the research community. GOMS, of course,

is the most notable example (Card, Moran, & Newell, 1983; Olson & Olson,

1990; Polson, 1987). Yet, as Carroll (e.g., l989a,b) and others have pointed

out, there are very few examples where substantive theory per se has had a major

and direct impact on design. On this last practical criterion, cognitive science can

more readily provide examples of impact through the application of empirical

methodologies and the data they provide and through the direct application of

psychological reasoning in the invention and demonstration of design concepts

(e.g., see Anderson & Skwarecki, 1986; Card & Henderson, 1987; Carroll,

1989a,b; Hammond & Allinson, 1988; Landauer, 1987).

As this research life cycle in HCI matures, fundamental questions are being

asked about whether or not simple deductions based on theory have any value at

all in design (e.g. Carroll, this volume), or whether behavior in human-computer

interactions is simply too complex for basic theory to have anything other than a

minor practical impact (e.g., see Landauer, this volume). As the next cycle of

research develops, the vision of a strong theoretical input to design runs the risk

of becoming increasingly marginalized or of becoming another fossilized

laboratory curiosity. Making use of a framework for understanding different

research paradigms in HCI, this chapter will discuss how theory-based research

might usefully evolve to enhance its prospects for both adequacy and impact.

Bridging Representations

In its full multidisciplinary context, work on HCI is not a unitary enterprise.

Rather, it consists of many different sorts of design, development, and research

activities. Long (1989) provides an analytic structure through which we can

characterize these activities in terms of the nature of their underlying concepts

and how different types of concept are manipulated and interrelated. Such a

framework is potentially valuable because it facilitates specification of,

comparison between, and evaluation of the many different paradigms and

practices operating within the broader field of HCI.

With respect to the relationship between basic science and its application,

Long makes three points that are fundamental to the arguments to be pursued in

this and subsequent sections. First, he emphasizes that the kind of

understanding embodied in our science base is a representation of the way in

which the real world behaves. Second, any representation in the science base

can only be mapped to and from the real world by what he called “intermediary”

representations. Third, the representations and mappings needed to realize this

kind of two-way conceptual traffic are dependent upon the nature of the activities

they are required to support. So the representations called upon for the purposes

of software engineering will differ from the representations called upon for the

purposes of developing an applicable cognitive theory.

Long’s framework is itself a developing one (1987, 1989; Long & Dowell,

1989). Here, there is no need to pursue the details; it is sufficient to emphasize

that the full characterization of paradigms operating directly with artifact design

differs from those characterizing types of engineering support research, which,

in turn, differ from more basic research paradigms. This chapter will primarily

be concerned with what might need to be done to facilitate the applicability and

impact of basic cognitive theory. In doing so it will be argued that a key role

needs to be played by explicit “bridging” representations. This term will be used

to avoid any possible conflict with the precise properties of Long’s particular

conceptualization.

Following Long (1989), Figure 7.1 shows a simplified characterization of an

applied science paradigm for bridging from the real world of behavior to the

science base and from these representations back to the real world. The blocks

are intended to characterize different sorts of representation and the arrows stand

for mappings between them (Long’s terminology is not always used here). The

real world of the use of interactive software is characterized by organisational,

Basic Theories and the Artifacts of HCI

group, and physical settings; by artifacts such as computers, software, and

manuals; by the real tasks of work; by characteristics of the user population; and

so on. In both applied and basic research, we construct our science not from the

real world itself but via a bridging representation whose purpose is to support

and elaborate the process of scientific discovery.

Obviously, the different disciplines that contribute to HCI each have their

own forms of discovery representation that reflect their paradigmatic

perspectives, the existing contents of their science base, and the target form of

their theory. In all cases the discovery representation incorporates a whole range

of explicit, and more frequently implicit, assumptions about the real world and

methodologies that might best support the mechanics of scientific abstraction. In

the case of standard paradigms of basic psychology, the initial process of

analysis leading to the formation of a discovery representation may be a simple

observation of behavior on some task. For example, it may be noted that

ordinary people have difficulty with particular forms of syllogistic reasoning. In

more applied research, the initial process of analysis may involve much more

elaborate taxonomization of tasks (e.g., Brooks, this volume) or of errors

observed in the actual use of interactive software (e.g., Hammond, Long, Clark,

Barnard, & Morton, 1980).

Conventionally, a discovery representation drastically simplifies the real

world. For the purposes of gathering data about the potential phenomena, a

limited number of contrastive concepts may need to be defined, appropriate

materials generated, tasks selected, observational or experimental designs

determined, populations and metrics selected, and so on. The real world of

preparing a range of memos, letters, and reports for colleagues to consider

before a meeting may thus be represented for the purposes of initial discovery by

 

an observational paradigm with a small population of novices carrying out a

limited range of tasks with a particular word processor (e.g., Mack, Lewis, &

Carroll, 1983). In an experimental paradigm, it might be represented

noninteractively by a paired associate learning task in which the mappings

between names and operations need to be learned to some criterion and

subsequently recalled (e.g., Scapin, 1981). Alternatively, it might be

represented by a simple proverb-editing task carried out on two alternative

versions of a cut-down interactive text editor with ten commands. After some

form of instructional familiarization appropriate to a population of computernaive

members of a Cambridge volunteer subject panel, these commands may be

used an equal number of times with performance assessed by time on task,

errors, and help usage (e.g., Barnard, Hammond, MacLean, & Morton, 1982).

Each of the decisions made contributes to the operational discovery

representation.

The resulting characterizations of empirical phenomena are potential

regularities of behavior that become, through a process of assimilation,

incorporated into the science base where they can be operated on, or argued

about, in terms of the more abstract, interpretive constructs. The discovery

representations constrain the scope of what is assimilated to the science base and

all subsequent mappings from it.

The conventional view of applied science also implies an inverse process

involving some form of application bridge whose function is to support the

transfer of knowledge in the science base into some domain of application.

Classic ergonomics-human factors relied on the handbook of guidelines. The

relevant processes involve contextualizing phenomena and scientific principles

for some applications domain – such as computer interfaces, telecommunications

apparatus, military hardware, and so on. Once explicitly formulated, say in

terms of design principles, examples and pointers to relevant data, it is left up to

the developers to operate on the representation to synthesize that information

with any other considerations they may have in the course of taking design

decisions. The dominant vision of the first life cycle of HCI research was that

this bridging could effectively be achieved in a harder form through engineering

approximations derived from theory (Card et al., 1983). This vision essentially

conforms to the full structure of Figure 7.1

The Chasm to Be Bridged

The difficulties of generating a science base for HCI that will support effective

bridging to artifact design are undeniably real. Many of the strategic problems

theoretical approaches must overcome have now been thoroughly aired. The life

cycle of theoretical enquiry and synthesis typically postdates the life cycle of

products with which it seeks to deal; the theories are too low level; they are of

restricted scope; as abstractions from behavior they fail to deal with the real

context of work and they fail to accommodate fine details of implementations and

interactions that may crucially influence the use of a system (see, e.g.,

discussions by Carroll & Campbell, 1986; Newell & Card, 1985; Whiteside &

Basic Theories and the Artifacts of HCI 107

Wixon, 1987). Similarly, although theory may predict significant effects and

receive empirical support, those effects may be of marginal practical consequence

in the context of a broader interaction or less important than effects not

specifically addressed (e.g., Landauer, 1987).

Our current ability to construct effective bridges across the chasm that

separates our scientific understanding and the real world of user behavior and

artifact design clearly falls well short of requirements. In its relatively short

history, the scope of HCI research on interfaces has been extended from early

concerns with the usability of hardware, through cognitive consequences of

software interfaces, to encompass organizational issues (e.g., Grudin, 1990).

Against this background, what is required is something that might carry a

volume of traffic equivalent to an eight-lane cognitive highway. What is on offer

is more akin to a unidirectional walkway constructed from a few strands of rope

and some planks.

In Taking artifacts seriously Carroll (1989a) and Carroll, Kellogg, and

Rosson in this volume, mount an impressive case against the conventional view

of the deductive application of science in the invention, design, and development

of practical artifacts. They point both to the inadequacies of current informationprocessing

psychology, to the absence of real historical justification for

deductive bridging in artifact development, and to the paradigm of craft skill in

which knowledge and understanding are directly embodied in artifacts.

Likewise, Landauer (this volume) foresees an equally dismal future for theorybased

design.

Whereas Landauer stresses the potential advances that may be achieved

through empirical modeling and formative evaluation. Carroll and his colleagues

have sought a more substantial adjustment to conventional scientific strategy

(Carroll, 1989a,b, 1990; Carroll & Campbell, 1989; Carroll & Kellogg, 1989;

Carroll et al., this volume). On the one hand they argue that true “deductive”

bridging from theory to application is not only rare, but when it does occur, it

tends to be underdetermined, dubious, and vague. On the other hand they argue

that the form of hermaneutics offered as an alternative by, for example,

Whiteside and Wixon (1987) cannot be systematized for lasting value. From

Carroll’s viewpoint, HCI is best seen as a design science in which theory and

artifact are in some sense merged. By embodying a set of interrelated

psychological claims concerning a product like HyperCard or the Training

Wheels interface (e.g., see Carroll & Kellogg, 1989), the artifacts themselves

take on a theorylike role in which successive cycles of task analysis,

interpretation, and artifact development enable design-oriented assumptions

about usability to be tested and extended.

This viewpoint has a number of inviting features. It offers the potential of

directly addressing the problem of complexity and integration because it is

intended to enable multiple theoretical claims to be dealt with as a system

bounded by the full artifact. Within the cycle of task analysis and artifact

development, the analyses, interpretations, and theoretical claims are intimately

bound to design problems and to the world of “real” behavior. In this context,

knowledge from HCI research no longer needs to be transferred from research

into design in quite the same sense as before and the life cycle of theories should

also be synchronized with the products they need to impact. Within this

framework, the operational discovery representation is effectively the rationale

governing the design of an artifact, whereas the application representation is a

series of user-interaction scenarios (Carroll, 1990).

The kind of information flow around the task – artifact cycle nevertheless

leaves somewhat unclear the precise relationships that might hold between the

explicit theories of the science base and the kind of implicit theories embodied in

artifacts. Early on in the development of these ideas, Carroll (1989a) points out

that such implicit theories may be a provisional medium for HCI, to be put aside

when explicit theory catches up. In a stronger version of the analysis, artifacts

are in principle irreducible to a standard scientific medium such as explicit

theories. Later it is noted that “it may be simplistic to imagine deductive relations

between science and design, but it would be bizarre if there were no relation at

all” (Carroll & Kellogg, 1989). Most recently, Carroll (1990) explicitly

identifies the psychology of tasks as the relevant science base for the form of

analysis that occurs within the task-artifact cycle (e.g. see Greif, this volume;

Norman this volume). The task-artifact cycle is presumed not only to draw upon

and contextualize knowledge in that science base, but also to provide new

knowledge to assimilate to it. In this latter respect, the current view of the task

artifact cycle appears broadly to conform with Figure 7.1. In doing so it makes

use of task-oriented theoretical apparatus rather than standard cognitive theory

and novel bridging representations for the purposes of understanding extant

interfaces (design rationale) and for the purposes of engineering new ones

(interaction scenarios).

In actual practice, whether the pertinent theory and methodology is grounded

in tasks, human information-processing psychology or artificial intelligence,

those disciplines that make up the relevant science bases for HCI are all

underdeveloped. Many of the basic theoretical claims are really provisional

claims; they may retain a verbal character (to be put aside when a more explicit

theory arrives), and even if fully explicit, the claims rarely generalize far beyond

the specific empirical settings that gave rise to them. In this respect, the wider

problem of how we go about bridging to and from a relevant science base

remains a long-term issue that is hard to leave unaddressed. Equally, any

research viewpoint that seeks to maintain a productive role for the science base in

artifact design needs to be accompanied by a serious reexamination of the

bridging representations used in theory development and in their application.

Science and design are very different activities. Given Figure 7.1., theorybased

design can never be direct; the full bridge must involve a transformation of

information in the science base to yield an applications representation, and

information in this structure must be synthesized into the design problem. In

much the same way that the application representation is constructed to support

design, our science base, and any mappings from it, could be better constructed

to support the development of effective application bridging. The model for

relating science to design is indirect, involving theoretical support for

Basic Theories and the Artifacts of HCI 109

engineering representations (both discovery and applications) rather than one

involving direct theoretical support in design.

The Science Base and Its Application

In spite of the difficulties, the fundamental case for the application of cognitive

theory to the design of technology remains very much what it was 20 years ago,

and indeed what it was 30 years ago (e.g., Broadbent, 1958). Knowledge

assimilated to the science base and synthesized into models or theories shoudl

reduce our reliance on purely empirical evaluations. It offers the prospect of

supporting a deeper understanding of design issues and how to resolve them.

Indeed, Carroll and Kellogg’s (1989) theory nexus has developed out of a

cognitive paradigm rather than a behaviorist one. Although theory development

lags behind the design of artifacts, it may well be that the science base has more

to gain than the artifacts. The interaction of sicnece and design nevertheless

should be a two-way process of added value.

Much basic theoretical work involves the application of only partially explicit

and incomplete apparatus to specific laboratory tasks. It is not unreasonable to

argue that our basic cognitive theory tends only to be successful for modeling a

particular application. That application is itself behavior in laboratory tasks. The

scope of the application is delimited by the empirical paradigms and the artifacts

it requires – more often than not these days, computers and software for

presentation of information and response capture. Indeed, Carroll’s task-artifact

and interpretation cycles could very well be used to provide a neat description of

the research activities involved in the iterative design and development of basic

theory. The trouble is that the paradigms of basic psychological research, and

the bridging representations used to develop and validate theory, typically

involve unusually simple and often highly repetitive behavioral requirements

atypical of those faced outside the laboratory.

Although it is clear that there are many cases of invention and craft where the

kinds of scientific understanding established in the laboratory play little or no

role in artifact development (Carroll, 1989b), this is only one side of the story.

The other side is that we should only expect to find effective bridging when what

is in the science base is an adequate representation of some aspect of the real

world that is relevant to the specific artifact under development. In this context it

is worth considering a couple of examples not usually called into play in the HCI

domain.

Psychoacoustic models of human hearing are well developed. Auditory

warning systems on older generations of aircraft are notoriously loud and

unreliable. Pilots don’t believe them and turn them off. Using standard

techniques, it is possible to measure the noise characteristics of the environment

on the flight deck of a particular aircraft and to design a candidate set of warnings

based on a model of the characteristics of human hearing. This determines

whether or not pilots can be expected to “hear” and identify those warnings over

the pattern of background noise without being positively deafened and distracted

(e.g., Patterson, 1983). Of course, the attention-getting and discriminative

110 Barnard

properties of members of the full set of warnings still have to be crafted. Once

established, the extension of the basic techniques to warning systems in hospital

intensive-care units (Patterson, Edworthy, Shailer, Lower, & Wheeler, 1986)

and trains (Patterson, Cosgrove, Milroy, & Lower, 1989) is a relatively routine

matter.

Developed further and automated, the same kind of psychoacoustic model

can play a direct role in invention. As the front end to a connectionist speech

recognizer, it offers the prospect of a theoretically motivated coding structure that

could well prove to outperform existing technologies (e.g., see ACTS, 1989).

As used in invention, what is being embodied in the recognition artifact is an

integrated theory about the human auditory system rather than a simple heuristic

combination of current signal-processing technologies.

Another case arises out of short-term memory research. Happily, this one

does not concern limited capacity! When the research technology for short-term

memory studies evolved into a computerized form, it was observed that word

lists presented at objectively regular time intervals (onset to onset times for the

sound envelopes) actually sounded irregular. In order to be perceived as regular

the onset to onset times need to be adjusted so that the “perceptual centers” of the

words occur at equal intervals (Morton, Marcus, & Frankish, 1976). This

science base representation, and algorithms derived from it, can find direct use in

telecommunications technology or speech interfaces where there is a requirement

for the automatic generation of natural sounding number or option sequences.

Of course, both of these examples are admittedly relatively “low level.” For

many higher level aspects of cognition, what is in the science base are

representations of laboratory phenomena of restricted scope and accounts of

them. What would be needed in the science base to provide conditions for

bridging are representations of phenomena much closer to those that occur in the

real world. So, for example, the theoretical representations should be topicalized

on phenomena that really matter in applied contexts (Landauer, 1987). They

should be theoretical representations dealing with extended sequences of

cognitive behavior rather than discrete acts. They should be representations of

information-rich environments rather than information-impoverished ones. They

should relate to circumstances where cognition is not a pattern of short repeating

(experimental) cycles but where any cycles that might exist have meaning in

relation to broader task goals and so on.

It is not hard to pursue points about what the science base might incorporate

in a more ideal world. Nevertheless, it does contain a good deal of useful

knowledge (cf. Norman, 1986), and indeed the first life cycle of HCI research

has contributed to it. Many of the major problems with the appropriateness,

scope, integration, and applicability of its content have been identified. Because

major theoretical prestroika will not be achieved overnight, the more productive

questions concern the limitations on the bridging representations of that first

cycle of research and how discovery representations and applications

representations might be more effectively developed in subsequent cycles.

An Analogy with Interface Design Practice

Basic Theories and the Artifacts of HCI

Not surprisingly, those involved in the first life cycle of HCI research relied very

heavily in the formation of their discovery representations on the methodologies

of the parent discipline. Likewise, in bridging from theory to application, those

involved relied heavily on the standard historical products used in the verification

of basic theory, that is, prediction of patterns of time and/or errors. There are

relatively few examples where other attributes of behavior are modeled, such as

choice among action sequences (but see Young & MacLean, 1988). A simple

bridge, predictive of times of errors, provides information about the user of an

interactive system. The user of that information is the designer, or more usually

the design team. Frameworks are generally presented for how that information

might be used to support design choice either directly (e.g., Card et al., 1983) or

through trade-off analyses (e.g., Norman, 1983). However, these forms of

application bridge are underdeveloped to meet the real needs of designers.

Given the general dictum of human factors research, “Know the user”

(Hanson, 1971), it is remarkable how few explicitly empirical studies of design

decision making are reported in the literature. In many respects, it would not be

entirely unfair to argue that bridging representations between theory and design

have remained problematic for the same kinds of reasons that early interactive

interfaces were problematic. Like glass teletypes, basic psychological

technologies were underdeveloped and, like the early design of command

languages, the interfaces (application representations) were heuristically

constructed by applied theorists around what they could provide rather than by

analysis of requirements or extensive study of their target users or the actual

context of design (see also Bannon & BØdker, this volume; Henderson, this

volume).

Equally, in addressing questions associated with the relationship between

theory and design, the analogy can be pursued one stage further by arguing for

the iterative design of more effective bridging structures. Within the first life

cycle of HCI research a goodly number of lessons have been learned that could

be used to advantage in a second life cycle. So, to take a very simple example,

certain forms of modeling assume that users naturally choose the fastest method

for achieving their goal. However there is now some evidence that this is not

always the case (e.g., MacLean, Barnard, & Wilson, 1985). Any role for the

knowledge and theory embodied in the science base must accommodate, and

adapt to, those lessons. For many of the reasons that Carroll and others have

elaborated, simple deductive bridging is problematic. To achieve impact,

behavioral engineering research must itself directly support the design,

development, and invention of artifacts. On any reasonable time scale there is a

need for discovery and application representations that cannot be fully justified

through science-base principles or data. Nonetheless, such a requirement simply

restates the case for some form of cognitive engineering paradigm. It does not in

and of itself undermine the case for the longer-term development of applicable

theory.

Just as impact on design has most readily been achieved through the

application of psychological reasoning in the invention and demonstration of

artifacts, so a meaningful impact of theory might best be achieved through the

invention and demonstration of novel forms of applications representations. The

development of representations to bridge from theory to application cannot be

taken in isolation. It needs to be considered in conjunction with the contents of

the science base itself and the appropriateness of the discovery representations

that give rise to them.

Without attempting to be exhaustive, the remainder of this chapter will

exemplify how discovery representations might be modified in the second life

cycle of HCI research; and illustrate how theory might drive, and itself benefit

from, the invention and demonstration of novel forms of applications bridging.

Enhancing Discovery Representations

Although disciplines like psychology have a formidable array of methodological

techniques, those techniques are primarily oriented toward hypothesis testing.

Here, greatest effort is expended in using factorial experimental designs to

confirm or disconfirm a specific theoretical claim. Often wider characteristics of

phenomena are only charted as and when properties become a target of specific

theoretical interest. Early psycholinguistic research did not start off by studying

what might be the most important factors in the process of understanding and

using textual information. It arose out of a concern with transformational

grammars (Chomsky, 1957). In spite of much relevant research in earlier

paradigms (e.g., Bartlett, 1932), psycholinguistics itself only arrived at this

consideration after progressing through the syntax, semantics, and pragmatics of

single-sentence comprehension.

As Landauer (1987) has noted, basic psychology has not been particularly

productive at evolving exploratory research paradigms. One of the major

contributions of the first life cycle of HCI research has undoubtedly been a

greater emphasis on demonstrating how such empirical paradigms can provide

information to support design (again, see Landauer, 1987). Techniques for

analyzing complex tasks, in terms of both action decomposition and knowledge

requirements, have also progressed substantially over the past 20 years (e.g.,

Wilson, Barnard, Green, & MacLean, 1988).

A significant number of these developments are being directly assimilated

into application representations for supporting artifact development. Some can

also be assimilated into the science base, such as Lewis’s (1988) work on

abduction. Here observational evidence in the domain of HCI (Mack et al.,

1983) leads directly to theoretical abstractions concerning the nature of human

reasoning. Similarly, Carroll (1985) has used evidence from observational and

experimental studies in HCI to extend the relevant science base on naming and

reference. However, not a lot has changed concerning the way in which

discovery representations are used for the purposes of assimilating knowledge to

the science base and developing theory.

In their own assessment of progress during the first life cycle of HCI

research, Newell and Card (1985) advocate continued reliance on the hardening

of HCI as a science. This implicitly reinforces classic forms of discovery

representations based upon the tools and techniques of parent disciplines. Heavy

Basic Theories and the Artifacts of HCI 113

reliance on the time-honored methods of experimental hypothesis testing in

experimental paradigms does not appear to offer a ready solution to the two

problems dealing with theoretical scope and the speed of theoretical advance.

Likewise, given that these parent disciplines are relatively weak on exploratory

paradigms, such an approach does not appear to offer a ready solution to the

other problems of enhancing the science base for appropriate content or for

directing its efforts toward the theoretical capture of effects that really matter in

applied contexts.

The second life cycle of research in HCI might profit substantially by

spawning more effective discovery representations, not only for assimilation to

applications representations for cognitive engineering, but also to support

assimilation of knowledge to the science base and the development of theory.

Two examples will be reviewed here. The first concerns the use of evidence

embodied in HCI scenarios (Young & Barnard, 1987, Young, Barnard, Simon,

& Whittington, 1989). The second concerns the use of protocol techniques to

systematically sample what users know and to establish relationships between

verbalizable knowledge and actual interactive performance.

Test-driving Theories

Young and Barnard (1987) have proposed that more rapid theoretical advance

might be facilitated by “test driving” theories in the context of a systematically

sampled set of behavioral scenarios. The research literature frequently makes

reference to instances of problematic or otherwise interesting user-system

exchanges. Scenario material derived from that literature is selected to represent

some potentially robust phenomenon of the type that might well be pursued in

more extensive experimental research. Individual scenarios should be regarded

as representative of the kinds of things that really matter in applied settings. So

for example, one scenario deals with a phenomenon often associated with

unselected windows. In a multiwindowing environment a persistent error,

frequently committed even by experienced users, is to attempt some action in

inactive window. The action might be an attempt at a menu selection. However,

pointing and clicking over a menu item does not cause the intended result; it

simply leads to the window being activated. Very much like linguistic test

sentences, these behavioral scenarios are essentially idealized descriptions of

such instances of human-computer interactions.

If we are to develop cognitive theories of significant scope they must in

principle be able to cope with a wide range of such scenarios. Accordingly, a

manageable set of scenario material can be generated that taps behaviors that

encompass different facets of cognition. So, a set of scenarios might include

instances dealing with locating information in a directory entry, selecting

alternative methods for achieving a goal, lexical errors in command entry, the

unselected windows phenomenon, and so on (see Young, Barnard, Simon, &

Whittington, 1989). A set of contrasting theoretical approaches can likewise be

selected and the theories and scenarios organized into a matrix. The activity

involves taking each theoretical approach and attempting to formulate an account

114 Barnard

of each behavioral scenario. The accuracy of the account is not at stake. Rather,

the purpose of the exercise is to see whether a particular piece of theoretical

apparatus is even capable of giving rise to a plausible account. The scenario

material is effectively being used as a set of sufficiency filters and it is possible to

weed out theories of overly narrow scope. If an approach is capable of

formulating a passable account, interest focuses on the properties of the account

offered. In this way, it is also possible to evaluate and capitalize on the

properties of theoretical apparatus and do provide appropriate sorts of analytic

leverage over the range of scenarios examined.

Traditionally, theory development places primary emphasis on predictive

accuracy and only secondary emphasis on scope. This particular form of

discovery representation goes some way toward redressing that balance. It

offers the prospect of getting appropriate and relevant theoretical apparatus in

place on a relatively short time cycle. As an exploratory methodology, it at least

addresses some of the more profound difficulties of interrelating theory and

application. The scenario material makes use of known instances of humancomputer

interaction. Because these scenarios are by definition instances of

interactions, any theoretical accounts built around them must of necessity be

appropriate to the domain. Because scenarios are intended to capture significant

aspects of user behavior, such as persistent errors, they are oriented toward what

matters in the applied context. As a quick and dirty methodology, it can make

effective use of the accumulated knowledge acquired in the first life cycle of HCI

research, while avoiding some of the worst “tar pits” (Norman, 1983) of

traditional experimental methods.

As a form of discovery bridge between application and theory, the real world

is represented, for some purpose, not by a local observation or example, but by a

sampled set of material. If the purpose is to develop a form of cognitive

architecture , then it may be most productive to select a set of scenarios that

encompass different components of the cognitive system (perception, memory,

decision making, control of action). Once an applications representation has

been formed, its properties might be further explored and tested by analyzing

scenario material sampled over a range of different tasks, or applications

domains (see Young & Barnard, 1987). At the point where an applications

representation is developed, the support it offers may also be explored by

systematically sampling a range of design scenarios and examining what

information can be offered concerning alternative interface options (AMODEUS,

1989). By contrast with more usual discovery representations, the scenario

methodology is not primarily directed at classic forms of hypothesis testing and

validation. Rather, its purpose is to support the generation of more readily

applicable theoretical ideas.

Verbal Protocols and Performanc

One of the most productive exploratory methodologies utilized in HCI research

has involved monitoring user action while collecting concurrent verbal protocols

to help understand what is actually going on. Taken together these have often

Basic Theories and the Artifacts of HCI 115

given rise to the best kinds of problem-defining evidence, including the kind of

scenario material already outlined. Many of the problems with this form of

evidence are well known. Concurrent verbalization may distort performance and

significant changes in performance may not necessarily be accompanied by

changes in articulatable knowledge. Because it is labor intensive, the

observations are often confined to a very small number of subjects and tasks. In

consequence, the representatives of isolated observations is hard to assess.

Furthermore, getting real scientific value from protocol analysis is crucially

dependent on the insights and craft skill of the researcher concerned (Barnard,

Wilson, & MacLean, 1986; Ericsson & Simon, 1980).

Techniques of verbal protocol analysis can nevertheless be modified and

utilized as a part of a more elaborate discovery representation to explore and

establish systematic relationships between articulatable knowledge and

performance. The basic assumption underlying much theory is that a

characterization of the ideal knowledge a user should possess to successfully

perform a task can be used to derive predictions about performance. However,

protocol studies clearly suggest that users really get into difficulty when they

have erroneous or otherwise nonideal knowledge. In terms of the precise

relationships they have with performance, ideal and nonideal knowledge are

seldom considered together.

In an early attempt to establish systematic and potentially generalizable

relationships between the contents of verbal protocols and interactive

performance, Barnard et al., (1986) employed a sample of picture probes to elicit

users’ knowledge of tasks, states, and procedures for a particular office product

at two stages of learning. The protocols were codified, quantified, and

compared. In the verbal protocols, the number of true claims about the system

increased with system experience, but surprisingly, the number of false claims

remained stable. Individual users who articulated a lot of correct claims

generally performed well, but the amount of inaccurate knowledge did not appear

related to their overall level of performance. There was, however, some

indication that the amount of inaccurate knowledge expressed in the protocols

was related to the frequency of errors made in particular system contexts.

A subsequent study (Barnard, Ellis, & MacLean, 1989) used a variant of the

technique to examine knowledge of two different interfaces to the same

application functionality. High levels of inaccurate knowledge expressed in the

protocols were directly associated with the dialogue components on which

problematic performance was observed. As with the earlier study, the amount of

accurate knowledge expressed in any given verbal protocol was associated with

good performance, whereas the amount of inaccurate knowledge expressed bore

little relationship to an individual’s overall level of performance. Both studies

reinforced the speculation that is is specific interface characteristics that give rise

to the development of inaccurate or incomplete knowledge from which false

inferences and poor performance may follow.

Just as the systematic sampling and use of behavioral scenarios may facilitate

the development of theories of broader scope, so discovery representations

designed to systematically sample the actual knowledge possessed by users

116 Barnard

should facilitate the incorporation into the science base of behavioral regularities

and theoretical claims that are more likely to reflect the actual basis of user

performance rather than a simple idealization of it.

Enhancing Application Representations

The application representations of the first life cycle of HCI research relied very

much on the standard theoretical products of their parent disciplines.

Grammatical techniques originating in linguistics were utilized to characterize the

complexity of interactive dialogues; artificial intelligence (A1)-oriented models

were used to represent and simulate the knowledge requirements of learning;

and, of course, derivatives of human information-processing models were used

to calculate how long it would take users to do things. Although these

approaches all relied upon some form of task analysis, their apparatus was

directed toward some specific function. They were all of limited scope and made

numerous trade-offs between what was modeled and the form of prediction made

(Simon, 1988).

Some of the models were primarily directed at capturing knowledge

requirements for dialogues for the purposes of representing complexity, such as

BNF grammars (Reisner, 1982) and Task Action Grammars (Payne & Green,

1986). Others focused on interrelationships between task specifications and

knowledge requirements, such as GOMS analyses and cognitive-complexity

theory (Card et al. 1983; Kieras & Polson, 1985). Yet other apparatus, such as

the model human information processor and the keystroke level model of Card et

al. (1983) were primarily aimed at time prediction for the execution of error-free

routine cognitive skill. Most of these modeling efforts idealized either the

knowledge that users needed to possess or their actual behavior. Few models

incorporated apparatus for integrating over the requirements of knowledge

acquisition or use and human information-processing constraints (e.g., see

Barnard, 1987). As application representations, the models of the first life cycle

had little to say about errors or the actual dynamics of user-system interaction as

influenced by task constraints and information or knowledge about the domain of

application itself.

Two modeling approaches will be used to illustrate how applications

representations might usefully be enhanced. They are programmable user

models (Young, Green, & Simon, 1989) and modeling based on Interacting

Cognitive Subsystems (Barnard, 1985). Although these approaches have

different origins, both share a number of characteristics. They are both aimed at

modeling more qualitative aspects of cognition in user-system interaction; both

are aimed at understanding how task, knowledge, and processing constraint

intersect to determine performance; both are aimed at exploring novel means of

incorporating explicit theoretical claims into application representations; and both

require the implementation of interactive systems for supporting decision making

in a design context. Although they do so in different ways, both approaches

attempt to preserve a coherent role for explicit cognitive theory. Cognitive theory

is embodied, not in the artifacts that emerge from the development process, but

Basic Theories and the Artifacts of HCI 117

in demonstrator artifacts that might emerge from the development process, but in

demonstrator artifacts that might support design. This is almost directly

analogous to achieving an impact in the marketplace through the application of

psychological reasoning in the invention of artifacts. Except in this case, the

target user populations for the envisaged artifacts are those involved in the design

and development of products.

Programmable User Models (PUMs)

The core ideas underlying the notion of a programmable user model have their

origins in the concepts and techniques of AI. Within AI, cognitive architectures

are essentially sets of constraints on the representation and processing of

knowledge. In order to achieve a working simulation, knowledge appropriate to

the domain and task must be represented within those constraints. In the normal

simulation methodology, the complete system is provided with some data and,

depending on its adequacy, it behaves with more or less humanlike properties.

Using a simulation methodology to provide the designer with an artificial

user would be one conceivable tactic. Extending the forms of prediction offered

by such simulations (cf. cognitive complexity theory; Polson, 1987) to

encompass qualitative aspects of cognition is more problematic. Simply

simulating behavior is of relatively little value. Given the requirements of

knowledge-based programming, it could, in many circumstances, be much more

straightforward to provide a proper sample of real users. There needs to be

some mechanism whereby the properties of the simulation provide information

of value in design. Programmable user models provide a novel perspective on

this latter problem. The idea is that the designer is provided with two things, an

“empty” cognitive architecture and an instruction language for providing with all

the knowledge it needs to carry out some task. By programming it, the designer

has to get the architecture to perform that task under conditions that match those

of the interactive system design (i.e., a device model). So, for example, given a

particular dialog design, the designer might have to program the architecture to

select an object displayed in a particular way on a VDU and drag it across that

display to a target position.

The key, of course, is that the constraints that make up the architecture being

programmed are humanlike. Thus, if the designer finds it hard to get the

architecture to perform the task, then the implication is that a human user would

also find the task hard to accomplish. To concretize this, the designer may find

that the easiest form of knowledge-based program tends to select and drag the

wrong object under particular conditions. Furthermore, it takes a lot of thought

and effort to figure out how to get round this problem within the specific

architectural constraints of the model. Now suppose the designer were to adjust

the envisaged user-system dialog in the device model and then found that

reprogramming the architecture to carry out the same task under these new

conditions was straightforward and the problem of selecting the wrong object no

longer arose. Young and his colleagues would then argue that this constitutes

118 Barnard

direct evidence that the second version of the dialog design tried by the designer

is likely to prove more usable than the first.

The actual project to realize a working PUM remains at an early stage of

development. The cognitive architecture being used is SOAR (Laird, Newell, &

Rosenbloom, 1987). There are many detailed issues to be addressed concerning

the design of an appropriate instruction language. Likewise, real issues are

raised about how a model that has its roots in architectures for problem solving

(Newell & Simon, 1972) deals with the more peripheral aspects of human

information processing, such as sensation, perception, and motor control.

Nevertheless as an architecture, it has scope in the sense that a broad range of

tasks and applications can be modeled within it. Indeed, part of the motivation

of SOAR is to provide a unified general theory of cognition (Newell, 1989).

In spite of its immaturity, additional properties of the PUM concept as an

application bridging structure are relatively clear (see Young et al., 1989). First,

programmable user models embody explicit cognitive theory in the form of the

to-be-programmed architecture. Second, there is an interesting allocation of

function between the model and the designer. Although the modeling process

requires extensive operationalization of knowledge in symbolic form, the PUM

provides only the constraints and the instruction language, whereas the designer

provides the knowledge of the application and its associated tasks. Third,

knowledge in the science base is transmitted implicitly into the design domain via

an inherently exploratory activity. Designers are not told about the underlying

cognitive science; they are supposed to discover it. By doing what they know

how to do well – that is, programming – the relevant aspects of cognitive

constraints and their interactions with the application should emerge directly in

the design context.

Fourth, programmable user models support a form of qualitative predictive

evaluation that can be carried out relatively early in the design cycle. What that

evaluation provides is not a classic predictive product of laboratory theory, rather

it should be an understanding of why it is better to have the artifact constructed

one way rather than another. Finally, although the technique capitalizes on the

designer’s programming skills, it clearly requires a high degree of commitment

and expense. The instruction language has to be learned and doing the

programming would require the development team to devote considerable

resources to this form of predictive evaluation.

Approximate Models of Cognitive Activity

Interacting Cognitive Subsystems (Barnard, 1985) also specifies a form of

cognitive architecture. Rather than being an AI constraint-based architecture,

ICS has its roots in classic human information-processing theory. It specifies

the processing and memory resources underlying cognition, the organization of

these resources, and principles governing their operation. Structurally, the

complete human information-processing system is viewed as a distributed

architecture with functionally distinct subsystems each specializing in, and

supporting, different types of sensory, representational, and effector processing

Basic Theories and the Artifacts of HCI 119

activity. Unlike many earlier generations of human information-processing

models, there are no general purpose resources such as a central executive or

limited capacity working memory. Rather the model attempts to define and

characterize processes in terms of the mental representations they take as input

and the representations they output. By focusing on the mappings between

different mental representations, this model seeks to integrate a characterization

of knowledge-based processing activity with classic structural constraints on the

flow of information within the wider cognitive system.

A graphic representation of this architecture is shown in the right-hand panel

of Figure 7.2, which instantiates Figure 7.1 for the use of the ICS framework in

an HCI context. The architecture itself is part of the science base. Its initial

development was supported by using empirical evidence from laboratory studies

of short-term memory phenomena (Barnard, 1985). However, by concentrating

on the different types of mental representation and process that transform them,

rather than task and paradigm specific concepts, the model can be applied across

a broad range of settings (e.g., see Barnard & Teasdale, 1991). Furthermore,

for the purposes of constructing a representation to bridge between theory and

application it is possible to develop explicit, yet approximate, characterizations of

cognitive activity.

In broad terms, the way in which the overall architecture will behave is

dependent upon four classes of factor. First, for any given task it will depend on

the precise configuration of cognitive activity. Different subsets of processes

and memory records will be required by different tasks. Second, behavior will

be constrained by the specific procedural knowledge embodied in each mental

process that actually transforms one type of mental representation to another.

Third, behavior will be constrained by the form, content, and accessibility of any

memory records that are need in that phase of activity. Fourth, it will depend on

the overall way in which the complete configuration is coordinated and

controlled.

Because the resources are relatively well defined and constrained in terms of

their attributes and properties, interdependencies between them can be motivated

on the basis of known patterns of experimental evidence and rendered explicit.

So, for example, a complexity attribute of the coordination and control of

cognitive activity can be directly related to the number of incompletely

proceduralized processes within a specified configuration. Likewise, a strategic

attribute of the coordination and control of cognitive activity may be dependent

upon the overall amount of order uncertainty associated with the mental

representation of a task stored in a memory record. For present purposes the

precise details of these interdependencies do not matter, nor does the particularly

opaque terminology shown in the rightmost panel of Figure 7.2 (for more

details, see Barnard, 1987). The important point is that theoretical claims can be

specified within this framework at a high level of abstraction and that these

abstractions belong in the science base.

Although these theoretical abstractions could easily have come from classic

studies of human memory and performance, there were in fact motivated by

experimental studies of command naming in text editing (Grudin & Barnard,

120 Barnard

1984) and performance on an electronic mailing task (Barnard, MacLean, &

Hammond, 1984). The full theoretical analyses are described in Barnard (1987)

and extended in Barnard, Grudin, and MacLean (1989). In both cases the tasks

were interactive, involved extended sequences of cognitive behavior, involved

information-rich environments, and the repeating patterns of data collection were

meaningful in relation to broader task goals not atypical of interactive tasks in the

real world. In relation to the arguments presented earlier in this chapter, the

information being assimilated to the science base should be more appropriate and

relevant to HCI than that derived from more abstract laboratory paradigms. It

will nonetheless be subject to interpretive restrictions inherent in the particular

form of discovery representation utilized in the design of these particular

experiments.

Armed with such theoretical abstractions, and accepting their potential

limitations, it is possible to generate a theoretically motivated bridge to

application. The idea is to build approximate models that describe the nature of

cognitive activity underlying the performance of complex tasks. The process is

actually carried out by an expert system that embodies the theoretical knowledge

required to build such models. The system “knows” what kinds of

configurations are associated with particular phases of cognitive activity; it

“knows” something about the conditions under which knowledge becomes

proceduralized, and the properties of memory records that might support recall

and inference in complex task environments. It also “knows” something about

the theoretical interdependencies between these factors in determining the overall

patterning, complexity, and qualities of the coordination and dynamic control of

cognitive activity. Abstract descriptions of cognitive activity are constructed in

terms of a four-component model specifying attributes of configurations,

procedural knowledge, record contents, and dynamic control. Finally, in order

to produce an output, the system “knows” something about the relationships

between these abstract models of cognitive activity and the attributes of user

behavior.

Basic Theories and the Artifacts of HCI

Figure 7.2. The applied science paradigm instantiated for the use of interacting cognitive subsystems as a theoretical basis for the development

Obviously, no single model of this type can capture everything that goes on

in a complex task sequence. Nor can a single model capture different stages of

user development or other individual differences within the user population. It is

therefore necessary to build a set of interrelated models representing different

phases of cognitive activity, different levels and forms of user expertise, and so

on. The basic modeling unit uses the four-component description to characterize

cognitive activity for a particular phase, such as establishing a goal, determining

the action sequence, and executing it. Each of these models approximates over

the very short-term dynamics of cognition. Transitions between phases

approximate over the short-term dynamics of tasks, whereas transitions between

levels of expertise approximate over different stages of learning. In Figure 7.2,

the envisaged application representation thus consists of a family of interrelated

models depicted graphically as a stack of cards.

Like the concept of programmable user models, the concept of approximate

descriptive modeling is in the course of development. A running demonstrator

system exists that effectively replicates the reasoning underlying the explanation

of a limited range of empirical phenomena in HCI research (see Barnard,

Wilson, & MacLean, 1987, 1988). What actually happens is that the expert

system elicits, in a context-sensitive manner, descriptions of the envisaged

interface, its users, and the tasks that interface is intended to support. It then

effectively “reasons about” cognitive activity, its properties, and attributes in that

applications setting for one or more phases of activity and one or more stages of

learning. Once the models have stabilized, it then outputs a characterization of

the probable properties of user behavior. In order to achieve this, the expert

system has to have three classes of rules: those that map from descriptions of

tasks, users, and systems to entities and properties in the model representation;

rules that operate on those properties; and rules that map from the model

representation to characterizations of behavior. Even in its somewhat primitive

current state, the demonstrator system has interesting generalizing properties.

For example, theoretical principles derived from research on rather antiquated

command languages support limited generalization to direct manipulation and

iconic interfaces.

As an applications representation, the expert system concept is very different

from programmable user models. Like PUMs, the actual tool embodies explicit

theory drawn from the science base. Likewise, the underlying architectural

concept enables a relatively broad range of issues to be addressed. Unlike

PUMs, it more directly addresses a fuller range of resources across perceptual,

cognitive, and effector concerns. It also applies a different trade-off in when and

by whom the modeling knowledge is specified. At the point of creation, the

expert system must contain a complete set of rules for mapping between the

world and the model. In this respect, the means of accomplishing and

expressing the characterizations of cognition and behavior must be fully and

comprehensively encoded. This does not mean that the expert system must

necessarily “know” each and every detail. Rather, within some defined scope,

the complete chain of assumptions from artifact to theory and from theory to

behavior must be made explicit at an appropriate level of approximation.

Basic Theories and the Artifacts of HCI 123

Equally, the input and output rules must obviously be grounded in the language

of interface description and user-system interaction. Although some of the

assumptions may be heuristic, and many of them may need crafting, both

theoretical and craft components are there. The how-to-do-it modeling

knowledge is laid out for inspection.

However, at the point of use, the expert system requires considerably less

precision than PUMs in the specification and operationalization of the knowledge

required to use the application being considered. The expert system can build a

family of models very quickly and without its user necessarily acquiring any

great level of expertise in the underlying cognitive theory. In this way, it is

possible for that user to explore models for alternative system designs over the

course of something like one afternoon. Because the system is modular, and the

models are specified in abstract terms, it is possible in principle to tailor the

systems input and output rules without modifying the core theoretical reasoning.

The development of the tool could then respond to requirements that might

emerge from empirical studies of the real needs of design teams or of particular

application domains.

In a more fully developed form, it might be possible to address the issue of

which type of tool might prove more effective in what types of applications

context. However, strictly speaking, they are not direct competitors, they are

alternative types of application representation that make different forms of tradeoff

about the characteristics of the complete chain of bridging from theory to

application. By contrast with the kinds of theory-based techniques relied on in

the first life cycle of HCI research, both PUMs and the expert-system concept

represent more elaborate bridging structures. Although underdeveloped, both

approaches are intended ultimately to deliver richer and more integrated

information about properties of human cognition into the design environment in

forms in which it can be digested and used. Both PUMs and the expert system

represent ways in which theoretical support might be usefully embodied in future

generations of tools for supporting design. In both cases the aim is to deliver

within the lifetime of the next cycle of research a qualitative understanding of

what might be going on in a user’s head rather than a purely quantitative estimate

of how long the average head is going to be busy (see also Lewis, this volume).

Summary

The general theme that has been pursued in this chapter is that the relationship

between the real world and theoretical representations of it is always mediated by

bridging representations that subserve specific purposes. In the first life cycle of

research on HCI, the bridging representations were not only simple, they were

only a single step away from those used in the parent disciplines for the

development of basic theory and its validation. If cognitive theory is to find any

kind of coherent and effective role in forthcoming life cycles of HCI research, it

must seriously reexamine the nature and function of these bridging

representations as well as the content of the science base itself.

124 Barnard

This chapter has considered bridging between specifically cognitive theory

and behavior in human-computer interaction. This form of bridging is but one

among many that need to be pursued. For example, there is a need to develop

bridging representations that will enable us to interrelate models of user cognition

with the formed models being developed to support design by software

engineers (e.g., Dix, Harrison, Runciman, & Thimbleby, 1987; Harrison,

Roast, & Wright, 1989; Thimbleby, 1985). Similarly there is a need to bridge

between cognitive models and aspects of the application and the situation of use

(e.g., Suchman, 1987). Truly interdisciplinary research formed a large part of

the promise, but little of the reality of early HCI research. Like the issue of

tackling nonideal user behavior, interdisciplinary bridging is now very much on

the agenda for the next phase of research (e.g., see Barnard & Harrison, 1989).

The ultimate impact of basic theory on design can only be indirect – through

an explicit application representation. Alternative forms of such representation

that go well beyond what has been achieved to date have to be invented,

developed, and evaluated. The views of Carroll and his colleagues form one

concrete proposal for enhancing our application representations. The design

rationale concept being developed by MacLean, Young, and Moran (1989)

constitutes another potential vehicle for expressing application representations.

Yet other proposals seek to capture qualitative aspects of human cognition while

retaining a strong theoretical character (Barnard et al., 1987; 1988; Young,

Green, & Simon, 1989).

On the view advocated here, the direct theory-based product of an applied

science paradigm operating in HCI is not an interface design. It is an application

representation capable of providing principled support for reasoning about

designs. There may indeed be very few examples of theoretically inspired

software products in the current commercial marketplace. However, the first life

cycle of HCI research has produced a far more mature view of what is entailed in

the development of bridging representations that might effectively support design

reasoning. In subsequent cycles, we may well be able to look forward to a

significant shift in the balance of added value within the interaction between

applied science and design. Although future progress will in all probability

remain less than rapid, theoretically grounded concepts may yet deliver rather

more in the way of principled support for design than has been achieved to date.

Acknowledgments

The participants ant the Kittle Inn workshop contributed greatly to my

understanding of the issues raised here. I am particularly indebted to Jack

Carroll, Wendy Kellogg, and John Long, who commented extensively on an

earlier draft. Much of the thinking also benefited substantially from my

involvement with the multidisciplinary AMODEUS project, ESPRIT Basic

Research Action 3066.

References

ACTS (1989). Connectionist Techniques for Speech (Esprit Basic Research

ACtion 3207), Technical Annex. Brussels: CEC.

Basic Theories and the Artifacts of HCI 125

AMODEUS (1989). Assimilating models of designers uses and systems (ESprit

Basic Research Action 3066), Technical Aneex. Brussels; CEC.

Anderson, J. R., & Skwarecki, E. 1986. The automated tutoring of

introductory computer programming. Communications of the ACM, 29,

842-849.

Barnard, P. J. (1985). Interacting cognitive subsystems: A psycholinguistic

approach to short term memory. In A. Ellis, (Ed.), Progress in the

psychology of language (Vol. 2, chapter 6, pp. 197-258. London:

Lawrenece Erlbaum Associates.

Barnard, P. J. (1987). Cognitive resources and the learning of human-computer

dialogs. In J.M. Carroll (Ed.), Interfacing thought: Cognitive aspects of

human-computer interaction (pp. 112-158). Cambridge MA: MIT Press.

Barnard, P. J., & Harrison, M. D. (1989). Integrating cognitive and system

models in human-computer interaction. In A. Sutcliffe & L. Macaulay,

(Ed.), People and computers V (pp. 87-103). Cambridge: Cambridge

University Press.

Barnard, P. J., Ellis, J., & MacLean, A. (1989). Relating ideal and non-ideal

verbalised knowledge to performance. In A. Sutcliffe & L. Macaulay

(Eds.), People and computers V (pp. 461-473). Cambridge: Cambridge

University Press.

Barnard, P. J., Grudin, J., & MacLean, A. (1989). Developing a science base

for the naming of computer commands. In J. B. Long & A. Whitefield

(Eds.), Cognitive ergonomics and human-computer interaction (pp. 95-

133). Cambridge: Cambridge University Press.

Barnard, P. J., Hammond, N., MacLean, A., & Morton, J. (1982). Learning

and remembering interactive commands in a text-editing task. Behaviour

and Information Technology, 1, 347-358.

Barnard, P. J., MacLean, A., & Hammond, N. V. (1984). User representations

of ordered sequences of command operations. In B. Shackel (Ed.),

Proceedings of Interact ’84: First IFIP Conference on Human-Computer

Interaction, (Vol. 1, pp. 434-438). London: IEE.

Barnard, P. J., & Teasdale, J. (1991). Interacting cognitive subsystems: A

systematic approach to cognitive-affective interaction and change.

Cognition and Emotion, 5, 1-39.

Barnard, P. J., Wilson, M., & MacLean, A. (1986). The elicitation of system

knowledge by picture probles. In M. Mantei & P. Orbeton (Eds.),

Proceedings of CHI ’86: Human Factors in Computing Systems (pp.

235-240). New York: ACM.

Barnard, P. J., Wilson, M., & MacLean, A. (1987). Approximate modelling of

cognitive activity: Towards an expert system design aid. In J. M. Carroll

& P. P. Tanner (Eds.), Proceedings of CHI + GI ’87: Human Factors in

Computing Systems and Graphics Interface (pp. 21-26). New York:

ACM.

Barnard, P. J., Wilson, M., & MacLean, A. (1988). Approximate modelling of

cognitive activity with an Expert system: A theory based strategy for

developing an interactive design tool. The Computer Journal, 31, 445-

456.

Bartlett, F. C. (1932). Remembering: A study in experimental and social

psychology. Cambridge: Cambridge University Press.

Broadbent, D. E. (1958). Perception and communication. London: Pergamon

Press.

Card, S. K., & Henderson, D. A. (1987). A multiple virtual-workspace

interface to support user task-switching. In J. M. Carroll & P. P. Tanner

(Eds.), Proceedings of CHI + GI ’87: Human Factors in Computing

Systems and Graphics Interface (pp. 53-59). New York: ACM.

Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of humancomputer

interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.

Carroll, J. M. (1985). What’s in a name? New York: Freeman.

Carroll, J. M. (1989a). Taking artifacts seriously. In S. Maas & H. Oberquelle

(Eds.), Software-Ergonomie ’89 (pp. 36-50). Stuttgart: Teubner.

Carroll, J. M. (1989b). Evaluation, description and invention: Paradigms for

human-computer interaction. In M. C. Yovits (Ed.), Advances in

computers (Vol. 29, pp. 44-77). London: Academic Press.

Carroll, J. M. (1990). Infinite detail and emulation in an ontologically

minimized HCI. In J. Chew & J. Whiteside (Eds.), Proceedings of CHI

’90: Human Factors in Computing Systems (pp. 321-327). New York:

ACM.

Carroll, J. M., & Campbell, R. L. (1986). Softening up hard science: Reply to

Newell and Card. Human-Computer Interaction, 2, 227-249.

Carroll, J. M., & Campbell, R. L. (1989). Artifacts as psychological theories:

The case of human-computer interaction. Behaviour and Information

Technology, 8, 247-256.

Carroll, J. M., & Kellogg, W. A. (1989). Artifact as theory-nexus:

Hermaneutics meets theory-based design. In K. Bice & C. H. Lewis

(Eds.), Proceedings of CHI ’89: Human Factors in Computing Systems

(pp. 7-14). New York: ACM.

Chomsky, N. (1957). Syntactic structures. The Hague: Mouton.

Dix, A. J., Harrison, M. D., Runciman, C., & Thimbleby, H. W. (1987).

Interaction models and the principled design or interactive systems. In

H. Nicholls & D. S. Simpson (Eds.), European software engineering

conference, (pp. 127-135). Berlin: Springer Lecture Notes.

Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data.

Psychological Review, 87, 215-251.

Grudin, J. T. (1990). The computer reaches out: The historical continuity of

interface design. In J. Chew & J. Whiteside (Eds.), Proceedings of CHI

’90: Human Factors in Computing Systems (pp. 261-268). New York:

ACM.

Grudin, J. T., & Barnard, P. J. (1984). The cognitive demands of learning

command names for text editing. Human Factors, 26, 407-422.

Hammond, N., & Allinson, L. 1988. Travels around a learning support

environment: rambling, orienteering or touring? In E. Soloway, D.

Basic Theories and the Artifacts of HCI 127

Frye, & S. B. Sheppard (Eds.), Proceedings of CHI ’88: Human

Factors in Computing Systems (pp. 269-273). New York: ACM.

Hammond, N. V., Long, J., Clark, I. A., Barnard, P. J., & Morton, J. (1980).

Documenting human-computer mismatch in interactive systems. In

Proceedings of the Ninth International Symposium on Human Factors in

Telecommunications (pp. 17-24). Red Bank, NJ.

Hanson, W. (1971). User engineering principles for interactive systems.

AFIPS Conference Proceedings , 39, 523-532.

Harrison, M. D., Roast, C. R., & Wright, P. C. (1989). Complementary

methods for the iterative design of interactive systems. In G. Salvendy

& M. J. Smith (Eds.), Proceedings of HCI International ’89 (pp. 651-

658). Boston: Elsevier Scientific.

Kieras, D. E., & Polson, P. G. (1985). An approach to formal analysis of user

complexity. International Journal of Man- Machine Studies, 22, 365-

394.

Laird, J.E., Newell, A., & Rosenbloom, P. S. (1987). SOAR: An architecture

for general intelligence. Artificial Intelligence, 33, 1-64.

Landauer, T. K. (1987). Relations between cognitive psychology and computer

systems design. In J. M. Carroll (Ed.), Interfacing thought: Cognitive

aspects of human-computer interaction (pp. 1-25). Cambridge, MA:

MIT Press.

Lewis, C. H. (1988). Why and how to learn why: Analysis-based

generalization of procedures. Cognitive Science, 12, 211-256.

Long, J. B. (1987). Cognitive ergonomics and human-computer interaction. In

P. Warr (Ed.), Psychology at Work (erd ed.). Harmondsworth,

Middlesex: Penguin.

Long, J. B. (1989). Cognitive ergonomics and human-computer interaction: An

introduction. In J. B. Long & A. Whitefield (Eds.), Cognitive

ergonomics and human-computer interaction (pp. 4-34). Cambridge:

Cambridge University Press.

Long, J. B., & Dowell, J. (1989). Conceptions of the discipline of HCI: Craft,

applied science and engineering. In A. Sutcliffe & L. Macaulay (Eds.),

People and computers V (pp. 9-32). Cambridge: Cambridge University

Press.

MacLean, A., Barnard, P., & Wilson, M. (1985). Evaluating the human

interface of a data entry system: User choice and performance measures

yield different trade-off functions. In P. Johnson & S. Cook (Eds.),

People and computers: Designing the interface (pp. 172-185).

Cambridge: Cambridge University Press.

MacLean, A., Young, R. M., & Moran, T. P. (1989). Design rationale: The

argument behind the artefact. In K. Bice & C.H. Lewis (Eds.),

Proceedings of CHI ’89: Human Factors in Computing Systems (pp.

247-252). New York: ACM.

Mack, R., Lewis, C., & Carroll, J.M. (1983). Learning to use word

processors: Problems and prospects. ACM Transactions on Office

information Systems, 1, 254-271.

Morton, J., Marcus, S., & Frankish, C. (1976). Perceptual centres: P-centres.

Psychological Review, 83, 405-408.

Newell, A. (1989). Unified Theories of Cognition: The 1987 William James

Lectures . Cambridge, MA: Harvard University Press.

Newell, A., & Card, S. K. (1985). The prospects for psychological science in

human computer interaction. Human-Comuter Interaction, 1, 209.242.

Newell, A., & Simon, H. A. (1972). Human Problem Solving. Englewood

Cliffs, NJ: Prentice-Hall.

Norman, D. A. (1983). Design principles for human-computer interaction. In

Proceedings of CHI ’83: Human Factors in Computing Systems (pp. 1-

10). New York: ACM.

Norman, D. A. (1986). Cognitive engineering. In D. A. Norman & S. W.

Draper (Eds.), User centered system design: New perspectives on

human-computer interaction (pp. 31-61). Hillsdale, NJ: Lawrence

Erlbaum Associates.

Olson, J. R., & Olson, G. M. (1990). The growth of cognitive modelling since

GOMS. Human Computer Interaction 5, 221-265.

Patterson, R. D. (1983). Guidelines for auditory warnings on civil aircraft: A

summary and prototype. In G. Rossi (Ed.), Noise as a Public Health

Problem (Vol. 2, pp. 1125-1133). Milan: Centro Richerche e Studi

Amplifon.

Patterson, R. D., Cosgrove, P., Milroy, R., & Lower, M.C. (1989). Auditory

warnings for the British Rail inductive loop warning system. In

Proceedings of the Institute of Acoustics, Spring Conference (Vol. 11,

pp. 5-51-58). Edinburgh: Institute of Acoustics.

Patterson, R. D., Edworthy, J., Shailer, M.J., Lower, M.C., & Wheeler, P. D.

(1986). Alarm sounds for medical equipment in intensive care areas and

operting theatres. Institute of Sound and Vibration (Research Report AC

598).

Payne, S., & Green, T. (1986). Task action grammars: A model of the mental

representation of task languages. HumanComputer Interaction, 2, 93-

133.

Polson, P. (1987). A quantitative theory of human-computer interaction. In J .

M. Carroll (Ed.), Interfacing thought: Cognitive aspects of humancomputer

interaction (pp. 184-235). Cambridge, MA: MIT Press.

Reisner, P. (1982). Further developments towards using formal grammar as a

design tool. In Proceedings of Human Factors in Computer Systems

Gaithersburg (pp. 304-308). New York: ACM.

Scapin, D. L. (1981). Computer commands in restricted natural language: Some

aspects of memory and experience. Human Factors, 23, 365-375.

Simon, T. (1988). Analysing the scope of cognitive models in human-computer

interaction. In D. M. Jones & R. Winder (Eds.), People and computers

IV (pp. 79-93). Cambridge: Cambridge University Press.

Suchman, L. (1987). Plans and situated actions: The problem of humanmachine

communication. Cambridge: Cambridge University Press.

Basic Theories and the Artifacts of HCI 129

Thimbleby, H. W. (1985). Generative user-engineering principles for user

interface design. In B. Shackel (Ed.), Human computer interaction:

Interact ’84 (pp. 661-665). Amsterdam: North-Holland.

Whiteside, J., & Wixon, D. (1987). Improving human-computer interaction: A

quest for cognitive science. In J. M. Carroll (Ed.), Interfacing thought:

Cognitive aspects of human-computer interaction (pp. 353-365).

Cambridge, MA: MIT Press.

Wilson, M., Barnard, P. J., Green, T. R. G., & MacLean, A. (1988).

Knowedge-based task analysis for human-computer systems. In G. Van

der Veer, J-M Hoc, T. R. G. Green, & D. Murray (Eds.), Working with

computers (pp. 47-87). London: Academic Press.

Young, R. M., & Barnard, P. J. (1987). The use of scenarios in humancomputer

interaction research: Turbocharging the tortoise of cumulative

science. In J. M. Carroll & P. P. Tanner (Eds.), Proceedings of CHI +

GI ’87: Human Factors in Computing Systems and Graphics Interface

(Toronto, April 5-9) (pp. 291-296). New York: ACM.

Young, R. M., Barnard, P.J., Simon, A., & Whittington, J. (1989). How

would your favourite user model cope with these scenarios? SIGCHI

Bulletin, 20( 4), 51-55.

Young, R. M., Green, T. R. G., & Simon, T. (1989). Programmable user

models for predictive ev aluation of interface designs. In K. Bice and C.

H. Lewis (Eds.), Proceedings of CHI ’89: Human Factors in Computing

Systems (pp. 15-19). New York: ACM.

Young, R.M., & MacLean, A. (1988). Choosing between methods: Analysing

the user’s decision space in terms of schemas and linear models. In E.

Soloway, D. Frye, & S. B. Sheppard (Eds.), Proceedings of CHI ’88:

Human Factors in Computing Systems (pp. 139-143). New York:

ACM.

Introduction

 

 

The Hexagon-Spindle Model for Ergonomics

 

 

Rachel Benedyk¹ and Andrée Woodcock²

 

 

¹ University College London Interaction Centre, UK 

r.benedyk@ucl.ac.uk

² Design and Ergonomics Applied Research Group, Coventry University, UK 

adx974@coventry.ac.uk

 

 

 

 

 

The concept of ‘work’ in ergonomics may now be applied to the satisfactory completion of any task. Using a systems view of the process, the task consists of a number of transformative interactions, performed simultaneously or successively, to a given level of effectiveness. Such interactions take place at the interface, between the user (or operator) and key components of the system; these encompass not only traditional hardware, software and workstations, but social, managerial, infrastructure and peripheral factors. This paper proposes a new model for Ergonomics, known as the Hexagon – Spindle Model, which represents the multiple factors influencing the effectiveness of interactions at the interface. The model is holistic, multi-dimensional, task-related and transferable across a range of settings. It extends to characterise a time base for serial and simultaneous tasks and highlights areas where operator/system conflicts may arise. The paper illustrates analysis tools for the application of the model in evaluation and design.

 

 

Introduction

 

Traditional work systems approaches in ergonomics (as described originally by Singleton, 1967 and 1974, and more recently by Dowell and Long, 1998) may be applied to the satisfactory completion of any task, not just those conducted as part of ‘formal work’. Using a systems view of the process, the task consists of a number of transformative interactions, performed simultaneously or successively, to a given level of effectiveness. The tasks are accomplished in a ‘workplace’ – the immediate environment in which the user and the work system interact. Each task is defined as one or more processes undertaken by a user or operator while in a dynamic exchange of information (an ‘interaction’) with another or with the key design features of the interface. Here the wider definition of an interface is used, referring to the ‘point of communication between two or more processes, persons, or other (physical) entities’ (Santoro, 2001). This would include concrete or virtual objects such as tools, equipment, networks, furniture, and formal and informal organisational structures.

 

Task interactions occur in a particular setting, whether in formal spaces such as offices or factories, or informal ones such as sports venues, coffee shops, cyber cafes, or the home. They are task dependent, user specific, and subject to influence from each component of the system; not only from traditional hardware, software and workstations, but also social, managerial, infrastructure and peripheral factors. The overall accomplishment of the task is achieved by the successful and effective completion of task interactions, each of which may be influenced by wider factors beyond the immediate control of the user. The challenge for ergonomists is to find the optimal conditions of all these factors to maximise this effectiveness.

 

Previous ergonomic models

 

Traditional models of ergonomics have either represented the interactive nature of the interface or the multiple sources of influential factors. The earliest model, commonly known as Concentric Rings (Shackel, 1969; Galer, 1987), represented an operator-centred focus, and the influence on the operator of factors at levels moving out from there; however, there is no further elaboration in the model of these factors in the system. Girling and Birnbaum (1988) took this further with the Elaborated Concentric Rings model, in which the many influential factors in the outer rings are seen to originate from management, situational or personal sources. These divisions facilitate the diagnosis of multiple sources of design problems and the directions needed for rectification. The enhanced model has since been applied in design problems, ensuring all relevant contributing factors are included (eg Dubois and Benedyk, 1991) and in evaluation problems, providing a holistic framework for scoping the evaluation (eg Naki and Benedyk, 2002).

 

In parallel, the development of the systems approach to ergonomics has emphasised the purpose (ie the task) and the functions to support that purpose; thus Singleton’s work (1967, 1974) began the ‘closed loop’ system representation of the task interaction at the interface. This exchange of data or information between operator and interface holds true whether operator/machine or practitioner/patient or user/tool. Later authors have developed detailed structures and behaviours to further understand the interface and the nature of this interaction (eg Dowell and Long, 1998).

 

Combining the two approaches to form a holistic representation would seem to have some advantage: as Wilson (2005, page 10) emphasises: “Human-machine interaction does not take place in a vacuum;…ergonomic techniques are needed to predict, investigate or develop each of the possible interactions, between person-task, person-process, person-environment, person-job, person-person, person-organisation and person-outside world.” In this regard, Singleton (1967) makes passing reference to groups of external factors that could ‘interrupt the closed loop’, and Grey (1987) expands on sources within the rings, placing the task as the innermost ring. Leamon’s (1980) model comes closest to integration, in that it places the Closed Loop inside the Concentric Rings, and attempts to indicate criteria for system effectiveness; however, it elaborates no further on the rings themselves.

 

In addition to the need for a holistic representation, there is a pressing need for an ergonomic model that can move beyond a systematic construct, to act as a useful tool for practitioners, able to demonstrate the utility of the ergonomic approach. Moreover, no models as yet embrace the complex ergonomic challenges arising from multiple users, from simultaneous or successive tasks, from peripatetic users or from changes in user characteristics over time. The Hexagon-Spindle model presented begins to address all these.

 

 

The Hexagon Model

 

The Hexagon Model has elaborated the Enhanced Concentric Rings model by firstly providing a clearer division of the sectors and the levels; secondly by making explicit the six directions for task interactions and the factors of influence; and thirdly, by differentiating between the constant characteristics of the user (common to any task), which are in the central hexagon of the model, and the individual variables that arise only for that task, which feature in the personal sector. The model was developed in order to elucidate the factors which effect task interactions in learning environments. Using this domain provided a series of specific tasks, interactions and multi-level factors which could be used to validate the model. These developments have been reported by Benedyk, Woodcock and Harder (2006, 2009), and Woodcock, Woolner and Benedyk (2009). In this paper, we wish to test the viability of the generic model by returning it to more traditional domains, using the example of a re-design of a cockpit simulator instructor workstation.

 

The generic form of the Hexagon Model is shown in Figure 1. This shows that for a specific task, the user performs transformative interactions (in line with his/her requirements) with the interface at the level  of  the  workstation. The workstation consists  of  all  the immediate  and relevant tools, materials,

 

 

Legend for Figure 1

 

Shaded areas represent levels

Dotted lines represent porous boundaries

Hashed area represents task interactions at the interface between

user and the workstation level

Sectors provide  groupings for sources of factors that potentially

influence the task interactions

 

Figure 1  A Generic Hexagon Model

 

technology, furniture, co-workers and other people in the vicinity, facilities and resources. This wider definition of the interface is compound and complex; it is not just limited to the human-machine interface, and its interactions are subject to influence (both positive and negative) from multiple factors at the workplace, work setting and external levels. The subdivisions to the outer rings intend to ensure a sharp focus of analysis, greater rigour and consistency across applications. The model further encourages breadth of consideration, so that all possible influences are addressed. Contextual factors need to be set each time the model is applied, specific to the application under consideration. For a more detailed description of the parts of the model, the reader is referred to Benedyk et al (2009).

 

To perform a Hexagon Analysis, which is the first stage of the tool for practice, the model is transformed into a matrix (Figure 2) and the key factors of potential ergonomic influence noted for each cell ie, any factor likely to impact on the effectiveness of the task interaction. Every cell in the Analysis Table must be given consideration; this is one of the benefits of the technique, as it encourages this holistic approach. Note that there is no attempt at quantification or comparison here; the inclusion of a factor simply gives it potential significance. If the analysis has an evaluation purpose, the factors noted may be rooted in actual observations of the task in context. However, more often the factors must be extracted from a focused creative discussion between ergonomist and user or designer.

 

As each analysis is task dependent, the choice and relative weight of factors in the different cells will vary. The acceptable level of effectiveness also needs to be set in advance, using relevant ergonomic criteria and priorities appropriate to that work system. The ergonomic practitioner’s judgment is needed here.

 

From the range of influencing factors identified, those most critical to task effectiveness are extracted, and an Application Format table (Figure 3) constructed, in which the potential ergonomic problems due to each factor are identified, and solutions suggested. The emphasis of the analysis is on a holistic approach to design and evaluation which recognizes where there are mismatches between ergonomic requirements for effective interaction, and features of the actual system. These issues can then be given professional consideration, with the application of ergonomic techniques as necessary, and recommendations for improvement put forward.

 

Figure 2 presents an exemplar of the Hexagon Analysis Table; the data here were used for the ergonomic evaluation and re-design of the physical ergonomic aspects of an instructor workstation in a cockpit motion simulator. Figure 3 presents an Application Format table, with an illustrative analysis for the same example.

 

 

The Spindle Model  

 

In the second stage of the analysis, multiple Hexagons are combined and compared, using a Spindle Model (Figure 4). The spindle allows the hexagon plates to be stacked or spaced, and the influencing factors on them compared, in order to integrate user requirements over different tasks over time.

 

Tasks are rarely undertaken singly. The user might move from task to task consecutively, or perform more than one task concurrently. These tasks have the same user in common, and often the same external environment, perhaps even the same organisational management, but they may require different equipment, different settings and entail different user requirements, at different times. Because they are not entirely independent of each other, and relate along a time dimension, it can be useful to see their hexagons arranged as ‘plates’ on a ‘time spindle’ as in Figure 4. The commonality of the user links them at the centre.

 

For serial tasks, the hexagons will be arranged spaced out along the time dimension as in Figure 4, and for concurrent tasks they will be ‘stacked’ at a single time point. Analysis involves a comparison of the key priority factors (the X’s in Figure 4) on each of the hexagons on the spindle, in order to identify conflict or congruence.  The spindle thus can hold user task-independent needs constant while varying user requirements by task-dependent capabilities.

For time t1	Organisational Sector	Contextual Sector	Personal Sector
	Management Factors	Infrastructure Factors	Task Factors	Tool factors	Individual Factors	Social Factors
External Level	Businessfunding for training; aviation training policy and standards	Developments in integration of instructor workstations in simulators	Best practice in simulator training facilities and in instructor facilities	Trends and standards in use of integrated cockpit instructor workstations	Instructor recruitment and training in relation to use of simulator workstations	Cultural (national and business) expectations of simulator facilities
TaskSetting Level (Pilot training centre)	Allocation of budgetfor simulator facility; costs and benefits ofupgrading	Provision of facilities for simulators; technical support	Syllabus and content of instruction sessions, in relation to use of equipment and interaction of instructor with pilot	Age and antiquity of simulator facilities; flexibility of use of space	Attitudes of instructors to use of different types of simulator workstations	Attitudes and preferences of pilots interacting with instructors in simulators
Task Workplace Level (Cockpit simulator)	Procurement and planning choices for design and facilities in training simulators	Overall space, accessibility,temperature, acoustics, lighting, vibration	Style of delivery of instruction for these tasks; equipment and sightline requirements for these tasks	Suitability of workplace configuration to accommodate instructor workstation; effect of environmental factors on workstation usability	Crowding factor; relation of instructor workstation to pilot workstation	Potential interference between instructor’s workstation and fidelity of cockpit simulation
TaskWorkstation Level (Instructor’s workstation)	Procurement choice of individual workstation eg adjustability, orientation and size – suitability for this cockpit	Installation, support and power supplies affecting individual workstation; task lighting	Instructor workstation and interface design functionality in relation to required task performance; frequency and extent of use	Instructor workstation and interface design features and their usability; fit to this cockpit; sightlines between instructor and pilot’ instruments	Posture, flexibility of posture, and static loading factors imposed by current configuration; effect of vibration on balance, vision and well-being during  these tasks	Sightlines and engagement (both required and unwanted) between instructor and pilot
Interaction Level (Task interface)
Interface User Level (Instructor)	Stable personal characteristics, eg anthropometry, skills, abilities, preferences, expectations and knowledge of particular instructor, in relation to these tasks in this cockpit simulator; individual susceptibility to effects of vibration

 

 

 

Figure 2 The Hexagon Analysis Table: an analysis table for a cockpit flight instructor’s tasks – this figure shows a partial analysis, giving examples filled into the table for physical ergonomics factors relevant to the ergonomic design of a flight instructor workstation

 

Note the hashed boundaries indicating porosity of influence.

Note also the stable user characteristics (those that are not task dependent) underpinning all sector columns.

Note the interaction level occurs between the user and the workstation, in a direct sense, but is under influence from factors at any level and from any sector.

 

 

 

 

Levels:	External	Work-Setting	Workplace	Workstation	Operator
Examples of influencing factors at time t1	Clients from different nations hire this training facility and require different configurations of cockpit to suit.	Motion Simulator instruction must replicate Static Simulator Instruction	Simulator must be as similar to cockpit as possible, so space for instructor is very limitedMovement of flight causes vibration	Instructor must monitor both instructor displays and pilot displaysInstruction takes place in darkness and during flight motion	Instructor workstation is used by many different instructors on different occasions
Examples of ergonomics issues at time t1	Different configurations impose variations in requirements which if not met could impact optimal use of the facility by some users	Do the ergonomic requirements for the instructor in the different facilities cause a conflict?	Are postures too constrained? Could emergency egress be compromised?Can the operator still make fine eye and finger movements without error?	Can the instructor keep sightlines and focus on both displays despite obstructions and distance?Do the darkness, flight motion and isolation of this workplace cause ill effects for these operators?	Given the constrained space in the work area, can the full range of sizes of people who may be instructors be accommodated?
Examples of ergonomics approaches to solutions	Ensure use of instructor workstation is compatible with all possible client-preferred cockpit configurations without loss of task quality	Ensure comparison of Motion to Static simulator is not compromised by ergonomic interventions	Assess potential risk of postures and problems with egress, and reduce risk by operator adjustment of layout.Design displays and controls that do not require fine movements. Provide limb supports.Ensure rest breaks are taken outside the work environment.	Carry out user trials to test that proposed design and layout meet visual task requirements, and build in adjustability where possible.Ensure potential users test their tolerance to the environmental conditions. Ensure operators are adapted to reduce motion sickness.	Ensure anthropometry applied to this design is valid for this user population, and build in adjustability where possible.
Figure 3 – The Application Format for the Hexagon-Spindle model: an illustrative example, using the exemplar from Figure 2

 

 

Legend for Figure 4

 

t= point in time

x= critical factors on each hexagon

hashed area = common task interactions

 

Figure 4 A Spindle Model

 

Provision of generic facilities can sometimes compromise the specific facilities the user needs to perform a variety of tasks. Thus, moving from one task to another or attempting consecutive tasks in the same workstation may entail different user requirements  – with the result that the workstation or the materials may be ergonomically optimal for this user for one task but not for others. Similarly, moving a peripatetic workstation (such as a laptop) from one setting or context to another can reveal conflicting requirements for optimal use, even for the same user.

 

The same analysis technique, stacking the hexagons, can compare different users doing the same task in the same workstation, and accommodate individual differences; highlighting the need for adjustability. For example, in the analysis in Figure 3, if there were two Cockpit Instructors using the same workstation at different times, their individual needs would both have to be accommodated by the workstation design. This perspective can also allow us to return to the wider focus at the workplace level. A single workplace supports not only the tasks of the system user, but also the tasks of the supervisor, the cleaning and maintenance tasks of the janitor, etc; one environment, several users. They have contrasting tasks, and thus demand contrasting user requirements from that environment. The Hexagon-Spindle model would allow us to identify points of conflict that would lead to ergonomic concern (such as increased risk) in such shared environments.

 

Further, the model can examine the same user over time by comparing factors in the central user hexagon, and point to the changes in the interface necessary to accommodate time-dependant factors such as fatigue, aging, or pregnancy.

 

Any of the factors in the model can be held constant, and any others varied, and the effect on the task interaction quality or the costs to the user can be examined.

 

 

Conclusion

 

The Hexagon-Spindle model provides a generic framework that enables a practitioner to take a multi-factorial, holistic approach to the design and evaluation of different forms of task material, aids, devices and environments whilst taking into account the requirements of different types of users and tasks. Its approach will lead to a more integrated understanding of each scenario, and an increased confidence in the likely effectiveness of ergonomic interventions and design changes. This is particularly critical in complex settings with multiple tasks, wide ranges of users and budget-limited facilities, which can lack a holistic ergonomic focus.

 

The flexibility of the spindle provides an opportunity to think outside the rigid box that can sometimes limit other models. A working day for many people now comprises sets of activities, which may or may not occur in the same place, with the same or different colleagues or using the same or different equipment. To create a truly effective working environment requires not just one, but a series of effective environments to be constructed.

 

The Hexagon-Spindle model explicitly focuses on the effectiveness of the interactions for a single task, thus clarifying conflicts between different task needs. It evaluates the whole complex, multifactorial space as defined by the extended concentric rings and the influence of all factors, whether supportive or limiting, on the task interactions. The Spindle allows for the inclusion of an extra dimension in which to explore and resolve conflicts and priorities over time, over space, or between tasks or users. Not only may the same environment have to cater for the needs of varying users, but also for the other work undertaken in the same environment by other stakeholders. Most importantly, the model extends to provide a usable ergonomics analysis and demonstration tool.

 

 

References

 

Benedyk, R., Woodcock, A. and Harder, A., 2006, Towards a new model of Educational Ergonomics. In: Pro. 2^nd Conf on Access and Integration in Schools, (Coventry University, UK)

Benedyk, R., Woodcock, A. and Harder, A., 2009, The Hexagon-Spindle Model for Educational Ergonomics, accepted for publication in Work Journal

Dowell, J and Long, J., 1998, Conception of the cognitive engineering design problem, Ergonomics, 41.2, 126-139.

Dubois, S. and Benedyk, R., 1991, An ergonomic review of the usability of the postal delivery bag. In: Designing for Everyone, I. Queinnec and F. Daniellou, Eds., (Taylor and Francis, London)

Galer, I., Ed., 1987, Applied Ergonomics Handbook, (Butterworths Scientific, Guildford)

Girling, B., and Birnbaum, R., 1988, An ergonomic approach to training for prevention of musculoskeletal stress at work, Physiotherapy 74.9

Grey, S., Norris, B. and Wilson, J., 1987, Ergonomics in the electronic retail environment (ICL UK Ltd, Slough, UK)

Leamon, T., 1980, The organisation of industrial ergonomics – a human machine model, Applied Ergonomics, 11, 223-226

Naki, M., and Benedyk, R., 2002, Integrating new technology into nursing workstations: can ergonomics reduce risks? In: Contemporary Ergonomics 2002, P. McCabe, Ed., (Taylor and Francis, London), 57-61.

Santoro, D., Snyder, R., Snyder E., 2001, FDA-Speak: A Glossary and Agency Guide, Informa Health Care, ISBN 1574911295, 9781574911299

Singleton, W., 1967, The systems prototype and his design problems, Ergonomics 10.2, 120-124

Singleton, W., 1974, Man-Machine Systems, Penguin.

Shackel, B., 1969, Workstation analysis – turning cartons by hand, Applied Ergonomics 1.1, 45-51

Woodcock, A., Woolner, A. and Benedyk, R., 2009, Applying the Hexagon-Spindle Model for Educational Ergonomics to the design of school environments for children with autistic spectrum disorders, accepted for publication in Work Journal

Wilson, J., 2005, Methods in the understanding of human factors, Chapter 1 in Wilson, J. and Corlett, N., (eds) Evaluation of Human Work, 3^rd Edition Taylor and Francis

 

Opportunities for Odor:

Experiences with Smell and Implications for Technology

Marianna Obrist1,2, Alexandre N. Tuch3,4, Kasper Hornbæk4

m.obrist@sussex.ac.uk | a.tuch@unibas.ch | kash@diku.dk

1Culture Lab, School of Computing Science

Newcastle University, UK

2School of Engineering and Informatics

University of Sussex, UK

3Department of Psychology,

University of Basel, CH

4Department of Computer Science,

University of Copenhagen, DK

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for profit or commercial advantage and that copies bear this notice and the full

citation on the first page. Copyrights for components of this work owned by others

than ACM must be honored. Abstracting with credit is permitted. To copy otherwise,

or republish, to post on servers or to redistribute to lists, requires prior specific

permission and/or a fee. Request permissions from Permissions@acm.org.

CHI 2014, April 26 – May 01 2014, Toronto, ON, Canada

Copyright 2014 ACM 978-1-4503-2473-1/14/04…$15.00.

http://dx.doi.org/10.1145/2556288.2557008
ABSTRACT

Technologies for capturing and generating smell are

emerging, and our ability to engineer such technologies and

use them in HCI is rapidly developing. Our understanding

of how these technologies match the experiences with smell

that people have or want to have is surprisingly limited. We

therefore investigated the experience of smell and the

emotions that accompany it. We collected stories from 439

participants who described personally memorable smell

experiences in an online questionnaire. Based on the stories

we developed 10 categories of smell experience. We

explored the implications of the categories for smellenhanced

technology design by (a) probing participants to

envision technologies that match their smell story and (b)

having HCI researchers brainstorm technologies using the

categories as design stimuli. We discuss how our findings

can benefit research on personal memories, momentary and

first time experiences, and wellbeing.

Author Keywords

Smell; smell experiences; odor; olfaction; user experience;

smell-enhanced technology; narratives; smell stories;

crowdsourcing; design brainstorming; designing for smell.

ACM Classification Keywords

H.5.2 Information interfaces and presentation (e.g., HCI):

Miscellaneous.

General Terms

Experimentation, Human Factors, Design.

INTRODUCTION

Smell plays an important role for memories and emotions.

Compared to other modalities, memories evoked by smell

give stronger feelings of being brought back in time, are

more emotionally loaded, are experienced more vividly,

feel more pleasant, and are autobiographically older

(ranging back to childhood) [15,33]. Smell is incredibly

powerful in connecting humans to past events and

experiences.

Matsukura et al. [22] recently proposed the Smelling

Screen, an olfactory display system that can distribute

smells. Earlier work in HCI has proposed other systems that

capture and generate smells. For example, Brewster et al.

[5] developed a smell-based photo-tagging tool, and Bodnar

et al. [4] showed smell to be a less disruptive notification

mechanism than visual and auditory modalities.

Thus, smell technologies are already emerging.

Our understanding of how these technologies match the

experiences with smell that people have or want to have is

surprisingly limited.

First, while technologies such as those

mentioned above are often evaluated, the results mainly

concern the perception of smell. The evaluations say little

about the general potential of smell technologies for

humans or their ability to generate particular experiences.

Second, whereas earlier work states that the subjective

experience of smell stimulation is crucial for the success of

a system (e.g., [5]), we are unaware of work in HCI that

studies the subjective experience of smell (though see [17]).

Third, several hundred receptors exist for smell and we

cannot rely on any primary smells to stimulate a particular

experience, as might be imagined for other human senses

[2].

Taken together, these points suggest that we can only

link smell tenuously to particular experiences or emotions.

This limits our ability to design for a spectrum of

experiences.

 

The present paper focuses instead on experiences and

emotions related to smell and links them to potential

technologies. Inspired by work on user experience [14,34],

we concentrate on personal memorable smell experiences

and their links to emotion. From the focus on experience we

developed design guidance for smell-enhanced

technologies. The goal is to contribute knowledge on

subjective smell experiences and their potential for design.

 

We collected 439 smell stories, that is, descriptions of

personal memorable experiences involving smell.We

distributed a questionnaire through crowdsourcing, ensuring

a large-scale coverage and variety of smell stories. We

analyzed the stories and identified 10 main categories and

36 sub-categories. Each category was described with

respect to its experiential and emotional characteristics and

specific smell qualities. Besides smell stories associated

with the past (e.g., memory of loved people, places, life

events) we identify stories where smell played an important

role in stimulating action, creating expectations, and

supporting change (e.g., of behavior, attitude, mood). Smell

can sometimes also be invasive and overwhelming, and can

affect people’s interaction and communication. Within the

categories, we identify common smell qualities and

emotions, which support the exploration of opportunities

for design. In particular, we discuss the implications for

technology based on feedback from participants and on a

brainstorming session with HCI researchers working on

smell technologies.

The main contributions of this paper are

(1) an experiencefocused understanding of smell experiences grounded in a

large sample of personal smell stories, which allowed us

(2) to establish a systematic categorization and description scheme for smell experiences, leading to

(3) the identification of technology implications by participants,and

(4) the exploration of design potentialities by HCI

researchers.

THE HUMAN SENSE OF SMELL

The sense of smell is the most complex and challenging

human sense.

Hundreds of receptors for smell exist and the

mixing of the sensations, in particular with our sense of

taste, is immense [2]. The sense of smell is further

influenced by other senses such as vision, hearing, and

touch; plays a significant role for memory and emotion; and

shows strong subjective preferences. Willander and Larsson

[33] showed that autobiographical memories triggered by

smell were older (mostly from the first decade of life) than

memories associated with verbal and visual cues (mostly

from early adulthood). Moreover, smell-evoked memories

are associated with stronger feelings of being brought back

in time, are more emotionally loaded, and are experienced

more vividly than memories elicited through other

modalities [15,33]. No other sensory system makes the

direct and intense contact with the neural substrates of

emotion and memory, which may explain why smellevoked

memories are usually emotionally potent [15].

The emotion-eliciting effect of smell is not restricted to the

context of autobiographical memories. Smell is particularly

useful in inducing mood changes because they are almost

always experienced clearly as either pleasant or unpleasant

[8]. For instance, Alaoui-Ismaïli et al. [1] used ‘vanilla’ and

‘menthol’ smells to trigger positive emotions in their

participants (mainly happiness and surprise) and ‘methyl

methacrylate’ and ‘propionic acid’ to trigger negative

emotions (mainly disgust and anger). Interestingly, Herz

and Engen [15] pointed out that almost all responses to

smell are based on associative learning principles. They

argued that only smells learned to be positive or negative

can elicit the corresponding hedonic response and that

people, therefore, should not have any hedonic preference

for novel smells. The only exceptions are smells of

irritating quality that strongly stimulate intranasal

trigeminal structures. Such smells often indicate toxicity.

While neuroscientists and psychologists have established a

detailed understanding of the human sense of smell, insight

into the subjective characteristics of smell and related

experiences is lacking.

The exploration of this subjective layer of smell is often understood as going beyond the

interest of these disciplines, but is highly relevant for HCI

and user experience research.

SMELL IN HUMAN-COMPUTER INTERACTION

Ten years ago, Kaye [17] encouraged the HCI community

to think about particular topics that need to be studied and

understood about smell. While some attempts have been

made to explore smell during recent years, the potential of

smell in HCI remains underexplored.

Most work on smell in HCI focuses on developing and

evaluating smell-enhanced technologies.

Brewster et al. [5]

used smell to elicit memories, and developed a smell-based

photo-tagging tool (Olfoto). Bodnar et al. [4] showed smell

to be less disruptive as a notification mechanism than visual

and auditory modalities. Emsenhuber et al. [9] discussed

scent marketing, highlighting the technological challenges

for HCI and pervasive technologies. Ranasinghe et al. [24]

further investigated the use of smell for digital

communication, enabling the sharing of smell over the

Internet. More examples of smell-enhanced technologies

can be found in multimedia applications [13], games [16],

online search interfaces [19], health and wellbeing tools

(e.g., http://www.myode.org/), and ambient displays [22].

The exploration of smell-enhanced technologies is mostly

limited to development efforts and the evaluation of users’

smell perception of single smell stimuli. The smells used

are often arbitrary and not related to experiences. This is

because of the lack of knowledge pertaining to the

description and classification of smells required for HCI

[17]. Kaye points out that “There are specific ones

[classification and description schemes] for the perfume,

wine and beer industries, for example, but these do not

apply to the wide range of smells that we might want to use

in a user interface” (p. 653). Thus, previous work has a

general and quite simple usage of smell.

 

THE POTENTIAL OF STUDYING SMELL EXPERIENCES

In contrast to the work reported above, the present paper

focuses instead on experiences with smell and links them to

potential technologies. We do so through stories of

experiences with smell. Stories are increasingly used within

user experience research to explore personal memories of

past experiences, but also to facilitate communication in a

design process [3,34]. Stories are concrete accounts of

particular people and events, in specific situations [10],

and

are more likely to stimulate empathy and inspire design thinking than, for example, scenarios.

STUDY METHOD

We asked a large sample of participants to report smell

experiences that were personal and meaningful.

We refer to the description of these experiences as smell stories. These

stories were captured through a questionnaire described

below, which included inspirational examples of smellenhanced

technologies at its end. Based on the examples we

asked participants to reflect on their experience and future

technologies. The rationale of this approach was to begin

from smell experiences that matter to participants, instead

of starting from an application or a particular technology.

Questionnaire

We created a web-based questionnaire consisting of six

parts. We started with an open question to stimulate the

report of a personal memorable smell experience. This was

followed by closed questions aiming to elucidate the

relevant emotional and experiential characteristics, as well

as the smell qualities. Participants could freely choose the

story to report. The questionnaire was administered through

a crowdsourcing platform to obtain a large sample of smell

stories. Crowdsourcing provides valid and reliable data [20]

and has been used for capturing user experiences [31].

Part 1: Smell Story

The smell stories were elicited through an initial exercise,

where participants were asked to think about situations and

experiences where smell played an important role. The aim

was to get participants into the right frame of mind and

sensitize them to smell. Next, participants were asked to

describe one memorable smell experience in as much detail

as possible, inspired by the questioning approach used in

explicitation interviews [23]. This questioning technique is

used to reconstruct a particular moment and aims to place a

person back in a situation to relive and recount it. Part 1 of

the survey was introduced as follows: Bring to your mind

one particular memorable moment of a personal smell

experience. The experience can be negative or positive.

Please try to describe this particular smell experience in as

much detail as possible. You can use as many sentences as

you like, so we can easily understand why this moment is a

memorable experience involving smell for you.

Participants were asked to give a title to their story

(reflecting its meaning) and indicate if the experience was

positive, negative, or ambivalent (i.e., equally positive and

negative). They were also asked to indicate how personally

relevant the experience was (from ‘not personally relevant

at all’ to ‘very personally relevant’).

Part 2: Context

Part 2 asked participants to give further details of their

reported experience via open and closed follow-up

questions. There were four questions on the context of the

described experience, including the social context (who else

was present), the place (based on the categories used by

[26]), the location (as an open field), and the time when the

reported experience took place (days, weeks, months, or

years ago).

Part 3: The smell

Specific questions on the characteristics and qualities of the

smell were asked in Part 3. Participants characterized the

smell itself using a list of 72 adjectives (i.e., affective and

qualitative terms) derived from the ‘Geneva Emotion and

Odor Scale’ (GEOS) [7]. Participants could also add

descriptions to characterize the smell in an open feedback

box. In addition, they rated the smell with respect to its

perceived pleasantness, intensity, and familiarity.

Part 4: Experienced emotions

In Part 4 participants had to describe how they felt about

the experience as a whole, using a list of affective terms

(101 in total). They could go through the list and tick the

words that best described their emotions during the

experience. The words were derived from Scherer [27].

Participants could also add their own words in a free-text

field.

Part 5: Smell technologies

After the participants had selected, thought about, and

described a particular smell episode, Part 5 linked their

personal experience to technology. The participants were

engaged in a envisioning exercise inspired by work on

mental time travel [30]. They were shown six inspirational

examples of smell technologies, namely: Olfoto: searching

and tagging pictures (CHI, [5]); Smelling screen: ambient

displays (IEEE, [22]); Digital smell: Sharing smell over the

Web (ICST, [24]); Scent dress: interactive fabric with smell

stimulation (http://www.smartsecondskin.com/); Mobile

smell App: iPhone To Detect Bad Breath and Other Smells

(BusinessInsider 01/2013), and Smell-enhanced cinema:

Iron Man 3 Smell-Enhanced Screening (Wired 04/2013).

These six technologies cover areas of relevance for HCI

(mobile, ambient, wearable, personal, and entertainment

computing), give realistic examples of smell technologies

from research, and include recent, commercial examples.

We asked the participants to imagine any desirable change

that future smell technology might make (or not) with

respect to their personal smell experience.

We asked them the following questions: (1) How could your experience be

enhanced? (2) What technology are you thinking about? (3)

Why would such a combination of your experience and the

technology be desirable, or why would it not?

Finally, the participants could express any other ideas for smell

technology in a free-text field.

Part 6: Personal background

At the end of the questionnaire, participants answered

questions on their socio-demographic and cultural

background. The goal was to try to identify any

geographical and cultural influences on smell attitudes (as

found by Seo et al. [29]). The participants were also asked

to assess their own smell sensitivity.

All the questions, except for those on demographics, were

mandatory. On average, the survey took 16 minutes to

complete (SD = 7.57 minutes). Participants received US$

1.50 for completing the questionnaire, corresponding to an

hourly salary of 5.63 dollars.

Collected data and participants

A total of 554 participants began the questionnaire. Of

these, 480 completed the questionnaire and answered three

verification questions at its end. These questions required

participants to describe the purpose of the study without

being able to go back and look at the earlier questions or

guidelines. After data cleaning, 41 stories were excluded.

Fake entries (n = 11) were identified immediately, while

repeated entries (n = 10), incomplete stories (unfinished

sentences; n = 6), and incomprehensible stories (which did

not make sense on their own; n = 14), were excluded

iteratively throughout the coding process. This left us with

439 smell stories.

At the time of the study, all 439 participants (52.8% female)

lived in the US; most had grown up in the US (95%). The

participants’ age ranged from 18 to 67 years (M = 31.5, SD

= 10.0). A majority of participants (84%) indicated being

sensitive to smell (rating 4 or higher on a scale from 1 to 5).

Data analysis

The analysis process followed an open and exploratory

coding approach [25]. Two researchers conducted the

qualitative coding process. After coding an initial 25% of

the stories, two more coding rounds (to reach 33% and then

50% of the data), led to the establishment of an agreed

coding scheme. The coding scheme contained 10 main

categories and 36 sub-categories, and a category entitled

‘not meaningful’ for cases where smell did not seem to

have any relevance in the described experience. Based on

this coding scheme, one researcher coded the remaining

50% of the data, and the second researcher coded a subsample

of 25% of that data, resulting in a good inter-coder

agreement (Cohen’s kappa of κ = .68) [12].

Follow up design brainstorming

In addition to the feedback from our participants,

we also explored the design value of the smell stories with experts

in the field. We organized a two-hour design brainstorming

session with three HCI researchers, two working on smell

technologies and one working on advanced interface and

hardware design. None of them were from the same

organization as the authors and none were familiar with the

details of the study before the session.

The brainstorming session aimed to share and interpret the

smell stories and followed four stages [11]: (1) prompting,

(2) sharing, (3) selecting, and (4) committing. We selected

36 stories (one representative story for each sub category)

as brainstorming prompts. All 36 stories were printed on A6

sheets (including the story title, the smell story, context

information, and personal background). Each researcher

was asked to read through the stories individually before

discussing them together. They were asked the same

questions as our participants (e.g., how they might imagine

a connection between the experience and technology). Each

researcher chose the most interesting/inspiring stories to

share with the group, then they generated ideas as a group,

and selected three to four ideas to be developed in more

detail. The outcome of the brainstorming session is

presented in the implication section, after the description of

the findings from the smell stories.

FINDINGS ON EXPERIENCES WITH SMELL

In the following sections we present our findings according

to the 10 identified categories. The 439 smell stories were

organized via their primary category, as agreed by the

coders. This categorization does not define a strict line

between the categories, as they are not wholly independent,

but it does enable us to organize the material and generate a

useful dataset for design.

Below we provide for each category a rich description of the particularities of the

stories, excerpts from example stories, and their associated

smell qualities and emotions. Each category also contains

information about the participants’ own rating of the stories

as positive, negative, or ambivalent.

Category 1: Associating the past with a smell

This category is the largest and contains 157 stories. In

these stories, the participants described a past experience in

which a smell was encountered during a special event in life

(e.g., ‘Wedding Day’, ‘New House’), at a special location

(e.g., ‘The Smells of Paris’, ‘Grandma’s House’), or as part

of a tradition (e.g., ‘The Smell of Thanksgiving’ or

‘Christmas Eve’). In these stories the smell was described

as having a strong association to those particular moments

in the past, with no actual smell stimulus in the present. A

particularity of this category is the distinction between

stories describing personal memorable events versus

personal life events (e.g., ‘Disneyland’ versus ‘When my

mother died’). Smells were also associated with personal

achievement/success (e.g., ‘Scent of Published Book’,

‘New Car Smell’) and other important episodes of change,

such as “‘Fresh Start’: I was taking a job in a new city. …. I

took a plane trip across the country and the moment I took

a step off the plane and took a deep breath will always stick

with me. It felt so clean and the air actually smelled fresh

and new” [#488]. Within this story, the qualities of the

smell were for instance described as fresh, energetic, and

invigorating. Some of the emotions experienced at this

moment were courageousness and excitement. Although

this category is dominated by positive experiences (n =

127), negative experiences were also reported (n = 27),

such as ‘Car Crash’.

Category 2: Remembering through a smell

The 40 stories in this category described a recent

experience of a smell, which reminded participants about

past events, people, locations, or specific times in their life.

In contrast to the previous category (where stories describe

a direct link from the recollected past smell to the present;

e.g., the smell of ‘Grandma’s House’), this category

contains stories that describe an indirect link from the

present experienced smell stimulus

to the past event, person or place (e.g., the smell of chocolate cookies as sudden

reminder about grandma). Most stories in this category

contain reminders of childhood described as ‘sweet’,

‘reassuring’ and ‘nostalgic’ with respect to the qualities of

the smell. A sub-set of stories in this category (n = 10) also

highlight the particular power of smells to take a person

back in time. The description of such a flashback caused by

a sudden smell stimulus was described as: “‘My first love’:

It was the next day, when I was walking through the local

Macy’s that I smelled something that threw me back into

that situation, I could feel and see everything that had

happened the day before when I smelled a perfume in the

store” [#630]. Some of the qualities used to describe the

smell were attractive, erotic, and fresh. The experienced

emotions were described as amorous, aroused, excited,

hopeful, and interested. The stories in this category were

mainly positive (n = 37), except for three.

Category 3: Smell perceived as stimulating

The 62 stories in this category described experiences with a

unique, mostly unknown smell (all stories, except one, were

positive). The smells arose from different sources, such as

perfume, food, and nature. A particularity of this category is

the quality of ‘first time’ encounters with a smell across all

origins. One participant described the first time he was at a

beach: “The smell was very different from anything I had

ever experienced before. At first I was kind of grossed out

by the smell, but I grew to love it” [#921]. Another

participant described the smell of a tornado experienced for

the first time: “It was similar to the smell before rain but

had a certain sharpness to it, as if to warn of the incoming

danger. I felt like I knew this smell but at the same time, it

felt foreign to me. It wasn’t a bad smell, it was just slightly

unfamiliar” [#713]. The smell qualities and experienced

emotions were often described with mixed attributes (e.g.,

heavy, imitating, and stimulating; attentive, serious, and

calm), but still rated as positive experiences by participants.

Most of the other stories in this category reported on the

first experiences with food (e.g., ‘Slice of Heaven’) and

nature (e.g., ‘Grass’), and were described as desirable,

fresh, or pure, and provoked feelings of happiness at the

moment they occurred. Although specific memories were

established, including unique new associations (e.g.,

‘Tornado smell’), the stories in this category did not evoke

the kind of strong connections to the past as described in

Category 1 and 2.

Category 4: Smell creating desire for more

This category contains 48 stories (45 positive). Key to these

stories is that the smell grabbed the persons’ attention

unexpectedly. The smell was either associated with food

(triggering appetite), nature (triggering curiosity), or the

scent of other people (triggering attractiveness), which

motivated one to do or get something. In some stories smell

was described in relation to the sensation of newness (e.g.,

“‘The sweet smell of CPU’: …There was the smell of the

cardboard boxes it all arrived in, the smell of new metal–

perhaps it was a combination of these and other things, but

when the building was complete there was just a singular

smell that was unique to a new computer built by my own

hands” [#685]). The qualities of the smell in this story

included beneficial, heavy, sophisticated, energetic, and

pleasantly surprising. The experienced excitement was

expressed through words such as confident, delighted,

enthusiastic, impressed, or triumphant. This category also

contained one story where the smell at a funeral stimulated

reflection in the moment (e.g., ‘The scent of moving on’).

The story was rated as a positive experience and at the same

time the smell was described as clean, penetrating, and

persistent, and the participant indicated that she was afraid,

anxious, discontented, sad, tired, and uncomfortable.

Despite the negative situation described in this story, the

smell gave hope and a desire to live and move on, looking

into the future in contrast to the stories in Category 1 and 2.

Category 5: Smell allowing identification and detection

This category captures the enabling role of smell in certain

situations, such as allowing one to identify or detect a smell

(e.g., “‘Gas leak’: I was cooking something on a gas stove

and went out for a few minutes. When I came back, the fire

was extinguished but the gas was still on. My roommate

was sat at the table doing schoolwork, completely oblivious

to the poisonous gas that was filling the room. I told him to

get the hell up and open the windows and doors” [#951]).

The qualities used to describe the smell were distinguished,

penetrating, dirty, and light. The emotions related to this

situation were described as anxious, conscientious,

confident, and serious. Although the category is rather

small (n = 11), the lesson to be learnt from the shared

stories was the immediacy of the smell, allowing the

participant to act or prevent something.

Category 6: Overwhelming power of smell

This category includes 37 stories where the smell

overwhelmed the person in a positive way (n = 5; e.g., ‘The

Chocolate Factory’) but predominately in a negative way (n

= 30; e.g., ‘The Smell of Death’). In the latter case, people

described the smell as something disturbing, as something

that hit them suddenly on their way or during an activity. A

subset of the stories was recounted as traumatizing, so that

the person could still vividly remember the particular

moment in the past although years have passed and no

recent similar smell stimulation had occurred unlike in

Category 2 (e.g., “‘Visit to a local county jail’: My guide

warned me ahead of the time that it was going to be a little

foul in there, but nothing could have prepared me for the

obscenely acrid stench of hundreds of men crammed into

every available space of the jail, right down to windowless

storage rooms converted into more cells. … For days

afterwards, I couldn’t shake the smell…. There weren’t

enough showers to take it away. It’s been several years

since then, and my memory of that smell is just a strong as

ever” [#604]). In this category, the qualities of the smell

were described as heavy, penetrating, dirty, or sickening.

Amongst others, the experienced emotions were described

as alarmed, anxious, distressed, frustrated, or

uncomfortable. In contrast to Category 1 and 2 (where the

smell was associated with an event from the past or

triggered a specific memory), Category 6 is about the smell

as such during the experience and not about the memory

associated with this smell. As opposed to the first two

categories, in most stories forgetting – not remembering –

the smell was the key element.

Category 7: Smell invading private and public spaces

All the stories in this category (n = 32) described an

experience where one could not get rid of the smell. The

smell invaded private and public spaces. In contrast to the

previous category, the smell entered the person’s personal

space (the person did not enter the space where the smell

already existed) and took over the space. The loss of control

over the smell was linked to the lingering quality of the

smell (e.g., “‘Don’t want to smell that twice!’: I woke up

one morning suddenly confused and was hit with an odor so

horrible I couldn’t figure out what it was. … It was not like

the smell you get a whiff of when a skunk stinks up the

outdoors” [#530]). In the story the power of the smell,

causing them to leave the house for several hours, was

described with qualities such as foul, nauseous, penetrating,

and persistent. One of the experienced emotions was

surprisingly ‘amused’, however it was overruled by other

emotions including annoyed, anxious, disgusted, taken

aback, and uncomfortable. Despite the glimpse of humor in

some stories, this category mainly contains negative

experiences and underlines the power of the smell with its

sudden and lingering qualities.

Category 8: Social interaction is affected by the smell

Within this category, smell was related to a person’s own

smell or to the smell of others. Smell negatively affected

the interaction among people and their togetherness (e.g.,

“‘Dragon breath teacher’: Once a teacher yelled at me

during class. She got so close up into my face that I could

smell her bad breath. This made the experience much worse

because I wanted to get up and walk away but she was

grabbing me to keep me focused on her while she was

talking” [#744]). The smell qualities were described as

nauseous, penetrating, and sickening, and caused negative

emotions experienced as bitter, distressed, or insulted.

Despite frequent interactions among people, this category

only contains 11 stories. This set of stories (overall negative

experiences, apart from two) contains interesting elements

with respect to a person’s own awareness of body smell and

the overbearing effect of other peoples’ smell on one’s

comfort.

Category 9: Smell changes mood, attitude and behavior

This category contains 23 stories, which underlined the

power of smells to change a person’s mood, attitude, or

behavior. Stories reported the active regulating effect of

smells with respect to mood, but mostly (n = 14) the change

of behavior due to smells (e.g., ‘Accidental vegetarian’ or

‘Saved by the Smell!’). One story showed the active usage

of smells to change one’s mood. A participant had recently

been divorced and reported on the day her husband had

moved out: “‘White Lilac Sheets’: “I made the bed with my

lilac sheets and the atmosphere changed. I still remember

that scent and how I felt on that day. I was going to be

okay. The room didn’t look or feel or smell lonely anymore.

It looked and smelled fresh and clean and lovely and a bit

romantic and it was mine” [#526]. The qualities of the

smell were described as fresh, reassuring, and spring-like,

while the experienced emotions were determined, hopeful,

longing, tense, but also worried. Overall, the stories in this

category were reported as mainly positive (n = 12)

experiences, but also as negative (n = 7) and four stories

were rated ambivalent, neither positive nor negative.

Category 10: Smell builds up and changes expectations

This category shows the potential of smell to build up

expectations and to surprise. In the former case (11 stories)

the smell was building up expectations until the actual

contact with the trigger, such as food or a perfume (e.g.,

“‘The Smell of Hungry Anticipation’: “I was trying a new

soup for the first time. When it was brought to the table, the

soft smell of rosemary immediately hit my nostrils. …It

complimented the taste of the soup and built anticipation”

[#585]). The smell was described as mouthwatering,

healthy, and pleasantly surprising, and was further related

to emotions such as conscientious, expectant, and relaxed.

In other stories (n = 7), expectations were exceeded to the

extent that they surprised and diverted anticipations (e.g.,

‘PomVinegar Surprise’: “I could smell the pomegranate

and vinegar from about 10 steps away, and it was a very

pungent (thought not unpleasant) odor. I almost felt my

nose becoming runny and took out a tissue. When I tasted

the dish, however, the taste wasn’t nearly as sour as I

expected it to be from the smell” [#542]). The smell in this

story was described as distinguished and penetrating, and

was associated with emotions such as attentive and excited.

Key quantitative facts behind the smell stories

While the above-described categories can be used as an

inspiration and as a starting point for exploring design

opportunities for smell in HCI,

our quantitative data provides additional background information. Below, the

key quantitative information across all the collected smell

stories is summarized. The majority of the 439 collected

stories were positively valenced (n = 296), 112 were

negative, and 31 were ambivalent. On average, negative

stories tended to be slightly longer (M = 90 words) than

positive stories (M = 79 words), but the difference is

statistically not significant (U = 14600, p = .063, r = .09).

Contextual information

Most stories occurred in a context where one or more familiar persons were present (64.2%)

or where participants were alone (21.6%). The presence of

one or more strangers was reported less frequently (8.7%).

With regard to location, most of the experiences happened

at the participant’s or a friend’s home (38.1%) or in a public

building (20.7%). Quite a few participants reported that

their experience took place in the streets or another public

space (14.4%), in a natural setting (7.3%), or at work

(6.4%). The remaining participants (13.2%) indicated other

places (e.g., stranger’s home). On average the reported

experiences occurred 8.7 years ago (SD = 10.3), ranging

from 1 day to 58 years ago.

The qualities of smell

The most frequent smell qualities

reported in positive stories were pleasant (60%), fresh

(42%), sweet (38%), clean (31%), and mouthwatering

(30%). Smells in negative stories were described as

unpleasant (62%), penetrating (55%), heavy (54%), foul

(53%), and nauseous (51%). In ambivalent stories the smell

was perceived as fresh (39%), pleasant (32%), mouthwatering

(32%), attractive (26%), and penetrating (23%).

Experienced emotions

When asked to describe how they felt during their experience, participants’ used the affective

terms happy (63%), pleased (53%), joyous (42%), delighted

(41%), and excited (39%) in positive stories and

uncomfortable (55%), disgusted (51%), distressed (43%),

miserable (42%), and taken aback (29%) in negative stories.

Ambivalent experiences were most frequently described as

happy (42%), excited (39%), enthusiastic (35%), joyous

(32%), and serene (29%).

An overview of all 10 categories and 36 sub-categories

including qualitative and quantitative information

(including a full example for each sub-category, used in the

design brainstorming session) is provided as supplementary

material. All smell stories and related qualities of smell,

experienced emotions, and context information, are also

available at www.multisensory.info for further exploration.

IMPLICATIONS FOR TECHNOLOGY

This study focused first on experiences and second on the

implications for technology. This section turns to

technology. Below we summarize the feedback from the

participants on how technology would fit with their

experience, and describe input from a brainstorming session

with HCI researchers working on smell technologies and

advanced interface and hardware design, based on a sub-set

of the smell stories (one from each sub-category).

Ideas for technology from participants

Below we summarize the six areas and ideas for desirable

future smell technologies mentioned by participants in Part

5 of the questionnaire:

(1) To share smells with family/friends: allow one to

participate in a family event through remote smelling; share

smells of special moments such as the smell of a newborn

baby with distant relatives; share smells with people who

they know would appreciate it (such as through social

media); allow capturing and sharing of smells to create a

common understanding where you can’t explain it.

Participants also desired to be able to design and share new

smells from a personal database and create a personal smell

box/bottle.

(2) To support decision making: use smells for a quick

judgment in online shopping (like/dislike can be determined

easily); create smell profiles about holiday places and travel

destinations; smell match maker in dating websites for

allowing a better decision making about going on a date or

not with a person (smell enhanced profiles).

(3) To regulate mood actively/passively: smell to relive

good experiences whenever you want to get in a better

mood; to calm yourself down in stressful moments such as

in traffic jam or at work; as a reminder of past memories

you would have forgotten otherwise but that can cheer you

up when you feel depressed and life seems too difficult.

(4) To combine with other technologies and activities:

integrate smell into radio; combine smell with music such

as with ‘soundhound’ or ‘shazam’ apps; smell-enhanced

advertisement on TV (for food channels); enhance visits of

concerts, theater and performances with smell; allow underwater

smelling when diving.

(5) To combine with everyday objects: enhance wristband

with smell for keeping a preferred perfume lingering; have

special glasses to see and smell the beach; smell-enhanced

jewelry and clothes. One participant imagined her wedding

ring enhanced with the smell of that day.

(6) To make oneself and others aware about body smell: to

avoid embarrassing moments; provide invisible cues to a

person about her/his smell level; quick smell check after

sporting activities.

The first idea matches the experiences in Categories 1 and

2, where particular events/moments in life are associated

with a smell. The desire for capturing and sharing these

experiences enhanced with smell becomes prevalent and

suggests design implications for real-time smell-enhanced

technologies (e.g., mobile phone, photo or video camera).

The second idea can be linked to Category 5, allowing

people to identify and detect a smell. Moreover, smells are

seen as very powerful for supporting quick decision-making

(e.g., smell-enhanced website navigation and searching).

The third idea shows a direct link to Category 9 and the

potential of smell to change mood. Interestingly participants

whose story was in Category 1 or 2 were wishing for the

possibility to capture pleasant smells, for instance from

their childhood, and released to them in the present. This

desire for smell-enhanced technologies or products is also

apparent in the fourth and fifth ideas, where technology,

objects, or even activities can be enhanced through smells

from the past, or actual smells sourced through nature (e.g.,

diving in the ocean). Finally, the sixth idea is linked to

Category 8, aiming to avoid embarrassing moments in

social interactions.

Participants also expressed concerns about future smell

technologies. They were concerned about the possible

misuse of smell when sharing it through the Internet or

mobile phones (e.g., teasing people with smells, how to

trust a smell message), and about the potential manipulation

through smell (e.g., TV ads, online shopping). Some

participants were also afraid to get sick, catch an allergy, or

become addicted if they are exposed to chemical

stimulations from technology. Finally, one participant

raised the question of copyright and ownership of smells

(e.g., ‘can I share others’ smells?’).

Ideas for technology from HCI researchers

Below we outline the ideas that emerged from the two-hour

brainstorming session prompted by 36 smell stories (one

from each sub-category). Four groups of design ideas

emerged from this session and are described below:

(1) Smell-enhanced performance regulator: a technology

stimulating smell in order to structure the day, taking

activities and moods into account, and combining different

smells to avoid habituation (training and evaluation phase

needed). Smell as a reminder to take a break or as

motivation to keep going a bit longer to meet the deadline

[inspired by #526 ‘White Lilac Sheets’, Category 9].

(2) Autonomous smell agent: a technology spreading

ambient cues (e.g., a robot) to guide someone to a certain

place, to build up expectations, and motivate action. Smell

trails in the environment can also make hidden information

accessible, for instance, before entering a room (e.g., smell

warning: tense working atmosphere) [inspired by #801

‘Don’t forget to check your gas stove before you leave the

house’, Category 5].

(3) Reminder alert with smell: a technology to remind us

about important events, birthdays, and appointments.

Although we have reminders on mobile phones and

computers, they are often ignored, snoozed or in the worst

case forgotten about. A smell can provide a pleasant

reminder to say ‘it is time to call your mom’ by presenting

the smell of your favorite dish your mom makes for you.

On the other hand, if more critical, bad smells can be very

powerful as a reminder and are not easily ignored [inspired

by #530 ‘Don’t want to smell that twice’, Category 7].

(4) Smell-enhanced storytelling: a technology that

stimulates storytelling around a digitized version of an

incense stick. A stick was imagined with different layers,

representing smells related to a loved person who passed

away. When friends or family members come together, for

instance at an anniversary year, they can add new smells to

be shared in the group and thus trigger new stories about

the dead person. It is as if looking through a photo album,

telling the stories from the past, and using the smells as

anchor points for keeping the memory alive [inspired by

#672 ‘The Scent of Moving On’, Category 4].

We saw that the smell stories, even if they only provided

limited information (story, story title, context, gender, and

age), triggered vivid discussions, created empathy, and

stimulated the sharing of personal smell experiences. The

four ideas described above provide a starting point for

exploring smell in HCI. The categorization along with

additional background information on smell qualities and

experienced emotions (see supplementary material) can

inspire further explorations of smell technologies.

DISCUSSION

Our findings about experiences with smell in combination

with the ideas for technologies just presented show several

design opportunities for smell. Below we do not provide

solutions for smell-enhanced technology designs, rather we

illustrate where our findings might be relevant to stimulate

novel designs in existing areas of interest within HCI. We

see three anchor points for smell-enhanced technology.

First, the smell stories in Categories 1 and 2 suggest design

opportunities for remembering and recalling the past. Our

findings might enrich ongoing research on the design for

personal memories. Apart from enhancing research

supporting the capturing and sharing of personal

experiences (e.g., in family relationships [18]) through

smell, our findings support research to support people who

are living with memory loss (e.g., patients with dementia

[32]), where smell can play an important role in

remembering the past. An increasing body of research also

explores the potential of digital technologies to support our

memory in everyday tasks (e.g., reminder systems), to

recall past events and experiences (e.g., life-logging tools),

to design end-of-life technologies allowing reminiscence of

passed away people [21], and to record and reproduce

smells [35]. All this research shows the potential of smell to

enrich experiences, for instance by enhancing personal

memories such as photos or videos with smell. Based on

their study of a smell-based photo-tagging system, Brewster

et al. [5] stated that participants asked for personal smells to

be added. The information on how to classify such smells

was still missing; the present analysis allows us to relate

smell qualities to particular types of experiences.

When designing with smell, as for any memory-based

technology, access to such memories has to be considered

carefully to preserve their uniqueness. One participant

wrote: “I could see it being desirable in that it would allow

me to experience the scent whenever I want, but it’s kind of

a two edged sword in that experiencing that scent time and

time again will make it common place” [#513]. The power

of smell might not persist if always available, thus

participants suggested to either restrict the access and

retrieval of smells to special times (e.g., at ‘grandmas

birthday’) or to link them to a certain social setting (e.g.,

smelling only in company with ‘your sister’). This way the

uniqueness of the smell can be preserved.

Second, the stories in Categories 3 to 8 as well as 10 draw

the attention away from past memories and suggest design

opportunities for the present moment. Designing for in-situ

stimulation, the ability to capture and share smells in the

moment, and the capability to mask and neutralize bad

smells creates a vast space for smell interaction design. One

suggestion made by participants was the combination of

smell and social media, such as “An app that would allow

me to store smells, send smells, or attach smells to a picture

that I could post on social media or Instagram or

something”. This supports existing research on the delivery

of smells through the Internet [24]. We draw attention to

three additional design directions concerned with (1) first

time experiences with smell, (2) the power of smell to build

up expectations, and (3) the potential of designing for bad

smells. User experience designers put a lot of effort into

designing ‘out-of-the-box’ and first time experiences to

create positive experiences [13]. Our categorization not

only provides designers with rich descriptions of such first

time experiences, but also describes the related qualities of

smells in combination with descriptors of the experienced

emotions. This can be used to stimulate positive smellenhanced

experiences with technology, build up

expectations, and create anticipation as studied within

experience research [33]. Typically this anticipation stage is

influenced by a variety of aspects (e.g., advertisements,

product descriptions, accounts from existing customers).

Smell stories in Category 10 provide evidence for the

power of smell to build up expectations, create surprise by

exceeding anticipated experiences, and enhance momentary

experiences through capturing and sharing pleasant smells.

Categories 6 to 8 contain stories about bad smells, which

are wished to be neutralized or masked to change the

experience from something negative to something positive.

While the idea of outbalancing smells seems to be

desirable, the design brainstorming session stimulated a

discussion on the usage of bad smell in design, particularly

as part of the design idea (3) Reminder alert with smell.

Designing for bad smells might not seem appropriate at

first, but through intensity manipulation it can open up an

interesting space for design. Similar to a snooze function,

which slowly increases volume, smell stimulations could be

added to certain events (e.g., reminder for mother’s

birthday). Starting with a pleasant smell, it could turn

slowly into something unpleasant if you did not act.

Category 8 also contained stories recounting social

experiences with smell, where the smell of a person or of

other people caused embarrassment or discomfort. Despite

the importance and frequency of social contact in everyday

life, few such stories were shared. They might not seem

meaningful enough to be memorable or to be shared. Yet,

this set of stories holds potential for personal and social

smell-enhanced awareness systems, as well as for wearable

technologies, and smart fabrics. Technology could, as stated

by a participant, “…make the people in those settings feel

more comfortable if I interact with them… My holding my

nose could be insulting and impede communication.”

Third, the smell stories in Category 9 suggest design

opportunities reaching out to the future through positive

stimulation, with potential relevance for wellbeing and

behavior change research in HCI. The stories shared in this

category were about the power of smell to regulate mood,

change attitudes, and behavior. Designing for smell could

be combined with behavior change research in HCI (e.g.,

tools to support healthy nutrition and diet), and thought of

in relation to positive psychology and research on

wellbeing. Seligman and Csikszentmihalyi [28] suggested

that happiness can be learned and cultivated and that

positive psychology can help change how a person feels.

They point to the power of positive emotions for our health,

happiness and wholeness. We would suggest that our

findings add an understanding of the positive emotional

impact of smells that might be a valuable research strategy

in wellbeing research (e.g., for regulating mood).

Smell can have a regulating impact on a person’s mood and can, as in

one case explicitly reported, be used to regulate mood

(‘White Lilac Sheets’). The participant wrote, “I guess the

experience could have been enhanced by some kind of

mood moderator. Something that would have sensed my

sadness and filled the room or house with comforting

scents” [#526]. The participant pointed out that technology

would not change the situation to something more positive,

as it just was not a happy time at all, but that it could

support the sad moments in this transitional period of life.

Limitations

We would like to acknowledge three limitations of this

work. First, by using Amazon Mechanical Turk for

recruiting and asking participants to describe personal

relevant experiences, we were limited to the US and do not

know to what extent the smell stories are representative of

more general experiences with smell. We are aware about

cultural and geographical differences (as described by Seo

et al. [29]), which require further studies with a more

diverse group of participants. Second, collecting narratives

by means of an online questionnaire has an influence on

how people narrate their experience and deprives us of the

advantages of an interview situation where we can engage

in a dialogue with the participant to explore the meaning

behind the shared experience in more depth as described by

Bruner [6]. We tried to collect information beyond the

initial trigger of the shared smell stories in order to allow

the establishment of meaningful categorizations and the

creation of a basic understanding of experiences with smell

in HCI.

Third, our approach provides an overview on the

emerging field of smell-enhanced technologies. Future

studies will, we hope, lead to in-depth research into

experiences with smell inspired by our identified categories.

 

CONCLUSIONS

Despite interactive technologies increasingly disappearing

into our environment (in ubiquitous and mobile computing)

and becoming essential in everyday life, the senses used to

interact with technology are still limited.

We have discussed the opportunities for smell in HCI based on an

analysis of 439 smell stories. We identified 10 primary

categories for stories about experiences with smell, which

help discuss the potential implications for technology.

Implications were drawn from feedback from our

participants envisaging desired connections between their

own personal experience and future smell technology. The

implications for designing for smell were further enriched

through ideas from an initial brainstorming session with

HCI researchers. Our findings provide guidance for smell

enhanced technology design, not only giving a

categorization of the role of smell in personal experiences,

but also extracting the qualities of smell across the smell

stories and the experienced emotions. We argue that this

research enriches existing technology driven research on

smell in HCI and provides a fruitful starting point when

designing for experiences with smell.

ACKNOWLEDGMENTS

This work is supported by the Marie Curie IEF Action of

the European Union (FP7-PEOPLE-2010-IEF) and the

Swiss National Science Foundation (PBBSP1 144196). We

thank our participants and especially Annika Haas for her

valuable support in designing the supplementary material.

REFERENCES

1. Alaoui-Ismaïli, O., Robin, O., Rada, H., Dittmar, A.,

Vernet-Maury, E. Basic Emotions Evoked by Odorants:

Comparison Between Autonomic Responses and Self-

Evaluation. Physiology & Beh. 62, 4 (1997), 713–720.

2. Bakalar, N. Sensory science: Partners in flavour. Nature

486, 7403 (2012), S4–S5.

3. Baumeister, R.F., Newman, L.S. How Stories Make

Sense of Personal Experiences: Motives that Shape

Autobiographical Narratives. Personality and Social

Psychology Bulletin 20, 6 (1994), 676–690.

4. Bodnar, A., Corbett, R., Nekrasovski, D. AROMA:

ambient awareness through olfaction in a messaging

application. Proc. ICMI, (2004), 183–190.

5. Brewster, S., McGookin, D., Miller, C. Olfoto:

designing a smell-based interaction. Proc. CHI (2006),

653–662.

6. Bruner, J. Life as narrative. Social Research: An

International Quarterly 71, 3 (2004), 691–710.

7. Chrea, C., Grandjean, D., Delplanque, S., Cayeux, I., Le

Calvé, B., Aymard, L., Velazco, MI., Sander, D.,

Scherer, KR. Mapping the Semantic Space for the

Subjective Experience of Emotional Responses to

Odors. Chem. Senses 34, 1 (2009), 49–62.

8. Ehrlichman, H., Bastone, L. The use of odour in the

study of emotion. In S. Van and G.H. Dodd, (Eds.),

Fragrance: The psychology and biology of perfume.

Elsevier, (1992), 143–159.

9. Emsenhuber, B. Scent Marketing: Making Olfactory

Advertising Pervasive. In J. Müller, F. Alt and D.

Michelis (Eds.), Pervasive Advertising. Springer,

(2011), 343–360.

10. Erickson, T. Design as storytelling. Interactions 3, 4

(1996), 30–35.

11. Faste, H., Rachmel, N., Essary, R., Sheehan, E.

Brainstorm, Chainstorm, Cheatstorm, Tweetstorm: new

ideation strategies for distributed HCI design. Proc. CHI

(2013), 1343–1352.

12. Fleiss, J.L., Levin, B., Paik, M.C. Statistical Methods

for Rates and Proportions. John Wiley & Sons, (2013).

13. Ghinea, G., Ademoye, O. The sweet smell of success:

Enhancing multimedia applications with olfaction.

TOMCCAP 8, 1 (2012), 2.

14. Hassenzahl, M. Experience Design: Technology for All

the Right Reasons. Syn. Lect. on HCI 3, 1 (2010), 1–95.

15. Herz, R.S., Engen, T. Odor memory: review and

analysis. Psy. Bulletin & Review. 3, 3 (1996), 300–313.

16. Jonsson, F., Verhagen, H. Senses working overtime: on

sensuous experiences and public computer game play.

Proc. ACE (2011), 56:1–56:8.

17. Kaye, J. “Jofish.” Making Scents: aromatic output for

HCI. Interactions 11, 1 (2004), 48–61.

18. Kazakos, K., Howard, S., Vetere, F. Revisiting the

relationship between reunion and technology-mediated

separation in periodically transitioning families. Proc.

CSCW (2013), 1157–1168.

19. Loumakis, F., Stumpf, S., Grayson, D. This image

smells good: effects of image information scent in

search engine results pages. CIKM (2011), 475–484.

20. Mason, W., Suri, S. Conducting behavioral research on

Amazon’s Mechanical Turk. Behavior Research

Methods 44, 1 (2012), 1–23.

21. Massimi, M., Moncur, W., Odom, W., Banks, R., Kirk,

D. Memento mori: technology design for the end of life.

Proc. CHI EA (2012), 2759–2762.

22. Matsukura, H., Yoneda, T., Ishida, H. Smelling Screen:

Development and Evaluation of an Olfactory Display

System for Presenting a Virtual Odor Source. IEEE

TVCG 19, 4 (2013), 606–615.

23. Petitmengin, C. Describing one’s subjective experience

in the second person: An interview method for the

science of consciousness. Phenomenology and the

Cognitive Sciences 5, 3-4 (2006), 229–269.

24. Ranasinghe, N., Karunanayaka, K., Cheok, A.D.,

Fernando, O.N.N., Nii, H., Gopalakrishnakone, P.

Digital taste and smell communication. Proc. BodyNets,

(2011), 78–84.

25. Saldana, J. The Coding Manual for Qualitative

Researchers. SAGE (2012).

26. Scherer, K.R. Appraisal considered as a process of

multilevel sequential checking. Appraisal processes in

emotion: Theory, methods, research 92, (2001), 120.

27. Scherer, K.R. What are emotions? And how can they be

measured? Social Science Inf. 44, 4 (2005), 695–729.

28. Seligman, M.E.P., Csikszentmihalyi, M. Positive

psychology: An intro. Am. Psy. 55, 1(2000), 5–14.

29. Seo, H.-S., Guarneros, M., Hudson, R., et al. Attitudes

toward Olfaction: A Cross-regional Study. Chem.

Senses 36, 2 (2011), 177–187.

30. Suddendorf, T., Corballis, M.C. The evolution of

foresight: What is mental time travel, and is it unique to

humans? Beh. & Brain Sciences 30, 3 (2007), 299–313.

31. Tuch, A.N., Trusell, R., Hornbæk, K. Analyzing users’

narratives to understand experience with interactive

products. Proc. CHI (2013), 2079–2088.

32. Wallace, J., Wright, P.C., McCarthy, J., Green, D.P.,

Thomas, J., Olivier, P. A design-led inquiry into

personhood in dementia. Proc. CHI (2013), 2617–2626.

33. Willander, J., Larsson, M. Smell your way back to

childhood: Autobiographical odor memory. Psy.

Bulletin & Review 13, 2 (2006), 240–244.

34. Wright, P., McCarthy, J. Experience-Centered Design:

Designers, Users, and Communities in Dialogue. Syn.

Lect. on HCI 3, 1 (2010), 1–123.

35. Wyszynski, B., Yamanaka, T., Nakamoto, T. Recording

and reproducing citrus flavors using odor recorder.

Sensors&Actuators B: Chemical 106, 1(2005), 388–393.