ITPDP2026- WEEK 3: DESIGN PROCESSES, PROJECT MANAGEMENT, AND DESIGN ETHICS

Clemens Nylandsted Klokmose Department of Computer Science Human-Centered Computing Section clemens@cs.au.dk

AARHUS UNIVERSITY

https://studypedia.au.dk/haandter-pensum/laesestrategier

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

2

AARHUS UNIVERSITY

PLAN

› Design processes

› Involving users

› Project management

› Design ethics

==> picture [222 x 222] intentionally omitted <==

› GDPR

==> picture [47 x 40] intentionally omitted <==

==> picture [222 x 146] intentionally omitted <==

----- Start of picture text -----
ITPDP 2026
----- End of picture text -----

AARHUS UNIVERSITY

PEOPLE AND PROTOTYPES

• › Chapter in Moggridge (2006) describes IDEO's methods

• › What is design? (Covered in FIT-DES)

• › It is important to understand the needs and desires of users

• › Observation and participation

• › Often tacit and implicit knowledge that can only be uncovered experimentally

• › Many versions of prototypes are needed (Later lecture)

• › Prototypes are tangible and visible proposals

• › User can "experience" a prototype and thus better evaluate proposed solutions

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

THE GOOD DESIGN?

• Transparency and tacit knowledge (Polanyi, Bødker, and more)

• › Fluid use without breakdowns

• › Leverages the users' intuition* (that is uncovered experimentally)

• › Scientific verification is often long and complex

Examples of assessment criteria for design projects:

• The height of creativity/innovation

• Aesthetics/quality

• Whether human factors/values are taken into account

• Performance and technology

• Finish and presentation

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

THE GOOD DESIGN: AFFORDANCES (BILL GAVER)

› Perceptible possibilities (Gibson, 1979)

› We sense immediately

• › That one can walk up a flight of stairs

› Sitting on a chair

› Tilting a door handle › Turning on faucet

• › Computer user interfaces should be designed with equally clear affordances...

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

AFFORDANCES

==> picture [47 x 40] intentionally omitted <==

==> picture [190 x 279] intentionally omitted <==

==> picture [225 x 226] intentionally omitted <==

2026

ITPDP

7

AARHUS UNIVERSITY

Need to repair a design that does not ”afford" the right action possibilities to the user

==> picture [47 x 40] intentionally omitted <==

==> picture [504 x 378] intentionally omitted <==

----- Start of picture text -----
ITPDP 2026
----- End of picture text -----

DESIGN DISCIPLINES AND TECHNIQUES

How do we understand the problem area and the needs of users?

AARHUS UNIVERSITY

LIMITATIONS WITH INCREASING COMPLEXITY

› For a holistic understanding of groups, organizations, society and the globe

›

› From the facts of human proportions and physics

ITPDP 2026

›

AARHUS UNIVERSITY

ANALYSIS METHODS

==> picture [116 x 121] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

ITERATIVE DESIGN PROCESS

• › Same type of activity is repeated to reduce uncertainty about the design

• › Many cross-cutting jumps between activities

• › From limitations to idea generation over prototyping less uncertainty back to remaining limitations

==> picture [47 x 40] intentionally omitted <==

==> picture [304 x 341] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

IDEO: 51 WAYS TO LEARN ABOUT USERS › IDEO Method cards

• › 4 Categories

• › Learn – from facts that can be collected

• › Look – at what users do

• › Ask – about their contributions;

• › Try – out ideas

• › The entire collection of 51 cards is available as a book/card box

› https://stoutbooks.com/products/ideo-method-cards-51-ways-to-inspire-design-61457

• › In the chapter, only 4 examples from each category

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUSUNIVERSITY LEARN

› Analyze the information you’ve collected to identify patterns and insights.

› FLOW ANALYSIS

• › How Represent the flow of information or activity through all phases of a system or process.

• › Why This is useful for identifying bottlenecks and opportunities for functional alternatives.

• › Example Designing an online advice Web site, flow analysis helped the team to gain a clearer sense of how to make it easy to find your way around the site.

› COGNITIVE TASK ANALYSIS

• › How List and summarize all of a user’s sensory inputs, decision points, and actions.

• › Why This is good for understanding users’ perceptual, attentional, and informational needs and for identifying bottlenecks where errors may occur.

• › Example Logging the commands that would be involved in controlling a remotely operated camera helped the team establish priorities among them.

› HISTORICAL ANALYSIS

• › How Compare features of an industry, organization, group, market segment or practice through various stages of development.

• › Why This method helps to identify trends and cycles of product use and customer behavior and to project those patterns into the future.

• › Example A historical view of chair design helped to define a common language and reference points for the team members from the client and consultancy.

› AFFINITY DIAGRAMS

• › How Cluster design elements according to intuitive relationships, such as similarity, dependence, proximity, and so forth.

• › Why This method is a useful way to identify connections among issues and to reveal opportunities for innovation.

• › Example This affinity diagram shows what’s involved in transporting young children, and helps to identify the opportunities to improve the design of a stroller.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUSUNIVERSITY LOOK

• › Observe people to discover what they really do—not what they say they do.

› FLY ON THE WALL

• › How Observe and record behavior within its context, without interfering with people’s activities.

• › Why It is useful to see what people do in real contexts and time frames, rather than accept what they say they did after the fact.

• › Example By spending time in the operating room, the designers were able to observe and understand the information that the surgical team needed.

› A DAY IN THE LIFE

• › How Catalog the activities and contexts that users experience for an entire day.

• › Why This is a useful way to reveal unanticipated issues inherent in the routines and circumstances people experience daily.

• › Example For the design of a portable communication device, the design team followed people throughout the day, observing moments at which they would like to be able to access information.

› SHADOWING

• › How Tag along with people to observe and understand their day-to-day routines, interactions, and contexts.

• › Why This is a valuable way to reveal design opportunities and show how a product might affect or complement user’s behavior.

• › Example The team accompanied truckers on their routes in order to understand how they might be affected by a device capable of detecting drowsiness.

› PERSONAL INVENTORY

• › How Document the things that people identify as important to them as a way of cataloging evidence of their lifestyles.

• › Why This method is useful for revealing people’s activities, perceptions, and values as well as patterns among them.

• › Example For a project to design a handheld electronic device, people were asked to show the contents of their purses and briefcases and explain how they use the objects that they carry around everyday.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUSUNIVERSITY ASK

› Enlist people’s participation to elicit information relevant to your project.

› CONCEPTUAL LANDSCAPE

• › How Ask people to diagram, sketch, or map the aspects of abstract social and behavioral constructs or phenomena.

• › Why This is a helpful way to understand people’s mental models of the issues related to the design problem.

• › Example Designing an online university, the team illustrated the different motivations, activities, and values that prompt people to go back to school.

› COLLAGE

• › How Ask participants to build a collage from a provided collection of images and to explain the significance of the images and arrangements they choose.

• › Why This illustrates participants’ understanding and perceptions of issues and helps them verbalize complex or unimagined themes.

• › Example Participants were asked to create a collage around the theme of sustainability to help the team understand how new technologies might be applied to better support people’s perceptions.

› FOREIGN CORRESPONDENTS

• › How Request input from coworkers and contacts in other countries and conduct a crosscultural study to derive basic international design principles.

• › Why This is a good way to illustrate the varied cultural and environmental contexts in which the products are used.

• › Example A global survey about personal privacy helped to quickly compile images and anecdotes from the experiences of the correspondents.

› CARD SORT

• › How On separate cards, name possible features, functions, or design attributes. Ask people to organize the cards spatially, in ways that make sense to them.

• › Why This helps to expose people’s mental models of a device or system. Their organization reveals expectations and priorities about the intended functions.

• › Example In a project to design a new digital phone service, a card-sorting exercise enabled potential users to influence the final menu structure and naming.

2026

ITPDP

AARHUSUNIVERSITY TRY

• › Create simulations and prototypes to help empathize with people and to evaluate proposed designs.

› EMPATHY TOOLS

• › How Use tools like clouded glasses and weighted gloves to experience processes as though you yourself have the abilities of different users.

• › Why This is an easy way to prompt an empathic understanding for users with disabilities or special conditions.

• › Example Designers wore gloves to help them evaluate the suitability of cords and buttons for a home health monitor designed for people with reduced dexterity and tactile sensation.

› SCENARIOS

• › How Illustrate a character-rich storyline describing the context of use for a product or service.

• › Why This process helps to communicate and test the essence of a design idea within its probable context of use. It is especially useful for the evaluation of service concepts.

• › Example Designing a community Web site, the team drew up scenarios to highlight the ways particular design ideas served different user needs.

› NEXT YEAR’S HEADLINES

• › How Invite employees to project their company into the future, identifying how they want to develop and sustain customer relations.

• › Why Based on customer-focused research, these predictions can help to define which design issues to pursue for development.

• › Example While designing an Intranet site for information technologists, the team prompted the client to define and clarify their business targets for immediate and future launches.

› INFORMANCE

• › How Act out an “informative performance” scenario by role-playing insights or behaviors that you have witnessed or researched.

• › Why This is a good way to communicate an insight and build a shared understanding of a concept and its implications.

• › Example A performance about a story of mobile communications shows the distress of a frustrated user.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

REMEMBER THE EXTREMES

› ”Extreme characters”

Example

• › Extremes in IT design for the home

• › The homeless living in a shopping cart

› The film actor with uniformly decorated apartments in New York, Paris, Tokyo and LA

==> picture [47 x 40] intentionally omitted <==

==> picture [344 x 273] intentionally omitted <==

2026

ITPDP

IDEATION

AARHUS UNIVERSITY

IDEO - IDEA GENERATION

• › 8-10 participants – responsible for documentation appointed

• › 50-100 ideas in an hour

• › Rules

• › No critical assessments

• › Stimulate wild ideas

• › Build on other people's ideas

• › Stay focused on the topic

• › Hold on to one "thread" at a time

• › Really good ideas can stop the process and restart it somewhere new

• › Ideas are taken over into an "envisionment" activity, where it is visible and tangible

›

2026

ITPDP

AARHUS UNIVERSITY

FUTURE WORKSHOPS

› Critique phase

› Brainstorming problems in current practice

› No discussion – just get problems on the board › Group issues and prioritize importance

› Fantasy phase

› Brainstorm wild/utopian ideas (that can solve the problems identified)

› No discussion – just get ideas on the board

› Group ideas and prioritize them in terms of value creation

› Realization phase

› Take the high-priority ideas › Delimit to realistic visions

• › Prepare concrete proposals for realization

==> picture [47 x 40] intentionally omitted <==

(Jungk & Müllert, 1987;Kensing & Halskov, 1991)

2026 21

ITPDP

THE ROLE OF THE DESIGNER & THOUGHTFUL INTERACTION DESIGN

Löwgren and Stolterman

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

Places the designer at the core of the process

L&S argue that the responsibility for the vision at the designer (p.34ff)

L&S argue that the responsibility for the design process is at the designer (p.38ff)

L&S argue that the designer should engage and manage the relations in the design process (p.32ff).

==> picture [241 x 190] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN Designing the design process

› Design starts earlier than project owners may think

› Select appropriate methods/techniques

• › Pay attention to and care for a common vision

• › Pay attention to roles and stakeholders

==> picture [240 x 190] intentionally omitted <==

• › Pay attention to design as a project

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

Divergence

“Designer expands her thinking to cover broader issues, find alternatives, and explore more opportunities” (L&S, p. 29)

Convergence

“Convergence is about focusing on a specific solution or a synthesis of several ideas” (Ibid.)

• What is the primary issue?

• • Who to involve and how?

• How would the shape look like?

• • What is the interaction modality?

• • What kind of feedback it give?

==> picture [79 x 76] intentionally omitted <==

Design choices

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

• › Vision : The first organising principle that help the designer respond to the situation at hand

• › Operative Image : The first (and consecutive) externalisations of the vision

• › Specification : The final “design”

==> picture [214 x 202] intentionally omitted <==

• specification

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

› Leaping between detail and the whole

• › Focusing on dilemmas in the domain

• › Alternatives and contraditions

• › Get the dilemmas and

trade offs on the table early in the Vision activity

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

INVOLVING USERS

AARHUS UNIVERSITY

PERSPECTIVES ON PEOPLE AND MACHINES

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

USER INVOLVEMENT

• None, very little, and/or only at the end

• User-centred design

• Participatory design

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

USER-CENTERED DESIGN

• Involvement of users in all parts of the design process

• Focus groups for ideation

• Evaluation of low-fidelity prototypes

• Evaluation of new features through AB testing and interviews

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

PARTICIPATORY DESIGN

• More radical approach to user involvement than user-centred design

• Users as direct design partners and active first-class members of the product design team

• Developed in Scandinavia in the 70s and 80s (Aarhus University was a key player)

• Methodology developed laid the foundation for user-centred design

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY UTOPIA PROJECT

Early participatory design project

• Alliance between typesetters union and IT researchers

• How to empower instead of replace typesetters with computers

• Design of computer systems based on the people on the shop floor rather than the management

• Introduced low-fi prototyping in systems design

==> picture [47 x 40] intentionally omitted <==

==> picture [220 x 99] intentionally omitted <==

==> picture [220 x 149] intentionally omitted <==

ITPDP 2026

Morten Kyng & Susanne Bødker

AARHUS UNIVERSITY

PARTICIPATORY DESIGN TECHNIQUES

• › Ethnographic field studies

• › Observations, interview and video analysis

› "Fictional inquiries”

• › Playful analysis in a fictional setting

• › Structured brainstorming

• › Future Workshop, Metaphorical Design, Inspiration Cars, Organizational Games

• › Scenarios

› Descriptions, tableau, video

› Mock-ups

• › Physical models, paper windows

==> picture [155 x 114] intentionally omitted <==

==> picture [155 x 110] intentionally omitted <==

==> picture [155 x 100] intentionally omitted <==

› Video prototyping

› Stop-motion, blue studio techniques

• › Prototyping

• › Exploratory, experimental, evolutionary, cooperative

==> picture [23 x 8] intentionally omitted <==

----- Start of picture text -----
2026
----- End of picture text -----

ITPDP

AARHUS UNIVERSITY

INTERNATIONAL BOOKS AND ARTICLES ON SCANDINAVIAN PARTICIPATORY DESIGN

• › Bødker, S., Grønbæk, K., & Kyng, M. (1995). Cooperative Design: Techniques and Experiences from the Scandinavian Scene. In R. M. Baecker, J. Grudin, & W. A. S. Buxton (Eds.), Readings in Human-Computer Interaction: Toward the Year 2000 . San Francisco: Morgan Kaufmann Publishers, Inc., 215-224.

• › G. Bjerknes, P. Ehn, & M. Kyng (Eds.) (1987) Computers and Democracy . Aldershot: Avebury. › Greenbaum, J., & Kyng, M. (1991). Design at Work: Cooperative Design of Computer Systems . Hillsdale, NJ: Lawrence Erlbaum Associates.

• › D. Schuler & A. Namioka (Eds.) (1993) Participatory Design: Principles and Practices . Hillsdale, New Jersey: Lawrence Erlbaum Associates, 157-175.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

==> picture [471 x 266] intentionally omitted <==

==> picture [214 x 282] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

36

AARHUS

UNIVERSITY

==> picture [47 x 40] intentionally omitted <==

ITPDP

2026

37

PROJECT MANAGEMENT

AARHUS UNIVERSITY

UNCERTAINTY ABOUT THE PRODUCT SHOULD BE REDUCED

==> picture [449 x 178] intentionally omitted <==

----- Start of picture text -----
Now Deadline Exam
----- End of picture text -----

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

PROJECT MANAGEMENT: SYSTEMATIC APPROACH

==> picture [14 x 6] intentionally omitted <==

----- Start of picture text -----
Now
----- End of picture text -----

==> picture [113 x 6] intentionally omitted <==

----- Start of picture text -----
Deadline Exam
----- End of picture text -----

==> picture [465 x 39] intentionally omitted <==

==> picture [436 x 77] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

PROJECT MANAGEMENT

Project management

› Focus on the project, starting point, purpose, budget etc.

› Focus on deadlines, deliveries, quality etc.

› Focus on progress, evaluation, success/failure Leadership

› Focus on competencies and roles

› Focus on performance and well-being

• › Focus on the team over time (and more projects)

• Self-management

• › Focus on your own tasks, satisfaction, prioritization, progression!

›

2026

ITPDP

AARHUS UNIVERSITY

PITFALLS

› Technical Rationality (Gedenryd 1998)

› Believe that you can follow a linear process

• › Optimistic estimation (Brooks 1975)

› Software is highly malleable compared to other materials

• › Brooks Law (Brooks 1975)

• › Believe that you can finish faster by putting more people on a project

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY ILLUSION OF TECHNICAL RATIONALITY

• Most straightforward model of a project • Most projects to some degree or the other follows this model • Pitfall • Paralysis by fear of wrong requirements can halt the process • Mistakes are expensive too fix late in the process

==> picture [47 x 40] intentionally omitted <==

Schön 1987; Gedenryd 1998

2026

ITPDP

AARHUS UNIVERSITY

AGILE INTERACTION DESIGN / AGILE DEVELOPMENT

• Break design process down in small iterations each involving all phases

• Iteratively develop software in working (and deployable) increments

• The software is never finished (for good … and for ill)

• Affords extensible software architectures that enables rapid prototyping of new features

==> picture [47 x 40] intentionally omitted <==

==> picture [520 x 146] intentionally omitted <==

----- Start of picture text -----
ITPDP
----- End of picture text -----

2026

AARHUS UNIVERSITY

MAKE A GOOD PLAN WITH ROOM FOR ERRORS AND ITERATIONS

==> picture [14 x 6] intentionally omitted <==

----- Start of picture text -----
Now
----- End of picture text -----

==> picture [113 x 6] intentionally omitted <==

----- Start of picture text -----
Deadline Exam
----- End of picture text -----

==> picture [465 x 39] intentionally omitted <==

==> picture [436 x 77] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

ETHICS

AARHUS UNIVERSITY

WHY DO WE EVEN TALK ABOUT ETHICS

› We build things

› … that affects people’s lives

• › … potentially a lot of people

• › ... that change their perspectives on things

• › … even their possibilities of action, self-understanding and daily life

==> picture [145 x 217] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

UNETHICAL TECHNOLOGY?

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

UNETHICAL TECHNOLOGY?

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

52

AARHUS UNIVERSITY

THREE SCHOOLS OF ETHICS

› Consequentialism (da: nytteetik )

› Cares for consequences: “The truth can hurt”

› Deontology (da: pligtetik )

• › Cares for rules: “You must not lie”

› Virtue ethics (da: dydsetik )

› Cares for principles: “I always tell the truth”

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

ETHICS

Verbeek’s Materializing Morality

AARHUS UNIVERSITY

VEERBEEK’S CLAIM

› ‘If technology mediates how we perceive and act in the world, it can also be designed to mediate perception and action in ethical or unethical ways.’

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

55

AARHUS UNIVERSITY

MEDIATION OF PERCEPTION

• › Simple : Me -> World

› Mediated : Me -> Technology -> World

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

56

AARHUS UNIVERSITY

==> picture [279 x 241] intentionally omitted <==

==> picture [314 x 241] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

57

AARHUS UNIVERSITY

MEDIATION OF ACTION

• › Inscription

› ‘The designer, who can be seen as the inscriber of scripts.’ › When we design

• › Scripts

› The influence of artifacts on human actions is a “script”

› Typical patterns of action

• › Translation

• › To new (or less) action possibilities (e.g., citizen+gun)

› Typical(/possible) outcomes

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

58

AARHUS UNIVERSITY

MEDIATION OF ACTION - EXAMPLES

==> picture [47 x 40] intentionally omitted <==

==> picture [216 x 295] intentionally omitted <==

2026

ITPDP

59

AARHUS UNIVERSITY

MEDIATION OF ACTION - EXAMPLES

==> picture [612 x 156] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

60

ETHICS

Dark Patterns

AARHUS UNIVERSITY

DARK PATTERNS

• › Basic assumption that UX features can be linked to similar user behavior

• › (Dark) patterns as a way to describe design → ‘scripts’ (cf Verbeek)

• › Pattern use suggests a causal relationship between intention → feature → behavior

• › Gray et al paper

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

62

AARHUS UNIVERSITY

DARK PATTERNS

==> picture [660 x 160] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

63

AARHUS UNIVERSITY

DARK PATTERNS

==> picture [331 x 261] intentionally omitted <==

==> picture [234 x 175] intentionally omitted <==

==> picture [276 x 129] intentionally omitted <==

2026

ITPDP

64

AARHUS UNIVERSITY

DARK PATTERNS & SOCIAL MUSIC THEME?

==> picture [306 x 305] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

65

GDPR

AARHUS UNIVERSITY

GDPR

• › General Data Protection Regulation is an EU regulation aimed at strengthening and harmonising the protection of personal data in the European Union.

• › Must protect the individual's rights and processing of personal data – consent, security, the right of access and the right to be forgotten, etc.

• › It is something we must relate to when we involve others than ourselves in the design

• process

• › Until now, you have mostly been the 'data subject' – now you will potentially also be data responsible!

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

67

AARHUS UNIVERSITY

GDPR RULES

• › Consent must be clearly obtained independently of other requests.

› Consent must be obtained with clear information about scope, purpose, responsibility and contact persons

• › In case of security breach, participants must be informed no later than 72 hours after discovery

• › The right to be forgotten must be implemented as a procedure in the process

• › A responsibility to be taken seriously ( but no need for further concern )

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

68

AARHUS UNIVERSITY

GDPR IN STUDENT PROJECTS

• › https://studerende.au.dk/en/it-support/information-security/data-protectiongdpr/projects

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

69

AARHUS UNIVERSITY

DOCUMENTS

• › Consent statement is used to obtain consent from participants – customize template as needed

• › The register of purposes is used to explain the purpose of the data collection

• › Data responsibility is used, as a group, to enter into an internal agreement on joint data responsibility

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

70

AARHUS UNIVERSITY

DOCUMENTS

› You are responsible for the preparation of the documents

› You are responsible for storing the documents

› You are responsible for the storage of data and GDPR

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

71

AARHUS UNIVERSITY

TIPS

• › Just get it done and learn that it is part of the study and our practice

• › Don't collect data you don't know what you need for (sensor data?)

• › Try to anonymize and 'get away from' data as early as possible (Clemens → Respondent M1)

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

72

AARHUS UNIVERSITY

WHAT CAN WE HELP WITH?

› Read through the documents when they are finished (to help)

› Answer questions

› Not so much more – it's agreements and your responsibility

› TA session where you’ll look at it!

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

73

==> picture [97 x 41] intentionally omitted <==

EVALUATION ITPDP’26, L13, W15/19

Assoc Prof. Minna Pakanen Department of Digital Design and Information Studies mpakanen @cc.au.dk

6[TH] MAY 2026

MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

TODAY

› Ways of evaluating

• › Usability testing

• › Early development phase UX evaluations

• › Prototypes in early development phase evaluations

• › Data gathering and recording methods

• › Evaluation planning and roles in the evaluation

• › TA session: Planning of the evaluation of your own ITPDP product!

6[TH] MAY 2026

MINNA PAKANEN

WAYS TO EVALUATE

6[TH] MAY 2026

MINNA PAKANEN

4

WHY DO WE EVALUATE?

› We are not designing for ourselves › Feedback on the design process

› Iterative development › ”Fail early, fail often"

6[TH] MAY 2026

MINNA PAKANEN

“Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process, you are going to throw buckets of money down the drain .”

(Bruce Tognazzini)

6[TH] MAY 2026

MINNA PAKANEN

THREE WAYS TO EVALUATE

Usability testing/ UX evaluation

• Test and evaluation setting controlled by the evaluator

• With users

Field evaluation

• Evaluation in a natural environment with real users and their actions

• The user’s surroundings and tasks set the evaluation frame

Analytical evaluation

• No users are involved!

• Heuristic evaluation

You will learn this in the coming HCI course!

• Walkthroughs

Combination of these!

6[TH] MAY 2026

MINNA PAKANEN

6[TH] MAY 2026 MINNA PAKANEN

USABILITY TESTING PLEASE DO NOT USE THIS IN YOUR EVALUATION!

6[TH] MAY 2026

MINNA PAKANEN

11

AIMS OF USABILITY TESTING

• › Interested of how easy it is for the user to use the system

• :

• › Usability is defined by 5 quality components

• Learnability : How easy is it for users to accomplish basic tasks the first time they encounter the design?

• Efficiency : Once users have learned the design, how quickly can they perform tasks?

• Memorability : When users return to the design after a period of not using it, how easily can they re-establish proficiency?

• Errors errors : How many do users make, how severe are these errors, and how easily can

• they recover from the errors?

• Satisfaction : How pleasant is it to use the design?

• (https://www.nngroup.com/articles/ten-usability-heuristics/)

6[TH] MAY 2026

MINNA PAKANEN

USABILITY TESTING

› Three ways of conducting:

• Laboratory-based user observations

• Controlled user testing

• › facilitator can help the user if the user does not know how to proceed, but this needs to be marked down in the observation notes!

• Expert inspection techniques

• › Analytical evaluation methods you will be learning more in the HCI course

(Greenberg & Buxton, 2008)

6[TH] MAY 2026

MINNA PAKANEN

USABILITY TESTING

› Controlled laboratory-based user (covert) observations & studies

==> picture [405 x 194] intentionally omitted <==

==> picture [284 x 194] intentionally omitted <==

==> picture [257 x 194] intentionally omitted <==

https://www.brighton.ac.uk/business-services/consultancy/user-centre-design-lab/index.aspx https://www.volkside.com/2009/12/tip-print-out-your-usability-testing-tasks/ https://www.testingtime.com/en/blog/in-house-usability-tests/ https://ux247.com/usability-lab-dead/

==> picture [224 x 49] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

What are the challenges of usability evaluations conducted in an early design phase according to Greenberg & Buxton (2008)?

6[TH] MAY 2026

MINNA PAKANEN

EARLY DESIGN IDEAS AND PROTOTYPES

• › Early designs are “sketches”

• Illustrate the essence of an idea, but have

• many underdeveloped aspects to it

• Can be represented in many forms

• Work as externalization of the idea and help the designer to iterate the design

• › Early prototypes

• Have many holes and undeveloped attributes

(Greenberg & Buxton, 2008)

==> picture [348 x 311] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

CHALLENGES OF USABILITY STUDIES IN EARLY DESIGN PHASE

› Usability study is not the best approach to evaluate early designs or prototypes because

• It focuses on negative aspects:

• Errors & bugs

• Task completion time: faster the better

• –> Abandoning a promising idea too early –> Focus on developing aspects that can be measured, usually something we are already familiar with

• –> Limit the number of alternative ideas

==> picture [392 x 273] intentionally omitted <==

(Greenberg & Buxton, 2008)

6[TH] MAY 2026

MINNA PAKANEN

EARLY PHASE USER EXPERIENCE EVALUATIONS

6[TH] MAY 2026

MINNA PAKANEN

19

USER EXPERIENCE EVALUATION

• › UX studies focus on lived experiences

• A holistic view of the user’s interaction with a product

  • Emotions, enjoyment & aesthetics

• Focuses on positive aspects of the use

  • Hedonic and non-instrumental aspects of use (non-task related)

• Dynamic and situational aspects

  • Different time spans (before, during, after the use)and momentary experiences (changes over time spans)

  • Context impacts the experience

(Bargas-Avila & Hornbæk, 2011)

6[TH] MAY 2026

MINNA PAKANEN

AIMS OF UX EVALUATIONS

• › To find out how the user feels about the system and interaction

• with it

• Qualitative or mixed-methods approach in research

  • Interviews, focus groups, observations

  • Questionnaires

  • Constructive and creative techniques (drawing & collages)

• UX studies can be conducted at any phase of the design process (concept ideas – – – –>final products)

(Bargas-Avila & Hornbæk, 2011)

6[TH] MAY 2026

MINNA PAKANEN

EARLY DEVELOPMENT PHASE UX EVALUATIONS

• › Early phase user experience evaluations are done to

• › help to choose the best design for the development

• › Evaluating that the development is on the right track

• › Examining if the final product can meet the set UX targets

(Stone et al. 2005)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED USER EXPERIENCE (AUX)

can ”Anticipated UX happens before the first use, or it happen also within the other time spans of UX, e.g. during and after the use and over time, as a person may imagine also during those time spans. A person can have indirect experience prior the first use through formed expectations brand of existing experience with related technologies, , advertisements , presentations, and other peoples’ opinions.”

(Roto et al., 2011)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED EXPERIENCE

”Anticipated user experience means the experiences and feelings that are expected to occur when the user is imagining using an interactive product.”

(Yogasara et al., 2011)

6[TH] MAY 2026

MINNA PAKANEN

PROBLEMS WITH THIS DEFINITION?

==> picture [467 x 380] intentionally omitted <==

----- Start of picture text -----
?
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

ANTICIPATED USER EXPERIENCE (AUX)

Users should not be put in a situation where they have to imagine their future needs without giving . concrete options for them

(Von Hippel, 1986)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED USER EXPERIENCE (AUX)

” needs wishes that result from Experiences, , and anticipated interaction with a concept of the product before the actual product exists.”

(Pakanen, 2015)

6[TH] MAY 2026

MINNA PAKANEN

PROTOTYPES IN EARLY DEVELOPMENT PHASE UX EVALUATIONS

6[TH] MAY 2026

MINNA PAKANEN

29

PROTOTYPES IN AUX EVALUATIONS

• › Prototypes should

• › evoke people’s dreams for the future (van den Hende 2010)

• › give a sense of experience before the actual artefact exists (Roto et al. 2011)

• › allow envisioning the concept and prevent unwanted confusion

  • (Kuutti et al. 2001, Gegner & Runonen 2012)

• › focus subject’s attention on studied things (Lim & Stolterman 2008)

• › allow utilizing concepts key characteristics directly to the design (Law 2011).

6[TH] MAY 2026

MINNA PAKANEN

MIXED FIDELITY PROTOTYPES

5 filter dimensions of prototypes

• Appearance

• size; color; shape; margin; form; weight; texture; proportion; hardness; transparency; gradation; haptic; sound

• The functionality

• system function; users’ functionality need

• The interactivity

• input behavior; output behavior; feedback behavior; information behavior

• The data

• data size; data type (e.g., number; string; media); data use; privacy type; hierarchy; organization

• The spatial structure

• arrangement of interface or information elements; relationship among interface or information elements— which can be either two- or three-dimensional, intangible or tangible, or mixed

(Lim & Stolterman, 2008)

6[TH] MAY 2026

MINNA PAKANEN

MANIFESTATION DIMENSIONS

==> picture [484 x 311] intentionally omitted <==

(Lim & Stolterman, 2008)

6[TH] MAY 2026

MINNA PAKANEN

FILTER-FIDELITY-PROFILES IN EVALUATION

5 filter-dimensions of prototypes

• Appearance

• Size, color, shape, weight, hardness, haptic, sound, & arrangement

• The functionality

• Breadth (functions realized) & depth (completeness)

==> picture [163 x 125] intentionally omitted <==

• The interactivity

• Action, reaction, input modality, & output modality

• The data

• Closeness to the reality, information architecture, data model, & amount and type of data

• The physicality

• Spatial position, coherence of tangibles, & tangible embodiment

(Kohler & Hochreuter, 2014)

==> picture [163 x 124] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

PROTOTYPES IN EARLY UX EVALUATIONS

==> picture [450 x 345] intentionally omitted <==

==> picture [503 x 346] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

COMPARABLE VISUAL MATERIALS IN AUX EVALUATIONS

Extending the existing prototype and comparing alternative visualization styles for GUIs When preparing comparable visual materials, make sure the comparable examples and items under evaluation are same in: 1) Relative size

• 2) Color scheme, unless you compare different color options

3) Level of detail and visualization style (sketch, line drawing, shaded, …)

Note that your alternative visualizations should all be presented in same application context (contextualization).

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE: VIRTUAL AVATARS FOR AR & VR IN COLLABORATIVE SETTING

==> picture [832 x 352] intentionally omitted <==

(Pakanen et al. 2022)

6[TH] MAY 2026 MINNA PAKANEN

EXAMPLE: RESEARCH/ BENCHMARKING

(Pakanen et al. 2022)

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE: EXAMPLES

• 1) Relative size

• 2) Color scheme

• 3) Level of detail & visualization style

==> picture [455 x 466] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

(Pakanen et al. 2022)

EXAMPLE: CONTEXT

Examples are shown both in AR & VR context

(Pakanen et al. 2022)

6[TH] MAY 2026

MINNA PAKANEN

COMPARABLE VISUAL MATERIALS

==> picture [722 x 370] intentionally omitted <==

----- Start of picture text -----
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

(Pakanen 2015)

PROTOTYPES IN EARLY UX EVALUATIONS

==> picture [642 x 321] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026 MINNA PAKANEN

VISUAL MATERIALS TO HELP EVALUATING PROTOTYPES

==> picture [422 x 299] intentionally omitted <==

==> picture [531 x 301] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the study in your report:

› How the visual materials were used?

› When they were used in the evaluation?

6[TH] MAY 2026 MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 1

Paula and her husband p lanned in the morning that she will buy groceries, and her husband will collect her from the shop. Hence, now prior leaving from the office, she pre-writes a message on her phone “At the checkout desk!” (1) and assigns it to a t urquoise color that she can remember.

When she arrives at t he checkout, she uses the bracelet device on her wrist to browse the messages that she has created with the phone (2). She finds the turquoise message (with inner ball) that she created at her office.

Then she selects the receiver, Matt, her husband (with outer ball) and sends it to him (3).

Matt is arriving to the parking lot, when Paula’s message arrives to his phone, he knows that it will only take a few minutes for Paula to come out of the shop, so rather than parking, he drives to waits at the front of the store (4).

==> picture [434 x 309] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

• Help user to see over limitations of the prototype by creating a model to show the appearance

• A nice rending/ visualization of the concept idea in use context held by a human (print/screen)

• A physical unfunctional mock-up (can be hold in hand by the user)

• A set of images + use case story (prints/ Powerpoint slides/video)

• Make sure you show this use case story before evaluating the partial and unfinished appearance (in this case size, weight, shape, materials), functionality (only two types of interactions were compared [interactivity]) or the prototype.

• Think of what aspects you need to realize as close to the final prototype (physicality as in this case tangible embodiment was almost realistically realized)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 2

Collect feedback for the further development

• 2 Altered visualizations of possible solutions presented on screenshots taken from the model

• Make sure you show the alternative vualizations after evaluating the partial and unfinished prototype

• appearance (no visualizations)

• functionality (only moving was realized in the prototype).

• Think of what aspects you need to realize as close to the final prototype (Appearance and the animations)

==> picture [199 x 393] intentionally omitted <==

(Kukka et al. 2017)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 4 Collect feedback for the further development

• One virtual environment with alternative interior and functionalities

• Make sure you let the participants see the alternative visualization after evaluating the partial and unfinished prototype

• appearance (no visualizations)

• functionality (only moving was realized in the prototype).

• Think of what aspects you need to realize as close to the final prototype (Appearance, functionalities, and the interactions)

==> picture [539 x 302] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

(Pakanen et al. 2020)

WIZARD OF OZ

6[TH] MAY 2026

MINNA PAKANEN

48

WIZARD OF OZ TECHNIQUE ALLEN MUNRO AND DON NORMAN, 1975

› The Wizard of Oz method is a moderated research method in which a user interacts with an interface that is not really working, but the system responses are initiated by a human operator

==> picture [817 x 236] intentionally omitted <==

----- Start of picture text -----
Video
camera User
Operator
https://www.nngroup.com/articles/wizard-of-oz/#:~:text=Definition%3A%20The%20Wizard%20of%20Oz,Norman%20at%20UC%20San%20Diego.
----- End of picture text -----

6[TH] MAY 2026

MINNA PAKANEN

WIZARD OF OZ TECHNIQUE AN EXAMPLE IN LAB SETTING

==> picture [385 x 267] intentionally omitted <==

==> picture [565 x 205] intentionally omitted <==

(Colley et al. 2016)

6[TH] MAY 2026

MINNA PAKANEN

WIZARD OF OZ TECHNIQUE

AN EXAMPLE IN FIELD WITH A MOBILE OPERATOR

==> picture [697 x 352] intentionally omitted <==

Pakanen et al. 2022

==> picture [44 x 6] intentionally omitted <==

----- Start of picture text -----
6 [TH] MAY 2026
----- End of picture text -----

==> picture [56 x 7] intentionally omitted <==

----- Start of picture text -----
MINNA PAKANEN
----- End of picture text -----

Reporting of the study in your report:

› What kind of Wizard of OZ evaluation was conducted?

› How was the Wizard of OZ method described in the paper? Paper: 3

MINNA PAKANEN

6[TH] MAY 2026

==> picture [377 x 326] intentionally omitted <==

----- Start of picture text -----
User
< 2m
----- End of picture text -----

WIZARD OF OZ TECHNIQUE MOBILE

==> picture [87 x 87] intentionally omitted <==

Operator

6[TH] MAY 2026

MINNA PAKANEN

PLANNING OF WIZARD OF OZ STUDY

Remember

› Assign group members to be:

• 1) Facilitator, introduces the tasks and interviews the participants

• 2) Operator/”observer” who operates the protype based on participants input/ other type of input. Make sure there is either direct or indirect (through live video feed) visibility to participants interaction with the

prototype with, so that you can operate the prototype well. If the operator is to be in the same space, then say to participants that this person is an observer

› Do not break the illusion!

• › Plan the tasks carefully and think all the possibilities what the user could do?

• If participant does unexpected things, facilitator should say: unfortunately the prototype does not function in that way, can you think any other way how it could work? (and at the end show how to use it).

• › Make sure the operator have a direct view to the participant and their interaction with the prototype

6[TH] MAY 2026

MINNA PAKANEN

USER STUDY DATA GATHERING METHODS

Blandford, Ann, Dominic Furniss, and Stephann Makri. ”Qualitative HCI research: going behind the scenes." Synthesis lectures on human-centered informatics 9, no. 1 (2016): 1-115.

6[TH] MAY 2026

MINNA PAKANEN

55

OBSERVATION

Observer observes

• › ”Fly on the wall” (Blandford et al. 2016)

• › What do people do and how do they do it?

==> picture [156 x 155] intentionally omitted <==

• › Difficulties or errors with the interaction?

• › What do they get enthusiastic about?

• › How do they operate the system?

• › Are their actions supporting what they say?

• › e.g., if they seem to have problems and they claim it is easy to use

6[TH] MAY 2026

MINNA PAKANEN

PLANNING OBSERVATION

Pay attention to

• › Select setting(s)

› It is not possible to observe everything, so decide what is to be documented in each observation.

› Plan an observation form for collecting observations for each task › When a participant does something very interesting, you can mark down the time, which helps finding that incident from the video or audio recording.

6[TH] MAY 2026

MINNA PAKANEN

THINK ALOUD

› Technique can be used in usability testing, user experience evaluation, and even in field studies › Start by explaining to the participant how to think aloud › It is not about what they do, but what they think while doing it › The tasks: chosen by participants (naturalistic) or defined by you › Thinking aloud does not come naturally to all

› You can prompt silent participant by asking: “What are you thinking” › You should politely steer too chatty participant back to the tasks › UX and usability studies have different rules

› Intervention in minimum (usability) | UX interventions for seeking clarifications

6[TH] MAY 2026

MINNA PAKANEN

INTERVIEW

› allows understanding people's perceptions and experiences

==> picture [401 x 314] intentionally omitted <==

› Different types

› Unstructured

• › not directed by a script (ethnographic)

• › Structured

• › tightly scripted, almost like a questionnaire

• › Semi-structured (most used)

• › guided by a script, interesting issues explored in more depth by asking for further details

MINNA PAKANEN

6[TH] MAY 2026

INTERVIEWING

The interviewer interviews the participant/s › Form the topics and questions: open, broad & narrow

• › Next slide what to avoid when forming questions

• › Opening the conversation

• › assure your interest in participants’ replies as an expert in the topic

› Ask the question and wait that the person gives a reply › Ask for clarifications if a person’s reply is not complete or if it is just yes or no/ good/bad

› You said that it is good, in what way/s it is good/ what features make it good?

You wished the information on the screen was clearer, in which way?

6[TH] MAY 2026 MINNA PAKANEN

INTERVIEW QUESTIONS

When creating questions avoid:

› Long questions

• › Compound sentences - split them into two

• › Jargon and language that the interviewee may not understand › What did you think of this AR feature? –> ….the feature where you can see items appearing on this real space we are in?

• › Leading questions that make assumptions

› Isn’t this feature good in…

• › Unconscious biases e.g., gender/age… stereotypes

• › As a/n elderly/woman/child you probably found this as complicated…

6[TH] MAY 2026

MINNA PAKANEN

DATA RECORDING

6[TH] MAY 2026

MINNA PAKANEN

62

VIDEO RECORDING

Pros and cons

› A rich way to capture interaction in the context + user comments are automatically in sync with user interaction

› Video recording can make participant anxious

› Placing the video camera in a more discrete location usually helps users to forget it

==> picture [86 x 86] intentionally omitted <==

==> picture [405 x 348] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

AUDIO RECORDING

==> picture [74 x 73] intentionally omitted <==

When and why

• › In some of the interviews it is fine to record audio only

• If you do not have any prototypes or visual materials, an audio recording is more discrete than a video.

› It is good to also record audio when you use video recording as sometimes it is hard to hear from the video what the user is saying especially if the camera is located far away from the user

6[TH] MAY 2026

MINNA PAKANEN

PHOTOGRAPHS

Records the moment › Capture moments as they happened. They help in reporting results and important moments in the interaction for example in project reports and publications

==> picture [73 x 73] intentionally omitted <==

==> picture [423 x 375] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

OBSERVATION NOTES

==> picture [80 x 81] intentionally omitted <==

Fast in situ note taking

› Observation notes are a good way of getting the most important things marked down, you can even mark the time when a user did or said something that was unexpected or interesting in some other way

› Notes can be used also as the basis of analysis (i.e., themes)

6[TH] MAY 2026

MINNA PAKANEN

LOG DATA

==> picture [74 x 73] intentionally omitted <==

Recording user interaction on the device › records user actions in the device log

› You can find errors and quantify your results easily with log data, but remember that you need the before-mentioned methods to understand why people made mistakes or did things in certain ways

6[TH] MAY 2026

MINNA PAKANEN

UX EVALUATION AND DATA CAPTURING METHODS

==> picture [709 x 475] intentionally omitted <==

----- Start of picture text -----
https://experienceresearchsociety.org/ux/evaluation-methods/
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

~~R~~ eporting of the study in your report: › What kind of data collection and recording methods were used?

› Were there some methods that were not described here?

6[TH] MAY 2026

MINNA PAKANEN

EVALUATION PLANNING TASKS, PROCEDURE, & INTERVIEW QUESTIONS

6[TH] MAY 2026

MINNA PAKANEN

78

PLANNING

1. Prepare procedure – steps, tasks…”script of the evaluation” 2. Prepare interview (+questionnaires) questions – background, task related, end

2. Pilot study – try out your procedure, tasks, forms, questions & how long the evaluation will be

3. Iterate your procedure – if something needs to be changed 5. 2[nd] Pilot – test if changes work better

4. Recruit participants – amount depends of the study, from 6-14-30-…, set also times for the studies and let them know the estimated duration

5. Prepare for the evaluation – cameras, video cameras, tripods, audio recording, print forms, get gifts for the participants, book rooms…

6. Conduct your study- follow your procedure, record the data and store it for analysis, remember to thank your participants after the evaluation! 6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE Applied from Leena Arhippainen, UX researcher and Minna’s Phd supervisor

6[TH] MAY 2026

MINNA PAKANEN

80

PROCEDURE

1. Consent from filling- participant agrees on the study (See L3: GDPR)

2. Intro to the study- who, why and what

3. Background info of the users- questionnaire or interview

4. Ice breaking task – easy task to get user familiar with the prototype 5. Actual task(s) with the prototype – design the task so that you are able to get feedback on right things

5. Post questionnaires- preferences and quantitative data

6. Semi-structured interview – get more detailed understanding how user felt using your prototype and what s/he thought about the design idea in general 6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE

Example of procedure:

• Everything is explained so it is easy to follow in the actual evaluation

1. Consent & background questionnaire form filling 2. Who are we and explanation of the purpose of the study (even if explained in the consent form in detail)

2. Actual tasks are written down so that each participant is instructed in a similar way

3. Questionnaire fillings are marked in the procedure 5. Interview questions for post-interview (can also be on different paper)

4. Thank and reward the participants

==> picture [348 x 492] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE

2. Introduction to the study

• Who are we

• Short description of the study aim

• Tell the user once more that they can withdraw from the study

  • whenever they feel like it

• Ask the user to think aloud during the tasks (+how to do it)

• In Usability evaluation: explain if you will not help the participant during the tasks

• Ask if the user has any questions

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

2. Introduction to the study

MINNA PAKANEN

6[TH] MAY 2026

Reporting of the study in your report:

› Was there differences in the example procedure that was given to your group?

MINNA PAKANEN

6[TH] MAY 2026

PROCEDURE

3. Background information

• What do you need to know about your user?

  • Age

  • Gender: male/female/nonbinary

  • Educational background/ profession

  • Prior experience with similar technology that you are evaluating

  • Prior experience with the use context (e.g., fab labs, pottery, chemical lab, etc.) of your system

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

3. Background questionnaire

Note that you do not necessarily need this, especially if you have only a few things to ask. The questions can also interviewed in the beginning or at the end of the evaluation!

==> picture [353 x 457] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

Reporting of the study in your report:

› How the participants were described in the paper, and what kind of information was given about them? › What way the description was written?

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

4. Icebreaking task

• Idea is to familiarize your user with the thing under evaluation

• Easy and short task

• Reduces nervousness of the participant

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

4. Icebreaking task

• Explore the prototype freely for a while

1. Familiarization with the current 3D City model

1. Familiarize yourselves with the 3D City model simply by looking at it and giving your initial thoughts before interacting with it.

2. Now you can start interacting with the model by controlling an avatar (WASD and arrows) and explore the model freely. While exploring the 3D model, please think aloud and comment on anything that caught your attention.

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

5. Actual task(s)

• The idea is to plan tasks that the user can do and by doing the tasks you will learn if your product is understandable for the user or how they do experience the prototype

  • Try to keep them short and not too complicated to conduct

  • Test how your core use case activities are working

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

5. Tasks

1st task:

1. First you want your handbag to match with these red shoes over here, what would you do?

2. Next you can try to make the bag imitate the fabric and color of this of this scarf here.

3. And then this shirt here.

Questions after the tasks are completed What did you think of this idea? Would it be useful for you?

Is the idea fun?

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

6. post questionnaire

• Sometimes you might need user feedback on different designs so you can use certain questionnaires to help assess different things.

• The attractiveness of the product: Attrakdiff https://www.attrakdiff.de/index-en.html (online service was discontinued from January 2025 L)

• Can be used for

• Single evaluation

• Comparison

• Before-after

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

6. Post questionnaire With Attrakdiff questionnaire

==> picture [547 x 189] intentionally omitted <==

==> picture [351 x 496] intentionally omitted <==

(Colley et al. 2016)

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

• › Visual presentation of Attrakdiff findings in the report

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

7. Semi-structured interview

• The last part of the evaluation is for asking questions from the user to find out their thoughts about the prototype and the interaction with it

  • What is good/bad in the design idea and the prototype?

  • What was difficult?

  • What would make it easier to use?

  • What would you like to change in it to make it more interesting to you?

  • Try to avoid questions that can be replied to just with yes or no!

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

7. Post-interview questions

• Describe your experiences with the system.

• What do you think of using this kind of technology for remote collaboration?

• What was easy/natural and what was difficult/unnatural in the interaction?

• Were there any critical moments in the interaction? If yes, describe it/them.

6[TH] MAY 2026

MINNA PAKANEN

HOW DO YOU DESCRIBE YOUR STUDY IN THE REPORT?

How procedure is reported in the paper?

› How is the procedure description in the paper differentiating from the study procedure, or is it doing that at all?

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the study in your report:

› How the procedure is reported in the paper?

› Is the procedure description in the paper differentiating from the study procedure, if so, how?

6[TH] MAY 2026

MINNA PAKANEN

ROLES IN THE EVALUATION

6[TH] MAY 2026

MINNA PAKANEN

102

ROLES IN THE EVALUATION

1. Facilitator/interviewer

• Facilitates the study, gives tasks, interviews, handles questionnaires

2. Observer

• Stays behind and observers what user does

• Takes notes on each task-can also write down user comments

3. Documenter (can be also facilitator)

• Videos/photos/audio recording

Pick a role that fits best for you!

6[TH] MAY 2026

MINNA PAKANEN

ROLES IN THE EVALUATION

An example

==> picture [369 x 283] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

QUALITATIVE DATA ANALYSIS Thematic Analysis

Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004

6[TH] MAY 2026

MINNA PAKANEN

105

PLEASE REVISIT MY SLIDES FROM L5

› https://brightspace.au.dk/d2l/le/lessons/202542/topics/256 9495 › There is detailed explanation how to conduct thematic analysis!

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the data analysis in your report:

› How the qualitative data analysis was reported in the papers?

6[TH] MAY 2026 MINNA PAKANEN

Reporting of the findings in your report:

› Check the example papers for tips how to report qualitative findings in your report ›How participants were identified? ›How participants quotes were written? › Pay attention to the story!

6[TH] MAY 2026

MINNA PAKANEN

PLANNING YOUR OWN EVALUATION TA SESSION FROM 13:15-> TODAY!

6[TH] MAY 2026

MINNA PAKANEN

132

TASK FOR TA

Start planning your evaluation

• Who are your participants? You need 5 participants

• Think of your research question

• Think what kind of tasks and questions would help you to answer to your research question

• Make an early draft of your procedure

• Think about the roles: facilitator, observer, video recorder…

6[TH] MAY 2026

MINNA PAKANEN

WHEN THE PROTOTYPE IS “READY”

Conduct a pilot evaluation

• Get participant/s from the other group/s

• Follow your procedure rigorously

• Act like you do not know your participant beforehand

• After the pilot you have an understanding whether your tasks are easy to understand or too complicated to pass and if your questions can reveal knowledge you were hoping for -> Iterate your procedure, tasks and questions, do another

• pilot study before you go to the actual evaluations!

6[TH] MAY 2026

MINNA PAKANEN

COURSE LITERATURE REFERENCES

• Blandford, Ann, Dominic Furniss, and Stephann Makri. "Qualitative HCI research: Going behind the scenes." Synthesis lectures on human-centered informatics 9, no. 1 (2016): 1-115.

• Greenberg, Saul, and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proc. CHI '08. ACM, New York, NY, USA, 111–120.

• Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Vaananen-Vainio-Mattila. 2010. User experience evaluation methods: current state and development needs. In Proc. NordiCHI '10. ACM, New York, NY, USA, 521–530. https://doi.org/10.1145/1868914.1868973

• Lim N-K & Stolterman E (2008) The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction 15(2): A7.

• Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004

6[TH] MAY 2026

MINNA PAKANEN

OTHER REFERENCES

• Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proc. CHI '11. ACM, New York, NY, USA, 2689–2698. https://doi.org/10.1145/1978942.1979336

• Ashley Colley, Minna Pakanen, Saara Koskinen, Kirsi Mikkonen, and Jonna Häkkilä. 2016. Smart Handbag as a Wearable Public Display - Exploring Concepts and User Perceptions. In Proceedings of the 7th Augmented Human International Conference 2016 (AH '16). Association for Computing Machinery, New York, NY, USA, Article 7, 1–8. https://doi.org/10.1145/2875194.2875212

• Gegner L & Runonen M (2012) For what it is worth: Anticipated experience evaluation. In: Brassett J, Hekkert P, Ludden G, Malpass M & McDonnell J (eds) Proceedings of the 8th International Conference on Design and Emotion. URI: https://reseda.taik. fi/Taik/jsp/taik/Publication_Types.jsp?id=23583813. Cited 2015/08/15.

• Hende, EA van den (2010) Really new stories the effect of early concept narratives on consumer understanding and attitudes. Doctoral dissertation. Delft University of Technology.

• Hippel E von (1986) Lead Users: A source of novel product concepts. Management Science 32(7): 791–805.

• Kuutti K, Battarbee K, Sade S, Mattelmaki T, Keinonen T, Teirikko T, & Tornberg A-M (2001) Virtual prototypes in usability testing. Proceedings of the 34th Hawaii International Conference on System Sciences. IEEE: 1–7.

6[TH] MAY 2026

MINNA PAKANEN

OTHER REFERENCES

• Law ELC (2011) The measurability and predictability of user experience. Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York NY, ACM: 1–10.

• Pakanen M (2015) Visual design examples in the evaluation of anticipated user experience at the early phases of research and development. Doctoral dissertation. University of Oulu.

• Pakanen, M., Alavesa, P., Van Berkel, N., Koskela, T., & Ojala, T. (2022). “Nice to see you virtually”: Thoughtful design and evaluation of virtual avatar of the other user in ar and vr based telexistence systems. Entertainment Computing, 40, 100457.

• Roto V, Law E, Vermeeren A, & Hoonhout J (eds) (2011) UX white paper. https//allaboutux.org/uxwhitepaper. Cited 2015/03/08.

• Stone D, Jarrett C, Woodroffe M, & and Minocha S (2005) User interface design and evaluation. Morgan Kaufmann Series in Interactive Technologies. San Francisco CA, Morgan Kaufman.

• Tognazzini, B. 1990. User Testing on the cheap. In Tog on Interface, Addison Wesley publ.

• Yogasara T, Popovic V, Kraal B, & Camorro-Koc M (2011) General characteristics of anticipated user experience (AUX) with interactive products. Proceedings of the 4th World Conference on Design Research: Diversity and Unity. Delft, Delft University of Technology: 1-11. URI: http://eprints.qut.edu.au/47035/. Cited 2015/06/15.

6[TH] MAY 2026

MINNA PAKANEN