~~TH~~ E DIARY STUDIES & CULTURAL PROBES + QUALITATIVE ANALYSIS

ITPDP’26, L5

Dr. Minna Pakanen,

Department of Digital Design and Information Studies mpakanen @cc.au.dk,

23[RD] FEB 2026

MINNA PAKANEN

TODAY

• The diary studies

• Media in the diary studies

• Cultural probes

• Qualitative data analysis

23[RD] FEB 2026

MINNA PAKANEN

23[RD] FEB 2026 MINNA PAKANEN

THE DIARY STUDY

==> picture [296 x 341] intentionally omitted <==

Carter, S., & Mankoff, J. (2005). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’2005). ACM, New York, 899-908.

23[RD] FEB 2026

MINNA PAKANEN

5

TYPICAL DATA COLLECTION MEANS IN THE DIARY STUDY

==> picture [288 x 288] intentionally omitted <==

==> picture [289 x 131] intentionally omitted <==

----- Start of picture text -----
+ +
----- End of picture text -----

==> picture [231 x 231] intentionally omitted <==

https://tinyurl.com/3ju6tmt2

https://penstore.dk/dk/ballograf/epoca-p-kulglepen

https://vshopg.tk/products.aspx?cname=custom+disposable+cameras&cid=109

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY

• Is a research method used to collect qualitative data about user behaviors, activities, and experiences in situ

• What, where, who?

• Longitudinal: from a few days to months (typically 3-14 days)

• Researchers are remote from participants:

  • Participants control the timing and means of capture

  • Researchers suggest the timing and means of capture

    • Guidance in the tasks description

    • Prompting by messages, emails, notifications

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLES OF DIARIES

==> picture [467 x 281] intentionally omitted <==

==> picture [485 x 281] intentionally omitted <==

Anna Luusua, Johanna Ylipulli, Marko Jurmu, Henrika Pihlajaniemi, Piia Markkanen, and Timo Ojala. 2015. Evaluation Probes. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM press, 85–94. https://doi.org/10.1145/2702123.2702466

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY ACTIVITIES TIMELINE

==> picture [74 x 73] intentionally omitted <==

Briefing

==> picture [73 x 73] intentionally omitted <==

Post-study interview

• Explain study • Discuss entries

• Provide materials and instructions

• Probe for additional insights

• Set expectations • Ask for clarification Pilot Actual diary study logging Data analysis Planning

• Select capturing methods • Evaluate entries as they come in • Create materials & instructions –> questions for post-study interview

• Recruit study participants • Check in with participants

• Select an analysis method • Conduct analysis rounds • Report findings

• Define the study timeline

• Schedule briefs and interviews

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY

• Can take place as:

• A feedback study

• An elicitation study

• Combination: results from the feedback study act as prompts for discussion in the elicitation study

23[RD] FEB 2026

MINNA PAKANEN

FEEDBACK STUDY

• Participants answer predefined questions about certain events

• Asynchronous communication between the researcher and participant

• Participant driven: participants answer questions about some event when it occurs

• Benefits:

• as questions are asked at the time of the event, or in situ –>provide accurate responses to questions that depend on recall of the event

• Drawbacks:

• Overburdening participants with questions, when the number of events reported is high because the act of answering questions is a significant distraction from their main task.

23[RD] FEB 2026

MINNA PAKANEN

ELICITATION STUDY

• Participants capture media that are then used as prompts for discussion in interviews

• Synchronous communication between the researcher and participant

• Participants merely capture some information about the event that will serve as a memory cue during a later interview

• Benefits:

• Less burdensome than feedback

  • rapid capture of prompts, such as a photograph, audio, tangible objects

• Drawbacks:

• Potentially inaccurate recall

23[RD] FEB 2026

MINNA PAKANEN

CONTENTS OF THE DIARY

• An introduction page with clear instructions and expectations for how often participants should fill in information

• Question pages for every day you expect your participant to be filling in the diary

• What did you do today?

• What did you like the most about [task/activity]?

• What did you like the least about [task/activity]?

• How could that have been made better for you?

• How did you feel about your experience [doing task/activity] today?

• Why did you feel that way?

• Did anything get in your way?

• Instructions for any photographs or videos you want them to take each day.

• A wrap-up page with concluding questions to be filled at the end of the study

https://outwitly.com/blog/research-methods-diary-study/

23[RD] FEB 2026

MINNA PAKANEN

What kind of questions related to the course topic you could add in a diary study?

Social music listening/ composing & music creation/ learning?

23[RD] FEB 2026

MINNA PAKANEN

==> picture [194 x 195] intentionally omitted <==

==> picture [170 x 169] intentionally omitted <==

ROLE OF MEDIA IN DIARY STUDIES

Carter, S., & Mankoff, J. (2005). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’2005). ACM, New York, 899-908. Kumpel, A.S. (2021). Using Messaging Apps in Audience Research: An Approach to Study Everyday Information and News Use Practices, Digital Journalism, DOI: 10.1080/21670811.2020.1864219

23[RD] FEB 2026

MINNA PAKANEN

15

FIND THE INFORMATION IN GROUPS

Open the article from Bs (Content -> Week 6/10 Diary Studies -> Before class–> Carter & Mankoff, 2005. When participants do the capturing: the role of media in diary studies)

Groups 1-3: Photo diary study Groups 4-6: Transit decision diary study Groups 7-9: Festival diary study What was the focus of the study? What kind of data was gathered? What were the results of the study?

23[RD] FEB 2026

MINNA PAKANEN

Photo/transit/festival diary study What was the focus of the study? What kind of data was gathered? What were the results of the study?

23[RD] FEB 2026

MINNA PAKANEN

MEDIA IN THE DIARY STUDIES

==> picture [628 x 358] intentionally omitted <==

----- Start of picture text -----
Easy recall of
the episode
Easy to Difficult to
capture capture
Difficult recall
of the episode
----- End of picture text -----

23[RD] FEB 2026

MINNA PAKANEN

MEDIA IN THE DIARY STUDIES

==> picture [75 x 75] intentionally omitted <==

==> picture [71 x 71] intentionally omitted <==

• Photos: the easiest to capture and recognize: best support of who and where recognition

Studies where details are important

• Audio clips: easy to capture events secretively & lightweight media appropriate for annotation, but suffer from recognition problem, but once recognized provides adequate support to participants to recall events

==> picture [73 x 74] intentionally omitted <==

• Tangible objects: do not lend themselves to who or where recognition, but elicit participants’ creative explanations of attitudes and beliefs

==> picture [75 x 75] intentionally omitted <==

• Raw location data: is not likely to lead to better recall of an episode. (Experience sampling method (ESM) works in some studies when the amount of events stays within a proper threshold)

23[RD] FEB 2026

MINNA PAKANEN

HOW TO SUPPORT RECALLING?

• To support recall of ambiguous events

• A brief annotation for each capture event is needed

• Suggested way:

==> picture [87 x 88] intentionally omitted <==

==> picture [87 x 88] intentionally omitted <==

==> picture [74 x 73] intentionally omitted <==

==> picture [74 x 74] intentionally omitted <==

thorough review by Lightweight lightweight capture tools in-situ annotation ex-situ annotation participants & researchers

23[RD] FEB 2026

MINNA PAKANEN

MESSAGING APP DIARY APPROACH +

• Using of messaging apps already used by people (WhatsApp, etc.)

• No additional apps needed –> convenience

• Easy to integrate into participants’ daily routines–> continuous participation & improved response rates

• Provides easy-to-use multimedia documentation solutions for the sharing of rich and context-sensitive data

• Allows researcher for an easy way for sending prompts & an instant feedback channel

• Allows participants to record their experiences, feelings, and thoughts “where they are”

(Kumpel, 2021) 23[RD] FEB 2026 MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

What could be the downsides of using WhatsApp in the diary studies?

23[RD] FEB 2026

MINNA PAKANEN

– MESSAGING APP DIARY APPROACH

• Needs constant attention from the researcher in the documentation phase –> time-consuming

• –>

• Apps are privately owned by international companies data security, GDPR of sensitive data

• Participants allow the researcher into their personal world–> mixing of personal life and participation in research

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY PIPELINE

• 1) A participant takes a photo

• 2) The participant annotates the photo with an audio recording

3) The participant uses a tool (an application) to log the photo and audio and add more annotations

• 4) The researcher provides feedback about the captured data

5) The researcher holds an elicitation interview with the participant using the captured media as prompts

23[RD] FEB 2026

MINNA PAKANEN

CULTURAL PROBES

Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: cultural probes. Interactions, 6(1), 21-29.

23[RD] FEB 2026

MINNA PAKANEN

26

USER-CENTERED INSPIRATION

• Data collection to inspire design

• Stimulate designers imagination

• Helps designers to familiarize with the design context and learn the culture

• Helps to establish a conversation with the target group of people that can last throughout the project

• Overcomes the distance

• Geographic and cultural + language barriers

• Respects the respondents

23[RD] FEB 2026

MINNA PAKANEN

CULTURAL PROBES

• Personal communication tool between the target group

• Designed for specific project, people & their environments

• Package containing:

• Maps + stickers

• Postcards

• Disposable camera

• Booklets: photo album, diary

• Is left behind and returned ‘filled’ to the designers!

==> picture [403 x 540] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

AESTHETICS

• Idea is to reduce the distance between the designers and target group

• Little bit abstract/alien design aesthetics

• Delightful, but not childish

• Not too professionally finished!

• Personal and informal feeling

• Revealing the energy put to the creation

• Revealing the tastes and interest of the designers –> making target group to reveal themselves to the designers

==> picture [305 x 358] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

==> picture [120 x 129] intentionally omitted <==

EXAMPLES CATHERINE LEGROS

==> picture [654 x 331] intentionally omitted <==

==> picture [120 x 104] intentionally omitted <==

==> picture [120 x 104] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

EXAMPLE: ITPDP PROJECT 2025

AESTHETICS RELATED TO THE THEME

==> picture [522 x 332] intentionally omitted <==

• © Hook’d team: Agnes Niewald de Place, Kamilla Nørgaard Nielsen, and Julian Philipp Leimbeck

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLE: BSC PROJECT 2020

PACKAGE AND SURPRISE

Keep the interest alive during one week study

==> picture [792 x 233] intentionally omitted <==

• © Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

MAP IN PROBES

INQUIRING ATTITUDES ABOUT THE ENVIRONMENT

• Map, questions and different coloured dot stickers to mark the answers on where:

• They would go to meet people?

• They would go to be alone?

• They liked to daydream?

• They would like to go but can’t?

• To emphasise individuality:

• Printed on different kinds of papers and cut into envelope forms –> easy to fold together and put into mail

==> picture [415 x 491] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

POSTCARD IN PROBES

• Inquiring about target groups’ attitudes towards their lives, cultural environments, and technology

• Informal and friendly way of communication

• Place image at front & 1 question at back:

• Please tell us a piece of advice or insight that has been important to you. (attitude)

• What do you dislike about (city/building)? (context)

• What place does (x) have in your life? (attitude)

• Tell us about your favourite device (technology)

• Oblique wording and evocative images to open space for possibilities

• Name and address of the designer + pre-stamping for return!

==> picture [354 x 456] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

==> picture [355 x 456] intentionally omitted <==

Create a probes stcard with one ~~po~~ image on the front and one question about attitude/context/tech nology related to the Social music topic on the back.

MINNA PAKANEN

23[RD] FEB 2026

==> picture [97 x 41] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

Give your card to another team and wait for the card to return to your team’s desk later J

23[RD] FEB 2026

MINNA PAKANEN

CAMERA IN PROBES

• Should be repackaged to separate it from its commercial origins and to make it fit with the other probe materials

• Mark on the backside/instruction label the requests for pictures:

• Your home

• Something desirable

• Something boring

• Unassigned pictures –> whatever participants want to show to designers before mailing the camera back

==> picture [359 x 338] intentionally omitted <==

Disposable camera: https://tinyurl.com/47rej4mm Reusable camera: https://tinyurl.com/53rkmcus Instant camera: https://tinyurl.com/3bsws9jn

23[RD] FEB 2026

MINNA PAKANEN

BOOKLETS

• Photo album

• Pictures or text to tell a story about the participant

• Diary

• The entries are done daily, for a total of a week or other period of time

• Record use of something:

  • What?

  • With whom?

  • When?

  • Example: television/ radio use or calls

==> picture [257 x 393] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLE: BSC PROJECT 2020

TASKS INQUIRING INTO DAILY LIVES IN THE CENTER

My world Floor plan Style board Who am I? Where I like/ dislike being? My day Emotion bubbles Postcards Who am I interacting with and when? What does it mean to me?

Style board

What are my aesthetic preferences?

Who I will remember from here in five years?

© Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLE: ITPDP PROJECT 2025

5 TASKS

==> picture [522 x 332] intentionally omitted <==

Selected fishing gear and reasoning of the preferences?

What websites do you use for research and planning before fishing trip?

Map: Fishing location preferences?

Emotional reflection through images (before & after the fishing trip)?

• © Hook’d team: Agnes Niewald de Place, Kamilla Nørgaard Nielsen, and Julian Philipp Leimbeck

23[RD] FEB 2026

MINNA PAKANEN

LAUNCHING THE PROBES

• Personal introduction to the group of participants

• Present yourself & your intentions

• Answer questions

• Encourage the participants to take an informal, experimental approach to the materials

  • Introduce the types of things participants will find from the probes package

• Learn about the participants and create enthusiasm and trust!

23[RD] FEB 2026

MINNA PAKANEN

THE RETURNED PROBES & DATA

• Allows designers to

• Familiarize themselves with the design contexts and people who are they designing for

• Ground proposals in detailed textures of the local

• Inspiration rather than direct design proposal!

==> picture [526 x 323] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

RETURNED

==> picture [649 x 141] intentionally omitted <==

==> picture [46 x 46] intentionally omitted <==

==> picture [46 x 46] intentionally omitted <==

==> picture [351 x 264] intentionally omitted <==

==> picture [451 x 269] intentionally omitted <==

© Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

RETURNED

==> picture [515 x 224] intentionally omitted <==

==> picture [245 x 286] intentionally omitted <==

© Hook’d team: Agnes Niewald de Place, Kamilla Nørgaard Nielsen, and Julian Philipp Leimbeck 23[RD] FEB 2026 MINNA PAKANEN

QUALITATIVE DATA ANALYSIS

General about qualitative data analysis (Blandford et al. 2016), Affinity mapping (Scupin 1997), and thematic analysis (Braun and Clarke, 2012)

23[RD] FEB 2026

MINNA PAKANEN

49

LEARNED IN FITDES AFFINITY DIAGRAM ( )

• Affinity diagram help organize your data into groups of similar items

• Process of affinity mapping:

1. Record all notes or observations on individual cards or sticky notes (can be done in Miro.com as long as you have pseudonymized data and your “key file” is not stored in it, if you are unsure check here: https://studerende.au.dk/en/itsupport/information-security/data-protection-gdpr/projects

2. Look for patterns in notes or observations that are related and group them

3. Create a group for each pattern or theme

4. Give each theme or group a name

5. Read it through carefully and combine or break themes until saturation is reached –>Affinity diagram is ready

   • Create a statement of what you learned about each group –> provide your analysis or key insights! 23[RD] FEB 2026 MINNA PAKANEN

MINNA PAKANEN

EXAMPLE AFFINITY DIAGRAM

Create a statement of each category to present your findings

Each participant is presented with different color Scheme names

© Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

ANALYSIS/ PRESENTATION OF FINDINGS

Anonymous code

==> picture [401 x 311] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

THEMATIC ANALYSIS (TA)

› Systematic and sophisticated method for identifying, organizing and offering insight into patterns of meaning (themes) across a data set

• › Accessible, flexible, and popular method

• › Driven by research question

• › Identifying relevant patterns to answering to it

23[RD] FEB 2026

MINNA PAKANEN

FLEXIBILITY AND CHOICES IN TA

Many approaches to choose from, but for you these two are important at this point:

• › Inductive approach: “What is in the data”

› Deductive approach: “Researchers decide what is mapped”

› In reality, we always combine these two approaches!

23[RD] FEB 2026

MINNA PAKANEN

FLEXIBILITY AND CHOICES IN TA

Inductive approach

• › “What is in the data”

• › Bottom-up: the codes and themes derive from the data

• › What is mapped relates closely to the data

• › Experiential in its orientation

• › Essentialist in its theoretical framework:

  • › Assuming that there is a knowledgeable world

  • ”

  • › “Giving voice to experiences and meanings of that world as reported in the data

›

23[RD] FEB 2026

MINNA PAKANEN

FLEXIBILITY AND CHOICES IN TA

Deductive approach

• › “Researchers decide what is mapped”

• › Top-down: the codes and themes derive from researchers' ideas and concepts

• › What is mapped has a looser link to the data

• › Critical in its orientation

• › Constructionist in its theoretical framework:

  • › examining how the world is put together (i.e., constructed) and the ideas and assumptions that inform the data gathered.

23[RD] FEB 2026

MINNA PAKANEN

PREPARING DATA FOR ANALYSIS

Transcribing and working with the raw data

› Transcribe the audio data (Transcriber in Ucloud)/per participant (Px, where x is participant ID number, I= interviewer):

• › Verbatim = Capturing every sound made (e.g. throat clearing, laughter, and verbal pauses: “ah,” “um,” and “uh” + other noises (e.g. a phone ringing or a door slamming).

• › Edited= edited to be clear in grammatic (avoid this)

• › Intelligent verbatim = edits out distracting filler words & sounds, repetitions, ramblings, non-standard words, and irrelevant conversations

23[RD] FEB 2026

MINNA PAKANEN

HOW TO DO THEMATIC ANALYSIS?

6-Phase Approach

1) Familiarization

• 2) Generating Initial Codes

• 3) Searching Themes

4) Reviewing the Potential Themes 5) Defining and Naming the Themes 6) Producing the Report

In practice, these phases are done in parallel!

23[RD] FEB 2026

MINNA PAKANEN

1) FAMILIARIZATION

Start by familiarizing yourself with the data › Listening to the audio, watching videos, and reading the transcripts

• › Write down some initial thoughts while doing so in a file or in a notebook

• › Annotate the whole data transcripts by underlining, noting, or color highlights

• › Helps you to read the data as data

• › Focus on things related to your research question

23[RD] FEB 2026

MINNA PAKANEN

==> picture [57 x 12] intentionally omitted <==

----- Start of picture text -----
Themes
----- End of picture text -----

==> picture [47 x 12] intentionally omitted <==

----- Start of picture text -----
Codes
----- End of picture text -----

2) GENERATING INITIAL CODES

• Systematic analysis of the data › Codes are the building blocks of the analysis (tiles)

• › Semantic: descriptive, stay close to the data

• › Latent: identify the meaning behind the semantic surface

• › Usually, a mix of both

• › Code each data item in its entirety before coding another

• › Modify codes when necessary and read the data again! › Cut and paste data from all participants in another Word file under themes (keep the participant ID (Px) with the snippet)

23[RD] FEB 2026

MINNA PAKANEN

2) CODE EXAMPLES

23[RD] FEB 2026 MINNA PAKANEN

==> picture [57 x 12] intentionally omitted <==

----- Start of picture text -----
Themes
----- End of picture text -----

==> picture [47 x 12] intentionally omitted <==

----- Start of picture text -----
Codes
----- End of picture text -----

3) SEARCHING FOR THEMES

Themes (Walls & roof)

› Searching for themes is an active process

• › Reviewing codes and coded data to identify broad topics around the

• code clusters

• › Thinking of a particular story about your data, together the themes should provide a meaningful picture of your data!

• › In an 8000-10000-word article, there are typically 2-6 themes

23[RD] FEB 2026

MINNA PAKANEN

3) CODE EXAMPLES IN THEME SEARCH

23[RD] FEB 2026

MINNA PAKANEN

4) REVIEWING POTENTIAL THEMES

Quality checking

› The developing themes are reviewed in relation to the coded data and the entire data set

› Check if the themes work with data under them, if not relocate the data snippets under another theme

• › Redraw boundaries of themes to better describe the data under them

› Consider if it is a theme or code

• › Is there enough meaningful data to support the theme?

• › Is the data too diverse and wide-ranging/ does the theme lack coherence?

–>If not create additional themes/refine the existing ones

23[RD] FEB 2026

MINNA PAKANEN

5) DEFINING AND NAMING THEMES

What is unique and specific in each theme? › Write a few sentences to sum up what is the essence of the theme

• › Good themes:

• › They have a singular focus: do not try to do too much

• › Are related to each other, but do not overlap (can build on previous themes)

• › They should directly address your research question

› Note that together the themes should tell a coherent story about your data!

23[RD] FEB 2026

MINNA PAKANEN

5) DEFINING AND NAMING THEMES

What is unique and specific in each theme?

• › Deep analytic work in this phase

• › Descriptive: using data in an illustrative way (easy to start with)

• › Conceptual and interpretative: latent meanings in the data

Often a combination of both!

• › Telling the story:

  • › Data does not speak for itself! Don’t just paraphrase the content of data

  • › Your analytic narrative will tell the reader what in the extract is interesting and why

  • › The data needs to be interpreted in relation to your research question

• › Naming should be informative, concise, and catchy

23[RD] FEB 2026

MINNA PAKANEN

5) DEFINED & NAMED THEME EXAMPLES

23[RD] FEB 2026 MINNA PAKANEN

6) PRODUCING THE REPORT

Reporting the findings in the publication, thesis, or report › In qualitative analysis, the analysis phase is usually never completed before write-up, but writing and analysis are interwoven

• › Good writing comes with practice, but try to avoid repetition, paraphrasing, unnecessary complexity, and passive phrasing

› Order of the themes should create a coherent story:

• › Logical and meaningful

• › Building up on previous themes is possible

23[RD] FEB 2026

MINNA PAKANEN

6) FINAL THEME REPORTING EXAMPLE

23[RD] FEB 2026 MINNA PAKANEN

DOING THEMATIC ANALYSIS WELL

Try to avoid common errors:

› Providing data extracts with no or little analysis =no interpretation of what data tells us, what is important in it regarding the research question

• › Using data collection questions (interview questions) as themes

for data analysis –> better to focus on what participants say

• › Lack of evidence: provide examples that convince the reader that this pattern was evident in the data

23[RD] FEB 2026

MINNA PAKANEN

EDITING TRANSCRIBES IN THE RESULTS REPORTING PHASE

Partly intelligent verbatim, adding things to ease understanding and shortening the quote + how to indicate it?

“Participant quotes are in Italic.”

“When it first time popped open, I was looking over my shoulder, and a woman passed-by me, and it [bag] made a neeee! [makes a sound of the motor] and I was like oh shit! I felt very aware of it, [its] affect to people and [as] I am going to places where are people, so I was like noo [embarrassed sound and holds hand on the face], because I was like waiting it to [. . .] start moving. In terms of getting attention, I received that when it made sound.” (P2)

[your addition inside of these

brackets are not italic]

[…]

= something has been

removed

23[RD] FEB 2026

MINNA PAKANEN

DIRECT QUOTATION PRESENTATION ★ ▲… Pseudonymized (P1 or #1)/ first name (real or made up)/ code

==> picture [97 x 41] intentionally omitted <==

”For me, this lecture has been relevant in learning how to conduct / diary studies.” (P3) (#3)/ (Mia) One participant (P3)/ ( ★ )/ Mia stated: ”For me, this lecture has been relevant in learning how to conduct diary studies.”

★ / Star/ Mia stated: ”For me, this lecture has been relevant in learning how to conduct diary studies.”

★ &▲ found this A few participants (P3 & P6/ / Mia & Jake) 23[RD] FEB 2026 MINNA PAKANEN lecture as helpful in learning how to conduct diary studies.

REFERENCES

• Carter, S., & Mankoff, J. (2005). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’2005). ACM, New York, 899-908.

• Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: cultural probes. Interactions, 6(1), 2129.

• Kumpel, A.S. (2021). Using Messaging Apps in Audience Research: An Approach to Study Everyday Information and News Use Practices, Digital Journalism, DOI: 10.1080/21670811.2020.1864219

23[RD] FEB 2026

MINNA PAKANEN

REFERENCES: DATA ANALYSIS

• Blandford, A., Furniss, D., & Makri, S. (2016). Qualitative HCI research: Going behind the scenes. Morgan & Claypool Publishers.

• Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-00

• Scupin, R. (1997). The KJ Method: A Technique for Analyzing Data Derived from Japanese Ethnology. Human Organization, 56(2), 233–237. http://www.jstor.org/stable/44126786

23[RD] FEB 2026

MINNA PAKANEN

ITPDP2026- WEEK 3: DESIGN PROCESSES, PROJECT MANAGEMENT, AND DESIGN ETHICS

Clemens Nylandsted Klokmose Department of Computer Science Human-Centered Computing Section clemens@cs.au.dk

AARHUS UNIVERSITY

https://studypedia.au.dk/haandter-pensum/laesestrategier

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

2

AARHUS UNIVERSITY

PLAN

› Design processes

› Involving users

› Project management

› Design ethics

==> picture [222 x 222] intentionally omitted <==

› GDPR

==> picture [47 x 40] intentionally omitted <==

==> picture [222 x 146] intentionally omitted <==

----- Start of picture text -----
ITPDP 2026
----- End of picture text -----

AARHUS UNIVERSITY

PEOPLE AND PROTOTYPES

• › Chapter in Moggridge (2006) describes IDEO's methods

• › What is design? (Covered in FIT-DES)

• › It is important to understand the needs and desires of users

• › Observation and participation

• › Often tacit and implicit knowledge that can only be uncovered experimentally

• › Many versions of prototypes are needed (Later lecture)

• › Prototypes are tangible and visible proposals

• › User can "experience" a prototype and thus better evaluate proposed solutions

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

THE GOOD DESIGN?

• Transparency and tacit knowledge (Polanyi, Bødker, and more)

• › Fluid use without breakdowns

• › Leverages the users' intuition* (that is uncovered experimentally)

• › Scientific verification is often long and complex

Examples of assessment criteria for design projects:

• The height of creativity/innovation

• Aesthetics/quality

• Whether human factors/values are taken into account

• Performance and technology

• Finish and presentation

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

THE GOOD DESIGN: AFFORDANCES (BILL GAVER)

› Perceptible possibilities (Gibson, 1979)

› We sense immediately

• › That one can walk up a flight of stairs

› Sitting on a chair

› Tilting a door handle › Turning on faucet

• › Computer user interfaces should be designed with equally clear affordances...

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

AFFORDANCES

==> picture [47 x 40] intentionally omitted <==

==> picture [190 x 279] intentionally omitted <==

==> picture [225 x 226] intentionally omitted <==

2026

ITPDP

7

AARHUS UNIVERSITY

Need to repair a design that does not ”afford" the right action possibilities to the user

==> picture [47 x 40] intentionally omitted <==

==> picture [504 x 378] intentionally omitted <==

----- Start of picture text -----
ITPDP 2026
----- End of picture text -----

DESIGN DISCIPLINES AND TECHNIQUES

How do we understand the problem area and the needs of users?

AARHUS UNIVERSITY

LIMITATIONS WITH INCREASING COMPLEXITY

› For a holistic understanding of groups, organizations, society and the globe

›

› From the facts of human proportions and physics

ITPDP 2026

›

AARHUS UNIVERSITY

ANALYSIS METHODS

==> picture [116 x 121] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

ITERATIVE DESIGN PROCESS

• › Same type of activity is repeated to reduce uncertainty about the design

• › Many cross-cutting jumps between activities

• › From limitations to idea generation over prototyping less uncertainty back to remaining limitations

==> picture [47 x 40] intentionally omitted <==

==> picture [304 x 341] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

IDEO: 51 WAYS TO LEARN ABOUT USERS › IDEO Method cards

• › 4 Categories

• › Learn – from facts that can be collected

• › Look – at what users do

• › Ask – about their contributions;

• › Try – out ideas

• › The entire collection of 51 cards is available as a book/card box

› https://stoutbooks.com/products/ideo-method-cards-51-ways-to-inspire-design-61457

• › In the chapter, only 4 examples from each category

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUSUNIVERSITY LEARN

› Analyze the information you’ve collected to identify patterns and insights.

› FLOW ANALYSIS

• › How Represent the flow of information or activity through all phases of a system or process.

• › Why This is useful for identifying bottlenecks and opportunities for functional alternatives.

• › Example Designing an online advice Web site, flow analysis helped the team to gain a clearer sense of how to make it easy to find your way around the site.

› COGNITIVE TASK ANALYSIS

• › How List and summarize all of a user’s sensory inputs, decision points, and actions.

• › Why This is good for understanding users’ perceptual, attentional, and informational needs and for identifying bottlenecks where errors may occur.

• › Example Logging the commands that would be involved in controlling a remotely operated camera helped the team establish priorities among them.

› HISTORICAL ANALYSIS

• › How Compare features of an industry, organization, group, market segment or practice through various stages of development.

• › Why This method helps to identify trends and cycles of product use and customer behavior and to project those patterns into the future.

• › Example A historical view of chair design helped to define a common language and reference points for the team members from the client and consultancy.

› AFFINITY DIAGRAMS

• › How Cluster design elements according to intuitive relationships, such as similarity, dependence, proximity, and so forth.

• › Why This method is a useful way to identify connections among issues and to reveal opportunities for innovation.

• › Example This affinity diagram shows what’s involved in transporting young children, and helps to identify the opportunities to improve the design of a stroller.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUSUNIVERSITY LOOK

• › Observe people to discover what they really do—not what they say they do.

› FLY ON THE WALL

• › How Observe and record behavior within its context, without interfering with people’s activities.

• › Why It is useful to see what people do in real contexts and time frames, rather than accept what they say they did after the fact.

• › Example By spending time in the operating room, the designers were able to observe and understand the information that the surgical team needed.

› A DAY IN THE LIFE

• › How Catalog the activities and contexts that users experience for an entire day.

• › Why This is a useful way to reveal unanticipated issues inherent in the routines and circumstances people experience daily.

• › Example For the design of a portable communication device, the design team followed people throughout the day, observing moments at which they would like to be able to access information.

› SHADOWING

• › How Tag along with people to observe and understand their day-to-day routines, interactions, and contexts.

• › Why This is a valuable way to reveal design opportunities and show how a product might affect or complement user’s behavior.

• › Example The team accompanied truckers on their routes in order to understand how they might be affected by a device capable of detecting drowsiness.

› PERSONAL INVENTORY

• › How Document the things that people identify as important to them as a way of cataloging evidence of their lifestyles.

• › Why This method is useful for revealing people’s activities, perceptions, and values as well as patterns among them.

• › Example For a project to design a handheld electronic device, people were asked to show the contents of their purses and briefcases and explain how they use the objects that they carry around everyday.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUSUNIVERSITY ASK

› Enlist people’s participation to elicit information relevant to your project.

› CONCEPTUAL LANDSCAPE

• › How Ask people to diagram, sketch, or map the aspects of abstract social and behavioral constructs or phenomena.

• › Why This is a helpful way to understand people’s mental models of the issues related to the design problem.

• › Example Designing an online university, the team illustrated the different motivations, activities, and values that prompt people to go back to school.

› COLLAGE

• › How Ask participants to build a collage from a provided collection of images and to explain the significance of the images and arrangements they choose.

• › Why This illustrates participants’ understanding and perceptions of issues and helps them verbalize complex or unimagined themes.

• › Example Participants were asked to create a collage around the theme of sustainability to help the team understand how new technologies might be applied to better support people’s perceptions.

› FOREIGN CORRESPONDENTS

• › How Request input from coworkers and contacts in other countries and conduct a crosscultural study to derive basic international design principles.

• › Why This is a good way to illustrate the varied cultural and environmental contexts in which the products are used.

• › Example A global survey about personal privacy helped to quickly compile images and anecdotes from the experiences of the correspondents.

› CARD SORT

• › How On separate cards, name possible features, functions, or design attributes. Ask people to organize the cards spatially, in ways that make sense to them.

• › Why This helps to expose people’s mental models of a device or system. Their organization reveals expectations and priorities about the intended functions.

• › Example In a project to design a new digital phone service, a card-sorting exercise enabled potential users to influence the final menu structure and naming.

2026

ITPDP

AARHUSUNIVERSITY TRY

• › Create simulations and prototypes to help empathize with people and to evaluate proposed designs.

› EMPATHY TOOLS

• › How Use tools like clouded glasses and weighted gloves to experience processes as though you yourself have the abilities of different users.

• › Why This is an easy way to prompt an empathic understanding for users with disabilities or special conditions.

• › Example Designers wore gloves to help them evaluate the suitability of cords and buttons for a home health monitor designed for people with reduced dexterity and tactile sensation.

› SCENARIOS

• › How Illustrate a character-rich storyline describing the context of use for a product or service.

• › Why This process helps to communicate and test the essence of a design idea within its probable context of use. It is especially useful for the evaluation of service concepts.

• › Example Designing a community Web site, the team drew up scenarios to highlight the ways particular design ideas served different user needs.

› NEXT YEAR’S HEADLINES

• › How Invite employees to project their company into the future, identifying how they want to develop and sustain customer relations.

• › Why Based on customer-focused research, these predictions can help to define which design issues to pursue for development.

• › Example While designing an Intranet site for information technologists, the team prompted the client to define and clarify their business targets for immediate and future launches.

› INFORMANCE

• › How Act out an “informative performance” scenario by role-playing insights or behaviors that you have witnessed or researched.

• › Why This is a good way to communicate an insight and build a shared understanding of a concept and its implications.

• › Example A performance about a story of mobile communications shows the distress of a frustrated user.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

REMEMBER THE EXTREMES

› ”Extreme characters”

Example

• › Extremes in IT design for the home

• › The homeless living in a shopping cart

› The film actor with uniformly decorated apartments in New York, Paris, Tokyo and LA

==> picture [47 x 40] intentionally omitted <==

==> picture [344 x 273] intentionally omitted <==

2026

ITPDP

IDEATION

AARHUS UNIVERSITY

IDEO - IDEA GENERATION

• › 8-10 participants – responsible for documentation appointed

• › 50-100 ideas in an hour

• › Rules

• › No critical assessments

• › Stimulate wild ideas

• › Build on other people's ideas

• › Stay focused on the topic

• › Hold on to one "thread" at a time

• › Really good ideas can stop the process and restart it somewhere new

• › Ideas are taken over into an "envisionment" activity, where it is visible and tangible

›

2026

ITPDP

AARHUS UNIVERSITY

FUTURE WORKSHOPS

› Critique phase

› Brainstorming problems in current practice

› No discussion – just get problems on the board › Group issues and prioritize importance

› Fantasy phase

› Brainstorm wild/utopian ideas (that can solve the problems identified)

› No discussion – just get ideas on the board

› Group ideas and prioritize them in terms of value creation

› Realization phase

› Take the high-priority ideas › Delimit to realistic visions

• › Prepare concrete proposals for realization

==> picture [47 x 40] intentionally omitted <==

(Jungk & Müllert, 1987;Kensing & Halskov, 1991)

2026 21

ITPDP

THE ROLE OF THE DESIGNER & THOUGHTFUL INTERACTION DESIGN

Löwgren and Stolterman

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

Places the designer at the core of the process

L&S argue that the responsibility for the vision at the designer (p.34ff)

L&S argue that the responsibility for the design process is at the designer (p.38ff)

L&S argue that the designer should engage and manage the relations in the design process (p.32ff).

==> picture [241 x 190] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN Designing the design process

› Design starts earlier than project owners may think

› Select appropriate methods/techniques

• › Pay attention to and care for a common vision

• › Pay attention to roles and stakeholders

==> picture [240 x 190] intentionally omitted <==

• › Pay attention to design as a project

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

Divergence

“Designer expands her thinking to cover broader issues, find alternatives, and explore more opportunities” (L&S, p. 29)

Convergence

“Convergence is about focusing on a specific solution or a synthesis of several ideas” (Ibid.)

• What is the primary issue?

• • Who to involve and how?

• How would the shape look like?

• • What is the interaction modality?

• • What kind of feedback it give?

==> picture [79 x 76] intentionally omitted <==

Design choices

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

• › Vision : The first organising principle that help the designer respond to the situation at hand

• › Operative Image : The first (and consecutive) externalisations of the vision

• › Specification : The final “design”

==> picture [214 x 202] intentionally omitted <==

• specification

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

LÖWGREN & STOLTERMAN

› Leaping between detail and the whole

• › Focusing on dilemmas in the domain

• › Alternatives and contraditions

• › Get the dilemmas and

trade offs on the table early in the Vision activity

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

INVOLVING USERS

AARHUS UNIVERSITY

PERSPECTIVES ON PEOPLE AND MACHINES

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

USER INVOLVEMENT

• None, very little, and/or only at the end

• User-centred design

• Participatory design

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

USER-CENTERED DESIGN

• Involvement of users in all parts of the design process

• Focus groups for ideation

• Evaluation of low-fidelity prototypes

• Evaluation of new features through AB testing and interviews

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

PARTICIPATORY DESIGN

• More radical approach to user involvement than user-centred design

• Users as direct design partners and active first-class members of the product design team

• Developed in Scandinavia in the 70s and 80s (Aarhus University was a key player)

• Methodology developed laid the foundation for user-centred design

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY UTOPIA PROJECT

Early participatory design project

• Alliance between typesetters union and IT researchers

• How to empower instead of replace typesetters with computers

• Design of computer systems based on the people on the shop floor rather than the management

• Introduced low-fi prototyping in systems design

==> picture [47 x 40] intentionally omitted <==

==> picture [220 x 99] intentionally omitted <==

==> picture [220 x 149] intentionally omitted <==

ITPDP 2026

Morten Kyng & Susanne Bødker

AARHUS UNIVERSITY

PARTICIPATORY DESIGN TECHNIQUES

• › Ethnographic field studies

• › Observations, interview and video analysis

› "Fictional inquiries”

• › Playful analysis in a fictional setting

• › Structured brainstorming

• › Future Workshop, Metaphorical Design, Inspiration Cars, Organizational Games

• › Scenarios

› Descriptions, tableau, video

› Mock-ups

• › Physical models, paper windows

==> picture [155 x 114] intentionally omitted <==

==> picture [155 x 110] intentionally omitted <==

==> picture [155 x 100] intentionally omitted <==

› Video prototyping

› Stop-motion, blue studio techniques

• › Prototyping

• › Exploratory, experimental, evolutionary, cooperative

==> picture [23 x 8] intentionally omitted <==

----- Start of picture text -----
2026
----- End of picture text -----

ITPDP

AARHUS UNIVERSITY

INTERNATIONAL BOOKS AND ARTICLES ON SCANDINAVIAN PARTICIPATORY DESIGN

• › Bødker, S., Grønbæk, K., & Kyng, M. (1995). Cooperative Design: Techniques and Experiences from the Scandinavian Scene. In R. M. Baecker, J. Grudin, & W. A. S. Buxton (Eds.), Readings in Human-Computer Interaction: Toward the Year 2000 . San Francisco: Morgan Kaufmann Publishers, Inc., 215-224.

• › G. Bjerknes, P. Ehn, & M. Kyng (Eds.) (1987) Computers and Democracy . Aldershot: Avebury. › Greenbaum, J., & Kyng, M. (1991). Design at Work: Cooperative Design of Computer Systems . Hillsdale, NJ: Lawrence Erlbaum Associates.

• › D. Schuler & A. Namioka (Eds.) (1993) Participatory Design: Principles and Practices . Hillsdale, New Jersey: Lawrence Erlbaum Associates, 157-175.

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

==> picture [471 x 266] intentionally omitted <==

==> picture [214 x 282] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

36

AARHUS

UNIVERSITY

==> picture [47 x 40] intentionally omitted <==

ITPDP

2026

37

PROJECT MANAGEMENT

AARHUS UNIVERSITY

UNCERTAINTY ABOUT THE PRODUCT SHOULD BE REDUCED

==> picture [449 x 178] intentionally omitted <==

----- Start of picture text -----
Now Deadline Exam
----- End of picture text -----

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

PROJECT MANAGEMENT: SYSTEMATIC APPROACH

==> picture [14 x 6] intentionally omitted <==

----- Start of picture text -----
Now
----- End of picture text -----

==> picture [113 x 6] intentionally omitted <==

----- Start of picture text -----
Deadline Exam
----- End of picture text -----

==> picture [465 x 39] intentionally omitted <==

==> picture [436 x 77] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

PROJECT MANAGEMENT

Project management

› Focus on the project, starting point, purpose, budget etc.

› Focus on deadlines, deliveries, quality etc.

› Focus on progress, evaluation, success/failure Leadership

› Focus on competencies and roles

› Focus on performance and well-being

• › Focus on the team over time (and more projects)

• Self-management

• › Focus on your own tasks, satisfaction, prioritization, progression!

›

2026

ITPDP

AARHUS UNIVERSITY

PITFALLS

› Technical Rationality (Gedenryd 1998)

› Believe that you can follow a linear process

• › Optimistic estimation (Brooks 1975)

› Software is highly malleable compared to other materials

• › Brooks Law (Brooks 1975)

• › Believe that you can finish faster by putting more people on a project

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY ILLUSION OF TECHNICAL RATIONALITY

• Most straightforward model of a project • Most projects to some degree or the other follows this model • Pitfall • Paralysis by fear of wrong requirements can halt the process • Mistakes are expensive too fix late in the process

==> picture [47 x 40] intentionally omitted <==

Schön 1987; Gedenryd 1998

2026

ITPDP

AARHUS UNIVERSITY

AGILE INTERACTION DESIGN / AGILE DEVELOPMENT

• Break design process down in small iterations each involving all phases

• Iteratively develop software in working (and deployable) increments

• The software is never finished (for good … and for ill)

• Affords extensible software architectures that enables rapid prototyping of new features

==> picture [47 x 40] intentionally omitted <==

==> picture [520 x 146] intentionally omitted <==

----- Start of picture text -----
ITPDP
----- End of picture text -----

2026

AARHUS UNIVERSITY

MAKE A GOOD PLAN WITH ROOM FOR ERRORS AND ITERATIONS

==> picture [14 x 6] intentionally omitted <==

----- Start of picture text -----
Now
----- End of picture text -----

==> picture [113 x 6] intentionally omitted <==

----- Start of picture text -----
Deadline Exam
----- End of picture text -----

==> picture [465 x 39] intentionally omitted <==

==> picture [436 x 77] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

ETHICS

AARHUS UNIVERSITY

WHY DO WE EVEN TALK ABOUT ETHICS

› We build things

› … that affects people’s lives

• › … potentially a lot of people

• › ... that change their perspectives on things

• › … even their possibilities of action, self-understanding and daily life

==> picture [145 x 217] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

UNETHICAL TECHNOLOGY?

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

AARHUS UNIVERSITY

UNETHICAL TECHNOLOGY?

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

52

AARHUS UNIVERSITY

THREE SCHOOLS OF ETHICS

› Consequentialism (da: nytteetik )

› Cares for consequences: “The truth can hurt”

› Deontology (da: pligtetik )

• › Cares for rules: “You must not lie”

› Virtue ethics (da: dydsetik )

› Cares for principles: “I always tell the truth”

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

ETHICS

Verbeek’s Materializing Morality

AARHUS UNIVERSITY

VEERBEEK’S CLAIM

› ‘If technology mediates how we perceive and act in the world, it can also be designed to mediate perception and action in ethical or unethical ways.’

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

55

AARHUS UNIVERSITY

MEDIATION OF PERCEPTION

• › Simple : Me -> World

› Mediated : Me -> Technology -> World

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

56

AARHUS UNIVERSITY

==> picture [279 x 241] intentionally omitted <==

==> picture [314 x 241] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

57

AARHUS UNIVERSITY

MEDIATION OF ACTION

• › Inscription

› ‘The designer, who can be seen as the inscriber of scripts.’ › When we design

• › Scripts

› The influence of artifacts on human actions is a “script”

› Typical patterns of action

• › Translation

• › To new (or less) action possibilities (e.g., citizen+gun)

› Typical(/possible) outcomes

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

58

AARHUS UNIVERSITY

MEDIATION OF ACTION - EXAMPLES

==> picture [47 x 40] intentionally omitted <==

==> picture [216 x 295] intentionally omitted <==

2026

ITPDP

59

AARHUS UNIVERSITY

MEDIATION OF ACTION - EXAMPLES

==> picture [612 x 156] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

60

ETHICS

Dark Patterns

AARHUS UNIVERSITY

DARK PATTERNS

• › Basic assumption that UX features can be linked to similar user behavior

• › (Dark) patterns as a way to describe design → ‘scripts’ (cf Verbeek)

• › Pattern use suggests a causal relationship between intention → feature → behavior

• › Gray et al paper

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

62

AARHUS UNIVERSITY

DARK PATTERNS

==> picture [660 x 160] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

63

AARHUS UNIVERSITY

DARK PATTERNS

==> picture [331 x 261] intentionally omitted <==

==> picture [234 x 175] intentionally omitted <==

==> picture [276 x 129] intentionally omitted <==

2026

ITPDP

64

AARHUS UNIVERSITY

DARK PATTERNS & SOCIAL MUSIC THEME?

==> picture [306 x 305] intentionally omitted <==

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

65

GDPR

AARHUS UNIVERSITY

GDPR

• › General Data Protection Regulation is an EU regulation aimed at strengthening and harmonising the protection of personal data in the European Union.

• › Must protect the individual's rights and processing of personal data – consent, security, the right of access and the right to be forgotten, etc.

• › It is something we must relate to when we involve others than ourselves in the design

• process

• › Until now, you have mostly been the 'data subject' – now you will potentially also be data responsible!

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

67

AARHUS UNIVERSITY

GDPR RULES

• › Consent must be clearly obtained independently of other requests.

› Consent must be obtained with clear information about scope, purpose, responsibility and contact persons

• › In case of security breach, participants must be informed no later than 72 hours after discovery

• › The right to be forgotten must be implemented as a procedure in the process

• › A responsibility to be taken seriously ( but no need for further concern )

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

68

AARHUS UNIVERSITY

GDPR IN STUDENT PROJECTS

• › https://studerende.au.dk/en/it-support/information-security/data-protectiongdpr/projects

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

69

AARHUS UNIVERSITY

DOCUMENTS

• › Consent statement is used to obtain consent from participants – customize template as needed

• › The register of purposes is used to explain the purpose of the data collection

• › Data responsibility is used, as a group, to enter into an internal agreement on joint data responsibility

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

70

AARHUS UNIVERSITY

DOCUMENTS

› You are responsible for the preparation of the documents

› You are responsible for storing the documents

› You are responsible for the storage of data and GDPR

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

71

AARHUS UNIVERSITY

TIPS

• › Just get it done and learn that it is part of the study and our practice

• › Don't collect data you don't know what you need for (sensor data?)

• › Try to anonymize and 'get away from' data as early as possible (Clemens → Respondent M1)

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

72

AARHUS UNIVERSITY

WHAT CAN WE HELP WITH?

› Read through the documents when they are finished (to help)

› Answer questions

› Not so much more – it's agreements and your responsibility

› TA session where you’ll look at it!

==> picture [47 x 40] intentionally omitted <==

2026

ITPDP

73

==> picture [97 x 41] intentionally omitted <==

EVALUATION ITPDP’26, L13, W15/19

Assoc Prof. Minna Pakanen Department of Digital Design and Information Studies mpakanen @cc.au.dk

6[TH] MAY 2026

MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

TODAY

› Ways of evaluating

• › Usability testing

• › Early development phase UX evaluations

• › Prototypes in early development phase evaluations

• › Data gathering and recording methods

• › Evaluation planning and roles in the evaluation

• › TA session: Planning of the evaluation of your own ITPDP product!

6[TH] MAY 2026

MINNA PAKANEN

WAYS TO EVALUATE

6[TH] MAY 2026

MINNA PAKANEN

4

WHY DO WE EVALUATE?

› We are not designing for ourselves › Feedback on the design process

› Iterative development › ”Fail early, fail often"

6[TH] MAY 2026

MINNA PAKANEN

“Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process, you are going to throw buckets of money down the drain .”

(Bruce Tognazzini)

6[TH] MAY 2026

MINNA PAKANEN

THREE WAYS TO EVALUATE

Usability testing/ UX evaluation

• Test and evaluation setting controlled by the evaluator

• With users

Field evaluation

• Evaluation in a natural environment with real users and their actions

• The user’s surroundings and tasks set the evaluation frame

Analytical evaluation

• No users are involved!

• Heuristic evaluation

You will learn this in the coming HCI course!

• Walkthroughs

Combination of these!

6[TH] MAY 2026

MINNA PAKANEN

6[TH] MAY 2026 MINNA PAKANEN

USABILITY TESTING PLEASE DO NOT USE THIS IN YOUR EVALUATION!

6[TH] MAY 2026

MINNA PAKANEN

11

AIMS OF USABILITY TESTING

• › Interested of how easy it is for the user to use the system

• :

• › Usability is defined by 5 quality components

• Learnability : How easy is it for users to accomplish basic tasks the first time they encounter the design?

• Efficiency : Once users have learned the design, how quickly can they perform tasks?

• Memorability : When users return to the design after a period of not using it, how easily can they re-establish proficiency?

• Errors errors : How many do users make, how severe are these errors, and how easily can

• they recover from the errors?

• Satisfaction : How pleasant is it to use the design?

• (https://www.nngroup.com/articles/ten-usability-heuristics/)

6[TH] MAY 2026

MINNA PAKANEN

USABILITY TESTING

› Three ways of conducting:

• Laboratory-based user observations

• Controlled user testing

• › facilitator can help the user if the user does not know how to proceed, but this needs to be marked down in the observation notes!

• Expert inspection techniques

• › Analytical evaluation methods you will be learning more in the HCI course

(Greenberg & Buxton, 2008)

6[TH] MAY 2026

MINNA PAKANEN

USABILITY TESTING

› Controlled laboratory-based user (covert) observations & studies

==> picture [405 x 194] intentionally omitted <==

==> picture [284 x 194] intentionally omitted <==

==> picture [257 x 194] intentionally omitted <==

https://www.brighton.ac.uk/business-services/consultancy/user-centre-design-lab/index.aspx https://www.volkside.com/2009/12/tip-print-out-your-usability-testing-tasks/ https://www.testingtime.com/en/blog/in-house-usability-tests/ https://ux247.com/usability-lab-dead/

==> picture [224 x 49] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

What are the challenges of usability evaluations conducted in an early design phase according to Greenberg & Buxton (2008)?

6[TH] MAY 2026

MINNA PAKANEN

EARLY DESIGN IDEAS AND PROTOTYPES

• › Early designs are “sketches”

• Illustrate the essence of an idea, but have

• many underdeveloped aspects to it

• Can be represented in many forms

• Work as externalization of the idea and help the designer to iterate the design

• › Early prototypes

• Have many holes and undeveloped attributes

(Greenberg & Buxton, 2008)

==> picture [348 x 311] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

CHALLENGES OF USABILITY STUDIES IN EARLY DESIGN PHASE

› Usability study is not the best approach to evaluate early designs or prototypes because

• It focuses on negative aspects:

• Errors & bugs

• Task completion time: faster the better

• –> Abandoning a promising idea too early –> Focus on developing aspects that can be measured, usually something we are already familiar with

• –> Limit the number of alternative ideas

==> picture [392 x 273] intentionally omitted <==

(Greenberg & Buxton, 2008)

6[TH] MAY 2026

MINNA PAKANEN

EARLY PHASE USER EXPERIENCE EVALUATIONS

6[TH] MAY 2026

MINNA PAKANEN

19

USER EXPERIENCE EVALUATION

• › UX studies focus on lived experiences

• A holistic view of the user’s interaction with a product

  • Emotions, enjoyment & aesthetics

• Focuses on positive aspects of the use

  • Hedonic and non-instrumental aspects of use (non-task related)

• Dynamic and situational aspects

  • Different time spans (before, during, after the use)and momentary experiences (changes over time spans)

  • Context impacts the experience

(Bargas-Avila & Hornbæk, 2011)

6[TH] MAY 2026

MINNA PAKANEN

AIMS OF UX EVALUATIONS

• › To find out how the user feels about the system and interaction

• with it

• Qualitative or mixed-methods approach in research

  • Interviews, focus groups, observations

  • Questionnaires

  • Constructive and creative techniques (drawing & collages)

• UX studies can be conducted at any phase of the design process (concept ideas – – – –>final products)

(Bargas-Avila & Hornbæk, 2011)

6[TH] MAY 2026

MINNA PAKANEN

EARLY DEVELOPMENT PHASE UX EVALUATIONS

• › Early phase user experience evaluations are done to

• › help to choose the best design for the development

• › Evaluating that the development is on the right track

• › Examining if the final product can meet the set UX targets

(Stone et al. 2005)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED USER EXPERIENCE (AUX)

can ”Anticipated UX happens before the first use, or it happen also within the other time spans of UX, e.g. during and after the use and over time, as a person may imagine also during those time spans. A person can have indirect experience prior the first use through formed expectations brand of existing experience with related technologies, , advertisements , presentations, and other peoples’ opinions.”

(Roto et al., 2011)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED EXPERIENCE

”Anticipated user experience means the experiences and feelings that are expected to occur when the user is imagining using an interactive product.”

(Yogasara et al., 2011)

6[TH] MAY 2026

MINNA PAKANEN

PROBLEMS WITH THIS DEFINITION?

==> picture [467 x 380] intentionally omitted <==

----- Start of picture text -----
?
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

ANTICIPATED USER EXPERIENCE (AUX)

Users should not be put in a situation where they have to imagine their future needs without giving . concrete options for them

(Von Hippel, 1986)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED USER EXPERIENCE (AUX)

” needs wishes that result from Experiences, , and anticipated interaction with a concept of the product before the actual product exists.”

(Pakanen, 2015)

6[TH] MAY 2026

MINNA PAKANEN

PROTOTYPES IN EARLY DEVELOPMENT PHASE UX EVALUATIONS

6[TH] MAY 2026

MINNA PAKANEN

29

PROTOTYPES IN AUX EVALUATIONS

• › Prototypes should

• › evoke people’s dreams for the future (van den Hende 2010)

• › give a sense of experience before the actual artefact exists (Roto et al. 2011)

• › allow envisioning the concept and prevent unwanted confusion

  • (Kuutti et al. 2001, Gegner & Runonen 2012)

• › focus subject’s attention on studied things (Lim & Stolterman 2008)

• › allow utilizing concepts key characteristics directly to the design (Law 2011).

6[TH] MAY 2026

MINNA PAKANEN

MIXED FIDELITY PROTOTYPES

5 filter dimensions of prototypes

• Appearance

• size; color; shape; margin; form; weight; texture; proportion; hardness; transparency; gradation; haptic; sound

• The functionality

• system function; users’ functionality need

• The interactivity

• input behavior; output behavior; feedback behavior; information behavior

• The data

• data size; data type (e.g., number; string; media); data use; privacy type; hierarchy; organization

• The spatial structure

• arrangement of interface or information elements; relationship among interface or information elements— which can be either two- or three-dimensional, intangible or tangible, or mixed

(Lim & Stolterman, 2008)

6[TH] MAY 2026

MINNA PAKANEN

MANIFESTATION DIMENSIONS

==> picture [484 x 311] intentionally omitted <==

(Lim & Stolterman, 2008)

6[TH] MAY 2026

MINNA PAKANEN

FILTER-FIDELITY-PROFILES IN EVALUATION

5 filter-dimensions of prototypes

• Appearance

• Size, color, shape, weight, hardness, haptic, sound, & arrangement

• The functionality

• Breadth (functions realized) & depth (completeness)

==> picture [163 x 125] intentionally omitted <==

• The interactivity

• Action, reaction, input modality, & output modality

• The data

• Closeness to the reality, information architecture, data model, & amount and type of data

• The physicality

• Spatial position, coherence of tangibles, & tangible embodiment

(Kohler & Hochreuter, 2014)

==> picture [163 x 124] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

PROTOTYPES IN EARLY UX EVALUATIONS

==> picture [450 x 345] intentionally omitted <==

==> picture [503 x 346] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

COMPARABLE VISUAL MATERIALS IN AUX EVALUATIONS

Extending the existing prototype and comparing alternative visualization styles for GUIs When preparing comparable visual materials, make sure the comparable examples and items under evaluation are same in: 1) Relative size

• 2) Color scheme, unless you compare different color options

3) Level of detail and visualization style (sketch, line drawing, shaded, …)

Note that your alternative visualizations should all be presented in same application context (contextualization).

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE: VIRTUAL AVATARS FOR AR & VR IN COLLABORATIVE SETTING

==> picture [832 x 352] intentionally omitted <==

(Pakanen et al. 2022)

6[TH] MAY 2026 MINNA PAKANEN

EXAMPLE: RESEARCH/ BENCHMARKING

(Pakanen et al. 2022)

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE: EXAMPLES

• 1) Relative size

• 2) Color scheme

• 3) Level of detail & visualization style

==> picture [455 x 466] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

(Pakanen et al. 2022)

EXAMPLE: CONTEXT

Examples are shown both in AR & VR context

(Pakanen et al. 2022)

6[TH] MAY 2026

MINNA PAKANEN

COMPARABLE VISUAL MATERIALS

==> picture [722 x 370] intentionally omitted <==

----- Start of picture text -----
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

(Pakanen 2015)

PROTOTYPES IN EARLY UX EVALUATIONS

==> picture [642 x 321] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026 MINNA PAKANEN

VISUAL MATERIALS TO HELP EVALUATING PROTOTYPES

==> picture [422 x 299] intentionally omitted <==

==> picture [531 x 301] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the study in your report:

› How the visual materials were used?

› When they were used in the evaluation?

6[TH] MAY 2026 MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 1

Paula and her husband p lanned in the morning that she will buy groceries, and her husband will collect her from the shop. Hence, now prior leaving from the office, she pre-writes a message on her phone “At the checkout desk!” (1) and assigns it to a t urquoise color that she can remember.

When she arrives at t he checkout, she uses the bracelet device on her wrist to browse the messages that she has created with the phone (2). She finds the turquoise message (with inner ball) that she created at her office.

Then she selects the receiver, Matt, her husband (with outer ball) and sends it to him (3).

Matt is arriving to the parking lot, when Paula’s message arrives to his phone, he knows that it will only take a few minutes for Paula to come out of the shop, so rather than parking, he drives to waits at the front of the store (4).

==> picture [434 x 309] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

• Help user to see over limitations of the prototype by creating a model to show the appearance

• A nice rending/ visualization of the concept idea in use context held by a human (print/screen)

• A physical unfunctional mock-up (can be hold in hand by the user)

• A set of images + use case story (prints/ Powerpoint slides/video)

• Make sure you show this use case story before evaluating the partial and unfinished appearance (in this case size, weight, shape, materials), functionality (only two types of interactions were compared [interactivity]) or the prototype.

• Think of what aspects you need to realize as close to the final prototype (physicality as in this case tangible embodiment was almost realistically realized)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 2

Collect feedback for the further development

• 2 Altered visualizations of possible solutions presented on screenshots taken from the model

• Make sure you show the alternative vualizations after evaluating the partial and unfinished prototype

• appearance (no visualizations)

• functionality (only moving was realized in the prototype).

• Think of what aspects you need to realize as close to the final prototype (Appearance and the animations)

==> picture [199 x 393] intentionally omitted <==

(Kukka et al. 2017)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 4 Collect feedback for the further development

• One virtual environment with alternative interior and functionalities

• Make sure you let the participants see the alternative visualization after evaluating the partial and unfinished prototype

• appearance (no visualizations)

• functionality (only moving was realized in the prototype).

• Think of what aspects you need to realize as close to the final prototype (Appearance, functionalities, and the interactions)

==> picture [539 x 302] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

(Pakanen et al. 2020)

WIZARD OF OZ

6[TH] MAY 2026

MINNA PAKANEN

48

WIZARD OF OZ TECHNIQUE ALLEN MUNRO AND DON NORMAN, 1975

› The Wizard of Oz method is a moderated research method in which a user interacts with an interface that is not really working, but the system responses are initiated by a human operator

==> picture [817 x 236] intentionally omitted <==

----- Start of picture text -----
Video
camera User
Operator
https://www.nngroup.com/articles/wizard-of-oz/#:~:text=Definition%3A%20The%20Wizard%20of%20Oz,Norman%20at%20UC%20San%20Diego.
----- End of picture text -----

6[TH] MAY 2026

MINNA PAKANEN

WIZARD OF OZ TECHNIQUE AN EXAMPLE IN LAB SETTING

==> picture [385 x 267] intentionally omitted <==

==> picture [565 x 205] intentionally omitted <==

(Colley et al. 2016)

6[TH] MAY 2026

MINNA PAKANEN

WIZARD OF OZ TECHNIQUE

AN EXAMPLE IN FIELD WITH A MOBILE OPERATOR

==> picture [697 x 352] intentionally omitted <==

Pakanen et al. 2022

==> picture [44 x 6] intentionally omitted <==

----- Start of picture text -----
6 [TH] MAY 2026
----- End of picture text -----

==> picture [56 x 7] intentionally omitted <==

----- Start of picture text -----
MINNA PAKANEN
----- End of picture text -----

Reporting of the study in your report:

› What kind of Wizard of OZ evaluation was conducted?

› How was the Wizard of OZ method described in the paper? Paper: 3

MINNA PAKANEN

6[TH] MAY 2026

==> picture [377 x 326] intentionally omitted <==

----- Start of picture text -----
User
< 2m
----- End of picture text -----

WIZARD OF OZ TECHNIQUE MOBILE

==> picture [87 x 87] intentionally omitted <==

Operator

6[TH] MAY 2026

MINNA PAKANEN

PLANNING OF WIZARD OF OZ STUDY

Remember

› Assign group members to be:

• 1) Facilitator, introduces the tasks and interviews the participants

• 2) Operator/”observer” who operates the protype based on participants input/ other type of input. Make sure there is either direct or indirect (through live video feed) visibility to participants interaction with the

prototype with, so that you can operate the prototype well. If the operator is to be in the same space, then say to participants that this person is an observer

› Do not break the illusion!

• › Plan the tasks carefully and think all the possibilities what the user could do?

• If participant does unexpected things, facilitator should say: unfortunately the prototype does not function in that way, can you think any other way how it could work? (and at the end show how to use it).

• › Make sure the operator have a direct view to the participant and their interaction with the prototype

6[TH] MAY 2026

MINNA PAKANEN

USER STUDY DATA GATHERING METHODS

Blandford, Ann, Dominic Furniss, and Stephann Makri. ”Qualitative HCI research: going behind the scenes." Synthesis lectures on human-centered informatics 9, no. 1 (2016): 1-115.

6[TH] MAY 2026

MINNA PAKANEN

55

OBSERVATION

Observer observes

• › ”Fly on the wall” (Blandford et al. 2016)

• › What do people do and how do they do it?

==> picture [156 x 155] intentionally omitted <==

• › Difficulties or errors with the interaction?

• › What do they get enthusiastic about?

• › How do they operate the system?

• › Are their actions supporting what they say?

• › e.g., if they seem to have problems and they claim it is easy to use

6[TH] MAY 2026

MINNA PAKANEN

PLANNING OBSERVATION

Pay attention to

• › Select setting(s)

› It is not possible to observe everything, so decide what is to be documented in each observation.

› Plan an observation form for collecting observations for each task › When a participant does something very interesting, you can mark down the time, which helps finding that incident from the video or audio recording.

6[TH] MAY 2026

MINNA PAKANEN

THINK ALOUD

› Technique can be used in usability testing, user experience evaluation, and even in field studies › Start by explaining to the participant how to think aloud › It is not about what they do, but what they think while doing it › The tasks: chosen by participants (naturalistic) or defined by you › Thinking aloud does not come naturally to all

› You can prompt silent participant by asking: “What are you thinking” › You should politely steer too chatty participant back to the tasks › UX and usability studies have different rules

› Intervention in minimum (usability) | UX interventions for seeking clarifications

6[TH] MAY 2026

MINNA PAKANEN

INTERVIEW

› allows understanding people's perceptions and experiences

==> picture [401 x 314] intentionally omitted <==

› Different types

› Unstructured

• › not directed by a script (ethnographic)

• › Structured

• › tightly scripted, almost like a questionnaire

• › Semi-structured (most used)

• › guided by a script, interesting issues explored in more depth by asking for further details

MINNA PAKANEN

6[TH] MAY 2026

INTERVIEWING

The interviewer interviews the participant/s › Form the topics and questions: open, broad & narrow

• › Next slide what to avoid when forming questions

• › Opening the conversation

• › assure your interest in participants’ replies as an expert in the topic

› Ask the question and wait that the person gives a reply › Ask for clarifications if a person’s reply is not complete or if it is just yes or no/ good/bad

› You said that it is good, in what way/s it is good/ what features make it good?

You wished the information on the screen was clearer, in which way?

6[TH] MAY 2026 MINNA PAKANEN

INTERVIEW QUESTIONS

When creating questions avoid:

› Long questions

• › Compound sentences - split them into two

• › Jargon and language that the interviewee may not understand › What did you think of this AR feature? –> ….the feature where you can see items appearing on this real space we are in?

• › Leading questions that make assumptions

› Isn’t this feature good in…

• › Unconscious biases e.g., gender/age… stereotypes

• › As a/n elderly/woman/child you probably found this as complicated…

6[TH] MAY 2026

MINNA PAKANEN

DATA RECORDING

6[TH] MAY 2026

MINNA PAKANEN

62

VIDEO RECORDING

Pros and cons

› A rich way to capture interaction in the context + user comments are automatically in sync with user interaction

› Video recording can make participant anxious

› Placing the video camera in a more discrete location usually helps users to forget it

==> picture [86 x 86] intentionally omitted <==

==> picture [405 x 348] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

AUDIO RECORDING

==> picture [74 x 73] intentionally omitted <==

When and why

• › In some of the interviews it is fine to record audio only

• If you do not have any prototypes or visual materials, an audio recording is more discrete than a video.

› It is good to also record audio when you use video recording as sometimes it is hard to hear from the video what the user is saying especially if the camera is located far away from the user

6[TH] MAY 2026

MINNA PAKANEN

PHOTOGRAPHS

Records the moment › Capture moments as they happened. They help in reporting results and important moments in the interaction for example in project reports and publications

==> picture [73 x 73] intentionally omitted <==

==> picture [423 x 375] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

OBSERVATION NOTES

==> picture [80 x 81] intentionally omitted <==

Fast in situ note taking

› Observation notes are a good way of getting the most important things marked down, you can even mark the time when a user did or said something that was unexpected or interesting in some other way

› Notes can be used also as the basis of analysis (i.e., themes)

6[TH] MAY 2026

MINNA PAKANEN

LOG DATA

==> picture [74 x 73] intentionally omitted <==

Recording user interaction on the device › records user actions in the device log

› You can find errors and quantify your results easily with log data, but remember that you need the before-mentioned methods to understand why people made mistakes or did things in certain ways

6[TH] MAY 2026

MINNA PAKANEN

UX EVALUATION AND DATA CAPTURING METHODS

==> picture [709 x 475] intentionally omitted <==

----- Start of picture text -----
https://experienceresearchsociety.org/ux/evaluation-methods/
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

~~R~~ eporting of the study in your report: › What kind of data collection and recording methods were used?

› Were there some methods that were not described here?

6[TH] MAY 2026

MINNA PAKANEN

EVALUATION PLANNING TASKS, PROCEDURE, & INTERVIEW QUESTIONS

6[TH] MAY 2026

MINNA PAKANEN

78

PLANNING

1. Prepare procedure – steps, tasks…”script of the evaluation” 2. Prepare interview (+questionnaires) questions – background, task related, end

2. Pilot study – try out your procedure, tasks, forms, questions & how long the evaluation will be

3. Iterate your procedure – if something needs to be changed 5. 2[nd] Pilot – test if changes work better

4. Recruit participants – amount depends of the study, from 6-14-30-…, set also times for the studies and let them know the estimated duration

5. Prepare for the evaluation – cameras, video cameras, tripods, audio recording, print forms, get gifts for the participants, book rooms…

6. Conduct your study- follow your procedure, record the data and store it for analysis, remember to thank your participants after the evaluation! 6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE Applied from Leena Arhippainen, UX researcher and Minna’s Phd supervisor

6[TH] MAY 2026

MINNA PAKANEN

80

PROCEDURE

1. Consent from filling- participant agrees on the study (See L3: GDPR)

2. Intro to the study- who, why and what

3. Background info of the users- questionnaire or interview

4. Ice breaking task – easy task to get user familiar with the prototype 5. Actual task(s) with the prototype – design the task so that you are able to get feedback on right things

5. Post questionnaires- preferences and quantitative data

6. Semi-structured interview – get more detailed understanding how user felt using your prototype and what s/he thought about the design idea in general 6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE

Example of procedure:

• Everything is explained so it is easy to follow in the actual evaluation

1. Consent & background questionnaire form filling 2. Who are we and explanation of the purpose of the study (even if explained in the consent form in detail)

2. Actual tasks are written down so that each participant is instructed in a similar way

3. Questionnaire fillings are marked in the procedure 5. Interview questions for post-interview (can also be on different paper)

4. Thank and reward the participants

==> picture [348 x 492] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE

2. Introduction to the study

• Who are we

• Short description of the study aim

• Tell the user once more that they can withdraw from the study

  • whenever they feel like it

• Ask the user to think aloud during the tasks (+how to do it)

• In Usability evaluation: explain if you will not help the participant during the tasks

• Ask if the user has any questions

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

2. Introduction to the study

MINNA PAKANEN

6[TH] MAY 2026

Reporting of the study in your report:

› Was there differences in the example procedure that was given to your group?

MINNA PAKANEN

6[TH] MAY 2026

PROCEDURE

3. Background information

• What do you need to know about your user?

  • Age

  • Gender: male/female/nonbinary

  • Educational background/ profession

  • Prior experience with similar technology that you are evaluating

  • Prior experience with the use context (e.g., fab labs, pottery, chemical lab, etc.) of your system

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

3. Background questionnaire

Note that you do not necessarily need this, especially if you have only a few things to ask. The questions can also interviewed in the beginning or at the end of the evaluation!

==> picture [353 x 457] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

Reporting of the study in your report:

› How the participants were described in the paper, and what kind of information was given about them? › What way the description was written?

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

4. Icebreaking task

• Idea is to familiarize your user with the thing under evaluation

• Easy and short task

• Reduces nervousness of the participant

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

4. Icebreaking task

• Explore the prototype freely for a while

1. Familiarization with the current 3D City model

1. Familiarize yourselves with the 3D City model simply by looking at it and giving your initial thoughts before interacting with it.

2. Now you can start interacting with the model by controlling an avatar (WASD and arrows) and explore the model freely. While exploring the 3D model, please think aloud and comment on anything that caught your attention.

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

5. Actual task(s)

• The idea is to plan tasks that the user can do and by doing the tasks you will learn if your product is understandable for the user or how they do experience the prototype

  • Try to keep them short and not too complicated to conduct

  • Test how your core use case activities are working

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

5. Tasks

1st task:

1. First you want your handbag to match with these red shoes over here, what would you do?

2. Next you can try to make the bag imitate the fabric and color of this of this scarf here.

3. And then this shirt here.

Questions after the tasks are completed What did you think of this idea? Would it be useful for you?

Is the idea fun?

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

6. post questionnaire

• Sometimes you might need user feedback on different designs so you can use certain questionnaires to help assess different things.

• The attractiveness of the product: Attrakdiff https://www.attrakdiff.de/index-en.html (online service was discontinued from January 2025 L)

• Can be used for

• Single evaluation

• Comparison

• Before-after

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

6. Post questionnaire With Attrakdiff questionnaire

==> picture [547 x 189] intentionally omitted <==

==> picture [351 x 496] intentionally omitted <==

(Colley et al. 2016)

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

• › Visual presentation of Attrakdiff findings in the report

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

7. Semi-structured interview

• The last part of the evaluation is for asking questions from the user to find out their thoughts about the prototype and the interaction with it

  • What is good/bad in the design idea and the prototype?

  • What was difficult?

  • What would make it easier to use?

  • What would you like to change in it to make it more interesting to you?

  • Try to avoid questions that can be replied to just with yes or no!

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

7. Post-interview questions

• Describe your experiences with the system.

• What do you think of using this kind of technology for remote collaboration?

• What was easy/natural and what was difficult/unnatural in the interaction?

• Were there any critical moments in the interaction? If yes, describe it/them.

6[TH] MAY 2026

MINNA PAKANEN

HOW DO YOU DESCRIBE YOUR STUDY IN THE REPORT?

How procedure is reported in the paper?

› How is the procedure description in the paper differentiating from the study procedure, or is it doing that at all?

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the study in your report:

› How the procedure is reported in the paper?

› Is the procedure description in the paper differentiating from the study procedure, if so, how?

6[TH] MAY 2026

MINNA PAKANEN

ROLES IN THE EVALUATION

6[TH] MAY 2026

MINNA PAKANEN

102

ROLES IN THE EVALUATION

1. Facilitator/interviewer

• Facilitates the study, gives tasks, interviews, handles questionnaires

2. Observer

• Stays behind and observers what user does

• Takes notes on each task-can also write down user comments

3. Documenter (can be also facilitator)

• Videos/photos/audio recording

Pick a role that fits best for you!

6[TH] MAY 2026

MINNA PAKANEN

ROLES IN THE EVALUATION

An example

==> picture [369 x 283] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

QUALITATIVE DATA ANALYSIS Thematic Analysis

Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004

6[TH] MAY 2026

MINNA PAKANEN

105

PLEASE REVISIT MY SLIDES FROM L5

› https://brightspace.au.dk/d2l/le/lessons/202542/topics/256 9495 › There is detailed explanation how to conduct thematic analysis!

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the data analysis in your report:

› How the qualitative data analysis was reported in the papers?

6[TH] MAY 2026 MINNA PAKANEN

Reporting of the findings in your report:

› Check the example papers for tips how to report qualitative findings in your report ›How participants were identified? ›How participants quotes were written? › Pay attention to the story!

6[TH] MAY 2026

MINNA PAKANEN

PLANNING YOUR OWN EVALUATION TA SESSION FROM 13:15-> TODAY!

6[TH] MAY 2026

MINNA PAKANEN

132

TASK FOR TA

Start planning your evaluation

• Who are your participants? You need 5 participants

• Think of your research question

• Think what kind of tasks and questions would help you to answer to your research question

• Make an early draft of your procedure

• Think about the roles: facilitator, observer, video recorder…

6[TH] MAY 2026

MINNA PAKANEN

WHEN THE PROTOTYPE IS “READY”

Conduct a pilot evaluation

• Get participant/s from the other group/s

• Follow your procedure rigorously

• Act like you do not know your participant beforehand

• After the pilot you have an understanding whether your tasks are easy to understand or too complicated to pass and if your questions can reveal knowledge you were hoping for -> Iterate your procedure, tasks and questions, do another

• pilot study before you go to the actual evaluations!

6[TH] MAY 2026

MINNA PAKANEN

COURSE LITERATURE REFERENCES

• Blandford, Ann, Dominic Furniss, and Stephann Makri. "Qualitative HCI research: Going behind the scenes." Synthesis lectures on human-centered informatics 9, no. 1 (2016): 1-115.

• Greenberg, Saul, and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proc. CHI '08. ACM, New York, NY, USA, 111–120.

• Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Vaananen-Vainio-Mattila. 2010. User experience evaluation methods: current state and development needs. In Proc. NordiCHI '10. ACM, New York, NY, USA, 521–530. https://doi.org/10.1145/1868914.1868973

• Lim N-K & Stolterman E (2008) The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction 15(2): A7.

• Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004

6[TH] MAY 2026

MINNA PAKANEN

OTHER REFERENCES

• Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proc. CHI '11. ACM, New York, NY, USA, 2689–2698. https://doi.org/10.1145/1978942.1979336

• Ashley Colley, Minna Pakanen, Saara Koskinen, Kirsi Mikkonen, and Jonna Häkkilä. 2016. Smart Handbag as a Wearable Public Display - Exploring Concepts and User Perceptions. In Proceedings of the 7th Augmented Human International Conference 2016 (AH '16). Association for Computing Machinery, New York, NY, USA, Article 7, 1–8. https://doi.org/10.1145/2875194.2875212

• Gegner L & Runonen M (2012) For what it is worth: Anticipated experience evaluation. In: Brassett J, Hekkert P, Ludden G, Malpass M & McDonnell J (eds) Proceedings of the 8th International Conference on Design and Emotion. URI: https://reseda.taik. fi/Taik/jsp/taik/Publication_Types.jsp?id=23583813. Cited 2015/08/15.

• Hende, EA van den (2010) Really new stories the effect of early concept narratives on consumer understanding and attitudes. Doctoral dissertation. Delft University of Technology.

• Hippel E von (1986) Lead Users: A source of novel product concepts. Management Science 32(7): 791–805.

• Kuutti K, Battarbee K, Sade S, Mattelmaki T, Keinonen T, Teirikko T, & Tornberg A-M (2001) Virtual prototypes in usability testing. Proceedings of the 34th Hawaii International Conference on System Sciences. IEEE: 1–7.

6[TH] MAY 2026

MINNA PAKANEN

OTHER REFERENCES

• Law ELC (2011) The measurability and predictability of user experience. Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York NY, ACM: 1–10.

• Pakanen M (2015) Visual design examples in the evaluation of anticipated user experience at the early phases of research and development. Doctoral dissertation. University of Oulu.

• Pakanen, M., Alavesa, P., Van Berkel, N., Koskela, T., & Ojala, T. (2022). “Nice to see you virtually”: Thoughtful design and evaluation of virtual avatar of the other user in ar and vr based telexistence systems. Entertainment Computing, 40, 100457.

• Roto V, Law E, Vermeeren A, & Hoonhout J (eds) (2011) UX white paper. https//allaboutux.org/uxwhitepaper. Cited 2015/03/08.

• Stone D, Jarrett C, Woodroffe M, & and Minocha S (2005) User interface design and evaluation. Morgan Kaufmann Series in Interactive Technologies. San Francisco CA, Morgan Kaufman.

• Tognazzini, B. 1990. User Testing on the cheap. In Tog on Interface, Addison Wesley publ.

• Yogasara T, Popovic V, Kraal B, & Camorro-Koc M (2011) General characteristics of anticipated user experience (AUX) with interactive products. Proceedings of the 4th World Conference on Design Research: Diversity and Unity. Delft, Delft University of Technology: 1-11. URI: http://eprints.qut.edu.au/47035/. Cited 2015/06/15.

6[TH] MAY 2026

MINNA PAKANEN