~~TH~~ E DIARY STUDIES & CULTURAL PROBES + QUALITATIVE ANALYSIS

ITPDP’26, L5

Dr. Minna Pakanen,

Department of Digital Design and Information Studies mpakanen @cc.au.dk,

23[RD] FEB 2026

MINNA PAKANEN

TODAY

• The diary studies

• Media in the diary studies

• Cultural probes

• Qualitative data analysis

23[RD] FEB 2026

MINNA PAKANEN

23[RD] FEB 2026 MINNA PAKANEN

THE DIARY STUDY

==> picture [296 x 341] intentionally omitted <==

Carter, S., & Mankoff, J. (2005). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’2005). ACM, New York, 899-908.

23[RD] FEB 2026

MINNA PAKANEN

5

TYPICAL DATA COLLECTION MEANS IN THE DIARY STUDY

==> picture [288 x 288] intentionally omitted <==

==> picture [289 x 131] intentionally omitted <==

----- Start of picture text -----
+ +
----- End of picture text -----

==> picture [231 x 231] intentionally omitted <==

https://tinyurl.com/3ju6tmt2

https://penstore.dk/dk/ballograf/epoca-p-kulglepen

https://vshopg.tk/products.aspx?cname=custom+disposable+cameras&cid=109

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY

• Is a research method used to collect qualitative data about user behaviors, activities, and experiences in situ

• What, where, who?

• Longitudinal: from a few days to months (typically 3-14 days)

• Researchers are remote from participants:

  • Participants control the timing and means of capture

  • Researchers suggest the timing and means of capture

    • Guidance in the tasks description

    • Prompting by messages, emails, notifications

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLES OF DIARIES

==> picture [467 x 281] intentionally omitted <==

==> picture [485 x 281] intentionally omitted <==

Anna Luusua, Johanna Ylipulli, Marko Jurmu, Henrika Pihlajaniemi, Piia Markkanen, and Timo Ojala. 2015. Evaluation Probes. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM press, 85–94. https://doi.org/10.1145/2702123.2702466

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY ACTIVITIES TIMELINE

==> picture [74 x 73] intentionally omitted <==

Briefing

==> picture [73 x 73] intentionally omitted <==

Post-study interview

• Explain study • Discuss entries

• Provide materials and instructions

• Probe for additional insights

• Set expectations • Ask for clarification Pilot Actual diary study logging Data analysis Planning

• Select capturing methods • Evaluate entries as they come in • Create materials & instructions –> questions for post-study interview

• Recruit study participants • Check in with participants

• Select an analysis method • Conduct analysis rounds • Report findings

• Define the study timeline

• Schedule briefs and interviews

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY

• Can take place as:

• A feedback study

• An elicitation study

• Combination: results from the feedback study act as prompts for discussion in the elicitation study

23[RD] FEB 2026

MINNA PAKANEN

FEEDBACK STUDY

• Participants answer predefined questions about certain events

• Asynchronous communication between the researcher and participant

• Participant driven: participants answer questions about some event when it occurs

• Benefits:

• as questions are asked at the time of the event, or in situ –>provide accurate responses to questions that depend on recall of the event

• Drawbacks:

• Overburdening participants with questions, when the number of events reported is high because the act of answering questions is a significant distraction from their main task.

23[RD] FEB 2026

MINNA PAKANEN

ELICITATION STUDY

• Participants capture media that are then used as prompts for discussion in interviews

• Synchronous communication between the researcher and participant

• Participants merely capture some information about the event that will serve as a memory cue during a later interview

• Benefits:

• Less burdensome than feedback

  • rapid capture of prompts, such as a photograph, audio, tangible objects

• Drawbacks:

• Potentially inaccurate recall

23[RD] FEB 2026

MINNA PAKANEN

CONTENTS OF THE DIARY

• An introduction page with clear instructions and expectations for how often participants should fill in information

• Question pages for every day you expect your participant to be filling in the diary

• What did you do today?

• What did you like the most about [task/activity]?

• What did you like the least about [task/activity]?

• How could that have been made better for you?

• How did you feel about your experience [doing task/activity] today?

• Why did you feel that way?

• Did anything get in your way?

• Instructions for any photographs or videos you want them to take each day.

• A wrap-up page with concluding questions to be filled at the end of the study

https://outwitly.com/blog/research-methods-diary-study/

23[RD] FEB 2026

MINNA PAKANEN

What kind of questions related to the course topic you could add in a diary study?

Social music listening/ composing & music creation/ learning?

23[RD] FEB 2026

MINNA PAKANEN

==> picture [194 x 195] intentionally omitted <==

==> picture [170 x 169] intentionally omitted <==

ROLE OF MEDIA IN DIARY STUDIES

Carter, S., & Mankoff, J. (2005). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’2005). ACM, New York, 899-908. Kumpel, A.S. (2021). Using Messaging Apps in Audience Research: An Approach to Study Everyday Information and News Use Practices, Digital Journalism, DOI: 10.1080/21670811.2020.1864219

23[RD] FEB 2026

MINNA PAKANEN

15

FIND THE INFORMATION IN GROUPS

Open the article from Bs (Content -> Week 6/10 Diary Studies -> Before class–> Carter & Mankoff, 2005. When participants do the capturing: the role of media in diary studies)

Groups 1-3: Photo diary study Groups 4-6: Transit decision diary study Groups 7-9: Festival diary study What was the focus of the study? What kind of data was gathered? What were the results of the study?

23[RD] FEB 2026

MINNA PAKANEN

Photo/transit/festival diary study What was the focus of the study? What kind of data was gathered? What were the results of the study?

23[RD] FEB 2026

MINNA PAKANEN

MEDIA IN THE DIARY STUDIES

==> picture [628 x 358] intentionally omitted <==

----- Start of picture text -----
Easy recall of
the episode
Easy to Difficult to
capture capture
Difficult recall
of the episode
----- End of picture text -----

23[RD] FEB 2026

MINNA PAKANEN

MEDIA IN THE DIARY STUDIES

==> picture [75 x 75] intentionally omitted <==

==> picture [71 x 71] intentionally omitted <==

• Photos: the easiest to capture and recognize: best support of who and where recognition

Studies where details are important

• Audio clips: easy to capture events secretively & lightweight media appropriate for annotation, but suffer from recognition problem, but once recognized provides adequate support to participants to recall events

==> picture [73 x 74] intentionally omitted <==

• Tangible objects: do not lend themselves to who or where recognition, but elicit participants’ creative explanations of attitudes and beliefs

==> picture [75 x 75] intentionally omitted <==

• Raw location data: is not likely to lead to better recall of an episode. (Experience sampling method (ESM) works in some studies when the amount of events stays within a proper threshold)

23[RD] FEB 2026

MINNA PAKANEN

HOW TO SUPPORT RECALLING?

• To support recall of ambiguous events

• A brief annotation for each capture event is needed

• Suggested way:

==> picture [87 x 88] intentionally omitted <==

==> picture [87 x 88] intentionally omitted <==

==> picture [74 x 73] intentionally omitted <==

==> picture [74 x 74] intentionally omitted <==

thorough review by Lightweight lightweight capture tools in-situ annotation ex-situ annotation participants & researchers

23[RD] FEB 2026

MINNA PAKANEN

MESSAGING APP DIARY APPROACH +

• Using of messaging apps already used by people (WhatsApp, etc.)

• No additional apps needed –> convenience

• Easy to integrate into participants’ daily routines–> continuous participation & improved response rates

• Provides easy-to-use multimedia documentation solutions for the sharing of rich and context-sensitive data

• Allows researcher for an easy way for sending prompts & an instant feedback channel

• Allows participants to record their experiences, feelings, and thoughts “where they are”

(Kumpel, 2021) 23[RD] FEB 2026 MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

What could be the downsides of using WhatsApp in the diary studies?

23[RD] FEB 2026

MINNA PAKANEN

– MESSAGING APP DIARY APPROACH

• Needs constant attention from the researcher in the documentation phase –> time-consuming

• –>

• Apps are privately owned by international companies data security, GDPR of sensitive data

• Participants allow the researcher into their personal world–> mixing of personal life and participation in research

23[RD] FEB 2026

MINNA PAKANEN

DIARY STUDY PIPELINE

• 1) A participant takes a photo

• 2) The participant annotates the photo with an audio recording

3) The participant uses a tool (an application) to log the photo and audio and add more annotations

• 4) The researcher provides feedback about the captured data

5) The researcher holds an elicitation interview with the participant using the captured media as prompts

23[RD] FEB 2026

MINNA PAKANEN

CULTURAL PROBES

Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: cultural probes. Interactions, 6(1), 21-29.

23[RD] FEB 2026

MINNA PAKANEN

26

USER-CENTERED INSPIRATION

• Data collection to inspire design

• Stimulate designers imagination

• Helps designers to familiarize with the design context and learn the culture

• Helps to establish a conversation with the target group of people that can last throughout the project

• Overcomes the distance

• Geographic and cultural + language barriers

• Respects the respondents

23[RD] FEB 2026

MINNA PAKANEN

CULTURAL PROBES

• Personal communication tool between the target group

• Designed for specific project, people & their environments

• Package containing:

• Maps + stickers

• Postcards

• Disposable camera

• Booklets: photo album, diary

• Is left behind and returned ‘filled’ to the designers!

==> picture [403 x 540] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

AESTHETICS

• Idea is to reduce the distance between the designers and target group

• Little bit abstract/alien design aesthetics

• Delightful, but not childish

• Not too professionally finished!

• Personal and informal feeling

• Revealing the energy put to the creation

• Revealing the tastes and interest of the designers –> making target group to reveal themselves to the designers

==> picture [305 x 358] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

==> picture [120 x 129] intentionally omitted <==

EXAMPLES CATHERINE LEGROS

==> picture [654 x 331] intentionally omitted <==

==> picture [120 x 104] intentionally omitted <==

==> picture [120 x 104] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

EXAMPLE: ITPDP PROJECT 2025

AESTHETICS RELATED TO THE THEME

==> picture [522 x 332] intentionally omitted <==

• © Hook’d team: Agnes Niewald de Place, Kamilla Nørgaard Nielsen, and Julian Philipp Leimbeck

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLE: BSC PROJECT 2020

PACKAGE AND SURPRISE

Keep the interest alive during one week study

==> picture [792 x 233] intentionally omitted <==

• © Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

MAP IN PROBES

INQUIRING ATTITUDES ABOUT THE ENVIRONMENT

• Map, questions and different coloured dot stickers to mark the answers on where:

• They would go to meet people?

• They would go to be alone?

• They liked to daydream?

• They would like to go but can’t?

• To emphasise individuality:

• Printed on different kinds of papers and cut into envelope forms –> easy to fold together and put into mail

==> picture [415 x 491] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

POSTCARD IN PROBES

• Inquiring about target groups’ attitudes towards their lives, cultural environments, and technology

• Informal and friendly way of communication

• Place image at front & 1 question at back:

• Please tell us a piece of advice or insight that has been important to you. (attitude)

• What do you dislike about (city/building)? (context)

• What place does (x) have in your life? (attitude)

• Tell us about your favourite device (technology)

• Oblique wording and evocative images to open space for possibilities

• Name and address of the designer + pre-stamping for return!

==> picture [354 x 456] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

==> picture [355 x 456] intentionally omitted <==

Create a probes stcard with one ~~po~~ image on the front and one question about attitude/context/tech nology related to the Social music topic on the back.

MINNA PAKANEN

23[RD] FEB 2026

==> picture [97 x 41] intentionally omitted <==

23[RD] FEB 2026 MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

Give your card to another team and wait for the card to return to your team’s desk later J

23[RD] FEB 2026

MINNA PAKANEN

CAMERA IN PROBES

• Should be repackaged to separate it from its commercial origins and to make it fit with the other probe materials

• Mark on the backside/instruction label the requests for pictures:

• Your home

• Something desirable

• Something boring

• Unassigned pictures –> whatever participants want to show to designers before mailing the camera back

==> picture [359 x 338] intentionally omitted <==

Disposable camera: https://tinyurl.com/47rej4mm Reusable camera: https://tinyurl.com/53rkmcus Instant camera: https://tinyurl.com/3bsws9jn

23[RD] FEB 2026

MINNA PAKANEN

BOOKLETS

• Photo album

• Pictures or text to tell a story about the participant

• Diary

• The entries are done daily, for a total of a week or other period of time

• Record use of something:

  • What?

  • With whom?

  • When?

  • Example: television/ radio use or calls

==> picture [257 x 393] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLE: BSC PROJECT 2020

TASKS INQUIRING INTO DAILY LIVES IN THE CENTER

My world Floor plan Style board Who am I? Where I like/ dislike being? My day Emotion bubbles Postcards Who am I interacting with and when? What does it mean to me?

Style board

What are my aesthetic preferences?

Who I will remember from here in five years?

© Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

EXAMPLE: ITPDP PROJECT 2025

5 TASKS

==> picture [522 x 332] intentionally omitted <==

Selected fishing gear and reasoning of the preferences?

What websites do you use for research and planning before fishing trip?

Map: Fishing location preferences?

Emotional reflection through images (before & after the fishing trip)?

• © Hook’d team: Agnes Niewald de Place, Kamilla Nørgaard Nielsen, and Julian Philipp Leimbeck

23[RD] FEB 2026

MINNA PAKANEN

LAUNCHING THE PROBES

• Personal introduction to the group of participants

• Present yourself & your intentions

• Answer questions

• Encourage the participants to take an informal, experimental approach to the materials

  • Introduce the types of things participants will find from the probes package

• Learn about the participants and create enthusiasm and trust!

23[RD] FEB 2026

MINNA PAKANEN

THE RETURNED PROBES & DATA

• Allows designers to

• Familiarize themselves with the design contexts and people who are they designing for

• Ground proposals in detailed textures of the local

• Inspiration rather than direct design proposal!

==> picture [526 x 323] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

RETURNED

==> picture [649 x 141] intentionally omitted <==

==> picture [46 x 46] intentionally omitted <==

==> picture [46 x 46] intentionally omitted <==

==> picture [351 x 264] intentionally omitted <==

==> picture [451 x 269] intentionally omitted <==

© Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

RETURNED

==> picture [515 x 224] intentionally omitted <==

==> picture [245 x 286] intentionally omitted <==

© Hook’d team: Agnes Niewald de Place, Kamilla Nørgaard Nielsen, and Julian Philipp Leimbeck 23[RD] FEB 2026 MINNA PAKANEN

QUALITATIVE DATA ANALYSIS

General about qualitative data analysis (Blandford et al. 2016), Affinity mapping (Scupin 1997), and thematic analysis (Braun and Clarke, 2012)

23[RD] FEB 2026

MINNA PAKANEN

49

LEARNED IN FITDES AFFINITY DIAGRAM ( )

• Affinity diagram help organize your data into groups of similar items

• Process of affinity mapping:

1. Record all notes or observations on individual cards or sticky notes (can be done in Miro.com as long as you have pseudonymized data and your “key file” is not stored in it, if you are unsure check here: https://studerende.au.dk/en/itsupport/information-security/data-protection-gdpr/projects

2. Look for patterns in notes or observations that are related and group them

3. Create a group for each pattern or theme

4. Give each theme or group a name

5. Read it through carefully and combine or break themes until saturation is reached –>Affinity diagram is ready

   • Create a statement of what you learned about each group –> provide your analysis or key insights! 23[RD] FEB 2026 MINNA PAKANEN

MINNA PAKANEN

EXAMPLE AFFINITY DIAGRAM

Create a statement of each category to present your findings

Each participant is presented with different color Scheme names

© Rebecca Rauff Mouritsen and Christoffer Vorgaard Ashorn & Louise Mathiasen and Trine Eg Fredslund

23[RD] FEB 2026

MINNA PAKANEN

ANALYSIS/ PRESENTATION OF FINDINGS

Anonymous code

==> picture [401 x 311] intentionally omitted <==

23[RD] FEB 2026

MINNA PAKANEN

THEMATIC ANALYSIS (TA)

› Systematic and sophisticated method for identifying, organizing and offering insight into patterns of meaning (themes) across a data set

• › Accessible, flexible, and popular method

• › Driven by research question

• › Identifying relevant patterns to answering to it

23[RD] FEB 2026

MINNA PAKANEN

FLEXIBILITY AND CHOICES IN TA

Many approaches to choose from, but for you these two are important at this point:

• › Inductive approach: “What is in the data”

› Deductive approach: “Researchers decide what is mapped”

› In reality, we always combine these two approaches!

23[RD] FEB 2026

MINNA PAKANEN

FLEXIBILITY AND CHOICES IN TA

Inductive approach

• › “What is in the data”

• › Bottom-up: the codes and themes derive from the data

• › What is mapped relates closely to the data

• › Experiential in its orientation

• › Essentialist in its theoretical framework:

  • › Assuming that there is a knowledgeable world

  • ”

  • › “Giving voice to experiences and meanings of that world as reported in the data

›

23[RD] FEB 2026

MINNA PAKANEN

FLEXIBILITY AND CHOICES IN TA

Deductive approach

• › “Researchers decide what is mapped”

• › Top-down: the codes and themes derive from researchers' ideas and concepts

• › What is mapped has a looser link to the data

• › Critical in its orientation

• › Constructionist in its theoretical framework:

  • › examining how the world is put together (i.e., constructed) and the ideas and assumptions that inform the data gathered.

23[RD] FEB 2026

MINNA PAKANEN

PREPARING DATA FOR ANALYSIS

Transcribing and working with the raw data

› Transcribe the audio data (Transcriber in Ucloud)/per participant (Px, where x is participant ID number, I= interviewer):

• › Verbatim = Capturing every sound made (e.g. throat clearing, laughter, and verbal pauses: “ah,” “um,” and “uh” + other noises (e.g. a phone ringing or a door slamming).

• › Edited= edited to be clear in grammatic (avoid this)

• › Intelligent verbatim = edits out distracting filler words & sounds, repetitions, ramblings, non-standard words, and irrelevant conversations

23[RD] FEB 2026

MINNA PAKANEN

HOW TO DO THEMATIC ANALYSIS?

6-Phase Approach

1) Familiarization

• 2) Generating Initial Codes

• 3) Searching Themes

4) Reviewing the Potential Themes 5) Defining and Naming the Themes 6) Producing the Report

In practice, these phases are done in parallel!

23[RD] FEB 2026

MINNA PAKANEN

1) FAMILIARIZATION

Start by familiarizing yourself with the data › Listening to the audio, watching videos, and reading the transcripts

• › Write down some initial thoughts while doing so in a file or in a notebook

• › Annotate the whole data transcripts by underlining, noting, or color highlights

• › Helps you to read the data as data

• › Focus on things related to your research question

23[RD] FEB 2026

MINNA PAKANEN

==> picture [57 x 12] intentionally omitted <==

----- Start of picture text -----
Themes
----- End of picture text -----

==> picture [47 x 12] intentionally omitted <==

----- Start of picture text -----
Codes
----- End of picture text -----

2) GENERATING INITIAL CODES

• Systematic analysis of the data › Codes are the building blocks of the analysis (tiles)

• › Semantic: descriptive, stay close to the data

• › Latent: identify the meaning behind the semantic surface

• › Usually, a mix of both

• › Code each data item in its entirety before coding another

• › Modify codes when necessary and read the data again! › Cut and paste data from all participants in another Word file under themes (keep the participant ID (Px) with the snippet)

23[RD] FEB 2026

MINNA PAKANEN

2) CODE EXAMPLES

23[RD] FEB 2026 MINNA PAKANEN

==> picture [57 x 12] intentionally omitted <==

----- Start of picture text -----
Themes
----- End of picture text -----

==> picture [47 x 12] intentionally omitted <==

----- Start of picture text -----
Codes
----- End of picture text -----

3) SEARCHING FOR THEMES

Themes (Walls & roof)

› Searching for themes is an active process

• › Reviewing codes and coded data to identify broad topics around the

• code clusters

• › Thinking of a particular story about your data, together the themes should provide a meaningful picture of your data!

• › In an 8000-10000-word article, there are typically 2-6 themes

23[RD] FEB 2026

MINNA PAKANEN

3) CODE EXAMPLES IN THEME SEARCH

23[RD] FEB 2026

MINNA PAKANEN

4) REVIEWING POTENTIAL THEMES

Quality checking

› The developing themes are reviewed in relation to the coded data and the entire data set

› Check if the themes work with data under them, if not relocate the data snippets under another theme

• › Redraw boundaries of themes to better describe the data under them

› Consider if it is a theme or code

• › Is there enough meaningful data to support the theme?

• › Is the data too diverse and wide-ranging/ does the theme lack coherence?

–>If not create additional themes/refine the existing ones

23[RD] FEB 2026

MINNA PAKANEN

5) DEFINING AND NAMING THEMES

What is unique and specific in each theme? › Write a few sentences to sum up what is the essence of the theme

• › Good themes:

• › They have a singular focus: do not try to do too much

• › Are related to each other, but do not overlap (can build on previous themes)

• › They should directly address your research question

› Note that together the themes should tell a coherent story about your data!

23[RD] FEB 2026

MINNA PAKANEN

5) DEFINING AND NAMING THEMES

What is unique and specific in each theme?

• › Deep analytic work in this phase

• › Descriptive: using data in an illustrative way (easy to start with)

• › Conceptual and interpretative: latent meanings in the data

Often a combination of both!

• › Telling the story:

  • › Data does not speak for itself! Don’t just paraphrase the content of data

  • › Your analytic narrative will tell the reader what in the extract is interesting and why

  • › The data needs to be interpreted in relation to your research question

• › Naming should be informative, concise, and catchy

23[RD] FEB 2026

MINNA PAKANEN

5) DEFINED & NAMED THEME EXAMPLES

23[RD] FEB 2026 MINNA PAKANEN

6) PRODUCING THE REPORT

Reporting the findings in the publication, thesis, or report › In qualitative analysis, the analysis phase is usually never completed before write-up, but writing and analysis are interwoven

• › Good writing comes with practice, but try to avoid repetition, paraphrasing, unnecessary complexity, and passive phrasing

› Order of the themes should create a coherent story:

• › Logical and meaningful

• › Building up on previous themes is possible

23[RD] FEB 2026

MINNA PAKANEN

6) FINAL THEME REPORTING EXAMPLE

23[RD] FEB 2026 MINNA PAKANEN

DOING THEMATIC ANALYSIS WELL

Try to avoid common errors:

› Providing data extracts with no or little analysis =no interpretation of what data tells us, what is important in it regarding the research question

• › Using data collection questions (interview questions) as themes

for data analysis –> better to focus on what participants say

• › Lack of evidence: provide examples that convince the reader that this pattern was evident in the data

23[RD] FEB 2026

MINNA PAKANEN

EDITING TRANSCRIBES IN THE RESULTS REPORTING PHASE

Partly intelligent verbatim, adding things to ease understanding and shortening the quote + how to indicate it?

“Participant quotes are in Italic.”

“When it first time popped open, I was looking over my shoulder, and a woman passed-by me, and it [bag] made a neeee! [makes a sound of the motor] and I was like oh shit! I felt very aware of it, [its] affect to people and [as] I am going to places where are people, so I was like noo [embarrassed sound and holds hand on the face], because I was like waiting it to [. . .] start moving. In terms of getting attention, I received that when it made sound.” (P2)

[your addition inside of these

brackets are not italic]

[…]

= something has been

removed

23[RD] FEB 2026

MINNA PAKANEN

DIRECT QUOTATION PRESENTATION ★ ▲… Pseudonymized (P1 or #1)/ first name (real or made up)/ code

==> picture [97 x 41] intentionally omitted <==

”For me, this lecture has been relevant in learning how to conduct / diary studies.” (P3) (#3)/ (Mia) One participant (P3)/ ( ★ )/ Mia stated: ”For me, this lecture has been relevant in learning how to conduct diary studies.”

★ / Star/ Mia stated: ”For me, this lecture has been relevant in learning how to conduct diary studies.”

★ &▲ found this A few participants (P3 & P6/ / Mia & Jake) 23[RD] FEB 2026 MINNA PAKANEN lecture as helpful in learning how to conduct diary studies.

REFERENCES

• Carter, S., & Mankoff, J. (2005). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’2005). ACM, New York, 899-908.

• Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: cultural probes. Interactions, 6(1), 2129.

• Kumpel, A.S. (2021). Using Messaging Apps in Audience Research: An Approach to Study Everyday Information and News Use Practices, Digital Journalism, DOI: 10.1080/21670811.2020.1864219

23[RD] FEB 2026

MINNA PAKANEN

REFERENCES: DATA ANALYSIS

• Blandford, A., Furniss, D., & Makri, S. (2016). Qualitative HCI research: Going behind the scenes. Morgan & Claypool Publishers.

• Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-00

• Scupin, R. (1997). The KJ Method: A Technique for Analyzing Data Derived from Japanese Ethnology. Human Organization, 56(2), 233–237. http://www.jstor.org/stable/44126786

23[RD] FEB 2026

MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

EVALUATION ITPDP’26, L13, W15/19

Assoc Prof. Minna Pakanen Department of Digital Design and Information Studies mpakanen @cc.au.dk

6[TH] MAY 2026

MINNA PAKANEN

==> picture [97 x 41] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

TODAY

› Ways of evaluating

• › Usability testing

• › Early development phase UX evaluations

• › Prototypes in early development phase evaluations

• › Data gathering and recording methods

• › Evaluation planning and roles in the evaluation

• › TA session: Planning of the evaluation of your own ITPDP product!

6[TH] MAY 2026

MINNA PAKANEN

WAYS TO EVALUATE

6[TH] MAY 2026

MINNA PAKANEN

4

WHY DO WE EVALUATE?

› We are not designing for ourselves › Feedback on the design process

› Iterative development › ”Fail early, fail often"

6[TH] MAY 2026

MINNA PAKANEN

“Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process, you are going to throw buckets of money down the drain .”

(Bruce Tognazzini)

6[TH] MAY 2026

MINNA PAKANEN

THREE WAYS TO EVALUATE

Usability testing/ UX evaluation

• Test and evaluation setting controlled by the evaluator

• With users

Field evaluation

• Evaluation in a natural environment with real users and their actions

• The user’s surroundings and tasks set the evaluation frame

Analytical evaluation

• No users are involved!

• Heuristic evaluation

You will learn this in the coming HCI course!

• Walkthroughs

Combination of these!

6[TH] MAY 2026

MINNA PAKANEN

6[TH] MAY 2026 MINNA PAKANEN

USABILITY TESTING PLEASE DO NOT USE THIS IN YOUR EVALUATION!

6[TH] MAY 2026

MINNA PAKANEN

11

AIMS OF USABILITY TESTING

• › Interested of how easy it is for the user to use the system

• :

• › Usability is defined by 5 quality components

• Learnability : How easy is it for users to accomplish basic tasks the first time they encounter the design?

• Efficiency : Once users have learned the design, how quickly can they perform tasks?

• Memorability : When users return to the design after a period of not using it, how easily can they re-establish proficiency?

• Errors errors : How many do users make, how severe are these errors, and how easily can

• they recover from the errors?

• Satisfaction : How pleasant is it to use the design?

• (https://www.nngroup.com/articles/ten-usability-heuristics/)

6[TH] MAY 2026

MINNA PAKANEN

USABILITY TESTING

› Three ways of conducting:

• Laboratory-based user observations

• Controlled user testing

• › facilitator can help the user if the user does not know how to proceed, but this needs to be marked down in the observation notes!

• Expert inspection techniques

• › Analytical evaluation methods you will be learning more in the HCI course

(Greenberg & Buxton, 2008)

6[TH] MAY 2026

MINNA PAKANEN

USABILITY TESTING

› Controlled laboratory-based user (covert) observations & studies

==> picture [405 x 194] intentionally omitted <==

==> picture [284 x 194] intentionally omitted <==

==> picture [257 x 194] intentionally omitted <==

https://www.brighton.ac.uk/business-services/consultancy/user-centre-design-lab/index.aspx https://www.volkside.com/2009/12/tip-print-out-your-usability-testing-tasks/ https://www.testingtime.com/en/blog/in-house-usability-tests/ https://ux247.com/usability-lab-dead/

==> picture [224 x 49] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

What are the challenges of usability evaluations conducted in an early design phase according to Greenberg & Buxton (2008)?

6[TH] MAY 2026

MINNA PAKANEN

EARLY DESIGN IDEAS AND PROTOTYPES

• › Early designs are “sketches”

• Illustrate the essence of an idea, but have

• many underdeveloped aspects to it

• Can be represented in many forms

• Work as externalization of the idea and help the designer to iterate the design

• › Early prototypes

• Have many holes and undeveloped attributes

(Greenberg & Buxton, 2008)

==> picture [348 x 311] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

CHALLENGES OF USABILITY STUDIES IN EARLY DESIGN PHASE

› Usability study is not the best approach to evaluate early designs or prototypes because

• It focuses on negative aspects:

• Errors & bugs

• Task completion time: faster the better

• –> Abandoning a promising idea too early –> Focus on developing aspects that can be measured, usually something we are already familiar with

• –> Limit the number of alternative ideas

==> picture [392 x 273] intentionally omitted <==

(Greenberg & Buxton, 2008)

6[TH] MAY 2026

MINNA PAKANEN

EARLY PHASE USER EXPERIENCE EVALUATIONS

6[TH] MAY 2026

MINNA PAKANEN

19

USER EXPERIENCE EVALUATION

• › UX studies focus on lived experiences

• A holistic view of the user’s interaction with a product

  • Emotions, enjoyment & aesthetics

• Focuses on positive aspects of the use

  • Hedonic and non-instrumental aspects of use (non-task related)

• Dynamic and situational aspects

  • Different time spans (before, during, after the use)and momentary experiences (changes over time spans)

  • Context impacts the experience

(Bargas-Avila & Hornbæk, 2011)

6[TH] MAY 2026

MINNA PAKANEN

AIMS OF UX EVALUATIONS

• › To find out how the user feels about the system and interaction

• with it

• Qualitative or mixed-methods approach in research

  • Interviews, focus groups, observations

  • Questionnaires

  • Constructive and creative techniques (drawing & collages)

• UX studies can be conducted at any phase of the design process (concept ideas – – – –>final products)

(Bargas-Avila & Hornbæk, 2011)

6[TH] MAY 2026

MINNA PAKANEN

EARLY DEVELOPMENT PHASE UX EVALUATIONS

• › Early phase user experience evaluations are done to

• › help to choose the best design for the development

• › Evaluating that the development is on the right track

• › Examining if the final product can meet the set UX targets

(Stone et al. 2005)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED USER EXPERIENCE (AUX)

can ”Anticipated UX happens before the first use, or it happen also within the other time spans of UX, e.g. during and after the use and over time, as a person may imagine also during those time spans. A person can have indirect experience prior the first use through formed expectations brand of existing experience with related technologies, , advertisements , presentations, and other peoples’ opinions.”

(Roto et al., 2011)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED EXPERIENCE

”Anticipated user experience means the experiences and feelings that are expected to occur when the user is imagining using an interactive product.”

(Yogasara et al., 2011)

6[TH] MAY 2026

MINNA PAKANEN

PROBLEMS WITH THIS DEFINITION?

==> picture [467 x 380] intentionally omitted <==

----- Start of picture text -----
?
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

ANTICIPATED USER EXPERIENCE (AUX)

Users should not be put in a situation where they have to imagine their future needs without giving . concrete options for them

(Von Hippel, 1986)

6[TH] MAY 2026

MINNA PAKANEN

ANTICIPATED USER EXPERIENCE (AUX)

” needs wishes that result from Experiences, , and anticipated interaction with a concept of the product before the actual product exists.”

(Pakanen, 2015)

6[TH] MAY 2026

MINNA PAKANEN

PROTOTYPES IN EARLY DEVELOPMENT PHASE UX EVALUATIONS

6[TH] MAY 2026

MINNA PAKANEN

29

PROTOTYPES IN AUX EVALUATIONS

• › Prototypes should

• › evoke people’s dreams for the future (van den Hende 2010)

• › give a sense of experience before the actual artefact exists (Roto et al. 2011)

• › allow envisioning the concept and prevent unwanted confusion

  • (Kuutti et al. 2001, Gegner & Runonen 2012)

• › focus subject’s attention on studied things (Lim & Stolterman 2008)

• › allow utilizing concepts key characteristics directly to the design (Law 2011).

6[TH] MAY 2026

MINNA PAKANEN

MIXED FIDELITY PROTOTYPES

5 filter dimensions of prototypes

• Appearance

• size; color; shape; margin; form; weight; texture; proportion; hardness; transparency; gradation; haptic; sound

• The functionality

• system function; users’ functionality need

• The interactivity

• input behavior; output behavior; feedback behavior; information behavior

• The data

• data size; data type (e.g., number; string; media); data use; privacy type; hierarchy; organization

• The spatial structure

• arrangement of interface or information elements; relationship among interface or information elements— which can be either two- or three-dimensional, intangible or tangible, or mixed

(Lim & Stolterman, 2008)

6[TH] MAY 2026

MINNA PAKANEN

MANIFESTATION DIMENSIONS

==> picture [484 x 311] intentionally omitted <==

(Lim & Stolterman, 2008)

6[TH] MAY 2026

MINNA PAKANEN

FILTER-FIDELITY-PROFILES IN EVALUATION

5 filter-dimensions of prototypes

• Appearance

• Size, color, shape, weight, hardness, haptic, sound, & arrangement

• The functionality

• Breadth (functions realized) & depth (completeness)

==> picture [163 x 125] intentionally omitted <==

• The interactivity

• Action, reaction, input modality, & output modality

• The data

• Closeness to the reality, information architecture, data model, & amount and type of data

• The physicality

• Spatial position, coherence of tangibles, & tangible embodiment

(Kohler & Hochreuter, 2014)

==> picture [163 x 124] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

PROTOTYPES IN EARLY UX EVALUATIONS

==> picture [450 x 345] intentionally omitted <==

==> picture [503 x 346] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

COMPARABLE VISUAL MATERIALS IN AUX EVALUATIONS

Extending the existing prototype and comparing alternative visualization styles for GUIs When preparing comparable visual materials, make sure the comparable examples and items under evaluation are same in: 1) Relative size

• 2) Color scheme, unless you compare different color options

3) Level of detail and visualization style (sketch, line drawing, shaded, …)

Note that your alternative visualizations should all be presented in same application context (contextualization).

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE: VIRTUAL AVATARS FOR AR & VR IN COLLABORATIVE SETTING

==> picture [832 x 352] intentionally omitted <==

(Pakanen et al. 2022)

6[TH] MAY 2026 MINNA PAKANEN

EXAMPLE: RESEARCH/ BENCHMARKING

(Pakanen et al. 2022)

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE: EXAMPLES

• 1) Relative size

• 2) Color scheme

• 3) Level of detail & visualization style

==> picture [455 x 466] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

(Pakanen et al. 2022)

EXAMPLE: CONTEXT

Examples are shown both in AR & VR context

(Pakanen et al. 2022)

6[TH] MAY 2026

MINNA PAKANEN

COMPARABLE VISUAL MATERIALS

==> picture [722 x 370] intentionally omitted <==

----- Start of picture text -----
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

(Pakanen 2015)

PROTOTYPES IN EARLY UX EVALUATIONS

==> picture [642 x 321] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026 MINNA PAKANEN

VISUAL MATERIALS TO HELP EVALUATING PROTOTYPES

==> picture [422 x 299] intentionally omitted <==

==> picture [531 x 301] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the study in your report:

› How the visual materials were used?

› When they were used in the evaluation?

6[TH] MAY 2026 MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 1

Paula and her husband p lanned in the morning that she will buy groceries, and her husband will collect her from the shop. Hence, now prior leaving from the office, she pre-writes a message on her phone “At the checkout desk!” (1) and assigns it to a t urquoise color that she can remember.

When she arrives at t he checkout, she uses the bracelet device on her wrist to browse the messages that she has created with the phone (2). She finds the turquoise message (with inner ball) that she created at her office.

Then she selects the receiver, Matt, her husband (with outer ball) and sends it to him (3).

Matt is arriving to the parking lot, when Paula’s message arrives to his phone, he knows that it will only take a few minutes for Paula to come out of the shop, so rather than parking, he drives to waits at the front of the store (4).

==> picture [434 x 309] intentionally omitted <==

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

• Help user to see over limitations of the prototype by creating a model to show the appearance

• A nice rending/ visualization of the concept idea in use context held by a human (print/screen)

• A physical unfunctional mock-up (can be hold in hand by the user)

• A set of images + use case story (prints/ Powerpoint slides/video)

• Make sure you show this use case story before evaluating the partial and unfinished appearance (in this case size, weight, shape, materials), functionality (only two types of interactions were compared [interactivity]) or the prototype.

• Think of what aspects you need to realize as close to the final prototype (physicality as in this case tangible embodiment was almost realistically realized)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 2

Collect feedback for the further development

• 2 Altered visualizations of possible solutions presented on screenshots taken from the model

• Make sure you show the alternative vualizations after evaluating the partial and unfinished prototype

• appearance (no visualizations)

• functionality (only moving was realized in the prototype).

• Think of what aspects you need to realize as close to the final prototype (Appearance and the animations)

==> picture [199 x 393] intentionally omitted <==

(Kukka et al. 2017)

6[TH] MAY 2026

MINNA PAKANEN

VISUAL MATERIALS IN AUX EVALUATIONS

Example from paper 4 Collect feedback for the further development

• One virtual environment with alternative interior and functionalities

• Make sure you let the participants see the alternative visualization after evaluating the partial and unfinished prototype

• appearance (no visualizations)

• functionality (only moving was realized in the prototype).

• Think of what aspects you need to realize as close to the final prototype (Appearance, functionalities, and the interactions)

==> picture [539 x 302] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

(Pakanen et al. 2020)

WIZARD OF OZ

6[TH] MAY 2026

MINNA PAKANEN

48

WIZARD OF OZ TECHNIQUE ALLEN MUNRO AND DON NORMAN, 1975

› The Wizard of Oz method is a moderated research method in which a user interacts with an interface that is not really working, but the system responses are initiated by a human operator

==> picture [817 x 236] intentionally omitted <==

----- Start of picture text -----
Video
camera User
Operator
https://www.nngroup.com/articles/wizard-of-oz/#:~:text=Definition%3A%20The%20Wizard%20of%20Oz,Norman%20at%20UC%20San%20Diego.
----- End of picture text -----

6[TH] MAY 2026

MINNA PAKANEN

WIZARD OF OZ TECHNIQUE AN EXAMPLE IN LAB SETTING

==> picture [385 x 267] intentionally omitted <==

==> picture [565 x 205] intentionally omitted <==

(Colley et al. 2016)

6[TH] MAY 2026

MINNA PAKANEN

WIZARD OF OZ TECHNIQUE

AN EXAMPLE IN FIELD WITH A MOBILE OPERATOR

==> picture [697 x 352] intentionally omitted <==

Pakanen et al. 2022

==> picture [44 x 6] intentionally omitted <==

----- Start of picture text -----
6 [TH] MAY 2026
----- End of picture text -----

==> picture [56 x 7] intentionally omitted <==

----- Start of picture text -----
MINNA PAKANEN
----- End of picture text -----

Reporting of the study in your report:

› What kind of Wizard of OZ evaluation was conducted?

› How was the Wizard of OZ method described in the paper? Paper: 3

MINNA PAKANEN

6[TH] MAY 2026

==> picture [377 x 326] intentionally omitted <==

----- Start of picture text -----
User
< 2m
----- End of picture text -----

WIZARD OF OZ TECHNIQUE MOBILE

==> picture [87 x 87] intentionally omitted <==

Operator

6[TH] MAY 2026

MINNA PAKANEN

PLANNING OF WIZARD OF OZ STUDY

Remember

› Assign group members to be:

• 1) Facilitator, introduces the tasks and interviews the participants

• 2) Operator/”observer” who operates the protype based on participants input/ other type of input. Make sure there is either direct or indirect (through live video feed) visibility to participants interaction with the

prototype with, so that you can operate the prototype well. If the operator is to be in the same space, then say to participants that this person is an observer

› Do not break the illusion!

• › Plan the tasks carefully and think all the possibilities what the user could do?

• If participant does unexpected things, facilitator should say: unfortunately the prototype does not function in that way, can you think any other way how it could work? (and at the end show how to use it).

• › Make sure the operator have a direct view to the participant and their interaction with the prototype

6[TH] MAY 2026

MINNA PAKANEN

USER STUDY DATA GATHERING METHODS

Blandford, Ann, Dominic Furniss, and Stephann Makri. ”Qualitative HCI research: going behind the scenes." Synthesis lectures on human-centered informatics 9, no. 1 (2016): 1-115.

6[TH] MAY 2026

MINNA PAKANEN

55

OBSERVATION

Observer observes

• › ”Fly on the wall” (Blandford et al. 2016)

• › What do people do and how do they do it?

==> picture [156 x 155] intentionally omitted <==

• › Difficulties or errors with the interaction?

• › What do they get enthusiastic about?

• › How do they operate the system?

• › Are their actions supporting what they say?

• › e.g., if they seem to have problems and they claim it is easy to use

6[TH] MAY 2026

MINNA PAKANEN

PLANNING OBSERVATION

Pay attention to

• › Select setting(s)

› It is not possible to observe everything, so decide what is to be documented in each observation.

› Plan an observation form for collecting observations for each task › When a participant does something very interesting, you can mark down the time, which helps finding that incident from the video or audio recording.

6[TH] MAY 2026

MINNA PAKANEN

THINK ALOUD

› Technique can be used in usability testing, user experience evaluation, and even in field studies › Start by explaining to the participant how to think aloud › It is not about what they do, but what they think while doing it › The tasks: chosen by participants (naturalistic) or defined by you › Thinking aloud does not come naturally to all

› You can prompt silent participant by asking: “What are you thinking” › You should politely steer too chatty participant back to the tasks › UX and usability studies have different rules

› Intervention in minimum (usability) | UX interventions for seeking clarifications

6[TH] MAY 2026

MINNA PAKANEN

INTERVIEW

› allows understanding people's perceptions and experiences

==> picture [401 x 314] intentionally omitted <==

› Different types

› Unstructured

• › not directed by a script (ethnographic)

• › Structured

• › tightly scripted, almost like a questionnaire

• › Semi-structured (most used)

• › guided by a script, interesting issues explored in more depth by asking for further details

MINNA PAKANEN

6[TH] MAY 2026

INTERVIEWING

The interviewer interviews the participant/s › Form the topics and questions: open, broad & narrow

• › Next slide what to avoid when forming questions

• › Opening the conversation

• › assure your interest in participants’ replies as an expert in the topic

› Ask the question and wait that the person gives a reply › Ask for clarifications if a person’s reply is not complete or if it is just yes or no/ good/bad

› You said that it is good, in what way/s it is good/ what features make it good?

You wished the information on the screen was clearer, in which way?

6[TH] MAY 2026 MINNA PAKANEN

INTERVIEW QUESTIONS

When creating questions avoid:

› Long questions

• › Compound sentences - split them into two

• › Jargon and language that the interviewee may not understand › What did you think of this AR feature? –> ….the feature where you can see items appearing on this real space we are in?

• › Leading questions that make assumptions

› Isn’t this feature good in…

• › Unconscious biases e.g., gender/age… stereotypes

• › As a/n elderly/woman/child you probably found this as complicated…

6[TH] MAY 2026

MINNA PAKANEN

DATA RECORDING

6[TH] MAY 2026

MINNA PAKANEN

62

VIDEO RECORDING

Pros and cons

› A rich way to capture interaction in the context + user comments are automatically in sync with user interaction

› Video recording can make participant anxious

› Placing the video camera in a more discrete location usually helps users to forget it

==> picture [86 x 86] intentionally omitted <==

==> picture [405 x 348] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

AUDIO RECORDING

==> picture [74 x 73] intentionally omitted <==

When and why

• › In some of the interviews it is fine to record audio only

• If you do not have any prototypes or visual materials, an audio recording is more discrete than a video.

› It is good to also record audio when you use video recording as sometimes it is hard to hear from the video what the user is saying especially if the camera is located far away from the user

6[TH] MAY 2026

MINNA PAKANEN

PHOTOGRAPHS

Records the moment › Capture moments as they happened. They help in reporting results and important moments in the interaction for example in project reports and publications

==> picture [73 x 73] intentionally omitted <==

==> picture [423 x 375] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

OBSERVATION NOTES

==> picture [80 x 81] intentionally omitted <==

Fast in situ note taking

› Observation notes are a good way of getting the most important things marked down, you can even mark the time when a user did or said something that was unexpected or interesting in some other way

› Notes can be used also as the basis of analysis (i.e., themes)

6[TH] MAY 2026

MINNA PAKANEN

LOG DATA

==> picture [74 x 73] intentionally omitted <==

Recording user interaction on the device › records user actions in the device log

› You can find errors and quantify your results easily with log data, but remember that you need the before-mentioned methods to understand why people made mistakes or did things in certain ways

6[TH] MAY 2026

MINNA PAKANEN

UX EVALUATION AND DATA CAPTURING METHODS

==> picture [709 x 475] intentionally omitted <==

----- Start of picture text -----
https://experienceresearchsociety.org/ux/evaluation-methods/
6 [TH] MAY 2026 MINNA PAKANEN
----- End of picture text -----

~~R~~ eporting of the study in your report: › What kind of data collection and recording methods were used?

› Were there some methods that were not described here?

6[TH] MAY 2026

MINNA PAKANEN

EVALUATION PLANNING TASKS, PROCEDURE, & INTERVIEW QUESTIONS

6[TH] MAY 2026

MINNA PAKANEN

78

PLANNING

1. Prepare procedure – steps, tasks…”script of the evaluation” 2. Prepare interview (+questionnaires) questions – background, task related, end

2. Pilot study – try out your procedure, tasks, forms, questions & how long the evaluation will be

3. Iterate your procedure – if something needs to be changed 5. 2[nd] Pilot – test if changes work better

4. Recruit participants – amount depends of the study, from 6-14-30-…, set also times for the studies and let them know the estimated duration

5. Prepare for the evaluation – cameras, video cameras, tripods, audio recording, print forms, get gifts for the participants, book rooms…

6. Conduct your study- follow your procedure, record the data and store it for analysis, remember to thank your participants after the evaluation! 6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE Applied from Leena Arhippainen, UX researcher and Minna’s Phd supervisor

6[TH] MAY 2026

MINNA PAKANEN

80

PROCEDURE

1. Consent from filling- participant agrees on the study (See L3: GDPR)

2. Intro to the study- who, why and what

3. Background info of the users- questionnaire or interview

4. Ice breaking task – easy task to get user familiar with the prototype 5. Actual task(s) with the prototype – design the task so that you are able to get feedback on right things

5. Post questionnaires- preferences and quantitative data

6. Semi-structured interview – get more detailed understanding how user felt using your prototype and what s/he thought about the design idea in general 6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE

Example of procedure:

• Everything is explained so it is easy to follow in the actual evaluation

1. Consent & background questionnaire form filling 2. Who are we and explanation of the purpose of the study (even if explained in the consent form in detail)

2. Actual tasks are written down so that each participant is instructed in a similar way

3. Questionnaire fillings are marked in the procedure 5. Interview questions for post-interview (can also be on different paper)

4. Thank and reward the participants

==> picture [348 x 492] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

PROCEDURE

2. Introduction to the study

• Who are we

• Short description of the study aim

• Tell the user once more that they can withdraw from the study

  • whenever they feel like it

• Ask the user to think aloud during the tasks (+how to do it)

• In Usability evaluation: explain if you will not help the participant during the tasks

• Ask if the user has any questions

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

2. Introduction to the study

MINNA PAKANEN

6[TH] MAY 2026

Reporting of the study in your report:

› Was there differences in the example procedure that was given to your group?

MINNA PAKANEN

6[TH] MAY 2026

PROCEDURE

3. Background information

• What do you need to know about your user?

  • Age

  • Gender: male/female/nonbinary

  • Educational background/ profession

  • Prior experience with similar technology that you are evaluating

  • Prior experience with the use context (e.g., fab labs, pottery, chemical lab, etc.) of your system

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

3. Background questionnaire

Note that you do not necessarily need this, especially if you have only a few things to ask. The questions can also interviewed in the beginning or at the end of the evaluation!

==> picture [353 x 457] intentionally omitted <==

6[TH] MAY 2026 MINNA PAKANEN

Reporting of the study in your report:

› How the participants were described in the paper, and what kind of information was given about them? › What way the description was written?

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

4. Icebreaking task

• Idea is to familiarize your user with the thing under evaluation

• Easy and short task

• Reduces nervousness of the participant

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

4. Icebreaking task

• Explore the prototype freely for a while

1. Familiarization with the current 3D City model

1. Familiarize yourselves with the 3D City model simply by looking at it and giving your initial thoughts before interacting with it.

2. Now you can start interacting with the model by controlling an avatar (WASD and arrows) and explore the model freely. While exploring the 3D model, please think aloud and comment on anything that caught your attention.

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

5. Actual task(s)

• The idea is to plan tasks that the user can do and by doing the tasks you will learn if your product is understandable for the user or how they do experience the prototype

  • Try to keep them short and not too complicated to conduct

  • Test how your core use case activities are working

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

5. Tasks

1st task:

1. First you want your handbag to match with these red shoes over here, what would you do?

2. Next you can try to make the bag imitate the fabric and color of this of this scarf here.

3. And then this shirt here.

Questions after the tasks are completed What did you think of this idea? Would it be useful for you?

Is the idea fun?

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

6. post questionnaire

• Sometimes you might need user feedback on different designs so you can use certain questionnaires to help assess different things.

• The attractiveness of the product: Attrakdiff https://www.attrakdiff.de/index-en.html (online service was discontinued from January 2025 L)

• Can be used for

• Single evaluation

• Comparison

• Before-after

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

6. Post questionnaire With Attrakdiff questionnaire

==> picture [547 x 189] intentionally omitted <==

==> picture [351 x 496] intentionally omitted <==

(Colley et al. 2016)

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

• › Visual presentation of Attrakdiff findings in the report

(Pakanen et al. 2014)

6[TH] MAY 2026

MINNA PAKANEN

PROCEDURE

7. Semi-structured interview

• The last part of the evaluation is for asking questions from the user to find out their thoughts about the prototype and the interaction with it

  • What is good/bad in the design idea and the prototype?

  • What was difficult?

  • What would make it easier to use?

  • What would you like to change in it to make it more interesting to you?

  • Try to avoid questions that can be replied to just with yes or no!

6[TH] MAY 2026

MINNA PAKANEN

EXAMPLE

7. Post-interview questions

• Describe your experiences with the system.

• What do you think of using this kind of technology for remote collaboration?

• What was easy/natural and what was difficult/unnatural in the interaction?

• Were there any critical moments in the interaction? If yes, describe it/them.

6[TH] MAY 2026

MINNA PAKANEN

HOW DO YOU DESCRIBE YOUR STUDY IN THE REPORT?

How procedure is reported in the paper?

› How is the procedure description in the paper differentiating from the study procedure, or is it doing that at all?

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the study in your report:

› How the procedure is reported in the paper?

› Is the procedure description in the paper differentiating from the study procedure, if so, how?

6[TH] MAY 2026

MINNA PAKANEN

ROLES IN THE EVALUATION

6[TH] MAY 2026

MINNA PAKANEN

102

ROLES IN THE EVALUATION

1. Facilitator/interviewer

• Facilitates the study, gives tasks, interviews, handles questionnaires

2. Observer

• Stays behind and observers what user does

• Takes notes on each task-can also write down user comments

3. Documenter (can be also facilitator)

• Videos/photos/audio recording

Pick a role that fits best for you!

6[TH] MAY 2026

MINNA PAKANEN

ROLES IN THE EVALUATION

An example

==> picture [369 x 283] intentionally omitted <==

6[TH] MAY 2026

MINNA PAKANEN

QUALITATIVE DATA ANALYSIS Thematic Analysis

Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004

6[TH] MAY 2026

MINNA PAKANEN

105

PLEASE REVISIT MY SLIDES FROM L5

› https://brightspace.au.dk/d2l/le/lessons/202542/topics/256 9495 › There is detailed explanation how to conduct thematic analysis!

6[TH] MAY 2026

MINNA PAKANEN

Reporting of the data analysis in your report:

› How the qualitative data analysis was reported in the papers?

6[TH] MAY 2026 MINNA PAKANEN

Reporting of the findings in your report:

› Check the example papers for tips how to report qualitative findings in your report ›How participants were identified? ›How participants quotes were written? › Pay attention to the story!

6[TH] MAY 2026

MINNA PAKANEN

PLANNING YOUR OWN EVALUATION TA SESSION FROM 13:15-> TODAY!

6[TH] MAY 2026

MINNA PAKANEN

132

TASK FOR TA

Start planning your evaluation

• Who are your participants? You need 5 participants

• Think of your research question

• Think what kind of tasks and questions would help you to answer to your research question

• Make an early draft of your procedure

• Think about the roles: facilitator, observer, video recorder…

6[TH] MAY 2026

MINNA PAKANEN

WHEN THE PROTOTYPE IS “READY”

Conduct a pilot evaluation

• Get participant/s from the other group/s

• Follow your procedure rigorously

• Act like you do not know your participant beforehand

• After the pilot you have an understanding whether your tasks are easy to understand or too complicated to pass and if your questions can reveal knowledge you were hoping for -> Iterate your procedure, tasks and questions, do another

• pilot study before you go to the actual evaluations!

6[TH] MAY 2026

MINNA PAKANEN

COURSE LITERATURE REFERENCES

• Blandford, Ann, Dominic Furniss, and Stephann Makri. "Qualitative HCI research: Going behind the scenes." Synthesis lectures on human-centered informatics 9, no. 1 (2016): 1-115.

• Greenberg, Saul, and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proc. CHI '08. ACM, New York, NY, USA, 111–120.

• Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Vaananen-Vainio-Mattila. 2010. User experience evaluation methods: current state and development needs. In Proc. NordiCHI '10. ACM, New York, NY, USA, 521–530. https://doi.org/10.1145/1868914.1868973

• Lim N-K & Stolterman E (2008) The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction 15(2): A7.

• Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004

6[TH] MAY 2026

MINNA PAKANEN

OTHER REFERENCES

• Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proc. CHI '11. ACM, New York, NY, USA, 2689–2698. https://doi.org/10.1145/1978942.1979336

• Ashley Colley, Minna Pakanen, Saara Koskinen, Kirsi Mikkonen, and Jonna Häkkilä. 2016. Smart Handbag as a Wearable Public Display - Exploring Concepts and User Perceptions. In Proceedings of the 7th Augmented Human International Conference 2016 (AH '16). Association for Computing Machinery, New York, NY, USA, Article 7, 1–8. https://doi.org/10.1145/2875194.2875212

• Gegner L & Runonen M (2012) For what it is worth: Anticipated experience evaluation. In: Brassett J, Hekkert P, Ludden G, Malpass M & McDonnell J (eds) Proceedings of the 8th International Conference on Design and Emotion. URI: https://reseda.taik. fi/Taik/jsp/taik/Publication_Types.jsp?id=23583813. Cited 2015/08/15.

• Hende, EA van den (2010) Really new stories the effect of early concept narratives on consumer understanding and attitudes. Doctoral dissertation. Delft University of Technology.

• Hippel E von (1986) Lead Users: A source of novel product concepts. Management Science 32(7): 791–805.

• Kuutti K, Battarbee K, Sade S, Mattelmaki T, Keinonen T, Teirikko T, & Tornberg A-M (2001) Virtual prototypes in usability testing. Proceedings of the 34th Hawaii International Conference on System Sciences. IEEE: 1–7.

6[TH] MAY 2026

MINNA PAKANEN

OTHER REFERENCES

• Law ELC (2011) The measurability and predictability of user experience. Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York NY, ACM: 1–10.

• Pakanen M (2015) Visual design examples in the evaluation of anticipated user experience at the early phases of research and development. Doctoral dissertation. University of Oulu.

• Pakanen, M., Alavesa, P., Van Berkel, N., Koskela, T., & Ojala, T. (2022). “Nice to see you virtually”: Thoughtful design and evaluation of virtual avatar of the other user in ar and vr based telexistence systems. Entertainment Computing, 40, 100457.

• Roto V, Law E, Vermeeren A, & Hoonhout J (eds) (2011) UX white paper. https//allaboutux.org/uxwhitepaper. Cited 2015/03/08.

• Stone D, Jarrett C, Woodroffe M, & and Minocha S (2005) User interface design and evaluation. Morgan Kaufmann Series in Interactive Technologies. San Francisco CA, Morgan Kaufman.

• Tognazzini, B. 1990. User Testing on the cheap. In Tog on Interface, Addison Wesley publ.

• Yogasara T, Popovic V, Kraal B, & Camorro-Koc M (2011) General characteristics of anticipated user experience (AUX) with interactive products. Proceedings of the 4th World Conference on Design Research: Diversity and Unity. Delft, Delft University of Technology: 1-11. URI: http://eprints.qut.edu.au/47035/. Cited 2015/06/15.

6[TH] MAY 2026

MINNA PAKANEN