Interaction Design Association

qualitative and quantitative research

Christine Boese
May 30, 2008 10:30am

Post a Response | Jump to Most Recent (12)

Generally (and forgive me if you are already familiar with these methods) the way to go is with methods of content analysis such as sociolinguists use. I always used to think part of the beauty of Q-sort methods is that they have wonderful open-ended approaches, yet the data can be crunched.

But this is an area where invention and inventiveness could really push on boundaries of what is possible. The problem is, most researchers tend to have either a quantitative or qualitative bias. Multi-modal stuff is interesting, tho! Triangulate!

My own issues with sociolinguistic-style content analysis data (which is what you get when you take oral speak-aloud transcripts from video usability testing and have independent raters code the content according to a particular schema, checking for inter-rater reliability) is that it often turns into "counts" of certain keywords that show frequency, but fail to reveal true emphasis or weight.

For instance (and I'm dating myself here), look at a content analysis study of the discourse of an online forum or some other social or dialogic context. You code and count, crunch in SPSS etc and what can be revealed is a map of sorts of the ways that that conversation or dialogue went down.

But the blind spot is the ethos of the different speakers, known to the other speakers. So while many speakers may be chattering away in the forum, generating lines and lines of text, this one speaker can enter the conversation with just a few well-chosen lines and turn the direction of the entire discussion, or shut down one line of argument entirely with a pithy or significant counter-argument. A counting-style analysis of qualitative content would completely miss the emphasis and power of these few well-chosen and influential statements.

Now, go visual, in a less social interface design example. Supposed you are looking at one of those fun "red dot" studies on a web page, red dot representing mouse actions. Great data. Fascinating to look at. Or the eyeball studies, that show where eyeballs go on a page. In many ways, these are attempts to quantify what is essentially idiosyncratic, qualitative user mouse and eyeball behaviors. And they keep getting better. You could even do studies with these tools that are completely phenomenological, as open-ended and qualitative as you could possibly get.

But they end up coming back to counting something, and you can usually find blind spots there too, where quantity does not reveal the real strength and emphasis that should be put on a single under-counted thing, something that works in the gestalt of the experience, or sits on the periphery, and yet strongly influences and colors the experience far out of proportion to anything being counted. For some sites, that might be a compelling (not lame) interface metaphor. Others, it might be a style of photography that just makes the keyed image dazzling far out of proportion to anything else on the site. Or it might be an unenunciated emotional effect, a common reaction to a combination of colors on a particular community site that makes it feel like "home" to the people who come to live there, so that when those colors are changed, the community inexplicably loses its energy and center.

Anyway, I'm just throwing this out there, for something to think about.

Chris

On Fri, May 30, 2008 at 12:35 PM, christine chastain < chastain.christine at gmail.com wrote:

Good morning!
Does anyone have any experience with "making" qualitative research as quantitative as possible? This is a rather nebulous idea in my mind for the moment but I was thinking about something like using images from still or video footage and tagging those such that they could be coded in a quantitative way. Sort of like a heuristic evaluation on steroids. Just thinking.thanks!

 
christine chastain
May 30, 2008 10:40am

Thanks for your thoughtful input, Chris! Being a designer and ethnographer, I think what I have in my head is actually tagging behaviors, interactions, even materiality in video or other self-reported data and then coming up with a way to code those that would show patterns over time. So imagine taking all visual materials from a time-motion study, for example, and tagging all behaviors, interactions and things, feeding that into some magical formula that would allow you to cross tab and identify patterns from which you could then produce a lifestyle narrative. So awesome.time consuming but could be amazing. And you could still have "other" qualitative input overlayed onto that.

On Fri, May 30, 2008 at 11:30 AM, Christine Boese <christine.boese at gmail.com wrote:

Generally (and forgive me if you are already familiar with these methods) the way to go is with methods of content analysis such as sociolinguists use. I always used to think part of the beauty of Q-sort methods is that they have wonderful open-ended approaches, yet the data can be crunched. But this is an area where invention and inventiveness could really push on boundaries of what is possible. The problem is, most researchers tend to have either a quantitative or qualitative bias. Multi-modal stuff is interesting, tho! Triangulate!

 
lachica
May 30, 2008 11:43am

Change Sciences attempts to plug qualitative measures into a quantitative formula. They have a whitepaper that describes the process: http://www.changesciences.com/

Best, Julie

On Fri, May 30, 2008 at 11:35 AM, christine chastain < chastain.christine at gmail.com wrote:

Good morning!
Does anyone have any experience with "making" qualitative research as quantitative as possible? This is a rather nebulous idea in my mind for the moment but I was thinking about something like using images from still or video footage and tagging those such that they could be coded in a quantitative way. Sort of like a heuristic evaluation on steroids. Just thinking.thanks!

 
Jared Spool
May 30, 2008 2:30pm

On May 30, 2008, at 12:35 PM, christine chastain wrote:

Does anyone have any experience with "making" qualitative research as quantitative as possible? This is a rather nebulous idea in my mind for the
moment but I was thinking about something like using images from still or
video footage and tagging those such that they could be coded in a quantitative way. Sort of like a heuristic evaluation on steroids.
We've been doing it for years. Most of our research is based on it. It's hard work, but quite doable.

At one point, we had a sign in the office that read: "Scientific advancement through mind-numbing manual labor."

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e:
jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

 
Gretchen Anderson
May 30, 2008 3:10pm

Tables. Percentages. Pie Charts. I'm serious.

Sometimes I refer to this as being "scientistic".

I always approach research problems with a somewhat scientific framework (coding activities, keeping tallies). It's just important to be clear with yourself that the samples sizes you are dealing with generally mean that true quant claims are meaningless. And, heck even dangerous, sometimes.

That said, it's especially helpful for skeptical clients who are quant jocks. Just make sure to let anyone on the project who actually gets the difference know what's up.

-----

Original Message -----
From:
discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Jared Spool
Sent: Friday, May 30, 2008 3:31 PM
To: christine chastain
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] qualitative and quantitative research

On May 30, 2008, at 12:35 PM, christine chastain wrote:

Does anyone have any experience with "making" qualitative research as quantitative as possible? This is a rather nebulous idea in my mind for the
moment but I was thinking about something like using images from still or
video footage and tagging those such that they could be coded in a quantitative way. Sort of like a heuristic evaluation on steroids.
We've been doing it for years. Most of our research is based on it. It's hard work, but quite doable.

At one point, we had a sign in the office that read: "Scientific advancement through mind-numbing manual labor."

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

 
Itamar Medeiros
May 30, 2008 5:45pm

If you're talking about how to turn qualitative data into something "measureable", there are several etnography methods now that are moving in that direction.

You should check out -- for example -- the "Insight Matrix", developed by Vijay Kumar and Brandon Schauer at the IIT Institute of Design (http://www.id.iit.edu/568/).

{ Itamar Medeiros } Information Designer
http://designative.info/
http://www.autodesk.com/

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org
http://www.ixda.org/discuss?post=29646

 
Christine Boese
May 30, 2008 6:02pm

Hi Christine,

Yup, that sort of tagging is just like more formal methods of content analysis, such as like socio-linguists use. You might pick up one of those small paperback Sage publications that run through an overview of content analysis methods, just so you are rigorous in your tags and how you assign them, or perhaps look to multiple raters/coders/taggers and establish a baseline of inter-rater reliability.

See, the trouble with winging it is you could spend a lot of time on a taggin/coding schema, and you might discover (or worse, not discover and be oblivious) to the fact that your schema is giving you bad data, which then becomes bad conclusions. That would be a nightmare.

So you'd want a really STRONG pilot project, and lots and lots of feedback to make sure the method will yield both reliable and useful results. Hone the method out, THEN turn it into your hamburger grinder and see what kind of burgers you get.

And then trot your great new method out at the next IxDA conference and tell us all about it, so we can try it too, and replicate it, and further test the usefulness of the results!

I don't say this to put you off by the amount of work entailed. Following vigorous and careful research methods may seem like a huge mountain to climb, but it can also be creative, interesting, and really valuable. I'm hoping you do it! We could use some fresh and innovative methods.

Chris

On Fri, May 30, 2008 at 2:40 PM, christine chastain < chastain.christine at gmail.com wrote:

Thanks for your thoughtful input, Chris! Being a designer and ethnographer, I think what I have in my head is actually tagging behaviors, interactions, even materiality in video or other self-reported data and then coming up with a way to code those that would show patterns over time. So imagine taking all visual materials from a time-motion study, for example, and tagging all behaviors, interactions and things, feeding that into some magical formula that would allow you to cross tab and identify patterns from which you could then produce a lifestyle narrative. So awesome.time consuming but could be amazing. And you could still have "other" qualitative input overlayed onto that.
On Fri, May 30, 2008 at 11:30 AM, Christine Boese < christine.boese at gmail.com wrote:
Generally (and forgive me if you are already familiar with these methods) the way to go is with methods of content analysis such as sociolinguists use. I always used to think part of the beauty of Q-sort methods is that they have wonderful open-ended approaches, yet the data can be crunched. But this is an area where invention and inventiveness could really push on boundaries of what is possible. The problem is, most researchers tend to have either a quantitative or qualitative bias. Multi-modal stuff is interesting, tho! Triangulate!

 
christine chastain
May 30, 2008 6:25pm

I'm going to give it a go.;) I mean, how much worse can it be than a time-motion study of knee replacements including all players using various surgical methods for the purpose of creating new instrumentation? I'm a glutton for punishment.

Your suggestions are great!

On Fri, May 30, 2008 at 7:02 PM, Christine Boese <christine.boese at gmail.com wrote:

Hi Christine,
Yup, that sort of tagging is just like more formal methods of content analysis, such as like socio-linguists use. You might pick up one of those small paperback Sage publications that run through an overview of content analysis methods, just so you are rigorous in your tags and how you assign them, or perhaps look to multiple raters/coders/taggers and establish a baseline of inter-rater reliability.
See, the trouble with winging it is you could spend a lot of time on a taggin/coding schema, and you might discover (or worse, not discover and be oblivious) to the fact that your schema is giving you bad data, which then becomes bad conclusions. That would be a nightmare. So you'd want a really STRONG pilot project, and lots and lots of feedback to make sure the method will yield both reliable and useful results. Hone the method out, THEN turn it into your hamburger grinder and see what kind of burgers you get.
And then trot your great new method out at the next IxDA conference and tell us all about it, so we can try it too, and replicate it, and further test the usefulness of the results!
I don't say this to put you off by the amount of work entailed. Following vigorous and careful research methods may seem like a huge mountain to climb, but it can also be creative, interesting, and really valuable. I'm hoping you do it! We could use some fresh and innovative methods. Chris
On Fri, May 30, 2008 at 2:40 PM, christine chastain < chastain.christine at gmail.com wrote:
Thanks for your thoughtful input, Chris! Being a designer and ethnographer, I think what I have in my head is actually tagging behaviors, interactions, even materiality in video or other self-reported data and then coming up with a way to code those that would show patterns over time. So imagine taking all visual materials from a time-motion study, for example, and tagging all behaviors, interactions and things, feeding that into some magical formula that would allow you to cross tab and identify patterns from which you could then produce a lifestyle narrative. So awesome.time consuming but could be amazing. And you could still have "other" qualitative input overlayed onto that.
On Fri, May 30, 2008 at 11:30 AM, Christine Boese < christine.boese at gmail.com wrote:
Generally (and forgive me if you are already familiar with these methods) the way to go is with methods of content analysis such as sociolinguists use. I always used to think part of the beauty of Q-sort methods is that they have wonderful open-ended approaches, yet the data can be crunched. But this is an area where invention and inventiveness could really push on boundaries of what is possible. The problem is, most researchers tend to have either a quantitative or qualitative bias. Multi-modal stuff is interesting, tho! Triangulate!

 
Paul Eisen
May 30, 2008 6:32pm

Christine asked:
Does anyone have any experience with "making" qualitative research as quantitative as possible? I'm not sure if this is what you are looking for: Fuzzy logic is a mathematics that has been used to model imprecise values. It allows you to apply values from 0 to 1 to constructs that a researcher may be interested in such as, "Easy", "Engaging", etc. Fuzzy logic has been used with varying levels of success to represent subjective impressions, as we often like to do in user research. Check out Wikipedia if you want to learn more. A Google search will lead you to some examples of the use of fuzzy logic in usability-related research.

Regards,

Paul

Paul Eisen
Principal User Experience Architect
tandemseven

 
Jeff Howard
May 31, 2008 9:09am

christine wrote:
Does anyone have any experience with "making" qualitative research as quantitative as possible?
It is not worth the while to go round the world to count the cats in Zanzibar. --Thoreau

That being said, a couple books come to mind with examples of this type of research. William Whyte wrote a short book called the Social Life of Small Urban Spaces and Paco Underhill (who worked for Whyte on the Project for Public Spaces) wrote Why We Buy and calls his approach "the science of shopping."

They're both engaged in some pretty serious ethnography (IIRC, primarily shadowing and video analysis) and neither skimp on the quantification.

// jeff

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org
http://www.ixda.org/discuss?post=29646

 
mark schraad
May 31, 2008 1:56pm

As Jeff eludes, It is a mistake to put ethnography solely in the qualitative box. Agar, Maanen and many of the other early ethnographers took meticulous counts and quantified their findings.

On May 31, 2008, at 10:09 AM, Jeff Howard wrote:

They're both engaged in some pretty serious ethnography (IIRC, primarily shadowing and video analysis) and neither skimp on the quantification.

 
Jessica Enders
May 31, 2008 6:04pm

Gosh I'm loving the way IxDA is getting input from all sorts of other fields. This forum rocks!

I'm have a market research background and what you describe sounds to me like qualitative coding. If you do a search for this term, you will find heaps of references for:

  • software to assist the process (usually assuming you're working from transcripts of interviews or focus groups, so probably not so useful for you)
  • papers and books about how to do it
  • papers and books about its reliability and usefulness.
  • My experience in market research was that the subjective nature of qualitative coding was recognised and mechanisms were put in place to manage it. These included ensuring the same person or people coded, that the coding frame was discussed and reviewed by all parties, and that a random sample of coded information was quality checked to look for glaring inconsistencies in how the frame was applied.

    The thing to remember is that if you want to draw conclusions about patterns you are observing, then you need to be working with a complete or at least statistically representative sample. This is usually *not* the case in qualitative market research but it might be in your situation.

    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org
    http://www.ixda.org/discuss?post=29646

     
    Log In to Post a Response
    Re: qualitative and quantitative research

    Name

    E-mail Address

    Back to Top

    Copyright © 2004-2006 Interaction Design Association.