qualitative and quantitative research
May 30, 2008 10:30am
But this is an area where invention and inventiveness could really push on
boundaries of what is possible. The problem is, most researchers tend to
have either a quantitative or qualitative bias. Multi-modal stuff is
interesting, tho! Triangulate!
My own issues with sociolinguistic-style content analysis data (which is
what you get when you take oral speak-aloud transcripts from video usability
testing and have independent raters code the content according to a
particular schema, checking for inter-rater reliability) is that it often
turns into "counts" of certain keywords that show frequency, but fail to
reveal true emphasis or weight.
For instance (and I'm dating myself here), look at a content analysis study of the discourse of an online forum or some other social or dialogic context. You code and count, crunch in SPSS etc and what can be revealed is a map of sorts of the ways that that conversation or dialogue went down.
But the blind spot is the ethos of the different speakers, known to the
other speakers. So while many speakers may be chattering away in the forum,
generating lines and lines of text, this one speaker can enter the
conversation with just a few well-chosen lines and turn the direction of the
entire discussion, or shut down one line of argument entirely with a pithy
or significant counter-argument. A counting-style analysis of qualitative
content would completely miss the emphasis and power of these few
well-chosen and influential statements.
Now, go visual, in a less social interface design example. Supposed you are
looking at one of those fun "red dot" studies on a web page, red dot
representing mouse actions. Great data. Fascinating to look at. Or the
eyeball studies, that show where eyeballs go on a page. In many ways, these
are attempts to quantify what is essentially idiosyncratic, qualitative user
mouse and eyeball behaviors. And they keep getting better. You could even do
studies with these tools that are completely phenomenological, as open-ended
and qualitative as you could possibly get.
But they end up coming back to counting something, and you can usually find
blind spots there too, where quantity does not reveal the real strength and
emphasis that should be put on a single under-counted thing, something that
works in the gestalt of the experience, or sits on the periphery, and yet
strongly influences and colors the experience far out of proportion to
anything being counted. For some sites, that might be a compelling (not
lame) interface metaphor. Others, it might be a style of photography that
just makes the keyed image dazzling far out of proportion to anything else
on the site. Or it might be an unenunciated emotional effect, a common
reaction to a combination of colors on a particular community site that
makes it feel like "home" to the people who come to live there, so that when
those colors are changed, the community inexplicably loses its energy and
center.
Anyway, I'm just throwing this out there, for something to think about.
Chris
On Fri, May 30, 2008 at 12:35 PM, christine chastain <
chastain.christine at gmail.com wrote:
Good morning!
Does anyone have any experience with "making" qualitative research as
quantitative as possible? This is a rather nebulous idea in my mind for the
moment but I was thinking about something like using images from still or
video footage and tagging those such that they could be coded in a
quantitative way. Sort of like a heuristic evaluation on steroids.
Just thinking.thanks!
Welcome to the Interaction Design Association (IxDA)!
To post to this list . discuss at ixda.org
Unsubscribe http://www.ixda.org/unsubscribe
List Guidelines http://www.ixda.org/guidelines
List Help http://www.ixda.org/help
Thanks for your thoughtful input, Chris! Being a designer and ethnographer,
I think what I have in my head is actually tagging behaviors, interactions,
even materiality in video or other self-reported data and then coming up
with a way to code those that would show patterns over time. So imagine
taking all visual materials from a time-motion study, for example, and
tagging all behaviors, interactions and things, feeding that into some
magical formula that would allow you to cross tab and identify patterns from
which you could then produce a lifestyle narrative. So awesome.time
consuming but could be amazing. And you could still have "other" qualitative
input overlayed onto that.
On Fri, May 30, 2008 at 11:30 AM, Christine Boese <christine.boese at gmail.com
wrote:
Generally (and forgive me if you are already familiar with these methods)
the way to go is with methods of content analysis such as sociolinguists
use. I always used to think part of the beauty of Q-sort methods is that
they have wonderful open-ended approaches, yet the data can be crunched.
But this is an area where invention and inventiveness could really push on
boundaries of what is possible. The problem is, most researchers tend to
have either a quantitative or qualitative bias. Multi-modal stuff is
interesting, tho! Triangulate! Change Sciences attempts to plug qualitative measures into a quantitative
formula. They have a whitepaper that describes the process:
http://www.changesciences.com/
Best,
Julie
On Fri, May 30, 2008 at 11:35 AM, christine chastain <
chastain.christine at gmail.com wrote:
Good morning! Welcome to the Interaction Design Association (IxDA)!
To post to this list . discuss at ixda.org
On May 30, 2008, at 12:35 PM, christine chastain wrote:
Does anyone have any experience with "making" qualitative research as
quantitative as possible? This is a rather nebulous idea in my mind
for the
At one point, we had a sign in the office that read: "Scientific
advancement through mind-numbing manual labor."
Jared
Jared M. Spool
Tables. Percentages. Pie Charts. I'm serious.
Sometimes I refer to this as being "scientistic".
I always approach research problems with a somewhat scientific framework
(coding activities, keeping tallies). It's just important to be clear
with yourself that the samples sizes you are dealing with generally mean
that true quant claims are meaningless. And, heck even dangerous,
sometimes.
That said, it's especially helpful for skeptical clients who are quant
jocks. Just make sure to let anyone on the project who actually gets the
difference know what's up.
----- Original Message -----
On May 30, 2008, at 12:35 PM, christine chastain wrote:
Does anyone have any experience with "making" qualitative research as
quantitative as possible? This is a rather nebulous idea in my mind
for the
At one point, we had a sign in the office that read: "Scientific
advancement through mind-numbing manual labor."
Jared
Jared M. Spool
Welcome to the Interaction Design Association (IxDA)!
To post to this list . discuss at ixda.org If you're talking about how to turn qualitative data into something
"measureable", there are several etnography methods now that are
moving in that direction.
You should check out -- for example -- the "Insight Matrix",
developed by Vijay Kumar and Brandon Schauer at the IIT Institute of
Design (http://www.id.iit.edu/568/).
{ Itamar Medeiros } Information Designer
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
Hi Christine,
Yup, that sort of tagging is just like more formal methods of content
analysis, such as like socio-linguists use. You might pick up one of those
small paperback Sage publications that run through an overview of content
analysis methods, just so you are rigorous in your tags and how you assign
them, or perhaps look to multiple raters/coders/taggers and establish a
baseline of inter-rater reliability.
See, the trouble with winging it is you could spend a lot of time on a
taggin/coding schema, and you might discover (or worse, not discover and be
oblivious) to the fact that your schema is giving you bad data, which then
becomes bad conclusions. That would be a nightmare.
So you'd want a really STRONG pilot project, and lots and lots of feedback
to make sure the method will yield both reliable and useful results. Hone
the method out, THEN turn it into your hamburger grinder and see what kind
of burgers you get.
And then trot your great new method out at the next IxDA conference and tell
us all about it, so we can try it too, and replicate it, and further test
the usefulness of the results!
I don't say this to put you off by the amount of work entailed. Following
vigorous and careful research methods may seem like a huge mountain to
climb, but it can also be creative, interesting, and really valuable. I'm
hoping you do it! We could use some fresh and innovative methods.
Chris
On Fri, May 30, 2008 at 2:40 PM, christine chastain <
chastain.christine at gmail.com wrote:
Thanks for your thoughtful input, Chris! Being a designer and ethnographer,
I think what I have in my head is actually tagging behaviors, interactions,
even materiality in video or other self-reported data and then coming up
with a way to code those that would show patterns over time. So imagine
taking all visual materials from a time-motion study, for example, and
tagging all behaviors, interactions and things, feeding that into some
magical formula that would allow you to cross tab and identify patterns from
which you could then produce a lifestyle narrative. So awesome.time
consuming but could be amazing. And you could still have "other" qualitative
input overlayed onto that. I'm going to give it a go.;) I mean, how much worse can it be than a
time-motion study of knee replacements including all players using various
surgical methods for the purpose of creating new instrumentation? I'm a
glutton for punishment.
Your suggestions are great!
On Fri, May 30, 2008 at 7:02 PM, Christine Boese <christine.boese at gmail.com
wrote:
Hi Christine, Christine asked:
Regards,
Paul
Paul Eisen christine wrote:
That being said, a couple books come to mind with examples of this
type of research. William Whyte wrote a short book called the Social
Life of Small Urban Spaces and Paco Underhill (who worked for Whyte
on the Project for Public Spaces) wrote Why We Buy and calls his
approach "the science of shopping."
They're both engaged in some pretty serious ethnography (IIRC,
primarily shadowing and video analysis) and neither skimp on the
quantification.
// jeff
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
As Jeff eludes, It is a mistake to put ethnography solely in the
qualitative box. Agar, Maanen and many of the other early
ethnographers took meticulous counts and quantified their findings.
On May 31, 2008, at 10:09 AM, Jeff Howard wrote:
They're both engaged in some pretty serious ethnography (IIRC,
primarily shadowing and video analysis) and neither skimp on the
quantification. Gosh I'm loving the way IxDA is getting input from all sorts of other
fields. This forum rocks!
I'm have a market research background and what you describe sounds
to me like qualitative coding. If you do a search for this term, you
will find heaps of references for:
My experience in market research was that the subjective nature of
qualitative coding was recognised and mechanisms were put in place to
manage it. These included ensuring the same person or people coded,
that the coding frame was discussed and reviewed by all parties, and
that a random sample of coded information was quality checked to look
for glaring inconsistencies in how the frame was applied.
The thing to remember is that if you want to draw conclusions about
patterns you are observing, then you need to be working with a
complete or at least statistically representative sample. This is
usually *not* the case in qualitative market research but it might be
in your situation.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
christine chastain
May 30, 2008 10:40am
lachica
May 30, 2008 11:43am
Does anyone have any experience with "making" qualitative research as
quantitative as possible? This is a rather nebulous idea in my mind for the
moment but I was thinking about something like using images from still or
video footage and tagging those such that they could be coded in a
quantitative way. Sort of like a heuristic evaluation on steroids.
Just thinking.thanks!
Unsubscribe http://www.ixda.org/unsubscribe
List Guidelines http://www.ixda.org/guidelines
List Help http://www.ixda.org/help
Jared Spool
May 30, 2008 2:30pm
moment but I was thinking about something like using images from
still or
video footage and tagging those such that they could be coded in a
quantitative way. Sort of like a heuristic evaluation on steroids.
We've been doing it for years. Most of our research is based on it.
It's hard work, but quite doable.
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks
Gretchen Anderson
May 30, 2008 3:10pm
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of
Jared Spool
Sent: Friday, May 30, 2008 3:31 PM
To: christine chastain
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] qualitative and quantitative research
moment but I was thinking about something like using images from
still or
video footage and tagging those such that they could be coded in a
quantitative way. Sort of like a heuristic evaluation on steroids.
We've been doing it for years. Most of our research is based on it.
It's hard work, but quite doable.
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks
Unsubscribe http://www.ixda.org/unsubscribe
List Guidelines http://www.ixda.org/guidelines
List Help http://www.ixda.org/help
Itamar Medeiros
May 30, 2008 5:45pm
http://designative.info/
http://www.autodesk.com/
http://www.ixda.org/discuss?post=29646
Christine Boese
May 30, 2008 6:02pm
On Fri, May 30, 2008 at 11:30 AM, Christine Boese <
christine.boese at gmail.com wrote:
Generally (and forgive me if you are already familiar with these methods)
the way to go is with methods of content analysis such as sociolinguists
use. I always used to think part of the beauty of Q-sort methods is that
they have wonderful open-ended approaches, yet the data can be crunched.
But this is an area where invention and inventiveness could really push on
boundaries of what is possible. The problem is, most researchers tend to
have either a quantitative or qualitative bias. Multi-modal stuff is
interesting, tho! Triangulate!
christine chastain
May 30, 2008 6:25pm
Yup, that sort of tagging is just like more formal methods of content
analysis, such as like socio-linguists use. You might pick up one of those
small paperback Sage publications that run through an overview of content
analysis methods, just so you are rigorous in your tags and how you assign
them, or perhaps look to multiple raters/coders/taggers and establish a
baseline of inter-rater reliability.
See, the trouble with winging it is you could spend a lot of time on a
taggin/coding schema, and you might discover (or worse, not discover and be
oblivious) to the fact that your schema is giving you bad data, which then
becomes bad conclusions. That would be a nightmare.
So you'd want a really STRONG pilot project, and lots and lots of feedback
to make sure the method will yield both reliable and useful results. Hone
the method out, THEN turn it into your hamburger grinder and see what kind
of burgers you get.
And then trot your great new method out at the next IxDA conference and
tell us all about it, so we can try it too, and replicate it, and further
test the usefulness of the results!
I don't say this to put you off by the amount of work entailed. Following
vigorous and careful research methods may seem like a huge mountain to
climb, but it can also be creative, interesting, and really valuable. I'm
hoping you do it! We could use some fresh and innovative methods.
Chris
On Fri, May 30, 2008 at 2:40 PM, christine chastain <
chastain.christine at gmail.com wrote:
Thanks for your thoughtful input, Chris! Being a designer and
ethnographer, I think what I have in my head is actually tagging behaviors,
interactions, even materiality in video or other self-reported data and then
coming up with a way to code those that would show patterns over time. So
imagine taking all visual materials from a time-motion study, for example,
and tagging all behaviors, interactions and things, feeding that into some
magical formula that would allow you to cross tab and identify patterns from
which you could then produce a lifestyle narrative. So awesome.time
consuming but could be amazing. And you could still have "other" qualitative
input overlayed onto that.
On Fri, May 30, 2008 at 11:30 AM, Christine Boese <
christine.boese at gmail.com wrote:
Generally (and forgive me if you are already familiar with these methods)
the way to go is with methods of content analysis such as sociolinguists
use. I always used to think part of the beauty of Q-sort methods is that
they have wonderful open-ended approaches, yet the data can be crunched.
But this is an area where invention and inventiveness could really push
on boundaries of what is possible. The problem is, most researchers tend to
have either a quantitative or qualitative bias. Multi-modal stuff is
interesting, tho! Triangulate!
Paul Eisen
May 30, 2008 6:32pm
Does anyone have any experience with "making" qualitative research as quantitative as possible?
I'm not sure if this is what you are looking for: Fuzzy logic is a mathematics that has been used to model imprecise values. It allows you to apply values from 0 to 1 to constructs that a researcher may be interested in such as, "Easy", "Engaging", etc. Fuzzy logic has been used with varying levels of success to represent subjective impressions, as we often like to do in user research. Check out Wikipedia if you want to learn more. A Google search will lead you to some examples of the use of fuzzy logic in usability-related research.
Principal User Experience Architect
tandemseven
Jeff Howard
May 31, 2008 9:09am
Does anyone have any experience with "making" qualitative
research as quantitative as possible?
It is not worth the while to go round the world to count the cats in
Zanzibar. --Thoreau
http://www.ixda.org/discuss?post=29646
mark schraad
May 31, 2008 1:56pm
Jessica Enders
May 31, 2008 6:04pm
http://www.ixda.org/discuss?post=29646
Log In to Post a Response
Re: qualitative and quantitative research
Copyright © 2004-2006 Interaction Design Association.