Coming of Age in Ethnography

By Dave Rogers

I first heard of ethnography in Sociology 101. In his sonorous voice, our professor regaled us with tales of intrepid anthropologists immersing themselves in little-known cultures in exotic settings. We discussed Margaret Mead's seminal work, Coming of Age in Samoa. We examined the rigors of fieldwork, the tension between observation and participation and the challenge of analysis. It was a great class and I even opted for Soc 102.

And that was that. Ethnography faded into the recesses of my mind until reawakened with a start a few years ago when I began hearing it applied to Web design. And it scared me spitless.

Why? Because my sociology professor had emphasized the rigors of the discipline—especially the challenge of maintaining objectivity and the difficulty of setting aside what he called our personal "frames of reference." He had explained how what we call objective truth is socially constructed, mediated by our own experiences, our cultures, our personal and community narratives—that we really cannot stand apart as unbiased and objective observers of the world.

His lesson had stuck. So when I first heard ethnography applied to Web user analysis, my first instinct was to duck and cover. "Gee whiz," I whined to myself, "Here's something else I have to learn to do." I was a newcomer to information architecture at the time, barely keeping up with usability testing, card sorting, surveys and other user analysis methods. Adding what I saw as a time-consuming, complex and academic science seemed too much to ask.

With time, I came to my senses. I learned to distinguish design ethnography from its classic, anthropological forebears. I recognized that it could take many forms—from contextual inquiry to task analysis (something I had done many times as an instructional designer). I grew beyond the myth of the do-it-all IA. I chuckled at myself for failing to realize that I had initially reacted to design ethnography from an inadequate "frame of reference."

Most of all, I appreciated design ethnography's emphasis on exploring how people actually use Web sites and products in the contexts of their lives—and not merely by how they explain it.

I'm going to leave the in-depth discussions of design ethnography to the much more experienced authors in this edition of the gotoreport. Where I want to land today is on the paramount lesson I took from my sociology professor: recognizing the points of possible distortion when observing and analyzing the behavior of others.

I was thinking about this a few weeks ago while compiling the report for a usability test and card sorting exercise. "How do I know these findings are reliable?" I asked myself as I wrote the team's conclusions. As I reflected on the question, I went through a mental checklist of possible entry points for distortion:

• The sample of users: Is it truly representative of the spectrum of possible users of our site or product?

  In my recent experience, we took great care in recruiting test participants who meshed with our three archetypal personas. Even so, they came from within a limited geographical area and a fairly narrow socioeconomic range. For our purposes early in the design cycle, we decided this was acceptable—but there are many situations where casting a wider net is required. The key here is simply asking the question; if we're honest, the answer is forthcoming.

• The number of participants: Are there enough to capture the usability issues?

  Practitioners often default to Virzi's and Landauer and Nielsen's "five is enough" rubric. It's a convenient, cost- and time-efficient number. But there's enough doubt to consider including more participants. For our usability test, we had nine participants and test results were remarkably consistent. For card sorting, however, nine is on the low side of general recommendations. While our results were again consistent, more participants would have boosted my confidence level.

• The test location: Could it induce participants to behave differently than they would at work or home?

  I'm as certain as I can be that plopping people into a usability lab replete with video camera, computers and unseen observers behind a mirrored window is not a neutral act; they will interact with a site or product differently than they would in their cubicles or living rooms. Even a typical office conference room is not a user's natural territory. Such environments silently encourage participants to pay closer attention, to try a little harder than they would otherwise do—in short, to diverge from their usual behavior.

  Ethnography and contextual inquiry ride to the rescue here—but we still have to admit that even the subtle intrusion of a skillful observer modifies the user's environment.

• The methodology: Did the observation use proven techniques? Did it have specific (written) goals and protocols? Were interview questions up to snuff? Were card labels carefully vetted for clarity and meaning? Did participants receive a thorough orientation? Did every participant receive equal treatment?

  Compared to just five years ago, we have an embarrassment of user-centered riches, chockfull of guidance. (Many of them—online and real world—are listed in the gotomedia resource center.) With a reasonable investment, everyone can have world-class expertise at their fingertips.

• The researcher: Have all moderators and observers sincerely made the effort to set aside personal biases, orientations and preferences?

  As my sociology professor said, we all approach life with our own set of filters that mediate how we observe the behavior of others. We can't step outside of ourselves and become dispassionate, neutral observers. Yet we can grow in self-awareness of our biases and assumptions in ways that can check their sway over us.

  For example, I imagine that most of us enjoy users who are talkative and eager to engage with us and the tasks at hand. That's understandable. If we're not careful, however, this enjoyment can become a preference for extroverts that affects how we conduct our observations and thus distorts our findings. Knowing this about ourselves gives us a leg up in mitigating its effects.

  And our motives (conscious or otherwise) can be especially tricky. I'm sure we've all faced the temptation to overstate a usability test's successes (or underplay its failures). The real dangers are more subtle—for example, when we unconsciously shape a test's design to favor a particular conclusion.

Ultimately, it is once again all about the user. Whether it's a usability test, card sorting exercise or complete ethnographic survey, we want to understand everything we can about how users use our product or Web site. Keeping this front-of-mind during the design, delivery and analysis of user observations is the best strategy of all.