big data critique ethnography methodology qualitative research research methods sensemaking

Of methods and mindset, or toward a theory of impact

I’m working on building my vocabulary for why and how it matters that we reflect on our mindset, our methods, and most importantly, our reason for doing social research in the first place. This exercise/essay is part of a larger set of writing projects.

Of methods and mindset, or toward a theory of impact

Annette Markham


(In this piece (and more in the future), I’m working on building my vocabulary for why and how it matters that we reflect on our mindset, our methods, and most importantly, our reason for doing social research in the first place. This is part of a larger argument i’m working on for several different writing projects)

How much does our academic research really matter? Most of my career, I’ve thought “not much” in the large scale of things. Lately (or in the past three years, anyway), I’ve begun to realize I’m wrong. It matters in ways that are deeply hidden from our typical self evaluations, or at scales that are simultaneously too small or too broad to recognize, much less measure in any statistical fashion. What happens to my mindset and methods when we begin with a premise that recognizes and highlights the possible impact of our inquiry?

If we consider how units of cultural information travel and function in the broader ecosystem, and really start to lay out these paths in a more visible way, we can see the complication of trying to encapsulate, enclose, or explain. Beyond the problem of describing “what is going on here” (the classic ethnographic question), one must consider what happened before we saw this snapshot of the context. If we stop for a moment the flood and swirl of experience to look closer, we immediately extract the phenomenon from lived context. We also abstract the interplay of actions that constitute what was interesting and relevant for us in the first place. There’s no such thing as just in time research, since it’s mostly–if not always– after the fact.

One must also consider that to understand how anything came to be, one would need to replay or re-experience the flow of interactions. Even if we (when we) are able to do this, we are not experiencing it in real time, since it already happened. And at a more basic level, we could just go back to the initial problem (or fact) that this flow of moments is extracted and abstracted from context whenever we later think about it, measure it, or analyze it.

This is not a problem from a conceptual standpoint as it is inherent to the sensemaking process. But if we claim some sort of external validity of what is after the fact constituted as data, we can get distracted from the fact of its inherency by searching for a way to make it something it’s not and never will be. Instead, one can get more productively sidetracked by considering different qualities and measures of our own activities. Say, for instance: As I talk at conferences, teach in classrooms, document and disseminate the processes and products of my scientific inquiry, I produce and combine units of cultural information in unique ways. These function as remixes when experienced by others and in the process can wither from inattention or grow and spread, like memes. These cultural units of information play in many and larger fields, beyond my own awareness or comprehension.

As temporal representations and possibly measures of my value, these units of information may come to matter. For scholars, we must acknowledge that our audiences are no longer just our students or colleagues. This has always been the case, but our ideas are much less likely in the 21st Century to sit quietly in books on library shelves. This means our research matters, in that every action we take to focus on a phenomenon and then somehow transform our witnessing into something else through the interpretive ethnographic filtering process “reconfigures the world in its becoming” (Barad, 2007, p. 396).

If we believe we ought to produce and disseminate research in a world that has plenty of information already, the issue of impact becomes an important consideration. Rather than taking the common administrative view of ‘impact’ as quantifiable in number of publications, production of evidence-based results, or statistically measured impact ratings, we can take a more change-oriented view to ask: How can we make a difference that makes a difference? This is not just a political challenge but also an epistemological and interpretive one. Let’s work through these levels.

The interpretive challenge is to add something else to a lens that seeks to provide thick description. Especially knowing that ‘digital internet’ interweaves with all of social life in ways that cannot be untangled, qualitative inquiry of these phenomena requires shifting one’s lens to better attend to fields as flows and networks, where self-other relations and social forms are temporary informational assemblages. Getting at these flows is an interpretive challenge, primarily guided through reflexive practice. By paying close attention to the details of how we accomplish our studies through habitual, instinctive, and playful action of the embodied mind, we can critically examine these habits and experiment with other forms of generating material that later is remixed into findings, conclusions, or understandings.

Beyond the interpretive level, this is an epistemological challenge: When confronted with the fact that our research will have impact plus we can’t know everything and perhaps not even much, how could we possibly think about our sensemaking practices in the same way?  Here, epistemology should be considered within its more political frame rather than the ideal, given that we are not born with our epistemologies, but learn to have certain assumptions about how we (or how we ought to) attend to, make sense, or understand the world.

Take for example the swiftly moving worldwide trend toward datafication, which more precisely for ethnographers involves the quantification of human experience. We are urged in many ways to take up this particular mindset. After all, research funding is channeled toward ‘evidence-based’ research design and taxpaying publics demand measurable solutions to real problems. Within this shift, qualitative researchers are pressed to respond, likely in one of three ways: 1) Change one’s vocabulary to match the rhetoric of big data, 2) Change one’s methods to meet positivist criteria,  or 3) Do nothing, which risks further marginalization.

For any smart interpretive or postmodern ethnographer, these are three impossible options. An alternative response is to 4) Reclaim the power of the interpretive epistemology and demonstrate this more clearly as meeting the demands of ‘evidence-based findings’ or ‘impact’ without sacrificing the integrity of the close, local, subjective, and qualitative paths to meaning.

To challenge datafication is not merely a political move. It is an epistemological defense of the most fundamental processes of making sense of lived experience, or living effectively in communities and societies.

The strength of ethnographic inquiry is not about data, in any sense of the word used by computational scientists, statisticians, or economists. Of course we count things. Of course we can use large datasets to help us think about the cultural formations we study. We use computers to help us sort and manage the materials we get from our fieldwork. Whether we call this ‘getting’ a process of ‘generating’, as if it is our decisions that create data, or ‘collecting,’ as if data pre-existed, the act of interpretation is—no matter how much it might be aided by machines and machine learning—a human based set of decisions about what matters, or what a wink of a wink means (Geertz, 1973). It is not meant to be a data science. Especially as the trend toward treating humans (and their data) as data continues, the fundamental epistemologies that have always grounded interpretive ethnographer is an important antidote.

To develop a clear counterpoint to ongoing trends toward quantification and datafication, however, requires refocused attention to one’s mindset, attitude, and reason for doing research. This doesn’t mean we have to avoid the term ‘data’, but it does require us to remember to walk a fine line between using the term strategically and using positivist epistemologies that undergird this concept in the first place.

Perhaps the most important goal of impact-oriented research is focus on the purpose of research, which can allow one to actually loosen the grip of technique (e.g., participant observation) as well as representation (e.g., thick description or narrative account), which bind us in a particular stance with a particular type of focus. Allowing for subtleties of practice or examples whereby this bind may be broken, the norm of the qualitative or ethnographic is to describe or explain. This implies there is a ‘thing’ or ‘object’ or identifiable something to describe, or a ‘whole’ to explain. This implication emerges from the longstanding mistake of separating qualitative from quantitative methods, but goes deeper, situated in the continual positioning of positivism/modernism/functionalism as the primary or mainstream stance, whereby anything qualitative becomes alternative, rather than distinctly different. This dualist or binary positioning has long infected qualitative practice and products, which blocks the epistemological development of the interpretive approach. While rectified in part by strong interpretive schools of thought, these are less prominent outside the U.S., where they have flourished. Thus, it remains difficult yet crucial to step beyond (or peer around) the interpretive goal of deep understanding and consequent thick description to address such questions as: What is our role in the larger scale and scope of things? What are we producing as part of our intellectual energies and output?

For most of us, the goal behind all other goals is to change the world for the better, to influence the shape of possible futures. What influence does the ethnographic mindset have? What might constitute useful and meaningful products of inquiry if we anticipated our goal to be intervention rather than description; change rather than comprehension?  This is a question that cuts to the heart of ethnographic practice of course, and it’s not new. But the gap between our research and actual social change shrinks through the widespread sharing and remixing of knowledge that occurs through social media. We find ourselves in the amazing position of speaking to and learning from multiple audiences. We should not underestimate the extent to which we can have impact in more public arenas.

The ethnographer’s understanding and depiction of cultural complexity both counters and strengthens the statistical abstraction of computational analysis. Ethnography is distinctive, in that its methods enable audiences to hear the voices of individuals, learn about intensely localized meanings, and understand some of the ways culture is lived. Nobody is better positioned to help different publics adopt more flexible and living metaphors around the concept and rhetoric of data and quantification. It can actively contest or at least complicate the typical ways in which computational reasoning tends to reduce and flatten experience to data points.

As the ethnographic perspective counters datafication, it simultaneously helps those working in data sciences understand the particular strength of the computational approach.  This is by virtue of contrast. When we can showcase the qualities of thick description, or provoke affective responses through evocative ethnographic renderings, these become important elements that stand apart from quantified findings. Both enrich the other.

At some level, it is easy to accept the premise that the ethnographic mindset is instrumental in helping those who design our future interfaces and infrastructures understand the complexity of the human experience. When we move to the idea that we will have impact whether or not we want to, and therefore must take a political stance, the challenge of actualization becomes more salient, prompting particular questions. How do our methodological and epistemological assumptions about qualitative research encourage particular ways of knowing or ways of approaching and analyzing social problems? How can we adjust our expectations for outcomes and findings when we work with communities, especially when we move beyond involving them in our research projects to the point where we are actively enacting or experimenting with transformative practices? Are we still in a state of “studying” or “researching” in the typical sense?

More and more, research of this sort becomes a form of pedagogy and activism, enabling new processes and products, but also sparking new ethical and moral considerations. Once we connect our research goals to social change, we get closer to ethical questions of impact. How might our products be used as interventions rather than just descriptions, to encourage different structures for social practice? Silverstone (2007) contends that our moral challenge is to get better at seeing the way our research interweaves in larger structures of meaning. This translates directly into an ethic of future accountability. In other words, we don’t use simply ethics as something we’ve learned from past mistakes. Rather, we produce the ethics of the future as we go about our everyday academic lives of producing research.

Our findings are directly connected to frameworks that can shape how users, designers, and other researchers conceptualize the socio-technical ecologies within which we are saturated. The impact is tangible and real: Look at the way the qualitative researchers and ethnographers at Microsoft or Intel influence the way computer scientists design user interfaces, or the way computational biologists might conceptualize and mark racial categories in DNA sequences. Many scholars write regularly in mainstream news outlets, using their academic work as a foundation for swift and thoughtful responses to public issues, crises, and controversies. These are not just examples of applied research, special outreach efforts, or accidents. These are scholars who have made a deliberate choice to find ways to do research that is read by different publics and composed in formats that can be disseminated quickly and understood by people across many expertise areas.

But it’s not ethnography or qualitative findings that do this political or technological intervention work. It’s ethnographers and qualitative researchers. They speak with a particular interpretive authority. They speak at a particular depth. They focus on particular aspects of lived experience, reminding the world that it is impossible to capture all human experience in data form.  So when Tricia Wang writes that big data needs thick data, what she really means is that a world immersed in computational explanations of the world need stories to get at what it means to have actual lived experience. An ethic of future accountability means accepting the responsibility that we are the people who must push this agenda, over and over again.

Leave a Reply

Your email address will not be published. Required fields are marked *