Questions and answers
Benefits of more integrated approaches to design and analysis of qualitative research
Researchers are inherently drawn to thinking about the relationship between questions and answers and exploring the nuances of that relationship in each study they undertake. These can include things related to formulating questions for research, methodological preferences that influence how they go about gathering evidence, and even the finer details of how to analyse and synthesise that evidence.
In qualitative research there is a relationships between the design of the research and design of analysis that is critical to the effectiveness and efficiency of the study.
In qualitative research there is a close relationship between the design of the research (all things you need to execute the data collection) and design of the analysis (the way in which you go about processing that data). That relationships is critical to the effectiveness, or the degree of confidence and depth in answering the questions, and the efficiency, or the amount of time and resources you need to execute the study. This is largely because the qualitative data collected is unstructured (in form of interviews, stories, images, video and audio) and the questions can be open ended and exploratory. In short, if the research design and analysis design is not integrated well, it can lead to gross inefficiencies in focusing your analytical scope and preparing and formatting the data to allow the type of analysis that is right for what you are trying to achieve,
We have recently encountered an example of this relationship and the challenge it can pose for research and analysis that is worth highlighting. A government department used their own capability to design and launch a survey for a public consultation (a process by which local and national governments in the UK engage with the public to gather feedback on a proposed course of action). Simultaneously, an announcement was made that they are searching for a partner to help them analyse, what they estimated to be, “thousands” of responses to the tens of free-text format questions. No small task indeed as it would require some serious manpower to process all the responses manually.
This kind of phasing of research and analysis has some underlying implications to the efficiency and effectiveness of the study and thus carry a degree of risk. Even though the high-level goal of the analysis might be well articulated, it's difficult to deliver on this goal if the specific methods and metrics around data collection are not defined at the onset. In our example of the public consultation it was clear that the goal was to get a) an overall sentiment to the proposed programme; b) an understanding of the reasons behind why people would disagree with it; and c) any suggestions on what an alternative solution might be. It was unclear, however, how this information would be used in the public consultation process, whether the decision-makers favoured qualitative or quantitative assessments, both of which were possible but would require radically different approaches, and in what format the outcome of the analysis should be delivered.
A question we would ask ourselves would be…
...how to best articulate and document the overall goal of the study and the specific design of research and analysis to potential project partners?
A second implication of phasing design of studies that is important to highlight is that it leads to data processing issues which cause process inefficiencies and are really missed opportunities. Allow me to illustrate. Here is fictional example of an open-format question modelled after the public engagement study.
What are the metrics, and what threshold should we apply to identify and engage the organisations who have the capacity to produce at required scale, and to eliminate those that do not?
This type of question design leads to an open response that is very difficult to process for even the most advanced Natural Language Processing algorithms because the sub points in the question can be perceived as independent conditions thus leading to separate answers. In short, the design of the question does not consider the analytical method and with several thousand responses combined with an unlimited character field, this one question alone could be a huge task for researchers to analyse.
Good question to consider in this case would be…
...how can the design of data collection be optimised for evidence gathering, data processing constraints, and desired outcomes of qualitative analysis?
And finally, we must also consider that merging data from multiple sources of input is a giant undertaking that eats up project resources. In our public consultation example, the survey was being deployed as a web form, email response, phone call, and a written form in various locations. Taking inputs in such different formats and preparing them for a shared analysis of any kind would prove problematic and very labour intensive.
A clever research and analysis design would answer…
...what are the right data collection methods based on compatibility of data output and potential for usability and engagement of participants with the study?
New technologies are making deployment and analysis of qualitative studies possible in ways that could not have been imagined even a decade ago. With it comes a need for a more integrated approach to design of research and analysis especially with more and more pressure to match the efficiency and scale of quantitative data analysis. Researcher could do worse than consider a more integrated approach to their process and connecting the chain of activities between a question and an answer more intimately.
Mindset reminder
- A more integrated approach means iterating between designing both research and analysis from the very beginning
- Large scale qualitative data analysis could be made more efficient and effective using new data analytics approaches and informatics software
- Qualitative research is complicated and at times ambiguous and as researchers we need better mechanisms to define goals, design approaches and execute on according to plan