The Covid-19 crisis is affecting the way that we work and most of us are learning how to work remotely. The crisis has created additional challenges in terms of data collection efforts by social science researchers, evaluators and development organisations. As the recently launched website Evaluation in Crisis points out, all of a sudden, the majority of contexts now have become ‘hard to reach’. Because of physical distancing and movement restrictions, we have lost the opportunity to meet target groups and stakeholders face-to-face in interviews or focused group discussions or to observe activities and interventions in-situ. What can still be done in order to monitor our intervention(s) and ever-changing context?
1. Cause no (additional) harm – Many NGOs are responding to the challenges caused by COVID-19 by moving towards or increasing their use of remote data collection tools. A lot of guidance on how to do this properly is out there, for example, the Abdul Latif Jameel Poverty Action Lab’s post on best practices for phone surveys and ID Insight’s series of articles on the same topic on Medium here, here, here and here. There is also 60 Decibels’ Remote Survey Toolkit, developed in response to COVID-19. Furthermore, Feedback Labs has listed a number of platforms that offer remote data collection tools with pre-designed questions related to COVID-19, such as Social Suite and Upinion. This may be an interesting option for organisations that do not have the capacity or time to develop these tools themselves.
Clearly, the problem is not the lack of available tools to collect data at a distance. The challenges lie in the way we use these tools. As Adam Jowett points out, the first thing to bear in mind is that the health and wellbeing of participants and front-line data collectors should take priority over research and monitoring and evaluation timelines and deadlines. While it may be possible to change interviews from face-to-face to online, we should consider whether asking people to participate in research or monitoring activities at this time will put them under any additional unnecessary stress. In addition to that, we should be aware of the exclusionary factors surrounding the use of technology.
Feedback Labs urges organisations to consult constituents on the best way to reach them before any decision is made on what tool(s) to use (or what questions to ask); to give people time, space, and multiple options to answer in a way that makes the most sense for them; to be flexible on deadlines and to close the loop by telling people how you will be responding to their feedback. Heather Krause from We All Count, points out that remote data collection can serve multiple purposes, e.g. phone surveys can be used not just for data collection but also to maintain personal contact and to offer guidance and support.
2. Rethink rigour in a move towards “good enough” data – One of the most important skills required now, to navigate our current crisis as best as we can, is the ability to make decisions based on incomplete data. This is the polar opposite of how most people were taught to make decisions: based on the full facts. This will require a shift toward “lean data“. The emphasis here is on data use. As long as the data that is being collected enables you to answer key questions and inform your next step – you will be good to go. This entails a shift in mindset away from reporting and compliance toward creating value for an NGO and its stakeholders. What data does your organisation need on a regular basis to know what people are thinking and needing, what difference you are making and what needs to be improved? As Tim Bidey of Traverse points out here: “A sense of basic intended outcomes, simple stakeholder feedback or even reflective, anecdotal data from staff can all provide ‘good enough’ insights into the situation at hand – so long as people remain honest with themselves (and others) about how insights were generated and what limitations sit behind them”. As the Centre for Evaluation Innovation points out, one of the most striking and consistent findings from research is that even when data is available, it is rarely used by foundations, policymakers and nonprofits. The problem, they point out, is not always a lack of information, but a lack of relevant information and siloed learning and decision-making in organisations.
3. Shift away from periodic monitoring to “real time” data – In addition to the amount of data we collect, we need to reconsider the frequency of data collection and analysis. The emphasis now will need to be on ‘real-time’ data: data captured as we implement (vs. data captured at monthly or quarterly intervals). Such data, which will mainly be generated through observation, keen listening skills and reflective practice can allow for rapid course corrections and needs to be reflected on and made sense of on a daily (individually) and weekly (team) basis.
4. Throw in a pinch of analogue – Data collected by tech enabled tools is best for answering quantitative questions but is unlikely to capture the granularity needed to assess the quality of our work. As Prof. Lipton points out, when facilitating interviews, Focus Group Discussions or M&E workshops online, asking people to respond to creative writing prompts or to draw images might still apply. She believes that these old-style methods can be really creative and are enjoyed by participants. Kevin Kahkula’akea John Fong, stresses that when working online, we should always bring in ceremony – this can be as simple as reading a poem, asking people to stand up and shake out or taking three breaths together. He says: “Technology is so head oriented that we need to be extra mindful of brining in the heart and the belly (or soul)”.
5. Share your data as much as possible – Data Innovator points out that with reduced face-to-face interaction it is important to continue to share data within organisations and externally. Organisations that are able to package data and information with the help of dashboards and infographics, will be better able to use this data for rapid communication and decision making. With online interaction drastically increased during lockdown, Data Innovator advises organisations to share their data in online spaces to increase organisations visibility and the use of their data by others.