Data - why bother?
Collecting data has certainly had plenty of bad press over recent months. Thanks for nothing, Mr. Zuckerberg and co.
But like most things, the use of data can be used for good purposes as well as less beneficial purposes. I would like to outline why we are endeavouring to collect useful data about PETER-linked programmes and how we are using that data for improving what we do. Data for a positive purpose!
Most organisations collect data about their programmes. In many cases, this is done as a requirement by their funders. Essentially data is collected to ‘prove’ to the funder that the programme is being run as planned and usually there is some attempt to show what impact the programme is achieving (hopefully in terms of what was proposed to the funder).
Collecting data to show that the programme has been carried out is relatively straightforward. Usually it involves collecting numerical data about what happens - for example, how many people attended events, who they were (socio-demographic data such as age, gender, ethnicity) and how allocated budgets were spent in running the programmes.
Collecting data to prove what impact a programme is having is much more difficult. In some cases, there are ready-made measures available that can be used - how many participants pass a test or get a job. But many programmes aims are much more difficult to measure (improved self-confidence, reducing family violence) and it also requires trying to isolate the effects of the programme from other factors - for example, lowering unemployment rates can occur because of improved economic conditions rather than the effects of a specific programme for unemployed.
It is usually possible to devise ways to accurately measure these sorts of more complex outcomes, but involves greater cost to ensure valid data systems are set up, which is usually beyond most programmes’ budgets. Therefore, limited budgets often mean that outcomes are measured only in terms of self-report - simply asking participants and other key stakeholders involved about what they believe happened as a result of the programme rather than more complex external measures. Funders usually look for these sorts of results, but don’t always provide the level of funding it requires.
But there is also a third way that data can be used that the PETER collective believes makes it even more useful. This involves the looping of data back into the programmes to provide insights and information that help to inform the planning process and make them more effective. With this approach, data is a critical mirror to reflect (sorry about the pun) on what is happening as a result of the programme and then using these lessons learned to improve how it works in order to further increase its impact. So in addition to monitoring its impact, the data being collected helps identify why it is successful as well as the factors that are reducing its impact. This approach is sometimes referred to as developmental evaluation.
With PETER’s data system, we are endeavouring to implement this approach through each programme area developing:
Writing a Report Card that includes:
Programme goals (ideally specific ones that can be measured)
A justification of why the programme is important in relation to PETER’s aims
Research findings that justify the need for this type of programme (either local, national or international sources)
What programmes of this sort are currently available in Puketapapa, including any info about the participants
What data should be collected to show the programme’s impact and how it can be accessed
Identify any challenges in running the programmes
Writing a ‘Theory of Change’ that identifies how we think change will occur as a result of running PETER programmes including: Long-term outcomes, Short-term outcomes, Programme strategies and Data sources for each of these
Devising a data collection system in collaboration with the key stakeholders - what data will be collected, how will it be collected and who is responsible to collect/analyse it
Publishing the data in a transparent and readable format. At present this is mainly done through PETER’s website and informal reports to programme planning groups
A review process by the relevant programme planning group to review the data and identify changes needed to improve the programme
Review of the Theory of Change including the data system as a result of 3-5 above.
At present, this process is most advanced with the PCDS and Transitions programmes. Most of the documentation is available on the PETER Collective website.