What we measure
Measure changes in people, institutions, livelihoods and communities, not only activities delivered.
Activity is easy to count and easy to mistake for effect. A programme that reports workshops run, people attended and prototypes built has described its own effort, not its results. We measure the two separately.
We measure quality against two recognised frameworks:
- the World Bank’s principles: ask why, design at scale for all, empower teachers, engage the ecosystem and be data-driven[54]
- UNICEF’s pillars: safety first, impact on learning, designed for children, accessibility, and inclusivity and equity[53]
Assessing learners in maker programmes can be done and has been done before: most of the makerspaces surveyed already do it[21].
We are testing a longer chain of change: from access to learning, to capability (the skills, tools and support to do something), to innovation, to a livelihood or enterprise, to community impact, and finally to change in the wider system.
That chain is a hypothesis to test, and it is not compulsory for any participant. Employment, research, further education, community leadership, teaching and venture creation are all valid destinations.
From activity to impact: levels of result
Activities, attendances, prototypes, innovations, ventures and impacts are not interchangeable. A workshop run is not a skill gained. A skill gained is not a job. A prototype is not a solved problem.
Reports often break this rule, usually by moving a result one line up the table when the numbers on the correct line are disappointing.
Innovation ladder: from a project to a solved problem
In innovation work, four different things are often reported as one. We separate them and measures the drop-off between them: how many projects stop before the next rung. That drop-off is where the useful information is.
A unit launched, a programme run, a learner trained or a prototype built is not a solved local problem. Only rung four is.
Rung four is defined by three tests: the work comes from the community, a real user who is not on the project team has tested it, and someone depends on the outcome. Enthusiasm about a project does not count.
Awards are recognition, never proof of impact. A prize says a panel found the work impressive. It says nothing about whether anyone uses it.
The records and checks we are building
We build these records and checks before we add indicators (the specific numbers we track). We add an indicator only when that number will help with a real decision about a programme, how it is run or how it is funded.
Indicators added before these records are in place produce numbers without a baseline, the starting measurement needed to make sense of them.
Unique participant register
Consent, basic details about each person, and programme history, so that individual people can be counted separately from attendances.
Skills assessment at the start and the end
Tied to each curriculum. Measure the content, or do not claim it. We are adding this to our courses one by one. Until a course has it, we do not claim what learners gained: enrolment shows who took part, not what they learned.
Completion and leaver register
Where people go next, and why people leave. The people who leave are followed as carefully as the people who finish.
Innovation and venture register
Gates passed, status, users, revenue, jobs, survival.
Follow-up over time
At 3, 6, 12, 24 and 36 months, starting from each person’s work and income when they join. Without this, there is no way to tell whether people go on to earn more.
Community co-creation check
Who defined the challenge, who made the decisions, what conditions the community set, and who owns the result.
Bridging outcome register
Each connection linked to an opportunity or result that can be seen, so that the share of connections that led somewhere can be worked out.
School and teacher transfer
How well the work was put into practice, and whether teaching continues on its own after we step back.
Co-creating is the first word of our mission, and nothing we track yet measures it. For now, we have no evidence on how well we co-create.
Measuring how a community makes decisions and owns the result is difficult, and there is no method for it yet. We do not use a substitute number in the meantime.
The questions the mission is tested against
- Can it be shown that the community was underserved, in a specific, named way?
- Did the community take a real part in decisions?
- Did participants gain skills they can demonstrate?
- Did they use those skills after our direct involvement ended?
- Did they keep solving problems without depending on us?
At present we can answer the first question in part and the third only weakly. The answers to the fourth and fifth rest mainly on individual stories, not on information gathered about everyone. One case shows that something can happen, not how often it does. We are building the eight records and checks above to close that gap.
When measurement happens
| Point | What is captured |
|---|---|
| Intake | Profile, prior experience, education, employment, income, access barriers, starting capability |
| During the programme | Attendance, progression, hands-on assessments, projects, support received, safeguarding |
| Completion or exit | Capability, destination, satisfaction, project status, reason for leaving |
| 3 months | Continued practice, study, work, project use, immediate barriers |
| 6 months | Skill retention, employment, income, innovation use, venture activity |
| 12 months | Durable outcomes, school adoption, venture survival, community effects |
| 24-36 months | Changes in livelihood, leadership, adoption by institutions, replication elsewhere, lasting impact |
Following only the people who finish, or the visible successes, makes results look better than they are. A programme that teaches above the level its learners start at sees attendance fall away, and the good feedback of the few who finish can then look like success. We follow up everyone who starts.
Bridging: how connections are counted
Bridging means connecting local people and organisations with people, opportunities and resources elsewhere. It is often named as what sets DreamSpace Academy apart, yet we have found almost no outside research on how it leads to change.
So we have to gather that evidence ourselves. We count:
- Jobs and paid assignments that came through a connection
- Partnerships that produced an observable opportunity
- Local organisations that gained funding or visibility through a connection
- Research placements and university connections
- Partner organisations taking up our methods on their own
- Former participants becoming trainers, managers and directors, and the share of sessions led by local former participants
A partnership counts only when it produced a result that can be seen: a programme run, funding, a placement, employment, research, infrastructure, adoption by others or a change in policy.
And one-way bridging does not count. When value flows only inward, that is just sourcing, so what flows back the other way is counted too.
How we report results
- Count unique people separately from attendances.
- Record leavers and reasons at every stage.
- Report ventures as active, dormant, closed or transformed, separately.
- Do not infer community impact from a prototype or an award.
- Do not infer income improvement without an intake baseline and follow-up.
- Report stopped projects as well as successful ones.
- Keep apart what people say about themselves, what staff observe, what official records show, and what has been checked independently.
- Measure who stays and who drops off, not only the gain right after a programme.
- Where possible, adjust any claim that the work caused a change for attribution (how much of the change is due to us), deadweight (what would have happened anyway), displacement (gains that only moved from somewhere else) and drop-off (effects that fade).
- Collect only data that informs a real decision and can be protected responsibly.
How a result is presented
- 1Baseline
- 2Target
- 3Actual result
- 4Difference
- 5Explanation
- 6Corrective action
We set targets only after a reliable baseline exists, not before. Every indicator states:
- which level of result it measures, and how that links back to the work
- exactly what is counted, and out of what total
- whether it counts people or attendances
- its baseline and target
- where the data comes from, and which record holds it
- who collects it, and how often
- how the results must be broken down
- its consent and privacy rules
- how it is checked
- its known limits
Results are broken down by gender, disability, geography and economic situation. Age group is not asked for. Age does not decide where a learner is placed, and an age band on a sign-up form is how placement by age creeps back in.
Instead, we record one exact value, a participant’s birth year, kept privately and with consent, so that we can show who we actually reach and meet our duty of care. An age band is worked out from that only when a report asks for one, and the report states where the bands are cut.
Explorer-to-Maker and Maker-to-Innovator progression rates are useful measures. They are not a claim that learners must move through the stages in order.
There is a recommended path, but a learner can be placed at a different stage, and moves on when set conditions are met. The rates show the moves people actually made, never a course everyone was expected to finish.
What we ask funders for
The link to income is a hypothesis, and when we searched sceptically we found the evidence thin. We found no clear result linking a rural makerspace to income, and the effects found by the most rigorous skills trial fade over time[55, 56].
Outcomes in education, creativity and community impact are well evidenced[24, 29]. Income is the thing that has to be measured rather than asserted.
That shapes what we ask funders for.
Fund the work and the evidence
A programme that runs well and publishes nothing leaves the field exactly where it found it. A programme that runs well and publishes proper data that follows people over years, including the parts that did not work, adds evidence the field is short of.
Publish limitations alongside results
We publish failures, incomplete evidence and unresolved questions next to the positive results, and we intend to be held to that standard.