Measurement

What the academy measures

Measure changes in people, institutions, livelihoods and communities, not only activities delivered.

Activity is easy to count and easy to mistake for effect. A programme that reports workshops run, people attended and prototypes built has described its own effort, not its results. The academy measures the two separately.

The academy measures quality against two recognised frameworks:

• the World Bank’s principles: ask why, design at scale for all, empower teachers, engage the ecosystem and be data-driven[54]
• UNICEF’s pillars: safety first, impact on learning, designed for children, accessibility, and inclusivity and equity[53]

Assessing learners in maker programmes can be done and has been done before: most of the makerspaces surveyed already do it[21].

The academy is testing a longer chain of change: from access to learning, to capability (the skills, tools and support to do something), to innovation, to a livelihood or enterprise, to community impact, and finally to change in the wider system.

That chain is a hypothesis to test, and it is not compulsory for any participant. Employment, research, further education, community leadership, teaching and venture creation are all valid destinations.

From activity to impact: levels of result

Activities, attendances, prototypes, innovations, ventures and impacts are not interchangeable. A workshop delivered is not a skill gained. A skill gained is not a job. A prototype is not a solved problem.

Reports often break this rule, usually by moving a result one line up the table when the numbers on the correct line are disappointing.

Innovation ladder: from a project to a solved problem

In innovation work, four different things are often reported as one. The academy separates them and measures the drop-off between them: how many projects stop before the next rung. That drop-off is where the useful information is.

The records and checks being built

The academy builds these records and checks before it adds indicators (the specific numbers it tracks). It adds an indicator only when that number supports a defined programme, management or funding decision.

Indicators added before these records are in place produce numbers without a baseline, the starting measurement needed to make sense of them.

The questions the mission is tested against

1. Can it be shown that the community was underserved, in a specific, named way?
2. Did the community take a real part in decisions?
3. Did participants gain skills they can demonstrate?
4. Did they use those skills after the academy’s direct involvement ended?
5. Did they keep solving problems without depending on the academy?

At present the first question is answered partly and the third weakly. The fourth and fifth are answered mainly through individual cases, not through a system covering the whole organisation. One case shows that something can happen, not how often it does. The academy is building the eight records and checks to close that gap.

When measurement happens

Bridging: how connections are counted

Bridging means connecting local people and organisations with people, opportunities and resources elsewhere. It is often named as what sets the academy apart, yet there is essentially no outside research showing how it causes change.

The academy’s own record is therefore the only evidence that will exist, so these indicators carry the weight.

• Jobs and paid assignments that came through a connection
• Partnerships that produced an observable opportunity
• Local organisations that gained funding or visibility through a connection
• Research placements and university connections
• Partner organisations taking up the academy’s methods on their own
• Former participants becoming trainers, managers and directors, and the share of delivery led by local alumni

Two reporting rules apply. A partnership counts as an outcome only when it produced a result that can be seen: delivery, funding, a placement, employment, research, infrastructure, adoption or policy change.

And one-way bridging does not count. A relationship where value flows only inward is simply sourcing, so what flows back the other way is measured too.

How the academy reports results

1. Count unique people separately from attendances.
2. Record leavers and reasons at every stage.
3. Report ventures as active, dormant, closed or transformed, separately.
4. Do not infer community impact from a prototype or an award.
5. Do not infer income improvement without an intake baseline and follow-up.
6. Report stopped projects as well as successful ones.
7. Keep apart what people say about themselves, what staff observe, what official records show, and what has been checked independently.
8. Measure who stays and who drops off, not only the gain right after a programme.
9. Where possible, adjust any claim that the work caused a change for attribution (how much of the change is due to the academy), deadweight (what would have happened anyway), displacement (gains that only moved from somewhere else) and drop-off (effects that fade).
10. Collect only data that informs a real decision and can be protected responsibly.

How a result is presented

1. 1Baseline
2. 2Target
3. 3Actual result
4. 4Difference
5. 5Explanation
6. 6Corrective action

The academy sets targets only after a reliable baseline exists, not before. Every indicator states:

• which level of result it measures, and how that links back to the work
• exactly what is counted, and out of what total
• whether it counts people or attendances
• its baseline and target
• where the data comes from, and which record holds it
• who collects it, and how often
• how the results must be broken down
• its consent and privacy rules
• how it is checked
• its known limits

Results are broken down by gender, disability, geography and economic situation. Age group is not a field. Age does not decide where a learner is placed, and an age band on an intake form is how age-based placement creeps back in.

Instead, the academy records one exact value, a participant’s birth year, kept privately and with consent, so that it can show who it actually reaches and meet its duty of care. An age band is worked out from that only when a report asks for one, and the report states where the bands are cut.

What the academy asks funders for

The link to income is a hypothesis, and a sceptical search found the evidence thin. It turned up no clear result linking a rural makerspace to income, and the effects found by the most rigorous skills trial fade over time[55, 56].

Outcomes in education, creativity and community impact are well evidenced[24, 29]. Income is the thing that has to be measured rather than asserted.

That shapes what the academy asks funders for.