Program audit · Evaluation · Resource decisions
Program activity is evidence of delivery—not proof of impact.
Arcus helps leaders examine whether a program remains relevant, how it operates, what results it contributes to and which changes are justified. We scope the review around the decision, evidence and level of assurance you actually need.
The decision before the program
01 / Are you testing delivery, outcomes, value or controls?
02 / What else could explain the reported results?
03 / What evidence would justify continuing, redesigning or expanding?
Define the decision before commissioning the review
A fictional board is asked to expand a program because enrolment rose and most surveyed participants reported improvement. Yet the board does not know whether the program reached the intended population, whether nonrespondents improved or what would have happened without the service. More activity and positive experience are relevant, but they do not settle the expansion decision.
Arcus’s central thesis is that a useful program review links the decision to a credible explanation of results and the cost of achieving them. A logic model describes the expected pathway; evidence must test whether that pathway operated, under what conditions and with what unintended consequences.
The service title “Program Audit” can cover different requests. At scoping, distinguish a management review, program evaluation and a formal assurance audit against specified criteria and standards. Agree the mandate, independence requirements and appropriately qualified team before work begins. An evaluation should not be presented as an assurance opinion that it was not designed to provide.
Four explanations for encouraging results
Test competing explanations before recommending more funding or a wider rollout.
01 / The program made a difference
Activities may have reached the intended participants and activated the expected mechanism. Examine the sequence, implementation quality and outcome evidence, including where results differ across groups or locations.
02 / Participants were different at the outset
People who enrol, complete or answer follow-up questions may differ from those who do not. Examine eligibility, access, baseline conditions and attrition. A favourable result among completers cannot automatically be generalized to everyone eligible.
03 / External conditions changed
Other services, policy changes or economic and social conditions may explain part of the improvement. Identify plausible influences and the counterfactual: what would likely have happened without this program?
04 / The reporting system changed
Definitions, recording practice, incentives or survey coverage may have shifted. Verify whether the same result is being measured over time. More reported outcomes may reflect better documentation rather than more impact—or vice versa.
What authoritative evaluation guidance contributes
The Treasury Board of Canada Secretariat’s Evaluation 101 guidance distinguishes delivery evaluation from outcome evaluation and sets out the role of an evaluation framework, theory of change and evidence matrix. These tools help connect questions to indicators, methods and limitations. Federal policy requirements apply within their own scope; they should not be assumed to govern every nonprofit or other organization.
Treasury Board’s guidance on theory-based approaches explains how evaluators can test mechanisms, assumptions and context when examining a program’s contribution. Such analysis does not necessarily produce a numerical estimate of impact. A plausible narrative needs supporting evidence and scrutiny of alternative explanations.
The UK Treasury’s Magenta Book distinguishes process, impact and value-for-money questions and emphasizes proportionate evaluation design. Arcus’s interpretation is to match the strength and type of evidence to the decision. A service-improvement review and a major expansion decision may require different designs; neither should promise causal certainty beyond what its evidence supports.
The Arcus program decision diagnostic
Apply this original framework to one pending program decision. It is a management aid, not a validated scoring system.
01 / Need and reach
Identify the problem, intended population and who is actually served. Examine access barriers, unmet need and overlap with other services. Distinguish demand for an activity from evidence that it addresses the intended need.
02 / Delivery and mechanism
Map resources, activities and outputs, then explain how they should produce outcomes. Test implementation quality, assumptions and dependencies. Identify whether weak results reflect a flawed design or incomplete delivery.
03 / Contribution and cost
Assess outcomes, competing explanations and the credibility of the comparison. Include delivery, participant and relevant partner costs. Distinguish total outcomes from additional outcomes attributable to, or credibly supported as a contribution of, the program.
04 / Decision and conditions
Compare continuation, redesign, expansion, replacement or an orderly exit. Specify evidence limits, implementation requirements and safeguards. Assign an owner and a review point for conditions attached to the decision.
Worked example: which cost per outcome?
Suppose a fictional program costs $120,000 and serves 100 participants. Sixty achieve the measured outcome, yielding a simple cost of $2,000 per observed outcome. That ratio does not establish the cost of producing an additional outcome.
If a credible evaluation estimates that 40 comparable outcomes would have occurred without the program, the estimated additional outcomes are 20. Dividing the same $120,000 by 20 gives $6,000 per additional outcome. Whether that is good value depends on the outcome’s importance, alternative delivery options, other benefits and costs, and the uncertainty around the estimate.
These figures are illustrative, not a program finding. A valid comparison requires defensible design and assumptions. If the counterfactual cannot be estimated credibly, report that limitation and use an appropriate contribution analysis rather than invent a precise net-impact ratio. A low cost per outcome can also conceal unmet need or exclusion of harder-to-serve participants.
Choose the review that answers the question
Methods can be combined, but the purpose and limits should remain clear.
Evaluability and evidence-readiness review
Useful when objectives, measures or records are not ready for a full evaluation. Produces a clarified program theory, evidence gaps and a feasible study plan. It cannot retrospectively create a baseline or reliable records that never existed.
Process and delivery review
Useful for improving access, workflow, consistency and implementation. Requires staff, participant and operational evidence. Can explain delivery problems without establishing the size of the program’s causal impact.
Outcome and contribution evaluation
Useful for examining what changed, why and for whom. May combine comparison designs with theory-based analysis. Requires suitable data and transparent assumptions; participant satisfaction alone is not a substitute for outcome evidence.
Value-for-money or formal assurance work
Value-for-money analysis compares resources and consequences against alternatives. A formal assurance audit requires its own criteria, standards and independence arrangements. Scope these explicitly rather than treating both as a generic report on whether the program is “good.”
Build a study leaders can use
Agree the decision date, intended users, evaluation questions and how findings could change a choice. Include program staff, funders and affected participants appropriately, while keeping the analytical basis for conclusions clear. Stakeholder involvement should help surface evidence and context rather than predetermine the finding.
Review administrative data, financial records, service standards, participant characteristics and outcome definitions. Document missing information and changes over time. Interviews and case studies can explain mechanisms and experience; their ability to support broader prevalence claims depends on design and sampling.
Budget for evidence access, data preparation, fieldwork, analysis, participant accessibility and reporting. Plan privacy, consent and secure access before linking records. A proportionate design may focus on a few decision-critical uncertainties rather than collect every available indicator.
Compare practical responses to the findings. Redesign may require new eligibility, referral arrangements, workforce capability or partner agreements. Expansion may change the mix of participants and delivery conditions, so evidence from one setting need not transfer unchanged. Ending or replacing a service also requires continuity and transition planning.
Distinguish findings, conclusions and recommendations. A management response should identify accepted actions, owners, resources and dates, as well as unresolved decisions. Review whether those actions occurred and whether the expected improvement followed. A completed evaluation report is not the same as a completed improvement.
What Arcus can deliver
Scope can range from a focused evidence review to a broader program evaluation.
01 / Evaluation scope and framework
A decision brief, program logic and theory of change, evaluation questions and evidence matrix. The framework specifies what the study can credibly answer and which limitations need management attention.
02 / Evidence collection and analysis
Document and data review, appropriate interviews or surveys, and analysis of implementation, results and alternatives. Deliverables separate observations, causal or contribution claims and remaining uncertainty.
03 / Options and recommendations
A defensible assessment of program relevance, delivery, outcomes and resource implications within scope. Leaders receive practical options with trade-offs, evidence limits and implementation requirements.
04 / Management response and follow-through
An action roadmap, responsibilities and review measures. Arcus can help connect the findings with program redesign, performance measurement and the next resource-allocation decision.
Discuss the program decision you need to make
Tell Arcus whether the immediate question is improvement, renewal, expansion, value for money or assurance. Share the decision timeline and what evidence exists. We can define an appropriate review and identify where additional specialist or independent assurance expertise is required. Begin with a general description rather than participant records.
Related support includes nonprofit research, organizational review and cost-benefit analysis. View experience through the central consultant directory.
Selected historical client work
The retained examples describe Arcus’s broader consulting work. They are not presented as measured program-evaluation impacts or assurance opinions.
Below is a sample of the range of services that Arcus has provided to clients.
- A survey of 2,350 consumers and 1,320 business leaders for feedback on sustainability trends
- Architecting a multi-year change strategy for a Fortune 500 company
- Mentoring a CEO on organizational change
- Excellence transformation of a leading B2B services company
- Creating a new sales deployment model for a healthcare company
- Developing a position evaluation and compensation model for a professional medical association
- Improving services to customer segments by deepening their understanding of customer attitudes