First Nations Management & Leadership · Arcus perspective
How to Measure Whether First Nations Health Transformation Is Working
Transformation is a claim that requires several kinds of evidence
More funding, additional positions and a new governance structure can be necessary achievements without demonstrating that people experience better services. Conversely, an organization may improve access while a population-level health measure changes slowly or is affected by circumstances beyond the organization. A single score cannot fairly answer both questions.
Arcus’s management thesis is to organize measurement around explicit decisions and contribution claims. For each important change, define the intended benefit, the mechanism expected to produce it, the evidence that would support the claim and the competing explanations that could weaken it. Then specify who can change the operating model when the evidence is disappointing.
The Nation must define what success means and govern how its information is interpreted and shared. This article offers management design guidance, not a clinical indicator standard or a validated evaluation tool. Appropriate community, clinical, evaluation and information-governance expertise should determine the local measures and their use.
Use the evidence architecture without importing another system’s scorecard
Johnson and Sutherland’s NCCIH conceptual framework distinguishes health-system actions, experience, performance, and health and wellness outcomes. It also recognizes influences beyond health-service performance.[1] This supports keeping different kinds of evidence visible. As a conceptual framework, it does not establish that a specific intervention caused a local outcome.
CIHI’s 2025 account of the Indigenous-led Cultural Safety Measurement Collaborative describes a process for selecting indicators, with patient experience, governance, resources and other domains represented.[2] The account describes planned development and reporting as well as selection; it should not be read as proof that every indicator already has a complete comparable data series. It illustrates the importance of Indigenous-led priorities and feasible measurement.
FNIGC’s OCAP® guidance places First Nations authority over collection, interpretation, storage and sharing at the centre of information governance.[3] A technically convenient reporting system therefore cannot determine the purpose or permissions of the measurement program. Leadership needs an approved governance arrangement before commissioning collection or linking records.
Start with decisions, then specify the evidence
List the decisions leadership expects to make over the next operating cycle: whether to expand a service, change a referral process, invest in management capacity or renegotiate a partner commitment. Each decision should have an evidence question. If a proposed indicator has no foreseeable decision use, identify whether it is required reporting, community accountability or an optional measure that may not justify its burden.
For a hypothetical navigation service, contacts made describe activity. Accepted handoffs describe part of the process. The person’s account of whether support was respectful describes experience. Completed access to the intended service describes another result. None alone establishes a change in overall health. Keeping these distinctions explicit prevents an easy-to-count activity from becoming a substitute for the benefit that justified the investment.
Write a short indicator specification: purpose, population, numerator and denominator where relevant, inclusion and exclusion rules, source, frequency, owner and known limits. Define when the observation begins and ends. A waiting-time measure starting at referral acceptance will answer a different question from one starting when the person first sought help.
Pair a result measure with a measure of what might deteriorate. Faster handling may coexist with more repeat contacts or inappropriate closure. Greater local access may require additional staff travel and coverage. These balancing measures help leadership see whether an apparent gain shifts burden elsewhere.
Test competing explanations before claiming improvement or failure
A change after transformation is not necessarily a change caused by transformation. Check whether eligibility, recording practices, coverage, staffing, seasonality or external service availability changed during the same period. A rise in recorded concerns may reflect a more trusted reporting route, more harmful experiences, improved documentation, or a combination. The appropriate response depends on understanding the mechanism.
Establish a baseline that is sufficiently comparable to the new period. Where the baseline is weak, state the limitation and begin building a consistent series; do not manufacture precision by combining unlike data. Annotate major operational changes so later reviewers can interpret the pattern rather than guessing why a line moved.
Use several forms of evidence. Administrative data can describe recorded events, while appropriately gathered accounts can reveal why a pathway failed or why people avoided it. Neither automatically represents those who are missing from the collection process. Seek a locally appropriate way to understand non-participation without pressuring people to disclose experiences.
A comparison group or phased implementation may strengthen an evaluation when appropriate and ethically and operationally feasible. It must still be examined for differences in need, access and context. Where a credible causal estimate is not possible, report a bounded contribution judgment: what changed, why the initiative plausibly contributed, what alternatives remain and how confident the organization should be.
Make small populations and uneven coverage visible
Small numbers can produce large changes in percentages. Show the underlying counts and the size of the eligible group where disclosure rules permit, and avoid ranking communities from unstable estimates. Longer observation periods can help distinguish a persistent pattern, but can also conceal recent deterioration. Choose the period to fit the decision and describe the trade-off.
Do not treat missing information as a favourable result. A dashboard showing no unresolved cases may mean all cases were resolved, or that cases were not recorded. Check completeness and definitions before interpreting performance. Mark an unknown result as unknown and assign a proportionate plan to improve the evidence.
Protect people from identification through small cells, unusual circumstances or combined reports. The Nation’s governance and relevant privacy requirements should determine access, aggregation and release. A detailed operational review may be appropriate for an authorized team while the community report requires a different level of detail.
A regional average can conceal an improvement in one location and a decline in another. Examine variation where authorized, meaningful and safe. Treat the purpose as understanding operating conditions and equitable access, rather than constructing a league table that penalizes communities facing different constraints.
Choose a measurement model the organization can sustain
A focused operating dashboard provides frequent information for decisions within management’s control. It can be economical when it uses reliable existing records. Its weakness is that it may privilege readily available activity measures and overlook experience or longer-term outcomes. Add periodic deeper inquiry where the dashboard cannot answer the question.
Periodic independent or jointly governed evaluation can investigate mechanisms, competing explanations and difficult questions across partners. It requires a clear mandate, access and protected time for participants. Independence does not replace Nation-led governance, and a sophisticated report has limited value without a response owner and resources to act.
A shared regional measurement function can pool analytic skills and infrastructure. It also requires agreement on purposes, definitions, access, interpretation and the local value returned to each Nation. Standardization should be negotiated around useful comparability; it should not erase locally important priorities or create unauthorized regional access.
Community-led learning and review can bring experience and local definitions of progress into decisions that an administrative dashboard misses. Fund participation, preparation and follow-through. Avoid repeatedly asking people to share difficult experiences if the organization cannot explain what it will do with the information.
Many organizations will combine these approaches. Specify the role of each, the questions it answers and the questions it cannot answer. More indicators do not compensate for weak definitions or an inability to act.
Budget for interpretation and response, not just software
A measurement budget includes data stewardship, record-quality work, analysis, community participation, training, protected review time and appropriate technical and evaluation expertise. Separate initial design and configuration from recurring collection, maintenance and reporting costs. Include the workload imposed on service teams and partner organizations.
Compare a limited manual process, an improved existing system and a new platform against the actual questions, permissions and volume. A platform may improve consistency but will not resolve disagreement about definitions or authority. Test the smallest sustainable arrangement first where the risks and workload permit.
Connect findings to operating resources. If a review identifies a referral bottleneck but every responsible team lacks time to redesign it, the measurement program has diagnosed a problem without creating a response. Present corrective options with costs, responsible owners and consequences of delay. Keep routine monitoring proportionate so it does not consume the capacity needed to improve services.
Evidence-to-Decision Test
This original Arcus decision framework is a management aid for local adaptation. It is not a validated assessment instrument or a substitute for the Nation’s authority and relevant professional requirements.
Purpose
Question: Which authorized decision or accountability commitment uses this measure?
Decision evidence: Name the intended user and action that different results could trigger.
Meaning
Question: Does the measure describe activity, experience, performance or outcomes?
Decision evidence: State the definition, population and relevant denominator.
Credibility
Question: What else could explain the observed change?
Decision evidence: Review baseline comparability, coverage and competing explanations.
Authority
Question: Who governs collection, interpretation and release?
Decision evidence: Confirm Nation-approved permissions and appropriate safeguards.
Response
Question: Who can act, with what resources, and by when?
Decision evidence: Record the decision, owner, cost and follow-up evidence.
Turn the design into an operating commitment
Choose one consequential decision and assemble a compact evidence plan with the Nation’s authorized representatives and relevant service leaders. Agree the intended benefit, a small set of measures, definitions, permissions, sources and costs. Review a sample of existing records before promising a reporting frequency that the organization cannot sustain.
Run an initial reporting cycle as a learning exercise. Ask whether the evidence is complete enough to support the decision, whose experience is missing and what alternative explanation remains plausible. Correct definitions or collection methods transparently, retaining a record of changes so apparent trends are not confused with measurement revisions.
At each formal review, record the interpretation, uncertainty, decision and follow-up owner. Report back through the agreed community and governance channels in a usable form. Arcus can support measurement architecture, operating dashboards, evaluation scoping, governance arrangements and costed response planning, working with appropriate clinical, data and evaluation specialists.
Sources and limits of the evidence
- Johnson and Sutherland, NCCIH. A conceptual framework for Indigenous cultural safety measurement (2022). Conceptual distinctions and proposed relationships, especially pp. 19–25; not a local causal impact estimate.
- CIHI. First Nations, Inuit and Métis wellness, 2025. Account of the Collaborative’s indicator-selection process and planned development; not evidence that all selected indicators are already reported.
- FNIGC. The First Nations Principles of OCAP®. Nation-specific information governance guidance. OCAP® is a registered trademark of the First Nations Information Governance Centre (FNIGC).
Research reviewed October 17, 2026. Decision frameworks, options and hypothetical illustrations are Arcus management synthesis. No client outcomes or universal staffing, savings or impact benchmarks are claimed.
Health Transformation collection · Accountability in health governance · Organization design services · Arcus consultant directory