First Nations Management & Leadership · Arcus research
Designing Better Community Surveys
Design the evidence before writing the questions
A useful community survey connects a question leadership needs to resolve with information members can provide and a conclusion the results can actually support.
For Chief and Council, senior administrators and engagement leads, the decision is whether a proposed questionnaire will improve a service or planning choice enough to justify the time and information requested from members.
Arcus’s central insight is that each question needs an explicit chain from decision to measure to permitted claim. A well-presented questionnaire can still measure the wrong thing. More responses cannot repair a question that confuses service need, satisfaction and support for a proposed solution.
This article addresses management and service-planning surveys. It does not replace the Nation’s processes for elections, ratification, rights or legally required consultation. A survey is one source of evidence, not an automatic grant of decision authority.
What research tells us about questionnaire design
In a randomized experiment involving 4,581 U.S. adults in 2018, Pew Research Center compared separate yes/no questions with a select-all-that-apply list about undesirable experiences. Endorsement estimates averaged eight percentage points higher with separate questions, while the ordering of most- and least-endorsed experiences changed little.[1] The researchers’ accuracy interpretation depended on an assumption that underreporting was more likely than false reporting. This is evidence that format can affect measurement, not a First Nations effect estimate or a universal instruction to use one format.
FNIGC’s 2018 report on the First Nations Regional Health Survey Phase 3 describes a different, directly relevant design discipline: content decisions considered usefulness, comparability across survey rounds, cultural alignment, respondent burden and regional priorities. Its national design selected communities and then individuals, with documented sampling and analysis methods.[2] That substantial health-survey infrastructure should not be equated with an open community feedback form. Its transferable lesson is to connect purpose, population, governance and interpretation before collection.
Statistics Canada’s questionnaire guidance recommends clear concepts and time periods, suitable response categories and testing of the instrument.[3] Its survey-quality guidance distinguishes coverage, measurement and nonresponse errors from sampling uncertainty.[4] A small reported margin of sampling error does not resolve those other problems.
The practical tools below are Arcus analytical synthesis. They are not a validated questionnaire, a national benchmark or original empirical research. The appropriate design depends on the Nation’s question, population, resources and information-governance decisions.
Diagnose why the previous survey did not support a decision
The construct was unclear. A question about “better services” may blend availability, staff conduct, waiting time and desired expansion. Ask willing pilot participants to explain what they thought the question meant and how they chose an answer. If interpretations diverge, changing promotion or sample size will not repair the measure.
The wrong people or experiences were represented. A satisfaction question answered mainly by current users cannot explain why others never obtained the service. Compare the intended population with the actual invitation route and eligibility questions. Distinguish people who did not need a service from those who needed it but could not obtain it. Use the complementary Member Access Record when geographic reach is the central problem.
The answer could not influence the decision. Staff may collect broad aspirations while leadership needs to choose between two feasible service schedules. Trace each question to the decision brief. If no plausible answer changes an option, exposes a constraint or informs implementation, remove or justify the question before asking members to spend time on it.
Members reasonably disagree. Different circumstances can produce different preferences even with clear wording and appropriate coverage. Examine reasons and relevant service experiences rather than treating a split result as defective research. Do not redesign the question until it produces the sponsor’s preferred answer. Several causes can coexist; record evidence against the initial diagnosis as well as evidence supporting it.
Choose the right evidence method for the management question
A short open feedback survey can identify issues, experiences and possible improvements. It is useful when exploration matters more than estimating a population percentage. Describe findings as those of respondents and retain minority concerns. Invitations through social channels alone do not establish representative coverage.
An invitation to every eligible member can support broad participation in a defined, manageable population. It still requires a usable contact approach, accessible alternatives and honest treatment of nonresponse. Calling it a census does not make incomplete returns a complete account of the population.
A probability sample can support population estimates when there is an appropriate sampling frame, selection method and analysis. Seek qualified survey-method expertise where those estimates materially affect decisions. Design for the relevant subgroups and nonresponse risks; a total sample target alone is insufficient.
Interviews or facilitated discussion may be more useful when leaders need to understand unfamiliar concerns or work through complex trade-offs. Service records may answer questions about actual wait times more directly than asking people to recall them. Combining sources can clarify a decision, provided their different purposes and populations remain visible. Do not merge administrative events, survey respondents and meeting comments into one apparent vote.
Original Arcus decision tool
The Question-to-Decision Specification
Prepare one record for each essential measure. The sponsor, engagement lead and analyst should be able to complete the following six fields before launch. A missing field is a design decision to resolve, not a blank for the analyst to fill after results arrive.
Name the decision
Identify the open choice, who has authority and when evidence is needed. State which options are feasible and which depend on resources or further approval.
Define who and what is counted
Specify eligibility and the unit: individual, household, service user or service encounter. Decide how repeated responses or several members of one household will be treated.
Specify the experience or preference
Separate need, access, experience and preference. Set a meaningful reference period and identify the knowledge a respondent needs to answer.
Write and test the question
Use understandable wording and relevant response options. Provide appropriate routes for no experience, uncertainty or declining to answer. Test interpretation, not just spelling.
Define the denominator
Document who enters each calculation, treatment of missing answers and any justified weighting. Specify what subgroup comparisons can be responsibly reported.
Limit the claim and assign the action
State what the result can support, what alternative explanation remains and who will consider it. Agree how members will hear what changed or why a recommendation was not adopted.
Release the questionnaire when the essential records are coherent and the pilot supports interpretation. Revise an affected question when its wording or response options fail. Hold the affected measure when necessary information control or a defensible denominator is unresolved; continue other ready work where separation is practical.
Worked example: separate need from approval of a solution
Consider this invented question: “Do you support improving our inadequate office hours and online services?” It labels the current situation negatively, combines two changes and leaves the cost and proposed service unspecified. Agreement cannot tell leadership whether evening access, a better online route or something else is needed.
A fictional administration deciding whether to pilot one evening opening could instead ask whether a member tried to contact the office during the past three months, whether they obtained the help needed, and which barriers they encountered. People who did not try need a separate route: they may not have needed help, may have expected the service to be inaccessible or may have used another provider.
Then describe feasible alternatives in neutral, equivalent terms. Ask which would make access easier for that person: an evening opening, scheduled telephone appointments, an online request route, no change needed or another arrangement. Explain any relevant service trade-off before asking a preference. Do not invent an unfunded option and present it as available.
Report access experience among the people eligible for that question and preference among those answering the option question. A preference for evening access is not evidence of willingness to pay, the number of appointments likely to occur or authorization to cut another service. A bounded pilot and actual booking evidence may be needed before a recurring commitment.
Test whether separate questions about each barrier are worth the additional response effort. The Pew experiment supports taking format effects seriously; it does not establish that more separate questions always produce better local evidence. Preserve a clear “none” option where appropriate and never treat an unchecked item as a confirmed “no” without a justified design.
Interpret percentages without turning respondents into the whole community
Suppose, purely for illustration, 400 eligible people are invited, 120 give an answer to the proposed evening-opening question and 90 support it. Support is 75% of those answering that question. The other 280 people’s preferences remain unknown. The observed supporters are 22.5% of all eligible people; if every unanswered preference were supportive, the total would be 92.5%.
Those endpoints are arithmetic bounds under the stated assumptions, not a confidence interval, an estimate or evidence that either extreme is likely. Their width shows why the result alone cannot establish majority support across all 400 people. They also assume eligibility and one answer per person have been correctly established. Duplicate or ineligible responses would create another problem.
The sensible response depends on the decision. A reversible, limited service pilot may be justified by demonstrated unmet needs, available capacity and safeguards even when overall preferences remain uncertain. A consequential permanent change needs evidence and authority proportionate to its effects. Do not keep collecting responses merely until a desired percentage appears.
Statistics Canada explains that conventional statements of sampling uncertainty require a probability-sampling basis and do not account for all survey errors.[4] Avoid attaching a standard margin of error to an open opt-in survey. Report the invitation and fieldwork methods, question wording, dates, number answering each item and material limitations. For broader readiness to decide, use How Much Consultation Is Enough?
Budget for interpretation, testing and returning results
The survey platform is only one cost. Plan staff time for defining the decision, drafting, translation where needed, testing, invitations, assisted completion, data preparation, analysis, review and returning findings. Identify who has the skills and available hours; outsourcing fieldwork does not remove the Nation’s governance and interpretation responsibilities.
A transparent burden calculation can expose unnecessary questions. In a fictional exercise with 150 completed questionnaires, reducing an assumed 12-minute completion time to eight minutes releases 600 minutes, or ten hours, across participants. If 30 completions require staff assistance, the same four-minute reduction releases two assistance hours. These are planning assumptions to test, not predicted response rates or cash savings. The staff time is separate from participants’ time and should not be counted twice as payroll savings.
Keep a question when its decision value justifies its burden. Removing an essential access or eligibility question may make the form shorter while making the results misleading. Conversely, collecting detailed comments without capacity to code, review and respond creates an unfunded operating task. For recurring surveys, budget for maintaining definitions and documenting changes so that a new question format is not mistaken for a real trend.
Assign information control and a practical implementation sequence
Before fieldwork, the Nation should decide the permitted purpose, access, custody, retention, release and any further use of responses. FNIGC describes OCAP® as First Nations principles governing ownership, control, access and possession, applied through Nations’ own circumstances and protocols.[5] Do not assume that a vendor contract or the absence of a name field settles these choices. Small-group details and free text can identify a person.
The sponsor confirms the decision, authority and intended use. The engagement lead and analyst complete the specification and analysis plan. Authorized information advisers confirm the data arrangements and public explanation. Separate contact information from answers where appropriate, and collect only what the agreed purpose requires. Do not promise anonymity unless it can be supported.
The pilot lead tests interpretation with willing people representing relevant languages, experiences and completion routes. Ask them to explain the question in their own words and describe how they chose an answer. Then test the whole process with fictional records: routing, coding, denominators, charts and a draft decision brief. This is a design check, not a statistically representative survey.
The administrator approves available staff capacity before launch. During collection, review item nonresponse, completion difficulties and demand for assistance through proportionate authorized records. A high completion count alone is not success. After analysis, report whether the evidence changed an option, exposed an implementation constraint or left a material uncertainty unresolved. Return the findings and the decision to members on the promised schedule.
Keep channels and questionnaires consistent enough for interpretation, while providing appropriate assistance. The separate Digital Engagement for First Nations article addresses channel operations and support. Neither a digital tool nor this design framework establishes research ethics approval or legal compliance where those are required.
Arcus Advisory
Turn a proposed survey into a usable decision brief
Arcus can help leadership diagnose why existing survey evidence is inconclusive, compare survey and non-survey options, and decide what information is necessary. A defined engagement can produce a Question-to-Decision Specification, revised questionnaire, pilot plan, staff-capacity estimate and reporting structure.
Implementation support can clarify responsibilities, prepare the return of findings and evaluate whether the evidence supports the intended decision. Statistical sampling, specialized research ethics, privacy and technical assurance should be assigned to appropriately qualified advisers where required. The scope should state these dependencies clearly.
Related research and consulting
Arcus Indigenous consulting services · Community Engagement research collection · First Nations Knowledge Centre
Sources and research limitations
- Pew Research Center. When Online Survey Respondents Only ‘Select Some That Apply’. May 9, 2019; randomized U.S. experiment conducted in 2018. Topic- and population-specific, with an explicit assumption underlying the accuracy interpretation.
- FNIGC. National Report of the First Nations Regional Health Survey Phase 3, Volume One. 2018; survey development and methodology, printed pages 9–15. Documents a governed national health study; not validation of a local management questionnaire.
- Statistics Canada. Quality Guidelines: Questionnaire design. 2009 archived technical guidance. General design principles require local testing and adaptation.
- Statistics Canada. Questions to ask about surveys. Institutional guidance, reviewed October 20, 2026. Distinguishes sampling uncertainty from other errors.
- FNIGC. The First Nations Principles of OCAP®. Institutional guidance, reviewed October 20, 2026; Nation-specific application.
The specification, fictional questions, arithmetic bounds and burden model are original Arcus management analysis. They are not observed client results or a validated research instrument. OCAP® is a registered trademark of the First Nations Information Governance Centre (FNIGC).