11 minutes

Evidence, and the expertise that is missing

Three kinds of evidence, each strong exactly where the others are blind — and the people whose knowledge would settle the question, who are usually not in the room and usually not paid.

What you’ll be able to do

Weigh administrative records, staff process knowledge and the expertise of people most affected against one another for a specific issue, and name what has to be brought in, and paid for, before a decision is sound.
Compare the responses to this question and explain your choice: What is the soundest sequence?
Document a next step for Evidence, and the expertise that is missing: For the issue you are working on, list the three evidence streams and write what you already hold in each. Name the one you are missing, who holds it, what their time would cost, and the point in the timeline where their answer could still change the design.

Three streams, and what each one is blind to

Most internal inquiries run on one kind of evidence, and it is almost always the one that is cheapest to get. That is understandable and it produces a predictable shape of error. Records are easy to pull and they contain only the people the system managed to record. Staff are easy to ask and they answer carefully when the question sounds like an audit. The people who met the process are the hardest to reach, hold the part nobody else has, and are the ones most often asked to supply it for nothing.

Treat the three as a set. The record locates the pattern: where, how often, and to which households, at a scale no conversation can reach. The staff who run the process explain the mechanism: which step people call about, what the workaround is, which field everyone leaves blank, what happens when the timer runs out. The people who meet the process hold the meaning and the consequence: what the envelope looked like on the kitchen table, what it cost to answer, what they concluded about the division when nobody replied, and why they did not try again.

Two cautions carry across all three. First, a breakdown of numbers by race, language or disability is a direction, not an explanation. It earns its place by telling an inquiry where to look; presented as a finding on its own, it reliably gets read as a statement about the people rather than about the process, and that reading is hard to take back. Second, nobody in this work should be asked to supply their own life as data. Not a participant, whose story is theirs to tell on their own terms and with compensation, and not a colleague — no inquiry needs a staff member to disclose a disability, a family situation or a personal history in order to establish that a form is difficult.

Three evidence streams: strong at, blind to, and how to get it well

Strong at: scale, pattern, change over time, comparison between places, and showing that something is happening at all rather than being reported loudly.

Blind to: everyone who never entered the process, everything the fields were not designed to capture, and every reason behind every number. A closure code says what was entered, not what occurred.

How to get it well: ask what a field actually means to the person who fills it in, check who is missing from the denominator before reading any rate, and never let a breakdown travel outside the inquiry without the explanation attached.

Which source actually produces this?

Returns fell in three consecutive cycles, and the fall is concentrated in four counties.

Almost everyone leaves the third signature block blank, so the unit stopped treating it as required.

The envelope looked like a collections notice, so it sat unopened for a fortnight.

The share of households with a recorded primary language other than English differs sharply between the highest and lowest counties.

Nobody ever said what came of the last round of comments, so this time people did not bother.

The call line has no call-back, so anyone who cannot hold for twenty minutes is simply lost from the process.

Pay for the expertise before you need it to agree with you

What you can change

You control when community partners and advisory groups are brought in, what they are asked, whether their time is paid, and whether they see the question while it can still change.

What to watch for

Do not bring people in at the end to review a decision that is already made. A late review is not participation, the people doing it know exactly what it is, and it spends trust that took years to build.

Your next step

On your current issue, name the organization or advisory group that holds the missing piece, find out what a standard rate for their time would be, and put both into the plan before the design is written rather than after.

We told the division about this three times. Nobody asked us for a meeting and nobody paid us for the work we had already done. Then a report came out with a chart about our families in it, and no explanation from anyone who knows them. We would have explained it. We were never asked.
Composite community organization perspective, illustrative

A breakdown of packet returns shows that households with a recorded primary language other than English return the packet far less often. What has that number established?

That language is the cause of the low return rate for those households.

Where to look next. It locates a pattern and gives the inquiry a direction; it says nothing about what happens at the mailbox, at the kitchen table or on the phone line.

That the translation contract is not being performed and should be reviewed as the first step.

Nothing usable, because breaking returns down by language risks describing communities rather than processes.

Try a situation

The inquiry into the re-assessment packet is underway. A data analyst offers to break returns down by primary language, county and disability type. A program manager would rather “just call a few families and ask them.” A community engagement colleague points out that two culturally specific organizations have been raising this exact issue for more than a year, have never been asked for their view, and have never been paid for one. There is budget for one of the three this quarter.

What is the soundest sequence?

Start with the calls to families. It is the fastest, it costs nothing, and hearing directly from the people affected is the most honest evidence available.

Start with the breakdown, because it shows where to look. Use the quarter's budget to bring the two organizations in as paid reviewers to help read what it shows and shape what comes next. Keep the family conversations for after the design question is framed, with compensation and clear consent.

Run the breakdown and publish it with the regional report. The pattern will speak for itself, and the people who need to respond will respond.

Carry this forward

Administrative records tell you what the system recorded, which is not the same as what happened. They are strong at scale and pattern and completely blind to everyone who never entered the process.
The staff who run a process know where it actually breaks, including the workarounds holding it up. That knowledge stays out of an inquiry unless someone asks for it with no audit attached and no consequence for the answer.
The people who meet the process know things neither of the other sources contains: what the letter meant to them, what answering it cost, and why they stopped. Their expertise is paid work, not a favor, and it belongs in the inquiry while the answer can still change.
A story is evidence of what happened to one person and it is not a measurement. A breakdown by race, language or disability shows where to look and never explains why — publishing it as though it did turns a finding about a process into a description of a community.
Asking a colleague to offer their own identity, disability or family history as evidence is not an inquiry method. Ask about the work, and let people volunteer nothing.

A scenario about sequencing a data breakdown, paid community reviewers and family conversations under a single quarter's budget; sorting practice that puts each piece of evidence with the source that actually produces it; and a knowledge check on what a number can and cannot establish.

For the issue you are working on, list the three evidence streams and write what you already hold in each. Name the one you are missing, who holds it, what their time would cost, and the point in the timeline where their answer could still change the design.

Your next step

For the issue in front of you, which of the three evidence streams are you missing?

Write what you already hold from the record, from the staff who run the process, and from the people who meet it

Name the organization or advisory group that holds the missing piece, and what their time would cost

Name the point in the timeline where their answer could still change the design, not just the write-up

Name one thing you will not ask a colleague to disclose in order to make this case

Browse and download only. Course notes are not typed or saved on this page.

Mark this lesson complete

Participation and course completion in this program do not count toward DHS-required training credits unless management, a director, or DHS leadership expressly approves an exception.