11 minutes
Evidence that agrees with you Four ordinary shortcuts — looking for what fits, remembering the vivid case, anchoring on the first number, and never noticing who is missing from the data.
What you’ll be able to do Test one number, finding or story you rely on by naming who is missing from it and what evidence would change your conclusion. Compare the responses to this question and explain your choice: What should the evaluation team do before this number goes into the briefing? Document a next step for Evidence that agrees with you: Take one measure your unit reports. Write one sentence naming who is not in it, and attach that sentence to the number the next time it is presented. The data describes the people the system already worked for Public systems generate enormous amounts of information about the people inside them: applications processed, calls answered, cases opened, services authorized, surveys returned. That information is real, and it is also systematically incomplete in one direction. It records the people who got far enough in to be recorded. The person who read the notice, could not understand it, and put it in a drawer produces no record at all — or produces one that says “did not respond,” which is a sentence about them rather than about the notice.
This is the most consequential measurement problem in public administration, and it does not require anyone to be careless. It is built into where the data comes from. A process that excludes people effectively will report high satisfaction among the people it did not exclude, and the number will be accurate.
The remedy is not to distrust data. It is to state, every time, what the data covers and who is outside it — and to spend some of the unit's effort deliberately reaching the people the usual channels miss, with interpreting, plain language, paper, phone and in-person options, and with community organizations compensated for the work of reaching them.
Four shortcuts, and what each looks like in a state agency Looking for what fits The vivid case The first number Who is missing
You believe the new process is working, so the three success stories stick and the two complaints get explained away as unusual. In practice: a pilot report that quotes four positive comments and summarizes twenty critical ones as “mixed feedback.” The fix is to write down, before you look, what result would change your mind.
One difficult call two years ago becomes the reason for a verification step that costs every applicant twenty minutes. Rare, memorable events feel common because they are easy to recall. The fix is to ask how often this actually happens, and what the safeguard costs everyone else.
Someone offers a rough estimate early in a meeting. Every later discussion adjusts up or down from it, and nobody returns to whether it was sound. The fix is to say the basis and the limits of a number in the same breath as the number, and to ask for the basis when someone else offers one.
The measure covers people who completed the process. The people it failed are not in the denominator, so the failure appears as success. The fix is to name the population the measure actually covers every time it is reported, and to build one route that hears from people outside it.
Five questions to ask of any finding before it travels Who is counted here, and who could never have been counted?
What would I expect to see if the opposite were true, and did I look for it?
Is this measuring the thing, or something that correlates with being close to the agency?
How was this collected — which language, which channel, which device, which hours?
What sentence about the limits will travel with this number into the next meeting?
Our completion rate looked excellent. Then someone asked how many people started and stopped. We had never collected it. The number we were proud of only counted the people who made it through.
Composite DHS evaluation staff perspective, illustrative Private reflection, not recorded and not shared: think of a finding you have repeated in the last month. Who was outside it, and would you have noticed if no one had asked?
Carry this forward Administrative data describes the people the system already processed. Everyone who gave up, never heard about the program, could not reach the channel, or was screened out at the door is missing from it, and their absence looks like satisfaction.
The vivid case — the one difficult call everyone in the unit remembers — quietly becomes the design assumption for thousands of ordinary cases.
The first number on the table anchors the discussion even when everyone knows it is rough. Say the limits of a number in the same breath as the number, before it travels.
Naming what would change your mind, in advance, is the most practical protection against finding only what you expected to find.
A scenario about a satisfaction survey that reached only people the process already worked for, and a knowledge check on reading a finding honestly.
Take one measure your unit reports. Write one sentence naming who is not in it, and attach that sentence to the number the next time it is presented.
Browse and download only. Course notes are not typed or saved on this page.
Reset this lesson
Previous lesson Next lesson Carry this into practice How assumptions travel from a quick judgment into a routine, into a standard, and into the lives of people who never met you.
Return to the experience: What did you notice or try, whose perspective informed it, and what would you keep or adjust?
Participation and course completion in this program do not count toward DHS-required training credits unless management, a director, or DHS leadership expressly approves an exception.
