11 minutes
Reading a difference without jumping to a cause A gap is a question, not an answer. Denominators, collapsed categories, uneven records, and the difference between explaining a pattern and hiding it.
What you’ll be able to do Examine a reported outcome difference well enough to say what it does and does not establish, naming at least three ways the reading could be wrong before naming any cause. Compare the responses to this question and explain your choice: What should the analyst do? Document a next step for Reading a difference without jumping to a cause: Take one number in a report you use. Write down the denominator it actually uses, one group it collapses, and one group of people who are not in it at all because they never reached the step being counted. A gap is a question, not an answer A reported difference tells you that something is happening. It does not tell you what. The distance between those two sentences is where most examinations of a disparity go wrong, and they go wrong in one of two directions: the number is treated as proof of a cause somebody already believed, or it is adjusted, qualified and explained until there is nothing left to act on.
Both moves feel rigorous. Only one of them is a reading. A reading names what the number covers, what it collapses, who is missing from it, and which of the possible explanations the division can actually check. It ends with a question sharp enough to answer, not with a verdict.
There is also a line this work does not cross. A pattern across a group is never evidence about an individual, and it is never evidence of anyone's motive. A difference in approval rates does not tell you that a particular assessor is biased, and it tells you nothing at all about the person sitting in front of you. Keeping that line visible is not a softening. It is what makes the examination possible, because it removes the accusation that makes people defend a number instead of reading it.
Six ways a reading goes wrong The denominator answers a different question Share of the state's population, share of people likely eligible, share of people who applied, and share of applicants approved are four separate measures. A group can be over-represented in one and under-represented in the next. Write the denominator down before interpreting anything.
The categories collapse the communities inside them Broad race and ethnicity categories hold communities with very different histories, languages, arrival stories and relationships with public systems. A single figure for a broad category can look flat while two communities inside it move in opposite directions. Where the numbers allow, look inside the category. Where they do not, say so in the report.
The numbers are too small to carry the weight Small counts move a great deal on one or two cases. That is a reason to look across several cycles, to be careful about publishing anything that could identify a person, and to ask more precisely. It is not a reason to stop asking.
The information itself is uneven Race and language are sometimes self-reported, sometimes recorded by staff observation, sometimes left blank. Disability type is often recorded to fit the program rather than the person. A difference can be produced by how a field was filled in rather than by anything that happened to anybody.
Adjusting removes the thing you were looking for Controlling for county, income, prior contact or referral source can be genuinely useful. It can also hide the pattern, because those are frequently results of the same history. If you adjust, report both figures and say plainly what the adjustment took out of view.
The reading arrives with its cause already attached Everyone in the room has a theory before the meeting starts. Write the competing explanations down first — including the ones that point at the division's own steps — and only then ask which of them can be checked with something you are able to obtain.
A measure shows the approved-within-timeframe share by recorded group, across three cycles. What does it carry? That the approved share differs between two recorded groups, consistently, across all three cycles.
Choose a category What the number shows What the number only suggests What the number cannot show That the difference is wider now than in the earliest cycle available.
Choose a category What the number shows What the number only suggests What the number cannot show That having fewer assessors in a region is the reason for the difference.
Choose a category What the number shows What the number only suggests What the number cannot show That the assessment step is worth examining before the outreach step.
Choose a category What the number shows What the number only suggests What the number cannot show That the requirement for a recently signed form is part of what produced it.
Choose a category What the number shows What the number only suggests What the number cannot show Whether any individual assessor treated any individual applicant unfairly.
Choose a category What the number shows What the number only suggests What the number cannot show What a particular applicant experienced at the counter.
Choose a category What the number shows What the number only suggests What the number cannot show What anyone involved intended.
Choose a category What the number shows What the number only suggests What the number cannot show How many people never applied because someone told them informally that they would not qualify.
Choose a category What the number shows What the number only suggests What the number cannot show Check matches Reset Five denominators, five different questions Out of everyone in Minnesota
Compares the service's population with the state's. Useful for asking whether a service reaches the communities it is meant to reach. Poor at explaining anything about how the service treats the people already in it.
Out of everyone likely eligible
Compares who is served with who could be served. Harder to estimate, and far more informative about reach, outreach and the front door.
Out of everyone who applied
Isolates what happened after the door. Differences here point at requirements, assessment and processing rather than at outreach.
Out of everyone approved
Asks what happened next: level of service, setting, waiting time, continuity. Differences here point at decisions made after eligibility was settled.
Out of everyone who answered
The quiet denominator behind every survey figure. Report it every time, and always say who could not answer at all.
Ask for the reading, not the verdict What you can change You control what you ask for when a difference appears: a reading with denominators, categories and missing people named, or a one-line answer about whether the number is a problem.
What to watch for Do not ask an analyst to make a difference go away, and do not ask them to prove it is somebody's fault. Both requests produce a worse answer than the question deserved.
Your next step The next time a difference reaches you, ask three things: what is the denominator, who is not in this number, and which of the possible explanations sits inside a step we control.
A private reflection, for you alone: when you last saw a difference in a report, what did you assume had caused it before you read any further? Nothing you write here is collected, scored, or attached to your name.
Carry this forward The denominator decides the story. Share of the state's population, share of people likely eligible, share of people who applied, and share of applicants approved are four different questions with four different answers.
Broad categories hide the communities inside them. A single figure can sit flat while two communities inside the same category move in opposite directions.
Small numbers are a reason to look more carefully, across more cycles, and with more care for privacy. They are not a reason to stop asking.
Adjusting for county, income or referral source can be useful and can also remove the pattern from view, because those are often results of the same history. Report both figures and say what the adjustment took out of sight.
A pattern across a group is never evidence about an individual, and it is never evidence of anyone's motive. Holding that line is what makes an honest reading possible, because it removes the accusation that makes a room defend a number instead of reading it.
A scenario about a request to adjust a difference away before it reaches leadership, an accordion on six ways a reading goes wrong, sorting practice separating what a number shows from what it only suggests, flashcards on denominators, and a knowledge check on adjustment.
Take one number in a report you use. Write down the denominator it actually uses, one group it collapses, and one group of people who are not in it at all because they never reached the step being counted.
Browse and download only. Course notes are not typed or saved on this page.
Reset this lesson
Previous lesson Next lesson Participation and course completion in this program do not count toward DHS-required training credits unless management, a director, or DHS leadership expressly approves an exception.
