Where automated decision tools appear and what is at stake

Automated decision tools are already part of human services. Some are simple rules engines that apply eligibility criteria to application data. Some are budget or resource allocation formulas that turn assessment answers into a dollar amount or a service range. Some are predictive models that produce a risk score, such as a child welfare screening score or a fraud flag. Some sort work queues, decide which cases get a second look or trigger an automatic notice. Leaders do not need to be data scientists to govern these tools, but they do need to know where they are, what they decide and who they affect.

These tools are adopted for real reasons. They can apply rules consistently, process large volumes quickly and draw on more information than one worker can hold in mind. Staff decisions have their own inconsistencies and biases. In a published case study of the Allegheny Family Screening Tool in Pennsylvania, which gives call screeners a risk score for child maltreatment referrals, the developers argued that a well-designed and audited tool can help screeners, while acknowledging that relying on public administrative data means some communities, such as people in poverty or particular racial and ethnic groups, will be disadvantaged because they appear in those systems more often.

Critics have examined the same tool closely. An ACLU analysis argued that specific design choices built values into the tool: combining individual scores into a household score, drawing on criminal legal and behavioral health system data that carry their own disparities, and using historical features a family can never change. It reported that under the protocols it examined, Black households were labeled high risk at a substantially higher rate than non-Black households, and it described thresholds that developers themselves characterized as set by trial and error. You do not have to settle this debate to learn from it. The lesson for leaders is that technical choices are policy choices, and they should be made and reviewed as such.

It is also important to be clear about how much weight a tool carries. A tool that informs a worker who has time, training and authority to decide differently is different from one whose output is accepted almost automatically. People can come to defer to a computer's recommendation even when other evidence points elsewhere, a pattern often called automation bias. If staff rarely depart from a score, the tool is effectively making the decision, whatever the policy says.

Fictional case: A DSD unit uses a scoring tool to rank requests for a limited grant program. Staff can override the ranking with a written reason. Beatriz, the unit supervisor, reviews a year of decisions and finds that staff changed the ranking in fewer than one in a hundred cases. When she asks why, staff say they assumed the score was more reliable than their own reading of the request and that writing a reason took time they did not have.

List every automated tool your unit uses that scores, sorts, flags or decides something about people or providers. For each, note what it decides, who owns it and roughly how often staff depart from its output. Share the list with your manager.

Privately consider a tool, template or score you rely on at work. How often do you depart from what it suggests, and what makes that easy or hard?