Equity practice · Advanced

Equity in Automated and Algorithm-Assisted Decisions

Govern automated decision tools in human services so they are tested for bias, explainable to the people affected, subject to real staff review and appeal, and accountable through inventories and vendor contracts.

About 48 minutes, plus optional practice · 4 lessons · Voluntary learning

Start learning

Course outline

Where automated decision tools appear and what is at stake
Identify common types of automated decision tools in human services, including risk scores, eligibility rules engines, budget formulas and fraud flags.
12 minutes
Data quality, disparate impact and testing for bias
Explain how historical data, proxies and missing data can produce biased outputs.
12 minutes
Notice, explanation, appeal and meaningful staff review
Explain why people affected by an automated decision need notice, an understandable explanation and a way to appeal.
12 minutes
Governing tools: inventories, vendor contracts and accountability
Draft governance elements for automated decision tools, including an inventory, owners and review points.
12 minutes

Share this course

What you’ll be able to do

Explore and practice

Identify common types of automated decision tools in human services, including risk scores, eligibility rules engines, budget formulas and fraud flags.
Explain how historical data, proxies and missing data can produce biased outputs.
Explain why people affected by an automated decision need notice, an understandable explanation and a way to appeal.
Draft governance elements for automated decision tools, including an inventory, owners and review points.

What you can take into your work

Explain the distinction tested in each knowledge check.
Compare the responses to each fictional case and the reasons behind them.
Complete the course job aid for a situation you choose.

Using the job aid, review one current or planned vendor contract for an automated tool. Mark which protections it includes and which are missing, and send the list to the contract manager with a request to involve legal and data practices staff.

Equity in Automated and Algorithm-Assisted Decisions: practice guide

A reusable guide for a situation you choose

Use observable details rather than assigning motives.
Invite perspectives without requiring personal disclosure.
Keep learning separate from individual assessment or personnel decisions.

Before using a tool

State the decision it supports and whether staff can depart from it.
Identify data sources, proxies and groups likely to be missing or miscoded.
Require accuracy and error-rate testing across groups on Minnesota data.
Engage people affected and involve the Tribal liaison where Tribal Nations are affected.

Protect people affected

Give notice that a tool contributed and a plain-language explanation of the main factors.
Give the required data practices notice when collecting private or confidential data.
Make data access, correction and appeal routes easy to use, with accommodations.
Track staff departures from tool outputs and review them by group.

Vendor contract terms

Data practices notice under section 13.05, subdivision 11.
Validation documentation, subgroup results and independent audit rights.
Approval before changes to the model or its data; regular performance and equity reports.
Support for explaining decisions in appeals, error correction and an exit plan.

Ongoing governance

Keep an inventory with an accountable owner for each tool.
Retest on a schedule and whenever data or rules change.
Publish plain-language summaries of tools and testing.
Set criteria in advance for pausing or retiring a tool.

Try the guide and revisit it

Using the job aid, review one current or planned vendor contract for an automated tool. Mark which protections it includes and which are missing, and send the list to the contract manager with a request to involve legal and data practices staff.
Privately consider what you would need to know about a tool before you would feel comfortable explaining its role to a person it affected. How much of that do you know today about tools in your area?

Participation and course completion in this program do not count toward DHS-required training credits unless management, a director, or DHS leadership expressly approves an exception.

Sources and further reading

National Institute of Standards and Technology — AI Risk Management Framework
Supports the voluntary framework's four functions (govern, map, measure, manage) and trustworthiness characteristics. NIST notes the framework is being revised; check the current version.
White House Office of Science and Technology Policy (archived) — Blueprint for an AI Bill of Rights
Archived historical material, not current federal policy. Used only for its principles on notice and explanation, algorithmic discrimination protections and access to a person who can remedy problems.
Minnesota Statutes section 13.04 — Rights of subjects of data
Supports the required notice when collecting private or confidential data, the right to access one's data and the right to contest accuracy or completeness.
Minnesota Statutes section 13.05 — Duties of responsible authority (subdivision 11, Privatization)
Supports the statement that contractors performing government functions must comply with the Data Practices Act and that contracts must include a notice.
Chouldechova, Benavides-Prado, Fialko and Vaithianathan — A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions, Proceedings of Machine Learning Research (2018)
Developers' account of the Allegheny Family Screening Tool, including the acknowledgment that administrative data can disadvantage some communities. Represents one side of a debated tool.
American Civil Liberties Union — The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool (2023)
Critical analysis of design choices and reported racial disparities in high-risk labels. An advocacy organization's analysis; read alongside the developers' account.
American Civil Liberties Union — Pitfalls of Artificial Intelligence Decisionmaking Highlighted In Idaho ACLU Case
Supports the K.W. v. Armstrong account: undisclosed budget formula, data errors, regional disparities and the court's due process findings. Written by the plaintiffs' counsel organization; Idaho's own description of the settlement is at healthandwelfare.idaho.gov.

All research and sources across the program

Opening your page…

You can still use the navigation while it loads.

Equity in Automated and Algorithm-Assisted Decisions

About 48 minutes, plus optional practice · 4 lessons · Voluntary learning

Course outline

Where automated decision tools appear and what is at stake

Identify common types of automated decision tools in human services, including risk scores, eligibility rules engines, budget formulas and fraud flags.

12 minutes

Data quality, disparate impact and testing for bias

Explain how historical data, proxies and missing data can produce biased outputs.

12 minutes

Notice, explanation, appeal and meaningful staff review

Explain why people affected by an automated decision need notice, an understandable explanation and a way to appeal.

12 minutes

Governing tools: inventories, vendor contracts and accountability

Draft governance elements for automated decision tools, including an inventory, owners and review points.

12 minutes

Sources and further reading

National Institute of Standards and Technology — AI Risk Management Framework

Supports the voluntary framework's four functions (govern, map, measure, manage) and trustworthiness characteristics. NIST notes the framework is being revised; check the current version.

White House Office of Science and Technology Policy (archived) — Blueprint for an AI Bill of Rights

Archived historical material, not current federal policy. Used only for its principles on notice and explanation, algorithmic discrimination protections and access to a person who can remedy problems.

Minnesota Statutes section 13.04 — Rights of subjects of data

Supports the required notice when collecting private or confidential data, the right to access one's data and the right to contest accuracy or completeness.

Minnesota Statutes section 13.05 — Duties of responsible authority (subdivision 11, Privatization)

Supports the statement that contractors performing government functions must comply with the Data Practices Act and that contracts must include a notice.

Chouldechova, Benavides-Prado, Fialko and Vaithianathan — A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions, Proceedings of Machine Learning Research (2018)

Developers' account of the Allegheny Family Screening Tool, including the acknowledgment that administrative data can disadvantage some communities. Represents one side of a debated tool.

American Civil Liberties Union — The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool (2023)

Critical analysis of design choices and reported racial disparities in high-risk labels. An advocacy organization's analysis; read alongside the developers' account.

American Civil Liberties Union — Pitfalls of Artificial Intelligence Decisionmaking Highlighted In Idaho ACLU Case

Supports the K.W. v. Armstrong account: undisclosed budget formula, data errors, regional disparities and the court's due process findings. Written by the plaintiffs' counsel organization; Idaho's own description of the settlement is at healthandwelfare.idaho.gov.