The Politics Hidden in Risk Scores
How Technical Choices Encode Value Judgments
A risk score appears objective: input data, apply model, output number. The number seems to emerge from mathematics, not from human choices.
This appearance is deceptive. Every risk score encodes political decisions - choices about what to measure, how to weight it, and who bears the consequences of errors.
Where Politics Enters
Feature selection: Which variables predict risk? Including zip code may improve accuracy while encoding historical segregation. Excluding it may reduce accuracy while promoting fairness. Both are choices with consequences.
Outcome definition: What counts as the outcome we’re predicting? “Recidivism” could mean re-arrest, reconviction, or reincarceration - each measured differently, encoding different assumptions about what matters.
Threshold setting: Where do we draw the line between “high risk” and “low risk”? The threshold determines who gets loans, jobs, bail. Moving it trades off false positives against false negatives - a value judgment, not a mathematical fact.
Error cost weighting: Are false positives and false negatives equally bad? A loan denial system that minimizes defaults may maximize unfair denials. Whose errors count more?
The Objectivity Illusion
Risk scores feel objective because they’re calculated consistently. Given the same inputs, you get the same output. This consistency is real - but it’s procedural objectivity, not value neutrality.
The inputs were chosen. The model architecture was chosen. The training data was chosen. The threshold was chosen. Each choice reflected someone’s judgment about what matters.
Procedural consistency doesn’t eliminate these judgments. It obscures them.
Why This Matters for AI
As AI systems make more consequential decisions, this dynamic intensifies:
Scale amplifies impact. A biased human decision-maker affects their cases. A biased model affects everyone it scores.
Opacity hides choices. Complex models make it harder to see which features matter and why. The politics becomes less visible, not less present.
Automation creates distance. When a person denies a loan, there’s a human to question. When a score denies a loan, the decision feels inevitable - just what the math said.
Making Politics Visible
Better practice isn’t eliminating value judgments - that’s impossible. It’s making them explicit:
Document feature choices and rationale. Why these variables? What was excluded and why?
Specify outcome definitions precisely. What exactly is being predicted? What assumptions does that definition encode?
Examine threshold decisions. Who set the threshold? What tradeoffs does it embody? Who bears the costs of different threshold choices?
Analyze error patterns. Do errors fall disproportionately on particular groups? Is that distribution acceptable?
Create accountability. Who is responsible for the choices encoded in this system? Not “the algorithm” - which humans?
The Governance Implication
Technical teams often frame risk scores as engineering problems. They’re also governance problems: decisions about values that should involve stakeholders beyond data scientists.
Effective governance asks: Who should decide what risks matter? Who should set thresholds? Who should be accountable for error patterns?
These questions don’t have technical answers. They have political answers that should be made deliberately, not hidden in technical choices.
What value judgments are encoded in the risk scores your organization uses? Who made those choices? Who should have?
Related: 04-atom—data-governance, 04-molecule—data-quality-contextual-relationships