Trust as a Design Variable in Human-AI Systems

The Principle

Trust in AI systems isn’t binary, it’s a calibration problem. Effective HAIC requires designing for appropriate trust, not maximum trust.

Why This Matters

The failure modes of trust are asymmetric:

Over-trust creates automation bias. Users defer to AI even when their own judgment would be better. They stop verifying. They miss errors they would catch if working alone. In high-stakes domains (healthcare, finance), this gets people hurt.

Under-trust creates friction and waste. Users ignore valid AI recommendations. They duplicate work. They slow down processes the AI could handle. The AI investment fails to deliver value.

Neither extreme serves users or outcomes. The goal is calibrated trust, users who know when to rely on AI and when to verify, who develop accurate mental models of AI capabilities and limitations.

How to Design for Calibrated Trust

Make uncertainty visible. Don’t present all outputs with equal confidence. Signal when the AI is less certain, working outside its training distribution, or flagging for human review.

Provide provenance. Let users see where AI conclusions came from. What data? What reasoning? This helps them evaluate appropriateness for their specific context.

Create feedback loops. Help users learn where the AI succeeds and fails. Over time, they should develop intuitions about when to trust, but only if they get feedback on outcomes.

Support appropriate skepticism. Design for users to verify, not just accept. Make it easy to drill down, check sources, understand the reasoning chain.

Match transparency to stakes. High-stakes decisions need more explainability. Routine decisions can have lighter-weight trust signals.

When This Especially Matters

• Medical diagnosis where AI assists but clinicians decide
• Financial recommendations where users act on AI analysis
• Any domain where the AI handles most cases well but fails on edge cases
• Situations where user expertise varies widely

The Tension

There’s a real tradeoff: too much transparency can overwhelm users and slow them down. Too little leaves them unable to calibrate. The right level depends on the domain, the stakes, and the user’s expertise.

The instinct to inspire confidence in AI systems is misguided. The goal is appropriate confidence, high when the AI is reliable, low when it isn’t.

Related: 05-atom—uniform-confidence-problem, 04-atom—provenance-design