The Overconfident-Wrong Critical Case

The most damaging failure mode in human-AI collaboration occurs when the AI is incorrect but expresses high confidence.

In this scenario, users lack signals that would prompt them to override the AI’s recommendation. The combination of wrong + confident leads to worse outcomes than wrong + uncertain, because users who receive appropriate uncertainty signals can sometimes catch the error.

Behzad et al. (2025) found that awareness-based feedback, communicating how the model performs on similar cases, slightly reduced compliance with overconfident-wrong predictions compared to raw confidence scores alone. The effect was modest but directionally important: contextual awareness provides a different type of signal than mathematical confidence.

This observation has direct design implications. Systems should not only communicate uncertainty but also communicate the basis for that uncertainty in terms users can evaluate.

Related: 05-atom—confidence-is-not-awareness, 05-atom—uniform-confidence-problem, 01-atom—calibrated-trust-vs-high-trust