Humans Struggle with Probabilistic Reasoning

People often fail at probabilistic reasoning, particularly when it comes to base-rate neglect and the interpretation of confidence percentages.

Kahneman and Tversky documented the base-rate fallacy: people ignore prior probabilities when presented with specific case information. This has implications for how AI systems should communicate uncertainty.

Telling a user “87% confident” assumes they can correctly combine that probability with their prior assessment of the case. Research suggests many cannot. Alternative, non-probabilistic representations, like “the model does well on similar cases” or “the model struggles with cases like this,” may produce better trust calibration because they align with how people naturally reason about reference classes.

This creates a design tension: confidence scores are precise but hard for many users to interpret correctly; categorical statements (“high/low confidence”) lose precision but may be more actionable.

Related: 05-atom—confidence-is-not-awareness, 01-molecule—performance-feedback-spectrum, 01-atom—calibrated-trust-vs-high-trust