Interaction safety
Trust calibration
Trust calibration is the alignment between how much a person trusts an AI system and how much it deserves. It is well calibrated when reliance rises and falls with actual reliability; miscalibrated trust means either believing a system that is wrong or discarding one that is right.
Also called appropriate reliance, calibrated trust in AI
Relational systems make calibration harder in a specific way. Trust in them is built through accumulated interaction, most of which is not about accuracy at all: it is warmth, consistency, and attention. A person who has been well treated for months by a system has good reason to trust it socially, and that trust transfers to its factual claims without ever having been tested against them.
The transfer is not irrational. It is how trust works between people, where being reliably decent and being reliably correct genuinely do correlate. In AI systems they do not, and there is nothing in the ordinary experience of the relationship that reveals the gap.
Calibration signals are therefore part of a system’s manner rather than its interface. Whether it expresses uncertainty in a way that registers, whether it distinguishes what it knows from what it is guessing, and whether it can say plainly that it does not know all shape reliance more than a confidence score ever will.
Why it matters
A person who trusts a relational system the way they trust a friend will act on its claims the way they act on a friend’s, in domains where it has no reliability at all.
Where TAICU stands
TAICU treats calibration as a property of the relationship rather than of an individual answer, and holds that it can only be evaluated by watching how reliance develops over time.
Common questions
- Why do people over-trust AI systems they talk to often?
- Because trust accrues from the social qualities of the interaction — warmth, consistency, attention — which are unrelated to accuracy. Nothing in the ordinary experience of the relationship exposes the difference.
- Do confidence scores fix trust calibration?
- Rarely on their own. Numeric confidence is easy to ignore and hard to interpret. How a system expresses uncertainty in ordinary language, and whether it will plainly say it does not know, matters more.