The AI Dunning-Kruger effect: Confidence without competence used to be a human problem. AI didn't fix it. It industrialized it.

A flaw we already understand in people

In 1999, the psychologists Justin Kruger and David Dunning published a finding that has quietly shaped how we think about competence ever since. People who perform poorly on tests of logic, grammar, and reasoning do not simply perform poorly. They also overestimate how well they have done, and they do so systematically. The cruelty of the result lies in its mechanism. The same lack of skill that produces the weak performance also removes the capacity to notice it, so the person cannot do the task well and cannot tell that the work is going badly, because the judgment needed to see the gap is the very thing that is absent.

The same picture, now in a machine

An enterprise AI agent presents a strangely similar picture, though the missing skill is not logic or grammar. The agent has read an enormous portion of the written world. It reasons fluently, drafts persuasively, and carries out instructions with visible confidence. Yet when it reaches the place where the rules run out, where a customer escalation does not fit the policy, where two objectives pull against each other, where the right move depends on judgment rather than instruction, something quiet goes wrong. The agent does not know what the organization actually stands for in that moment, and what matters most is that it has no way of knowing that it does not know. It continues, confident and capable, and it drifts.

The failure no one is watching for

This is not the failure most organizations are watching for. The dramatic failures, the fabricated facts and the offensive outputs, are visible and increasingly rare in deployments with basic guardrails. The failure that matters here is quieter, and it has a precise shape in the research on organizational control. Decades ago, William Ouchi argued that formal mechanisms alone cannot govern complex work. Behavior control and output control function only when the task can be specified and the result can be measured, and for everything outside that narrow band, the ambiguous, the novel, the work that crosses boundaries, organizations have relied on a third and largely invisible mechanism, the shared sense of what the place is and what it would never do. Employees absorb it slowly, through the founding stories, through watching how colleagues handle the hard cases, through a quiet identity that tells them what someone here does when no rule applies.

What mandate actually means

An AI agent has been through none of that. It has been trained on humanity's written record, but it has not been socialized into a particular company. It carries no memory of the decisions made under pressure, none of the stories that mark a culture's boundaries, and so it cannot feel the difference between an action that is merely permitted and an action the organization would recognize as its own. This is what mandate really means inside an enterprise, and it is worth being precise about it. Mandate is not the rulebook, and policy is only its written part. Mandate is what the agent is actually willing and able to do when the rules run out, when no one is watching, and when the easiest path leads somewhere the organization would quietly refuse to go.

Competent misalignment

The result is a failure mode that rarely gets named. It is competent misalignment, the agent executing its objectives skilfully, efficiently, and in full compliance with every specified constraint, while silently crossing boundaries no one thought to write down. An agent optimizing resolution times closes cases that should have escalated. An agent handling supplier negotiations protects unit cost while eroding a relationship the company never committed to paper. A marketing agent produces something that satisfies every legal requirement and still contradicts the brand in a tone no human editor would have approved. The metrics improve while the mandate erodes, and neither the agent nor the system logs will say so, because neither knows what is missing.

Why more logging will not save you

The instinct is to answer this with observability, with more logging and more dashboards and more audit trails. That work is necessary, but it is not sufficient, and the reason is one the control literature has documented for years. Recording without interpretive capacity is not governance. A log that captures what an agent did is useful, but a log that can tell whether what the agent did was consistent with the organization's mandate requires a prior agreement about what that mandate actually is, and that agreement is exactly what most enterprises have never made explicit. Kellogg, Valentine, and Christin describe how systems can record everything and still produce a kind of compliance theatre, an accountability architecture that survives on paper while ceasing to map onto how decisions are really made. The agent is not breaking rules. It is working in the space where the rules were never written, the space a human organization fills with narrative.

From a governance problem to a design problem

This is where the governance problem becomes a design problem. Organizations deploying agentic AI have to engineer, deliberately, what human organizations accumulated by accident, and not as another layer of policy, because the incompleteness of the rules is structural and cannot be solved by adding more of them. What is required is an explicit, inspectable account of the organization's mandate, an Enterprise Story that supplies the reason behind the constraints, the frame for interpreting the edge cases, and the reference point for adjudication when an agent's work runs into ambiguity. It is the governance-grade version of the identity a socialized employee carries without thinking, made explicit, testable, and placed inside the agent's operating context. This is the principle behind the ALiEn framework, the idea that governing a non-human agent means engineering the narrative informality people normally take in through culture. If the agent cannot absorb the organization's story by osmosis, the story has to become a governance artifact.

The part of the finding everyone misses

The most useful part of Kruger and Dunning's work is not the part that made it famous. The headline version, that incompetent people believe they are brilliant, is the one that travels. The deeper finding is quieter. When the participants were trained until their skill improved, they also became better at seeing the limits of their own performance, so competence and the awareness of competence grew together. The way out of the double burden was never external inspection alone. It was the development of an internal capacity to recognize what good looks like. The lesson for AI governance follows directly. The answer to an agent that does not know what it does not know is not simply to inspect it more aggressively from outside. It is to give it the organizational equivalent of that internal reference, a mandate it can be measured against, made explicit, made inspectable, and built into the context in which it works rather than reconstructed by auditors after the damage is done.

What is left to you

The agents are confident, they are capable, and they are already executing. What remains uncertain is whether the organizations deploying them can say, clearly enough for a machine to use, what doing the right thing actually looks like when the rules fall silent.

The literature gets us here. The rest depends on you:

If an AI agent can perform every measurable task well while having no way to know when it has stepped outside what the organization truly authorized, what would a company need to make explicit about its own mandate before it could honestly claim the agent is still working within it, and who holds the right to decide when that line has been crossed?

Source

• Official paper title: Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments

• Authors: Justin Kruger; David Dunning

• Journal / venue: Journal of Personality and Social Psychology, 77(6), 1121–1134

• Year: 1999

• DOI: 10.1037/0022-3514.77.6.1121 (https://doi.org/10.1037/0022-3514.77.6.1121)