Leaving the Room
AI makes the generation of ideas cheap. What it does not make cheap (directly) is verifying the ideas against reality. That asymmetry is what this essay is about.
There are two basic types of “reality”:
- Physical world, as the objective reality (or what Searle called the “Brute Fact”).
- Human consensus (or what Searle called the “Institutional Fact”), is the collective recognition (social rules) that are accepted by a group of people12. It gives an object, action, or person a particular meaning or status.
Language, or the description of reality, is a projection of a human’s understanding of reality. The AI model can capture the structure of the language perfectly as long as there is enough data. However, printing letters that can follow the same structure of human language does not mean it can affect reality as expected. If you, like me, believe that some part of the great goal is to transform reality, then bear with me: the usefulness of whatever machine generates, depends on the cost and speed of the reality verification. Coding is an area where verification is the cheapest. For bounded programming tasks with executable tests, many claims can be checked automatically within seconds. However, even with the test verification only verifies what its author thought to assert, as a cache of anticipated failures based on the author’s understanding of the reality. The true verification comes with running the code in production to battle against both types of realities (for example, how the code performs on real hardware, and how users feel about the product). Mathematics shows a partial version of the same transition: once a statement and proof have been formalized, a proof assistant can check them mechanically, though the cost of formalization remains high. Beyond coding and Math, a legal opinion, a medical diagnosis, a business strategy, their verification all require the gradual expansion of reality. It takes a couple of finance quarters to verify the business strategy, and it takes a human body’s course of disease to verify the medical diagnosis. The destination of the verification is always the reality by itself, and the key separation between language and reality, determines that the verification cannot be fully done within language itself.
And among all the ways verification exits language, the hardest case, the one worth examining, is when the only available verifier is human first-hand experience.
In a famous 1982 thought experiment, philosopher Frank Jackson imagined Mary, a scientist who knows every physical fact about color (wavelengths, optics, the neuroscience of vision) but has lived her whole life in a black-and-white room. The question: when she finally steps outside and sees red, does she learn something new?
Philosophers have argued about the answer for forty years. But set the debate aside and notice what happens if we recast the story: the language model is Mary, a Mary whose room has no door. It has read every description of red ever written: it can recite the wavelengths, the associations, the metaphors. And it will never step outside, because its room is the language itself. We, meanwhile, are all people who have seen red. Framed this way, the machine’s fluency about color is neither new nor frightening. The interesting moment is when it gets one wrong: when it confidently misnames a color it has never seen. Who catches the error? Only someone who has been outside the room. This is quite the inversion the thought experiment hides, that the value of first-hand experience is not merely that it produces knowledge but that in some domains it supplies the final standard against which claims are checked. The experience is the verifier of last resort. Most physical reality verification is, in fact, being instrumented. Blood panels check diagnoses, strain gauges check bridges. So the claim is not that human experience monopolizes verification. The claim is narrower and harder: there is a class of domains where the standard of correctness is human experience: whether the dish is too salty, whether the pain has eased, whether the prose moves you. The problem is not that instruments lack precision; it is that they have nowhere to stand. You could build a delight-detector, but you could only calibrate it against human reports. If the answer bottoms out in a human report, then the human has not been removed from the loop, only moved to its end. When the instrument verifies that something should be classified as red, it is executing a cached human calibration.
The more interesting case is the areas where verification appears to happen entirely within language. It becomes an even harder problem especially for human consensus. The verification loop for a lot of human consensus is intrinsically what Merton called, a “self-fulfilling prophecy”. A bank is solvent as long as enough people believe it is; the act of verifying ‘this bank is failing’, by withdrawing, is what makes it fail. On March 9, 2023, Silicon Valley Bank faced approximately $42 billion withdrawal requests in a single day, with another $100 billion pending for the next day, a bank run that moved at the speed of group chats3. This is where AI gets really dangerous. Physical reality will punish the wrong language. If the model provides a wrong calculation to a bridge design, the bridge will break, and the physical reality would verify what is generated. Human consensus, however, has no immunity against the language machine because its verification is the belief. In reflexive social domains, a widely believed description can change the object until it becomes true. This critical danger existed before the introduction of AI, however, the cost of the generation of the language that can form human consensus, was very high. It used to take printing presses, parties, churches, newspapers, and decades, to form a new consensus. This high cost was a rough proof-of-work: the consensus that survived had survived collision with reality itself. The open question, how a society re-prices its consensus verification when language is free, deserves its own essay.
Whatever machine generates is only as useful as the verification behind it. It depends on the cost and speed of the reality verification. If that is the bottleneck, then the human work of the coming years is not to compete with machines at generating ideas (ideas, like coastlines, can be subdivided endlessly), it is to drive down the cost of verifying them. There are three ways to do it.
The first is to design for verifiability. Coding is not cheap to verify by accident; decades of engineering (type systems, unit tests, continuous integration, etc.) deliberately compressed the cost of checking a program from days to seconds. A lean, well-bounded abstraction is exactly this: an artifact whose claims are few and sharp enough for reality to check quickly. Other domains have barely begun this work. Build the correct abstraction (which assumes simplification), align others on it (an abstraction no one shares verifies nothing, and also see the next paragraph), and iterate fast against reality.
The second is to tend to the consensus. Consensus behaves like a cache of verification: you spend money without re-verifying what money is, sign contracts without re-deriving contract law, because society verified these once and cached the result. A language machine that pollutes consensus is poisoning that cache, and a society whose cache fails falls back to every person verifying everything alone, which no one can afford. The cache does not maintain itself; it is rebuilt, continuously, by people through social processes, and that work cannot be delegated to the very machines it guards against.
The third is to grow verifiers. If experience is the verifier of last resort4, then the point of education shifts from transferring knowledge alone (machines transfer knowledge for almost free now), towards giving people with enough first-hand experience with reality that they can check what machines say. Teach people to leave the room.
Cheap idea generation was never the goal. It was the bottleneck, and the bottleneck has moved. Verification is ours to build.
Because there are different groups of people, there could be different consensus realities in the world. ↩
Even though they rely on human recognition, it does not mean an individual can arbitrarily determine such reality. ↩
To be more clear, in the SVB example, consensus alone does not create the underlying vulnerability, but it acts as an accelerant and trigger within an already vulnerable financial structure. ↩
To be precise about the boundary of this example: machines are not condemned to the room. I am borrowing the architecture of Jackson’s thought experiment, not its argument about consciousness. A camera gives a machine an empirical channel to the world; it does not by itself settle whether the machine experiences color. ↩