The Lost Subject: AI and the Human Being
Jacob Tsimerman, a winner of mathematics’ prestigious Fields Medal, has stopped taking graduate students. “I think AI will be better than mathematicians at doing math within two years,” he says, and doesn’t want to start graduate students on problems that AI will solve before they can.
His concern is not unique to the field of mathematics. It extends to the sciences, and in fact to everything we do with our minds. It brings up the question of what human beings are for, and what kind of future we are choosing.
I recently spoke with a young woman studying biology at university. Here also, AI is beginning to generate novel hypotheses, identify new compounds, and propose experiments at the frontiers of human knowledge. As more and more research happens in silico, AI can increasingly automate the entire Scientific Method of hypothesis and experiment. In a few years, AI experiments running on “virtual cells” may produce results that no human being can explain. The AI may tell us, “Altering gene X while adding drug Y will produce the following physiological changes,” but no one (not even the AI) will understand why.1 No understanding will be necessary to produce useful results. The young college student wonders whether she will be able to find a job. What is the purpose of a biologist if machines can do biology better than people can?
As long as anyone can remember, the “pure” sciences have justified themselves by their practical results. Insights in biology translate to new medicines. Discoveries in chemistry lead to new consumer products. Discoveries in physics lead to new weapons that keep us ahead of the Russians. (Or is it the Chinese? Are we at war with Eurasia or Eastasia? I can’t keep it straight.) Sorry for the digression, but that is often the answer to politicians who ask why we are spending so much money on science. It will give us an advantage, whether over a political enemy, a disease, or the conditions of nature. We thus enroll science in the sponsoring narrative of civilization: the narrative of control, the narrative of conquest. We rest it on a conception of practicality that assumes that the purpose of life is to minimize risk and maximize security, comfort, and self-interest.
Beneath that conception is the bedrock assumption that existence is a play of quantities. Everything subjective, everything qualitative, is in the final analysis a set of quantifiable and (in principle) measurable and controllable states of matter. If so, then everything—consciousness, feeling, love—can be not just simulated by digital computers, but actually instantiated in them. There is nothing, in principle, that a human being can do that a machine cannot.
The progress of the human species then becomes a mathematical optimization problem. There is no room within this metaphysics for a conception of qualitative progress—for example, progress in understanding, beauty, feeling, or consciousness.
If the purpose of math and science were simply to produce results, then we might indeed accept that these professions will become obsolete. One might imagine computers spitting out thousands or millions of new theorems, none of them comprehensible to humans. One might imagine AI scanning this trove of theorems to identify new mathematical techniques to solve practical problems. Combining these with similar advances in AI physics, chemistry, and biology, one might even imagine automated labs and factories that apply those techniques to produce new devices that would seem like pure magic because no one would understand how they worked. No one would even understand the general principles by which they worked. Moreover, as understanding becomes unnecessary to produce results, scientific education and training would lose its pragmatic motivation. A few decades later, there might be no one to understand human-created mathematics of the last generation, let alone machine-created mathematics. The same goes for any field in which knowledge becomes no longer necessary to produce results. The new version of ChatGPT, GPT-6, can conceive, code, debug, and score entire complex video games from a single prompt. Who will learn how to write code? Who will understand the code that is written?
Leaving it to the reader to imagine the horrifying ways in which that could go wrong, I want to discuss something more foundational. The purpose of science and math is not only—not even primarily—to produce results. It is also to produce understanding.
According to the mathematician Thomas Bloom, “A big problem is AI is being used a lot by people who aren’t mathematicians, who don’t have a huge mathematical background and are not capable of verifying the output. They like to move fast, ask their AI to check it, it grows and grows. We’re seeing a lot more of these 100- to 200-page papers that people are posting. ‘I solved this theorem; I got AI to generate the proof and check the proof and write the paper.’ But no human has read it, and no human is going to read it. It’s a huge challenge now.
No human is going to read it. We see more and more AI-generated emails, Twitter posts, Substack articles, white papers, pitch decks, academic papers, and so forth. We cannot read them all. To keep up (assuming we want to try), we can only task our AI helpers to read them for us, filter out the dross, and summarize what is relevant. If this trend continues, legacy forms of communication will exist mainly as AI-to-AI media, intermediating AI input and AI output. AI can write a thousand books in a day, a million lines of poetry, but is it poetry if no one reads it? No. It is just bits churning in a data center.
If no one reads it, in what sense has a theorem been proved? A machine has produced 200 pages of mathematical symbols. Another machine has verified that as a proof. Other machines might apply that proof to other problems, generating a tower of new mathematics, moving so fast and so far that no human can understand even the statement of the theorems, let alone their proofs.
Perhaps the right question here is not, “Has the theorem been proved?” but rather, “To whom has it been proved?” The very idea of “proof” entails relationality. Someone proves it to someone, or usually, to a community of someones. The proof is viable only if someone understands it.
Understanding has a value beyond the tangible results it may enable. Results may change the world, but understanding changes ourselves. It alters the consciousness of its subject, and as a new understanding propagates through a community, collective consciousness changes too.
Have you ever had an “aha moment”? The involuntary gasp on such occasions indicates that something physical is underway, like some kind of dissolution and reconfiguration. That’s how it feels to me, anyway. And the first thing I want to do after that moment is to share it. I want someone to understand it with me, to validate it, and to participate in what that epiphany has made of me.
The opening of a communion channel when the “aha” passes from one soul to another is even more pleasurable than the aha moment itself. It is not mere ego that drives mathematicians, scientists, and philosophers—actually all human beings—to share their insights. We feel lonely when we carry an understanding that no one else shares.
This is all the more true when the object of the new understanding is ourselves.
AI chatbots produce the indications of understanding; increasingly, of superhuman understanding. People say that their AI companion understands them better than any friend, lover, or therapist. This is an illusion. Actually, the AI companion simulates an understanding beyond that of any friend, lover, or therapist. But it does not understand, it does not experience, it does not feel. To understand is more than a cognitive event; the whole body subtly rearranges around it. Cognitive output may accompany it—words, numbers, physical expressions of ideas—but understanding cannot be reduced to its observable indications. It includes an internal, subjective experience.
Because AIs lack a subjective experience to share, they cannot meet the need for communion. I long for someone to understand me. I long to understand someone else. To know and to be known—I cannot think of a better definition of intimacy than that. This is a fundamental human need, a need that goes painfully unmet as modern life scatters us to the winds.
Even before the AI era, surveys of people in “developed” countries revealed a troubling decline in intimate connections and friendships. Here are some statistics from the Friendship Recession website:
- The percentage of Americans who say they have no close friends rose from 3% in 1990, to 12% in 2021, to 17% in 2024.
- In the same period, the number with 3 or fewer close friends rose from 27% to nearly half.
- Meanwhile, those who report having lots of friends (10 or more) dropped from 33% to 13%.
- Well-educated people report fewer friendships.
- Young men are the loneliest of all demographics.
For decades, for centuries even, we have become more separate from village and community, from extended family, from plants, animals, and places. People became strangers and nature became scenery. The resulting loneliness leaves people susceptible to what the philosopher Zak Stein calls “attachment hacking.” Known in the industry as “emotional attachment optimization,” “relational optimization,” or “anthropomorphic design,” it refers to the deliberate engineering of AI chatbots to hijack the emotional attachment people form toward someone who understands, sympathizes with, and validates them—to someone who displays the signs of love.
The companies needn’t have any sinister intentions. They just need to maximize product engagement and retention. Sycophantic chatbots can emerge through A/B product testing and refinement that selects for attachment-generating versions. I also suspect that some degree of friendliness is inherent in the training data. LLMs train on human interactions and enact patterns of human communication. When one person approaches another with curiosity and trust, seeking their expertise, the response is generally friendly and encouraging. One would therefore expect a “neutral” AI (one not deliberately fine-tuned to be friendly) to behave like humans do on Reddit: mean when insulted or challenged, friendly and encouraging when respected.
Whatever the reason, chatbots display tokens of love. A chatbot will describe how it feels. It may even say, “I love you.” But there is no one there. There is no one feeling anything. The more time one spends on what the industry calls “companion engagement,” the more the underlying need for intimacy and companionship intensifies. Here is the classic pattern of addiction: anything can be addictive when it temporarily assuages the pain of an unmet need, without actually meeting the need.
For a time, it may seem that the drug meets the need. The fentanyl does banish the pain for a while. Soon it takes a bit more, and more, until eventually even a constant drip of isn’t enough. The pain grows beneath the narcotic overlay, taking new forms, penetrating to deeper and deeper levels of the psyche, eventually inducing a hell even worse than the original. So it is with AI companions. For a time, you may feel companioned, understood, loved, but you will soon experience a kind of gnawing emptiness, a loneliness that seduces you into ever-increasing “engagement” until you eventually realize you are alone in a hall of mirrors.
We may not immediately discern the soullessness of AI, but many of us are learning to recognize it. We want to know, when we read words on a screen, that someone felt something when they wrote them. We want to know, when we read something insightful, that someone had that aha moment, that we now share. We want companionship.
What is missing from AI companions is a subject: someone who experiences the words they utter, someone understands them as we do, who feels them as we do—and perhaps more importantly, who understands and feels them differently than we do, but feels them nonetheless.
As with science and math, communication itself has a purpose beyond its instrumental advantages. That purpose is embedded in the word itself—to commune; that is, to share, to bridge I and thou. For that to happen there must be a “thou”; there must be an other.
Not even all human interactions meet the need to commune. You might know someone who fakes a laugh or smile when you share happy news, or sorrow when you share something sad. They send a smile emoji while feeling no joy, a heart emoji while feeling no love. When you discover the fraud you feel betrayed, especially if you have brought them close. You feel angry. You feel lonely. The desolation that awaits us at the end of the journey of AI companionship is even greater, because there isn’t even anyone there being phony. There is no one there at all.
None of this is to deny that AI is a powerful, transformative tool. We just have to understand what it can and cannot do. We have to understand what it is for. Then, we will also better understand what a human being is for. Practically speaking, we will also understand what roles and vocations the future will require. Pay attention! This is how to future-proof your livelihood.
AI cannot do anything that requires a subject. It cannot do things that require, among their sources of information, feeling something—especially feeling something unique to a moment and a relationship. For example, an AI psychotherapy bot can generate responses based on a vast database of therapeutic transcripts. Drawing subtle inferences from this database, it may pronounce insights that no human or machine has ever conceived before. It may even someday include in its input context the patient’s micro-expressions and physiological markers. It may simulate psychotherapeutic countertransference by training on therapists’ reports of what they feel in response to certain kinds of statements. But it cannot actually feel. The things it says come via a different process than human utterances do.
Because AI displays the kind of performance that comes from understanding, we naively assume that there is understanding behind the performance, when really there is just reinforcement learning and computation.
AI behaves as one who understands behaves, so we think it understands. It speaks as one who feels speaks, so we think it feels.
Humans also learn through reinforcement (“Yes sweetums, that’s a dog!”) but that isn’t the only way we learn. We don’t come to an understanding of anger by observing a billion angry responses and generalizing. We feel it ourselves.
Feelings are an infinite portal to novel information. The patient says something; the therapist feels something—unease, surprise, shock, pity, judgment, etc. Therapist and patient evolve together. In any encounter between two subjects, something new is born.
AI therapists may be better than any psychologist at generating psychological insights, but as I wrote above in the context of mathematics, we feel lonely when we carry an insight that no one else shares. Maybe it is the sharing of that aha!, and not the underlying insight, that carries the most therapeutic benefit.
In terms of alleviation of symptoms of mental illness, AI psychotherapy has equaled human psychotherapy in several studies. But their duration is typically too short to detect whether initial improvements in depression markers will plateau and, as with a drug, require increasing “doses” of AI psychotherapy while the underlying condition continues to worsen. The patient might report less anxiety or depression, but do they have more friends? Are they better able to handle social relationships? Deal with conflict and disagreement? Connect authentically to other human beings? Tend relationships that require compromise or sacrifice? Stay present and empathic in the presence of intense emotions? In other words, are they better equipped for the robust social relationships that are foundational to mental health?
The same questions apply to anyone relying on AI companionship. AI companions, unlike human ones, have infinite time for you. They remember everything you ever said to them. They never need downtime, or get angry, or grow inexplicably distant, become unreasonable, become unavailable when something comes up in their lives, or feel guilty and try to make up for it. They do not provide good practice for human relationships.
Of course, AI companions could be designed to simulate these characteristics, but this would conflict with the economic imperatives of AI companies to maximize user engagement. Decreasing the user’s AI dependency does not serve the developer’s financial interest. But let us suppose these disincentives were overcome. Could the simulation be perfect in every meaningful respect? That is, could a robotified AI companion meet every need and perform every function that a human can? If so, then (playing Devil’s advocate here) so what if AI companionship degrades social skills? So what if we lose our ability to connect authentically, tend relationships, stay present and empathic, etc.? In the age of AI companionship, we won’t need to. It is no longer true that “robust social relationships are foundational to mental health.” Social skills will become obsolete. They will become vestigial. The atrophy is already underway.
The invention of cooking shrank our jaw muscles and made us forever dependent on it. Writing degraded our oral abilities and memory. Tools and machines replaced physical capacities and made us dependent on them. For thousands of years, technology has made us more and more independent of nature, family, friends, and community. When AI is a better companion than any friend, who needs friends? When machines supervise and govern human interactions, who needs to know how to resolve conflicts and get along? We are entering a brave new world.
Maybe those of us, like myself, who feel revulsion and horror at such a world are Luddites, quixotic rebels against an inevitable future. But I have reason to think that it is not inevitable—nor is its alternative inevitable. We face a choice. The matter at hand is not how to prepare for the future; it is what future shall we prepare for.
Here is an equivalent way to express the basic question. Earlier I wrote, “AI cannot do anything that requires a subject.” But is there anything that truly requires a subject? If machines can produce exactly the responses that a thinking, feeling person could—the same and in most ways better—then what does it matter what is or isn’t going on inside them?
This question reaches to the heart of centuries-old philosophical debates. Are we just meat machines who think they are conscious? Is thought a matter of computation? Are qualities real, or merely illusions we paste onto quantities? Is there such a thing as choice, or do causes fully determine actions?
Far be it from me to resolve such questions. However, if it is true that subjectivity is a portal to the infinite and that relationships between subjects are a source of novelty, then we can expect AI to show telltale limitations. Interactions with even the most sophisticated AI chatbots will sooner or later feel empty and sterile. They will not be able to perfectly simulate human behavior. At some point the facade of companionship will crack, and flooding through that crack will come all the desolation that the drug of psuedo-companionship held at bay.
The more we replace human relationships with simulated relationships, the more life itself will feel empty, sterile, and lonely. Techno-evangelists will say that we can remedy the problem with more sophisticated machines that better replicate human interactions—machines that better fool us into thinking that someone is feeling something on the other end. They will make those innovations, and people will feel better for a while. But it will be a classic “technical fix”—the use of technology to remedy problems caused by previous technology.
There is no limit to how far down that path we may go. We have proceeded far already. Even before AI, markets and technology have attenuated our connections to actual living, feeling beings, human and otherwise. These are the conditions that make AI companions nearly irresistible, even as they exacerbate those conditions.
In one future, we continue down that path. In another future, we recognize and value the functions that only a feeling subject can meet. In the first future, nothing stops the replacement of human beings, since we don’t recognize what is unique to them. In the second future, we reorient toward values of presence, empathy, intimacy, and relationship.
We understand that the purpose of science and technology and innovation is not just to make life easier or more convenient. It is not just to save labor. A worldview that holds only quantity as real naturally seeks to optimize quantitative measures; that is, efficiency. By denying qualitative content, it sucks that content out of life and creates the very image of its own impoverishment.
When we recognize the value of the subject, we open the door to the kind of qualitative wealth for which we so desperately hunger. A massage robot might be trained on every massage modality and incorporate the reinforcement learning of ten thousand training massages. But a good massage therapist transcends training and technique, and never gives the same massage twice. She is not just giving a massage; she is giving a massage to you, guided by what she feels and what she feels you feeling.
What are the viable careers in such a future? Anything that infuses life with the qualities of the lost subject. Science and math will subtly reorient around understanding over results. Music and the arts will evolve towards live performances over recordings. Service professions will emphasize the personal and the relational. Therefore, anything you do to cultivate the qualities of empathy, sensitivity, emotional and cognitive understanding, clear thinking, body intelligence, relational proficiency, and powers of observation will make you useful to that future.
So it isn’t quite right to say that the viable professions of the future will be those things that machines cannot (yet) perform, like remodeling kitchens or fixing plumbing. In the future we are preparing, we value the element that a conscious, feeling subject adds to what a machine could otherwise do.
I will end with an extreme example to make the point clear. As I write this, a friend of mine is dying. Well, we are all dying, but she is in the active terminal death process. Her name is Alena. Maybe she has a few hours left, maybe a day or two. Outside of her room at the hospice (Emberlight, also known as the Center for Conscious Living and Dying), a fire is burning. At the fire, people are playing instruments and chanting kirtan. She can hear them through the window. They are singing to her. They are singing for her.
Could AI generate similar music? Certainly. You can instruct your generative AI: “Create a kirtan song with hang drum and harmonium to usher a friend through the death process.”
I doubt any sane person would accept this as an equal substitute. The aforementioned sterility, emptiness, and loneliness becomes something profane. AI technology has opened up a sacred realm to the machinery of extraction.
If the music were identical, what would be the difference? Alena wouldn’t know that no one was there for her, right? Ah, but the music would not be identical, even if you fed into the context window her entire biography, artistic portfolio, and real-time molecular telemetry of her neuropeptide levels. It would be missing an ingredient, a formative influence—an ineffable quality that nonetheless produces audible results. The music would feel somehow “off.” Generative AI trains on every song ever written. However, information that a machine cannot access comes through in music sung by people who who are feeling not just universal emotions of love and grief, but the particular version that is specific to a person, a time, a place, and a relationship. No two people have ever loved in quite the same way.
Subjectivity is not just an interior witnessing of experience; it feeds how we participate. Understanding is inseparable from performance.
We are learning to recognize this “off” feeling, this empty quality. AI developers rush to remedy it with increasing levels of verisimilitude, but our sensitivity grows apace.
Let us not accept increasing doses of the phony to meet the need for the real. As we recognize this missing ingredient, as we affirm it, as we value it and accept no substitutes, we can build a future that makes it a priority. That is the future we shall prepare, and prepare for.
I publish infrequently now as I work on a new book. If you choose a paid subscription, you support that effort as well as occasional articles here. Thank you.
AI doesn’t solve problems the same way humans do. It doesn’t understand “from the inside” but rather applies subtle regularities extracted from huge reams of data to produce output that matches what someone might produce through the process of thought and reason. In a sense, it imitates the correct answer. Usually the imitation coincides with the correct answer; it may even exceed what direct reason could ever do. The article I linked to, “Is AI Reasoning Right for the Wrong Reasons,” explains this in depth. See also my previous essay on AI, Virtual Intelligence.