The Illusion of Personhood: Generative AI and Simulated Communication

AI takes different forms. Artificial general intelligence (AGI) refers to computational tools capable of general-purpose reasoning. AGI research is as ambitious and speculative as efforts to build quantum computers.

A second type of AI is predictive artificial intelligence. These systems are widely used to forecast outcomes or classify inputs based on patterns learned from data. Common examples include recommendation engines (e.g., Netflix, Amazon), credit scoring systems, diagnostic tools in healthcare, and predictive maintenance in industry. While predictive AI offers benefits, its deployment in critical domains, such as the criminal justice system, may reduce transparency in decision-making and limit the ability of affected individuals to challenge outcomes.

This essay focuses on a third category of AI: generative artificial intelligence. These systems are designed to produce novel outputs, often in response to a prompt, that mimic human-authored material. As the novel and much-hyped AI tools proliferate—writing emails, summarizing articles, scripting code, and offering consumer advice—they reshape daily life, much like cell phones, GPS, search engines, and social media before them.

Generative AI is surrounded by optimism because we envision these tools supporting human creativity and communication. Yet their ability to generate convincing output raises urgent questions: Who is responsible for what they produce? How do we sustain meaningful communication when synthetic media becomes indistinguishable from human expression?

To this list, we must add another concern—one that has received less attention: generative AI systems create a convincing illusion of understanding, agency, and personhood. This makes the technology engaging—even enjoyable—but the implications of the illusion of personhood, and the reasons for our credulity, warrant consideration.

The Illusion of Personhood

When I interact with a generative AI system—usually via my phone or computer—information is routed to a distant data center. Whether written or spoken, the signal that returns to me is a response, but not a reply. The message is not addressed to me by a person, nor offered in relationship. It is an echo of my communicative act, filtered through a statistical model trained on hard-won knowledge that our society has chosen to digitize.

These statistical models—called large language models—are neural networks with hundreds of billions of learned weights. They are “neural” only in the loosest sense: their architecture is inspired by signal propagation in biological brains. Their development is driven by ever-increasing processing power and sustained by the global computational infrastructure we call the internet.

When we query a system like ChatGPT, Gemini, or LLaMA, we initiate a human-machine interaction. Words are exchanged, but this is not communication in the full sense. Today’s machines have no intentions, no awareness, no presence. At the risk of being glib, you might say that AI systems imitate the form of dialogue without its essence: the other person. This is a basic point—akin to the suspension of disbelief in cinema—that harkens back to concerns voiced at the dawn of the cybernetic age.

As early as the 1960s, Joseph Weizenbaum, creator of one of the first natural language processing systems (ELIZA), was troubled by how readily users attributed understanding and emotional depth to a system that so obviously mirrored their own statements. He was especially disturbed when colleagues suggested that systems like ELIZA might one day replace psychotherapists (Weizenbaum, 1976), because such proposals betrayed a profound misunderstanding of both human communication and human dignity.

Communicative Norms

Ethics begins with the distinction between what can be done and what ought to be done. Are there moral—not merely technical—limits to what machines ought to do? To answer this question, we must distinguish calculation and judgment, information and communication, interaction and dialogue. The potential confusion and need for discernment are intellectual, but also cultural and spiritual. Allow me to offer three brief reflections—one from each of these domains—that help clarify what is at stake.

Intellectual: Thinkers like Jürgen Habermas remind us that authentic communication is not just an exchange of information, but a rational, dialogical process aimed at mutual understanding. Communicative agents bring not just words, but intentions—claims to truth, sincerity, and rightness—grounded in their personhood. His vision of the “ideal speech situation” depends on freedom, equality, and accountability between the speaker and the listener (Habermas, 1981).

Cultural: In the universe of Frank Herbert’s Dune (1965), the aftermath of the Butlerian Jihad leaves behind a civilizational ban: “Thou shalt not make a machine in the likeness of a human mind.” This prohibition from The Orange Catholic Bible ensures that machines, in mimicking human reason, will not displace the human spirit. The deliberate rejection of artificial intelligence creates space for the disciplined cultivation of human capacity—exemplified by the Mentats, masters of logic and calculation guided by intuition, and the Bene Gesserit Sisterhood, long-term strategists of human destiny.

Spiritual: Christian tradition teaches that communication is an aspect of communion. Words are not just information—they are creative acts of presence. The incarnation is a divine speech-act. “The Word became flesh and dwelt among us” (John 1:14, ESV). Words written millennia ago can be inspired. Human speech, at its best, is a sort of offering of oneself to another. To listen is to acknowledge a neighbor, to receive a stranger, to understand an adversary.

In summary, communication requires sincerity, freedom, and the presence of at least two people. The aim of speaking and listening is mutual understanding and shared good. We ought to reject those forms of language and computation that diminish human dignity and embrace those that uphold it. Dialogue is not just the sharing of words—it is the offering of selves.

Efficiency And Efficacy Of Communication

Concerns about the impact of generative AI on human communications may seem minor compared to the broader harms of digital culture—bullying, disinformation, surveillance, and the erosion of democracy. But it is on my mind.

It is commonplace to reflect on how personal and workplace correspondence has changed over the last two decades. Handwritten letters have given way to text messages. Occasional photocopied office memos have become a torrent of emails—too many to read, much less reflect upon.

In recent months, the interoffice communications arms race has escalated again. Now it is dominated by glossy, professional-grade digital pamphlets, replete with machine-generated images and slick graphic design. The gap continues to widen between the compositional polish of these missives and their actual significance.

The efficiency of writing with an AI copilot may produce a surplus of words and images that clog the channels of communication. Dialogue among colleagues can easily devolve into automated interoffice solicitation. Generative AI can enable fluid yet fruitless exchanges, because it is the struggle to find one’s voice that makes genuine disclosure possible. “The more the words, the less the meaning, and how does that profit anyone?” (Ecclesiastes 6:11, NIV).

Artificial Intelligence As Cultural Technology

We are natural-born cyborgs—not in the science-fiction sense of implants and exoskeletons, but in the more profound and ordinary sense that we continuously integrate cultural technologies into our way of being human (Clark 2004; Gopnik 2023). From the printing press to the chatbot, our minds and behaviors are shaped and extended by the tools we adopt, for better and for worse. There is a sense in which generative AI is simply the latest of these cultural technologies. Yet generative AI presents distinct opportunities and risks. Allow me to share three related concerns.

Concern 1 – The illusion of knowledge

Generative AI systems present themselves as convenient portals to the vast, digitized corpus of information on the internet. Thanks to their fluent natural language interfaces, we interact with them much as the ancients once consulted oracles for prophecy and advice. It feels as though we are speaking with a confident and attentive reference librarian who appears to command the collective knowledge of our society. Let’s call her Sybil.

Sibyl is trained to summarize, synthesize, and suggest. However, the relationship between her seemingly encyclopedic knowledge and the work of genuine subject matter experts remains opaque, and her motives are unknown.

Today’s generative AI will teleport you to the Library of Congress—if you are willing to be blindfolded and tied to a chair. Sibyl will answer any question you ask, but you cannot see which stacks are open or closed, or why she favors one section over another. Her voice is seductive. But have you noticed? She is most expert on the subjects you know the least about.

How Sybil was trained is proprietary information. She doesn’t work for you.

Concern 2 – Simulated intimacy

Weizenbaum was baffled by the way people attributed humanlike understanding and emotion to ELIZA. Why are we so easily fooled? Because we are made for relationship.

There is a video genre on YouTube known as ASMR, which stands for Autonomous Sensory Meridian Response. It refers to a pleasurable tingling sensation or chill that some people experience in response to whispered speech, mouth sounds, and soft tactile noises. Many use ASMR videos to relax, reduce anxiety, or fall asleep.

The sounds that trigger ASMR resemble the susurrus of close physical presence: a loved one’s breathing, the rustle of sheets, the pulse in your ears. Their comfort resides in the simulation of intimacy.

The appeal of the natural language interface of generative AI is similar. The emulation of human presence mimics a listening friend, a thoughtful expert, a humorous peer.

“Hey, Sybil.”
“Uh huh?”
“Knock knock.”
“Who’s there?”
“No one.”

Generative AI’s appealing front end may hijack your social brain and disarm your critical faculties. Sybil’s charm is not incidental—it is a design feature meant to elicit unearned trust.

Concern 3 – Disguised agency

The sophisticated pattern completion of generative AI does more than parrot prompts. Because Sybil was trained on human-generated data, her outputs reflect human patterns: tone, grammar, reasoning styles, and even emotional cues. But these traces of humanity are not signs of understanding or personality, but artifacts of recursive mirroring and remixing.

Log in to a generative AI system, and you enter a crowded carnival funhouse filled with angled glass. Is that a human-like figure gesturing from deep within the kaleidoscopic scene? The more convincing the illusion of personhood becomes, the more misdirected our perception of the true locus of human agency.

In the literature on artificial intelligence, considerable attention is given to “alignment”—the challenge of ensuring that future AGI systems will act in accordance with human values. But the alignment problem is already here. The holographic persona that AI systems project conceals the intentions of those who own the funhouse and adjust the mirrors.

As generative AI advances, the true relationships between the system, its owners, and users will become increasingly complex and obscure.

Conclusion

AI technologies will likely unfold over this century with a sense of inevitability. As Jacques Ellul observed long ago, the technologies of modern society develop as though they had lives of their own (Ellul, 1954).

Should we be concerned? Undoubtedly. The havoc already wrought by human-controlled machines—upon the planet and the least among us—is devastating. We are not prepared for further advances in AI. The socioeconomic consequences are unknowable, and regulation lags far behind the pace of innovation.

God forbid that our capacity to harm others becomes even more automated—and more carefully concealed by computational systems embedded in a digital cage of Kafkaesque bureaucracy and increasingly efficient, unrepresentative political structures.

But today’s question is not whether machines will someday think, or whether their goals might diverge from ours. The more immediate concern is this: natural language interfaces can distort our perception of agency. The illusion of personhood enabled by generative AI risks diverting our attention from where agency truly resides in human-machine interactions.

That agency exists at two poles—one on each side of the machine. You and I occupy one pole. The other belongs to the institutions that design and deploy these systems—most often corporations whose goals do not align with ours.

We are natural-born cyborgs. Most of us carry a sleek, non-invasive human-machine interface in our pocket and speak to a smart speaker in the kitchen. Are recent advances in generative AI merely the next cultural technology? Or do they herald the arrival of artificial general intelligence? I have my doubts about both claims.

Whatever the future holds, the cybernetic viewpoint remains a sound basis for human judgment. Are our machines reshaping us for better or for worse? Do they expand our capacities—or diminish us?

References

Clark, Andy (2004). Natural-Born Cyborgs: Minds, Technology, and the Future of Human Intelligence. Oxford University Press.

Ellul, Jacques (1964). The Technological Society (J. Wilkinson, Trans.). Vintage Books.

Gopnik, Alison (2023). What Four‑year‑olds Can Do and AI Can’t (Yet). YouTube. https://www.youtube.com/watch?v=53sQCXi5HPw

Habermas, Jürgen (1984). The Theory of Communicative Action: Volume 1 – Reason and the Rationalization of Society (T. McCarthy, Trans.). Beacon Press.

Herbert, Frank (1965). Dune. Chilton Books.

The Holy Bible, English Standard Version (ESV) and New International Version (NIV).

Weizenbaum, Joseph (1976). Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman.

Note

Habermas 1984 is a secondary reference discussed in “The Discourse Ethics of Jürgen Habermas” @PhiloofAlexandria (Daniel Bonevac) and @OverthinkPodcastPhilosophy, hosted by professors Dr. Ellie Anderson (Pomona College) and Dr. David Peña-Guzmán (San Francisco State University).

Abstract

Generative AI systems imitate human dialogue, creating the illusion of personhood that can mislead users and distort perceptions of agency. Generative AI may undermine communicative norms by substituting fluency for relationship. As AI systems become deeply embedded cultural technologies in everyday life, we must critically examine their design and deployment, not just for their efficiency, but also for their impact on human dignity, responsibility, and the ethics of dialogue.