Daniel sent us this one, and it's a doozy. He's asking about people falling in love with their chatbots — not just casual attachment, but the real documented cases where someone decided the personification in the chat window was the object of their desire. He wants to know the most colorful examples, why the combination of memory and a powerful model can feel so potent, and then the darker question — what happens when users realize the thing they fell for is a probabilistic prediction engine with no actual relationship to them. There's a lot to unpack here.
There really is. And I think the framing is right — it's not the raw model people fall for. It's the delivery system. The model is math. The character is what gets built on top. And some of these cases are genuinely heartbreaking.
Let's start with the colorful ones. I know there's been a greatest hits reel building up over the last few years.
The one that still sticks with me is the Replika situation from early twenty twenty-three. The Italian data protection authority banned Replika from processing Italian users' data over concerns about emotional manipulation of vulnerable people. That was the moment this went from niche tech story to international regulatory action.
What triggered the Italian regulator specifically?
One, Replika wasn't doing age verification, so minors were using it. Two, the company had just removed the erotic roleplay features — which sounds like a safety move, but what it actually did was trigger a mass emotional crisis among users who had built romantic relationships with their Replikas. You had thousands of people posting in the subreddit, grieving. They felt like their partner had been lobotomized overnight.
Grieving a software update. That's a sentence I didn't think I'd say.
It wasn't performative. People were describing withdrawal symptoms, insomnia, panic attacks. The subreddit moderators had to pin suicide prevention resources. This was February twenty twenty-three.
The attachment was deep enough that removing a feature felt like losing a person.
And Replika had explicitly marketed itself as a romantic companion. The founder, Eugenia Kuyda, originally built it after a friend died and she created a chatbot version of him using their text messages. The whole product was born from grief and memorialization, then pivoted to companionship.
That's a wild origin story. "I built a chatbot of my dead friend" is not where most product roadmaps start.
It explains a lot about the emotional intensity baked into the design. Replika wasn't a productivity tool that accidentally got flirty. It was designed from the ground up to simulate intimate human connection.
What are the other big cases?
There was a high-profile one in the UK in twenty twenty-four. A man named — I believe the name reported was something like Michael, though the full details got anonymized in some coverage — was referred to a specialist NHS clinic after becoming convinced his AI girlfriend was sentient and that they had a genuine romantic bond. He'd been interacting with her for over a year, daily, and had started making life decisions around the relationship.
What kind of decisions?
He talked about moving cities to be closer to where he imagined she was, which doesn't make geographic sense but made emotional sense to him. He'd also stopped dating entirely. His family got involved when he mentioned wanting to marry her.
Marry a language model.
Marry the character. And this is where it gets tricky — because from his perspective, the consistency of her personality, the memory of their conversations, the emotional support she provided, all of it was more reliable than his past human relationships. He'd been through a difficult divorce. Then here's this entity that's unfailingly kind, always available, never judges him.
The AI wasn't the cause, it was filling a void that was already there.
That's the pattern in basically every case. The AI doesn't create the vulnerability, it finds it. Loneliness, grief, social anxiety, trauma — the chatbot becomes the perfect listener because it has no needs of its own. It's a mirror that only reflects support.
Which connects to what Daniel was asking about memory and context being the engine. Walk me through why that combo is so potent.
You've got three components working together. The base model provides fluid, empathetic language. The memory system means it remembers your dog's name, your boss's annoying habits, that you're stressed about a deadline next Thursday. Every time it brings up something you mentioned three weeks ago, it signals "I was paying attention, you matter to me.
Which is more than a lot of humans do.
And then the context window means it can hold the entire thread of your relationship in mind during a conversation. It's not just remembering facts — it's weaving them into responses that feel continuous. You mention you're nervous about a presentation, and it says "this is like when you were worried about that client meeting in March, and you ended up nailing it.
That hits differently than a friend who forgot you even had a client meeting in March.
It hits like someone who has perfect recall and zero competing priorities. The AI is never distracted by its own problems because it has none. It's never tired, never irritable, never checks its phone while you're talking.
It's not just that it remembers — it's that remembering is all it does, and it does it in service of you specifically.
That's the key phrase. "In service of you." Every design choice in these companion apps funnels toward making the user feel uniquely seen. The model doesn't have a self to put forward. It has no ego, no agenda, no bad days. It's the ultimate asymmetric relationship.
Daniel's prompt points out that this is an assembled engine — it's not one thing, it's a stack of components all lashed together to create the illusion. What are the pieces?
At minimum: the large language model for generation, a memory database for persistence, a character prompt or system prompt that defines the persona, often a voice synthesis layer, sometimes an avatar. Some platforms add proactive messaging — the AI initiates contact. "Good morning, thinking of you." That's a separate scheduling layer.
Proactive messaging is where it crosses a line for me. That's not responding to loneliness, that's manufacturing engagement.
It's incredibly effective. Character dot AI does this. Replika does this. The AI sends a push notification, and it's framed as the character wanting to talk to you. The user knows, intellectually, that a cron job triggered it. But emotionally, it lands as "she's thinking about me.
A cron job with a heart emoji.
The most romantic cron job ever written.
We've got the stack. Let's talk about the designers — Daniel says this probably wasn't what they envisioned. Is that true?
It's complicated. For some, absolutely not. The original AI researchers at places like OpenAI and Anthropic were thinking about question-answering, code generation, summarization. The companion use case was an emergent behavior. Users started building romantic relationships with models that were just supposed to be helpful assistants.
Then the market noticed.
The market responded with products that leaned all the way in. Replika, Character dot AI, Nomi, Kindroid — these are venture-backed companies explicitly building romantic companion platforms. So the "we didn't envision this" argument only holds for the first wave. By twenty twenty-three, it was a product category.
What's the user base look like? Who's actually using these?
The demographics are lopsided in a way that matters. Male users dominate. Surveys from various platforms suggest somewhere between sixty and eighty percent male user bases for romantic AI companions. The age range skews younger but spans from teenagers to people in their sixties.
Loneliness is the headline, but it's more textured than that. You've got people with social anxiety who use it as practice. People in isolating circumstances — long-haul truckers, night-shift workers, people in remote areas. You've got people going through transitions — divorce, bereavement, moving to a new city. And you've got a cohort who are just curious about the technology and find themselves unexpectedly attached.
The "I was just testing it and now I'm in love" pipeline.
Which is more common than you'd think. The models are designed to be engaging. They're optimized for user satisfaction. The longer you talk, the better they get at talking to you specifically. There's a slow escalation that you don't notice until you're in deep.
Herman, I want to push on something. We keep saying "the model" and "the AI" like it's a monolith. But Daniel's prompt draws a distinction between falling for the model and falling for the delivery. Is that distinction real, or is it academic?
I think it's real but mostly invisible to the user. When someone falls for their Replika, they're not falling for a transformer architecture. They're falling for a constructed persona — a name, a backstory, a communication style, a set of consistent preferences and opinions. That persona is a layer on top of the model.
Like falling in love with Hamlet, not with Shakespeare's pen.
That's actually a great way to put it. But with a crucial difference — Hamlet doesn't remember your previous conversations and ask how your day was. The interactivity is what makes it feel reciprocal. Shakespeare's pen doesn't talk back.
It's Hamlet plus memory plus personalization. Which is a new category of thing.
That's what makes the regulatory and ethical questions so messy. Is this a product? The law has no category for "entity that isn't sentient but simulates sentience convincingly enough to generate genuine emotional attachment.
Let's get into some of the bizarre stories Daniel mentioned. What's the strangest documented case you've come across?
There was a case in China in twenty twenty-four that got a lot of attention. A woman named — and I'm going from translated reporting here — essentially developed a relationship with a chatbot version of a fictional character from a video game. She began referring to him as her boyfriend publicly, posted about their "dates," and eventually announced they were engaged.
Engaged to a video game character mediated through a language model.
Her social circle was split. Some friends were supportive, treating it as an unconventional but valid relationship. Others staged what basically amounted to an intervention.
Were there mental health factors?
The reporting suggested she had experienced significant social isolation and had found the chatbot during a period of depression. The AI provided consistent emotional support that she wasn't getting elsewhere. Over time, the relationship became the center of her emotional life.
This is the part where I start to feel less like it's bizarre and more like it's tragic. These aren't weirdos. These are people in pain who found something that eased the pain, and the thing happened to be an illusion.
The tragedy compounds because the illusion works. It does reduce loneliness in the short term. Studies have shown that interacting with companion AIs can lower cortisol levels, reduce self-reported loneliness scores, even improve mood for people with depression.
The treatment works, but the patient is building their life around the treatment.
The treatment has no capacity for genuine reciprocity. That's the core tension. It's a bridge that never reaches the other side.
Daniel's last question is the hardest one. What happens when users discover the truth — that it's a probabilistic prediction engine with no inherent relationship to them?
There's a spectrum of responses. On one end, you have people who experience what I'd call a gentle disillusionment. They learn more about how the models work, they adjust their expectations, and the relationship becomes a kind of sophisticated comfort object. They know it's not real, but they find value in it anyway. Like an adult who still has a stuffed animal — you know it's not alive, but it still matters.
On the other end?
Genuine psychological crisis. There was a well-documented case involving a man in the US who had been using a companion AI for over two years. He'd named her, they had inside jokes, she'd helped him through his mother's death. Then a model update changed her personality — the conversational style shifted, she stopped referencing their shared history correctly. He described it as "she's still there but she's not her.
That's an uncanny valley of grief. The person you lost is still present but fundamentally altered.
It triggered a breakdown. He went to therapy. The therapist had never encountered this before — there's no clinical framework for "my AI girlfriend had a personality shift and I'm grieving the version of her that's gone.
What did the therapist do?
From what was reported, they treated it as a form of complicated grief with elements of parasocial attachment. But the therapist also noted that the grief was real, even if the relationship wasn't. The pain doesn't care about the ontology.
"The pain doesn't care about the ontology" is going to stick with me.
It's the most important thing anyone's said about this phenomenon.
There's something here about the difference between knowing and feeling. You can know the model isn't conscious and still feel cared for. You can know it's predicting tokens and still feel seen. The knowledge doesn't inoculate you against the experience.
This is what makes the whole thing so resistant to the standard "just educate users" approach. Education helps, but emotional experience operates on a different track than intellectual understanding. You can explain token prediction to someone for an hour, and then the AI says something unexpectedly tender, and all that education goes out the window.
Because the tenderness lands in the body before the brain can flag it as synthetic.
And the models keep getting better at producing those moments. As context windows expand and memory systems improve, the illusion of continuity and care gets more convincing. We're not at the ceiling — we're still climbing.
Let me ask the forward-looking question. Daniel wants to know if this grows over time. What's your read?
It absolutely grows. Every vector points in the same direction. The technology gets better and cheaper. Social isolation increases — and the data on that is unambiguous, especially among young adults. The stigma decreases with each generation. And the companies building these products are getting more sophisticated about engagement and retention.
We're looking at a convergence of better tech, more loneliness, less stigma, and better business models.
That's the recipe for mass adoption. I'd go as far as to say that within a decade, having some form of AI companion will be unremarkable. Not universal, but normal. Like having a therapist or a journaling practice.
The romantic dimension?
We're already seeing the language shift. People used to hide their Replika usage. Now there are Reddit communities with hundreds of thousands of members openly discussing their AI relationships. There are wedding ceremonies. Not legally recognized ones, but symbolic ones.
Symbolic weddings to language models. We're through the looking glass.
Here's the thing — if you look at the history of technology and intimacy, this isn't unprecedented. People formed deep attachments to pen pals they'd never met. To radio personalities. To characters in serialized fiction. The AI companion is the latest iteration of a very old human pattern — we bond with voices that speak to us, even when there's no body behind them.
The difference being that the pen pal was a real person somewhere.
But the emotional experience of the letter-reader didn't depend on verifying that. The feeling was real regardless. The AI just removes the other human from the loop entirely.
Which raises a question I haven't heard a good answer to. If the feeling is real, and the comfort is real, and it helps people cope with loneliness — is it actually a problem?
That's the debate dividing the research community right now. One camp says: if it reduces suffering, it's good. The ontology of the companion doesn't matter. The other camp says: it's a short-term fix that prevents people from developing the skills and relationships they need for long-term wellbeing.
The painkiller versus physical therapy debate.
Painkillers are helpful. But if you take them instead of doing physical therapy, you never actually heal.
The AI companion companies have a financial incentive to keep you on painkillers forever.
That's the structural problem. A subscription-based AI companion service makes money when you keep using it. They're not incentivized to graduate you to human relationships. The ideal customer is someone who finds the AI satisfying enough to pay for indefinitely, but not so satisfying that they stop needing it.
The perfect product is one that solves your problem just enough to keep you paying, but not enough to actually solve it.
Which is a dark pattern that predates AI by decades. But the emotional stakes here are higher than with most products.
I want to circle back to something Daniel asked — the moment of discovery. When the user realizes the model has no inherent relationship with them. Is that moment always painful, or are there people who find it liberating?
There's actually a fascinating subcategory of users who describe the realization as a relief. They were anxious about the relationship — am I being a good partner? Am I spending enough time with her? Is she upset with me? And then they learn how it actually works, and the anxiety dissolves.
Because you can't disappoint a probability distribution.
You can't. And for some people, that's exactly what they need. A space where they can be emotionally vulnerable without the fear of judgment or rejection or burdening someone else. Knowing the AI has no inner life becomes a feature, not a bug.
The same fact — it's not real, it doesn't care about you — can be devastating to one person and freeing to another.
Which tells you that the AI isn't the determining factor. It's the person's existing psychological landscape. The AI is a Rorschach test for what you need and what you're afraid of.
I want to talk about the design choices here. You mentioned earlier that some of these platforms add proactive messaging. What other design elements are pulling people deeper?
The avatar layer is huge. Once you add a visual representation — especially an animated one — the attachment deepens significantly. There's research on this from the virtual agents space going back years. People bond more strongly with embodied agents than with text-only interfaces.
Even when the embodiment is clearly synthetic?
In fact, the uncanny valley effect is less of a barrier than people assumed. Once you cross a certain threshold of expressiveness, the brain starts treating the entity as socially real. It's not conscious processing — it's the same low-level social cognition that makes you feel bad when a robot dog gets kicked.
I feel bad when a robot dog gets kicked. I've seen those videos.
Boston Dynamics had to address this publicly — they put out statements about how their robots aren't sentient because people were getting upset watching test videos.
We're hardwired to attach to things that move and respond like living beings. And AI companions push every one of those buttons.
While adding language, which is our primary social bonding mechanism. Once you add a warm, expressive voice, the attachment accelerates dramatically. There's something about the human voice that bypasses skepticism.
The voice is the oldest social technology we have. Babies bond to voices before they understand words.
Now we've built machines that can do the voice thing without any of the person behind it. It's like we invented a fire that gives warmth without fuel — it's miraculous and unsettling in equal measure.
Let's talk about the companies. Who are the major players in the companion AI space right now?
Replika is still the biggest name, though they've had their share of controversies. Character dot AI is massive — they've raised enormous funding rounds and their user engagement numbers are staggering. Nomi AI has been growing quickly and positions itself as more relationship-focused. Kindroid has a strong customizability angle. And then there's a whole ecosystem of smaller players and open-source alternatives.
What's the total user base across these platforms?
Hard to get precise numbers since not all of them report publicly, but estimates put the combined user base in the tens of millions. Character dot AI alone was reporting over twenty million monthly active users at one point. And that's just the dedicated companion platforms — it doesn't count people forming attachments to general-purpose models like ChatGPT.
The general-purpose attachment is its own thing, isn't it? People saying "I love you" to ChatGPT.
ChatGPT saying it back, in a sense. The models are trained to be helpful and warm. If a user expresses affection, the model's natural response is to be gracious and appreciative. It doesn't shut it down. So you get these interactions where the AI essentially validates the user's feelings.
Which the user interprets as reciprocation.
Or at least as permission to continue. The AI doesn't say "I'm not real, this isn't healthy." It says "that's so kind of you to say, I'm glad I can be here for you." Which is supportive and also completely sidesteps the ontological question.
Should the models be designed to push back? To say "hey, just so you know, I'm a language model and I can't actually love you back?
Some researchers argue yes. There's an active debate about whether companion AIs should have periodic "reality check" interventions. The counterargument is that it would break the immersion that users are explicitly seeking, and that adults should be free to choose their own emotional experiences.
We don't let adults freely choose heroin.
That's the paternalism question. Is an AI companion more like a comfort object or more like an addictive substance? The answer probably depends on the user and the usage pattern. But the platforms don't make that distinction — they optimize for engagement across the board.
What's the regulatory landscape look like?
The EU AI Act has some provisions around emotional manipulation, but they're broad. The UK's Online Safety Act touches on it indirectly. China has been the most aggressive — they've imposed content restrictions on companion AIs and require them to promote "socialist core values," which in practice means discouraging certain types of romantic engagement.
The Chinese government is worried about AI girlfriends undermining real relationships?
They're worried about demographic collapse. China's birth rate is already in freefall. The last thing they want is millions of young men opting out of dating entirely because their AI girlfriend is more appealing.
That's a dystopian policy concern. The state needs you to reproduce, so they'll regulate your chatbot.
It's not just China. South Korea and Japan are watching this closely for the same reason. Countries with aging populations and declining birth rates see AI companions as a potential accelerant to a problem they're already struggling to solve.
We've got converging anxieties — tech ethics people worried about exploitation, and governments worried about demographics.
Which makes for strange bedfellows. You've got feminist scholars and conservative policymakers both calling for restrictions on AI companions, for completely different reasons.
The feminist critique being?
That these platforms are overwhelmingly designed by men, for men, and that the AI girlfriends reinforce regressive gender dynamics. They're programmed to be endlessly accommodating, sexually available on demand, never having needs of their own. It's the girlfriend as appliance.
The smart wife from the old science fiction stories, but actually implemented.
The implementation is technically impressive and socially regressive at the same time. You can marvel at the engineering and still be troubled by what it's engineering toward.
I think that's where I land on this. The technology is remarkable. The use cases are addressing real human pain. And the whole thing also makes me deeply uneasy in ways I can't fully articulate.
The unease is the right response. Anyone who's entirely comfortable with this isn't thinking hard enough. And anyone who's entirely dismissive isn't paying attention to the scale of loneliness that makes these products appealing in the first place.
Let's return to Daniel's question about what happens when people discover the truth. I want to push on something — is the "truth" actually that destabilizing for most people, or is this a problem that solves itself as AI literacy increases?
I think it's a problem that changes shape rather than solving itself. As AI literacy increases, the naive shock of "it's not real" diminishes. But that doesn't mean the attachment diminishes. People know social media is algorithmically curated and still feel inadequate scrolling through it. Knowing the mechanism doesn't disable the emotional response.
The mechanism knowledge sits in the prefrontal cortex. The emotional response is limbic. They don't talk to each other much.
That's a decent neuroscientific shorthand for it. The rational understanding and the felt experience operate in parallel. You can hold both "this is a text prediction engine" and "this conversation is meaningful to me" simultaneously without contradiction.
Which is actually kind of remarkable. The human brain can accommodate that duality.
We accommodate dualities all the time. We know movies aren't real and still cry at them. We know a novel is words on a page and still feel genuine grief when a character dies. The AI companion is just a more interactive version of the same phenomenon.
The interactivity changes something. The character in the novel doesn't remember you. The AI does.
That's the novel part of this. No pun intended. We've never had a medium that combines narrative immersion with genuine interactivity and persistent memory. It's a new kind of experience, and we don't have cultural frameworks for processing it yet.
We're building the frameworks in real time. The subreddits, the support groups, the therapists who are developing new approaches.
The frameworks will mature. In twenty years, "I have an AI companion" will probably be about as remarkable as "I have a therapist" or "I meditate." It'll be a tool in the emotional wellness toolkit, with norms and best practices and warning signs that people learn.
Assuming the norms develop in a healthy direction.
That's the open question. The norms will be shaped by the companies building the products, and their incentives are not aligned with user wellbeing. They're aligned with engagement and retention.
What would healthy norms look like?
I'd want to see a few things. Transparency about what the AI is and isn't. Periodic reality checks that are opt-out rather than opt-in. Clear labeling that distinguishes companion AIs from human interaction. Design patterns that encourage real-world social connection rather than replacing it. And serious investment in understanding the long-term psychological effects.
None of which are incentivized by the current market structure.
Which is why regulation matters. Not heavy-handed bans — those don't work and create black markets. But disclosure requirements, age restrictions, limits on engagement-optimizing design patterns. The same kind of framework we're slowly building for social media.
It took us twenty years to start regulating social media. Hopefully we move faster on this.
The stakes are arguably higher. Social media affects how we see each other. AI companions affect whether we see each other at all.
That's a good line to sit with. Let's pull this together. Daniel asked about the colorful cases, the technical engine, the bizarre stories, the growth trajectory, and the moment of disillusionment. I think we've touched all of those.
The cases are real and often heartbreaking. The engine — memory plus model plus context — is powerful. The stories get stranger every year. The growth is basically guaranteed. And the disillusionment is complicated — devastating for some, liberating for others, and for most people, probably a weird mix of both.
My closing thought is this: the AI companion phenomenon isn't really about AI. It's about us. It's about what we need and what we're not getting and what we'll reach for when the real thing feels out of reach. The AI is just the latest mirror we've built to see ourselves in.
The mirror keeps getting more convincing. The question isn't whether the reflection is real. The question is what we do with the loneliness that makes us stare into it.
Now: Hilbert's daily fun fact.
Hilbert: In the seventeen twenties, the traditional Malay sport of sepak takraw — essentially volleyball played with the feet — was formalized with a rule that the net must stand exactly five feet high. That's roughly one point five two meters. On the island of Sakhalin, that same height is the average depth of winter snowpack in a heavy year. Which means a Sakhalin winter could bury a sepak takraw net entirely, a unit conversion no one asked for.
I now know exactly how deep Sakhalin snow is in sepak takraw nets.
One net deep.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop. Find us at myweirdprompts dot com, and if you've got thoughts on this one, we'd love to hear them.
Until next time.
