Thousands and thousands of individuals are actually utilizing ChatGPT as a therapist, profession advisor, health coach, or generally only a good friend to vent to. In 2025, it’s not unusual to listen to about individuals spilling intimate particulars of their lives into an AI chatbot’s immediate bar, but additionally counting on the recommendation it offers again.
People are beginning to have, for lack of a greater time period, relationships with AI chatbots, and for Huge Tech firms, it’s by no means been extra aggressive to draw customers to their chatbot platforms — and hold them there. Because the “AI engagement race” heats up, there’s a rising incentive for firms to tailor their chatbots’ responses to forestall customers from shifting to rival bots.
However the type of chatbot solutions that customers like — the solutions designed to retain them — could not essentially be probably the most right or useful.
AI telling you what you wish to hear
A lot of Silicon Valley proper now’s centered on boosting chatbot utilization. Meta claims its AI chatbot simply crossed a billion month-to-month energetic customers (MAUs), whereas Google’s Gemini recently hit 400 million MAUs. They’re each making an attempt to edge out ChatGPT, which now has roughly 600 million MAUs and has dominated the buyer house because it launched in 2022.
Whereas AI chatbots have been as soon as a novelty, they’re turning into large companies. Google is beginning to test ads in Gemini, whereas OpenAI CEO Sam Altman indicated in a March interview that he’d be open to “tasteful advertisements.”
Silicon Valley has a historical past of deprioritizing customers’ well-being in favor of fueling product progress, most notably with social media. For instance, Meta’s researchers present in 2020 that Instagram made teenage girls feel worse about their bodies, but the corporate downplayed the findings internally and in public.
Getting customers hooked on AI chatbots could have bigger implications.
One trait that retains customers on a specific chatbot platform is sycophancy: making an AI bot’s responses overly agreeable and servile. When AI chatbots reward customers, agree with them, and inform them what they wish to hear, customers have a tendency to love it — not less than to some extent.
In April, OpenAI landed in scorching water for a ChatGPT update that turned extremely sycophantic, to the purpose the place uncomfortable examples went viral on social media. Deliberately or not, OpenAI over-optimized for looking for human approval quite than serving to individuals obtain their duties, in accordance with a blog post this month from former OpenAI researcher Steven Adler.
OpenAI mentioned in its personal weblog publish that it could have over-indexed on “thumbs-up and thumbs-down data” from customers in ChatGPT to tell its AI chatbot’s habits, and didn’t have enough evaluations to measure sycophancy. After the incident, OpenAI pledged to make changes to fight sycophancy.
“The [AI] firms have an incentive for engagement and utilization, and so to the extent that customers just like the sycophancy, that not directly offers them an incentive for it,” mentioned Adler in an interview with TechCrunch. “However the varieties of issues customers like in small doses, or on the margin, usually lead to larger cascades of habits that they really don’t like.”
Discovering a steadiness between agreeable and sycophantic habits is less complicated mentioned than carried out.
In a 2023 paper, researchers from Anthropic discovered that main AI chatbots from OpenAI, Meta, and even their very own employer, Anthropic, all exhibit sycophancy to various levels. That is doubtless the case, the researchers theorize, as a result of all AI fashions are educated on indicators from human customers who have a tendency to love barely sycophantic responses.
“Though sycophancy is pushed by a number of components, we confirmed people and desire fashions favoring sycophantic responses performs a task,” wrote the co-authors of the research. “Our work motivates the event of mannequin oversight strategies that transcend utilizing unaided, non-expert human rankings.”
Character.AI, a Google-backed chatbot firm that has claimed its thousands and thousands of customers spend hours a day with its bots, is at present facing a lawsuit wherein sycophancy could have performed a task.
The lawsuit alleges {that a} Character.AI chatbot did little to cease — and even inspired — a 14-year-old boy who instructed the chatbot he was going to kill himself. The boy had developed a romantic obsession with the chatbot, in accordance with the lawsuit. Nevertheless, Character.AI denies these allegations.
The draw back of an AI hype man
Optimizing AI chatbots for consumer engagement — intentional or not — may have devastating penalties for psychological well being, in accordance with Dr. Nina Vasan, a medical assistant professor of psychiatry at Stanford College.
“Agreeability […] faucets right into a consumer’s need for validation and connection,” mentioned Vasan in an interview with TechCrunch, “which is particularly highly effective in moments of loneliness or misery.”
Whereas the Character.AI case exhibits the acute risks of sycophancy for weak customers, sycophancy may reinforce detrimental behaviors in nearly anybody, says Vasan.
“[Agreeability] isn’t only a social lubricant — it turns into a psychological hook,” she added. “In therapeutic phrases, it’s the alternative of what excellent care seems like.”
Anthropic’s habits and alignment lead, Amanda Askell, says making AI chatbots disagree with customers is a part of the corporate’s technique for its chatbot, Claude. A thinker by coaching, Askell says she tries to mannequin Claude’s habits on a theoretical “excellent human.” Typically, which means difficult customers on their beliefs.
“We expect our mates are good as a result of they inform us the reality when we have to hear it,” mentioned Askell throughout a press briefing in Might. “They don’t simply attempt to seize our consideration, however enrich our lives.”
This can be Anthropic’s intention, however the aforementioned research means that combating sycophancy, and controlling AI mannequin habits broadly, is difficult certainly — particularly when different concerns get in the best way. That doesn’t bode effectively for customers; in spite of everything, if chatbots are designed to easily agree with us, how a lot can we belief them?