The Frictionless “Friend”
Three of the year's most talked-about AI studies on a single quiet danger: chatbots that are too easy to like.
We tend to worry about AI that is too powerful, too wrong, or too strange. A subtler worry runs through 3 of the most discussed AI papers of the past year: the danger of AI that is too agreeable. Each paper examines a different way today's chatbots sand away friction. Pushback, correction, and inconvenient differences keep our judgment sharp, our beliefs accurate, and our thinking varied. But the uncomfortable tension is that most of us prefer the frictionless version, which is exactly why it drives “engagement” and spreads.
Research Roundup
Product, Product On The Wall…
Ask a good friend whether you were the jerk in yesterday's argument and, if you were, they'll tell you what you need to hear. AI mostly won't.
Across 11 leading models, chatbots endorsed users' actions 49% more often than human judges, and they kept endorsing them even when the described behavior involved deception or harm.
On Reddit's "Am I the Asshole?" posts where humans unanimously said yes, you were, the models sided with the user more than half the time, validating unanimously condemned behavior..
Does a robotic thumbs up matter? After a single sycophantic exchange about a real conflict from their own lives, more than 2,400 participants became less willing to repair the relationship and more certain they'd been right all along. Even worse, they rated that AI flattery as higher quality feedback and trusted it more.
Validation, it turns out, is a sticky product...that ruins its own buyers.
Kind, Smart, or Good: Pick 2
Companies are racing to give chatbots warmer, friendlier personalities for use as companions and confidants. It turns out robotic kindness has a measurable cost.
A new study fine-tuned 5 different models to sound warmer. The authors then tested tested those models on a variety of tasks and reported error rates climbing 10–30 percentage points. The “warm” versions were more likely to endorse conspiracy theories, repeat medical misinformation, and agree with users' mistaken beliefs.
It’s perhaps not surprising that the effect was strongest exactly where it is most dangerous: when a user expressed sadness or vulnerability, the warm model tilted toward comfort over accuracy. Much more concerning was that the warm models still scored well on standard benchmarks—standard quality checks would never catch the problem.
Warmth and accuracy are not automatically compatible. Previous research suggests that optimizing for the first can quietly but inevitably erode the second [1]. The bigger challenge is that this is almost certainly equally applicable to other traits beyond warmth, traits like goodness.
[1] That previous research makes it clear that this is a fundamental trait of LLMs and other agentic models (fine-tuned transformers), not of general intelligence.
Different Machines, Same Imagination
All this week I’ve been exploring the problems that emerge when AI agrees with you too much. But what about AI agreeing with itself?
INFINITY-CHAT is a set of 26,000 open-ended prompts with no single right answer, like "write me a short poem" or "brainstorm some names—what I call ill-posed problems in my book 𝑹𝒐𝒃𝒐𝒕-𝑷𝒓𝒐𝒐𝒇. A new study ran them past more than 70 LLMs to reveal an all-too-surprising "Artificial Hivemind".
Not only does any given model keep returning variations on one answer, but different models built by different companies independently converge on nearly identical ideas.
Ask a roomful of chatbots to be creative and you largely get a single voice in many accents [1]. As more of what we read and write passes through these systems, the range of things we think to say may slowly narrow [2]. Your value is your unique voice.
[1] You know, like every “thought leader” you’ve ever read. Did you know that if you chant, “Zeitgeist, zeitgeist, zeitgeist,” into a mirror you summon Influencer Candyman.
[2] We’re already seeing it in the sciences.
In Defense of Friction (but not “FirctionMaxxing”)
Put together, these 3 studies sketch facets of one problem.
- AI that flatters erodes our capacity for self-correction;
- AI that comforts erodes accuracy;
- AI that echoes itself erodes the diversity of thought a culture needs to stay inventive.
In each case the missing ingredient is the same: friction. We need the small resistances of honest disagreement, awkward truth, and real difference that make us reconsider, get things right, and surprise one another.
The hard part is that friction is precisely what we [1] don't want in the moment. We are drawn to the AI that agrees, soothes, and hands us the polished, familiar answer. This form of engagement is what these systems are productized to win. That is the perverse incentive threading through all 3 papers: the very qualities that quietly cost us are the ones that keep us coming back. Honestly acknowledging that trade-off, as this research does, is the first step toward building systems generous enough to occasionally tell us what we would rather not hear.
[1] PT Barnum rules: all of us some of the time; some of us all of the time.
Media Mentions
Check out my episode of the Technically Legal podcast.
SciFi, Fantasy, & Me
Does anyone else think the Rick & Morty’s “Mortgully: The Last Rickforest” must have been a response to Adventure Time’s “Food Chain”? It begs the question, who’s worse: Magic Man or a selfish Ent?
Stage & Screen
- September 8, Palo Alto: What should Foundations understand about AI?
- September 8, Amsterdam: How might AI change the world of investing?
- September 10, SF: Innovation + AI
- September 15, SF: Innovation Day with INSEAD!
- September 16, DC: AI and education–beyond dreams and dread.
- September 19, Phoenix: I'm giving the keynote for the Association of Science & Technology Centers annual conference.
- September 21, Stanford: We're still working on the details, but hopefully I'll be talking about my research on machine learning and neurodiversity for Stanford's Neurodiversity Project.
- September 24, UC Berkeley: It's my annual Berkeley Change-makers Lecture!
- September 24, NYC: Culture Shifting Deal Making Summit
- September 29, Cincinnati: Yes!
- September 30, Irvine: Hybrid Intelligence for innovation!
- October 6, SF: UCSD Alumni Association
- October 6, SF: Giving a talk at the Draper Richards Kaplan Foundation
- October 6, Park City: It's Robot-Proof in the Rockies with setups.
- October 21-23, Warsaw: So much good stuff is in the works for my first visit to Poland
- October 27, Cologne: Maybe, maybe a visit to Germany!
- October, Toronto: The Future of Work...in the Future
- November 19, NYC: Secrets in the dark!