Innovation Is a Sparse Market
In "winner-take-all" fields—such as scientific research, popular music, and high-level innovation—the market structure inherently discourages breakthrough discovery. Unlike traditional markets where diversity is rewarded, sparse markets concentrate nearly all attention and capital on a few dominant "products", leaving the vast majority in total obscurity. I call these Sparse Markets.
In these systems, individual incentives diverge from the market’s purported goal of uncovering the best possible ideas. Because the cost of failure is absolute, the mathematically rational strategy for any participant is herding. Actors avoid the high-risk periphery where transformative discoveries reside, opting instead to cluster around safe, proven successes. By producing incremental variations of dominant theories or trends, they maximize their probability of being "purchased" by the market’s narrow window of attention.
The result is a profound market failure: rational mediocrity. While competition is intended to drive progress, the structural sparseness of these markets forces participants to prioritize survival over original exploration. Consequently, the most valuable and revolutionary ideas go undiscovered because the risk of deviation is too high for any individual to bear, leaving society with a surplus of the familiar and a deficit of the truly innovative.
Research Roundup
Competitions Aren't Markets
Science is a fun industry. A small number of highly influential papers and scientists dominate a given field. It’s much less a “marketplace of ideas” than winner-take-all competition. Does this competition drive innovation?
The title of a new paper offers a clear answer: “Race to the Bottom: Competition and Quality in Science”. The authors’ hypothesis is that “scientists trade off…higher-quality research against the…risk of being preempted. Projects with the highest scientific potential are the most competitive [and so] these projects are also the most rushed and lowest quality.”
Analysis of “data from the Protein Data Bank (PDB)” reveals that protein structures with higher “potential generate more competition, are completed faster, and are lower quality.” Scientists rushing to win the competition produce lower value work.
How much lower value? They estimate “the costs associated with improving these low-quality structures are between $1.5 and $8.8 billion since the PDB’s founding”.
Fortunately, not all scientists show this effect. Those “who—by nature of their employment position—are less focused on publication and priority“ produce higher quality structures regardless of incentives.
I just happened to have given a talk on “The Paradox of Incentive Insensitivity” at TED Med many years ago. My research looked at salespeople, but I also found that those who were the least sensitive to incentive were also the best salespeople. We see the same effect in students.
If you believe in markets, then understand that winner-take-all systems are not traditional markets. They are what I call sparse markets and they behave very differently. This leads to sometimes dramatic misalignment between the individual incentive of innovators and what we want innovation to do for us as a society.
This is exactly the process driving Industry AI development today. If we don’t fix this, we all lose.
Media Mentions
Check out that TED Med talk at https://www.youtube.com/watch?v=o4ckVXyruks
And BUY MY BOOK!!! Can find my research on innovation and collective intelligence…and so much more: Robot-Proof: When Machines Have All The Answers, Build Better People
SciFi, Fantasy, & Me
I read every book from Terry Pratchett, starting right in the middle of the Discworld series with Small Gods. (This was quite fortunate because it is a largely standalone novel.) Near the end of his life, his books changed. They were still funny but they became darker but also more ernest. They also changed in another important manner, because Terry was suffering from progressive and profound dementia.
Rather than review any of his books (there are so many; go read some), I’m sharing a new research study using language models to study how his prolific writing changed as the disease progressed; “Detecting Dementia Using Lexical Analysis: Terry Pratchett’s Discworld Tells a More Personal Story”.
Using large scale language analysis the authors found a “significant decrease in lexical diversity (TTR) was observed for nouns and adjectives in later works.”
Perhaps to make up for this growing loss of just the right word, he began using many more words to try and convey its meaning (Someone like me might call this dictionary incoherence, and we see it even in the activity of individual neurons firing more but with less nuance.)
This hidden signal in his writing can be seen “approximately ten years before Pratchett’s formal diagnosis.”
Science Fiction becomes Science and pays Fantasy a visit. What will we do with these tools?
Stage & Screen
- March 4, Basel: I'll be giving a keynote at the Health.Tech Global Conference 2026: "Robot-Proof: How Human Agency Drives Hybrid Intelligence & Discovery"
- March 8, LA: I'll be at UCLA talking about AI and teen mental health at the Semel Institute for Neuroscience and Human Behavior.
- March 12, Santa Barbara: Economic development on the Central Coast.
- March 14, Online: The book launch! Robot-Proof: When Machines Have All The Answers, Build Better People is will finally be inflicted on the world.
- Boston, NYC, DC, & Everywhere Along the Acela line: We're putting together a book tour for you! Stay tuned...
- Late March/Early April, UK & EU: Book Tour!
- March 30, Amsterdam: What else: AI and human I--together is better!
- plus London, Zurich, Basel, Copenhagen, and many other cities in development.
- April 14, Seattle: Ill be keynoting at the AACSB Business School Conference.
- May 12, Online: I'll be reading from Robot-Proof for the The Library Speakers Consortium.
- June, Stockholm: The Smartest Thing on the Planet: Hybrid Collective Intelligence
- October, Toronto: The Future of Work...in the Future