Together with L Rudolf L, we’ve published an essay series on what we call the intelligence curse. Most content is brand new, and all previous writing has been heavily reworked.
Visit intelligence-curse.ai for the full series.
Below is the introduction and table of contents.
We will soon live in the intelligence age. What you do with that information will determine your place in history.
The imminent arrival of AGI has pushed many to try to seize the levers of power as quickly as possible, leaping towards projects that, if successful, would comprehensively automate all work. There is a trillion-dollar arms race to see who can achieve such a capability first, with trillions more in gains to be won.
Yes, that means you’ll lose your job. But it goes beyond that: this will remove the need for regular people in our economy. Powerful actors—like states and companies—will no longer have an incentive to care about regular people. We call this the intelligence curse.
If we do nothing, the intelligence curse will work like this:
Powerful AI will push automation through existing organizations, starting from the bottom and moving to the top.
AI will obsolete even outlier human talent. Social mobility will stop, ending the social dynamism and progress that it drives.
Non-human factors of production, like capital, resources, and control over AI, will become overwhelmingly more important than humans.
This will usher in incentives for powerful actors around the world that break the modern social contract.
This could result in the gradual—or sudden—disempowerment of the vast majority of humanity.
But this prophecy is not yet fulfilled; we reject the view that this path is inevitable. We see a different future on the horizon, but it will require a deliberate and concerted effort to achieve it.
We aim to change the incentives driving the intelligence curse, maintaining human economic relevance and strengthening our democratic institutions to withstand what will likely be the greatest societal disruption in history.
To break the intelligence curse, we should chart a different path on the tech tree, building technology that lets us:
Avert AI catastrophes by hardening the world against them, both because it is good in itself and because it removes the security threats that drive calls for centralization.
Diffuse AI, to get it in the hands of regular people. In the short-term, build AI that augments human capabilities. In the long-term, align AI directly to individual users and give everyone control in the AI economy.
Democratize institutions, making them more anchored to the needs of humans even as they are buffeted by the changing incentive landscape and fast-moving events of the AGI transition.
In this series of essays, we examine the incoming crisis of human irrelevance and provide a map towards a future where people remain the masters of their destiny.
Chapters
1. Introduction
We will soon live in the intelligence age. What you do with that information will determine your place in history.
2. Pyramid Replacement
Increasingly powerful AI will trigger pyramid replacement: a systematic hollowing out of corporate structures that starts with entry-level hiring freezes and moves upward through waves of layoffs.
3. Capital, AGI, and Human Ambition
AI will make non-human factors of production more important than human ones. The result may be a future where today's power structures become permanent and frozen, with no remaining pathways for social mobility or progress.
4. Defining the Intelligence Curse
With AGI, powerful actors will lose their incentive to invest in regular people–just as resource-rich states today neglect their citizens because their wealth comes from natural resources rather than taxing human labor. This is the intelligence curse.
5. Shaping the Social Contract
The intelligence curse will break the core social contract. While this suggests a grim future, understanding how economic incentives reshape societies points to a solution: we can deliberately develop technologies that keep humans relevant.
6. Breaking the Intelligence Curse
Avert AI catastrophes with technology for safety and hardening without requiring centralizing control. Diffuse AI that differentially augments rather than automates humans and decentralizes power. Democratize institutions, bringing them closer to regular people as AI grows more powerful.
7. History is Yours to Write
You have a roadmap to break the intelligence curse. What will you do with it?
Both the resource curse and intelligence curse were mentioned by Tristan Harris in his interview with Sam Harris
https://www.youtube.com/watch?v=90irsXaKxZA
I think the intelligence curse series can also show why it's necessary for AIs that can automate away all of human labor to be aligned to human values, even though right now we don't need to align other people's value to ours, and rely on selfishness/trade to get benefits.
Because once non-1% rich people have no selfish value, there is no reason to trade with them or give them goods relevant to their survival, which means there's no reason to trade with them instead of forming your own economy.
More generally, alignment in values will be way more necessary for regular people to survive than in the modern era, which is a shift we are not prepared for.
Also, it's a very good argument against claims that we will peacefully trade with AIs/humans if they are misaligned, and this is enough to ensure our survival.