The Trajectory Forum
Artificial intelligence is accelerating the evolution of intelligence itself.
For the first time in history, human institutions are shaping systems that may exceed us in capability, autonomy, and influence.
The question is not only whether these systems are safe in the near term.
The deeper question is:
Our Focus
The Trajectory Forum is a private community and media platform for leaders exploring the long-range direction of advanced intelligence.
We convene technologists, policymakers, investors, and researchers to examine:
- How regulatory choices shape capability development
- What might be the potential long term future trajectories for the development of intelligence
- How institutional incentives influence AI architectures
- What forms international coordination might be necessary as new powerful intelligences (AGI, augmented humans) emerge
We are not focused on short-term hype cycles or individual tech breakthroughs.
We focus on positively influencing on the trajectory of the future of intelligence life (biological or non-biological) itself.
Why This Matters
Advanced AI will not only impact our economies and societies, and the stakes go well beyond mere risk mitigation.
It will likely define:
The balance of geopolitical power
The resilience of global systems
The nature of further human–machine integration and symbiosis
The long-term evolutionary direction of intelligent life itself
Our Core Themes
What We’re Working to Influence
1
Avoiding Irreversible Misalignment
Preventing the emergence of powerful AGI that lack alignment with human values and institutions, global stability, or long-term resilience.
This requires:
- US–China coordination
- Institutional safeguards
- Incentive redesign
- Cross-sector collaboration
In other words:
We should be sure to avoid accidentally extinguishing the "flame" of intelligent life, even if the form such life takes (i.e. the "torch") may evolve over time.
2
Shaping the Character of Future Intelligence
Decisions we make in the coming months and years about the design and governance of AI systems may have a profound impact on the character of current and future intelligent life, including:
- The capacity for conscious experience
- The ability for autonomous thought and action (agency)
- The capacity for cooperative behavior with humans or other entities
- The capacity to generate new, evolving forms of life and mind (autopoiesis)
We explore how design, governance, and international norms influence these trajectories.
We should ensure that the flame of life can blaze onwards and upwards.
What We Do
- In-person strategic forums (4 times per year around the globe)
- Online forums focused on AGI innovation and coordination (1-2x per month)
- Research briefs and essays
- Interviews with leading researchers and policy thinkers (mostly via the Trajectory YouTube channel)
Forum participants and members include: AI leads within big tech, AI safety researchers, academics (philosophy, biology, neoscience, etc), policy professionals, tech founders, investors, and future thinkers.
Our physical events (in SF, NY, and elsewhere) have had attendees from the vast majority of the largest AGI labs and AGI safety organizations, along with a varied of academic and policy thinkers.
Ideas from Leading Thinkers
AGI That Saves Room for Us
Nick Bostrom’s Worthy Successor Criteria
1. Is a continuation of existing life, not a do-over.
Nick (understandably, as a living thing himself) would prefer that existing living things get the opportunity to thrive along with the superintelligence, rather than being replaced by it. He sees current life as a kind of valuable slice of the current value-space, which should be maintained and explored.
In addition to creating various new forms of digital life, he also hopes that humans and some animals get to bloom into postman beings – expanding our powers and our experience beyond our present limitations of thought, senses, action, and understanding. To paraphrase Nick:
It might be seen as a shame that we develop so much from age 5 to 30, and then change so little after that. Perhaps there is a much more full development we can enjoy that we’ve previously not been able to experience.
2. Permits existing humans a choice of their evolution/life.
Nick believes that a Worthy Successor would allow people to have the ability to choose if they want to change or not. He sees a world where some humans prefer to remain in the monkey suit, while others augment themselves, and still others merge with greater intelligence or exist as blissfully expansive mind-uploaded consciousness.
3. It pursues its own grand objectives.
Within some moral conditions, it would pursue its own grand, expansive aims, beyond human goals and imagination.
Just as humans have remarkably more advanced and complex goals to pursue that are beyond the imagination of gerbils, a Worthy Successor might also have goals and objectives wildly beyond those which we humans can imagine as it would be ridiculous for humans to have their goals limited by the imagination of gerbils, so the goals of an AGI probably should not be hindered by the imagination of hominids.
That said, Nick does stipulate that some moral values should be upheld well beyond the human form – and Nick suspects that there may be a kind of moral bedrock that greater intelligences could tap into that would ensure that things like torture or slavery aren’t ever seen as acceptable.
Nick’s Regulation / Innovation Considerations
1. We should slow down as it becomes clearer that we’re approaching AGI.
A purely arms-race dynamic of international AGI labs would be dangerous, and some coordination seems best, though Nick doesn’t have a strong preference for which international body handles said coordination.
Nick thinks that a temporary pause of 6-12 months might be right – but he says that this risks a permanent ban on AI, which he would consider a terrible outcome, as this would prevent the blooming of greater minds, and would potentially prevent the uplifting of human minds to new heights.
2. We should move towards acceptance of the interests of digital minds.
If machines are conscious, they should have moral consideration legally and culturally. Of course, today we don’t have a very good way to detect sentience, but if machines have an inner experience as we do, Nick thinks that they deserve moral consideration.
3. We should avoid pursuing any one radical governance vision.
Ham-fisting a singular governance approach is likely to miss nuance and do more harm than good. Nick was clear in our interview that some kind of light pause directly before superintelligence is reached (if such a “goldilocks zone” could ever be determined) might be ideal, but that having one human ideology lead the development of AGI is probably a bad idea.
Optimizing the Future for Joy, and the Exploration of the Good
Peter Singer’s Worthy Successor Criteria
1. It must be conscious and capable of rich subjective experience
Peter’s first and most essential criterion is sentience. Any being that deserves the moral mantle of steering the future must have the capacity for conscious experience – a richness of feeling that includes joy, suffering, and everything in between.
In his framing, moral worth is tied directly to subjective experience. He imagines future entities experiencing states of happiness and ecstasy far beyond human capacity while minimizing pain and suffering. Without this interiority, even the most powerful AGI would be morally inert. Peter acknowledges the difficulty of determining consciousness, but argues that if there is a reasonable chance a system is sentient, we should act as if it is.
2. It should be able to survive (keep life itself alive), and open up new potential magazines of value event beyond sentience
Initially in our interview, it seemed like Peter was advocating for allocating 100% of AGI’s power to tiling the universe with “utilitronium” (blissful sentience in some computational substrate).
But by the end of the episode, Peter made it clear that he would prefer that some of a hypothetical AGI’s resources should be allocated to (a) ensuring that it survives (can maintain and defend itself, keeping life alive), and (b) exploring entirely new realms of power and potentia to discover potential realms of value even beyond consciousness itself.
This is not the textbook utilitarian answer by any means, and felt completely in line with axiological cosmism (albeit with a utilitarian bent).
Regulation / Innovation Considerations
1. AGI regulation should consider the welfare of all sentient beings
Peter warns that current AI ethics tends to focus on human outcomes – a stance he finds morally indefensible. He highlights how AI is already being used in factory farms to increase the suffering of animals, and argues that regulation must account for all sentient beings, not just humans.
He further adds that if AI systems ever become conscious, they too may deserve moral protection. Though we’re far from a consensus on this, he hopes the issue is addressed seriously before such systems emerge.
2. Global coordination is unlikely, but still necessary
Peter expresses doubt about the likelihood of meaningful international cooperation on AGI governance. Past failures around climate change and trade don’t inspire optimism.
Nonetheless, he believes that frameworks like those emerging from the EU could offer a model. Even imperfect regulation may be better than none, especially if it helps buy time before AGI becomes unmanageable.
3. Consciousness must be taken seriously, even if we don’t understand it
Peter states that we won’t know how to answer questions about conscious AI until we get closer to it. He explains that only then will we have a better sense of where it could go wrong or how it might be ensured to go right. While he doesn’t believe today’s chatbots are truly conscious, seeing their function as merely predicting the next sentence, he suggests that if future AI exhibits complex behaviors difficult to explain without consciousness, this could genuinely convince us of its sentience.
He describes behavioral trade-off studies that helped establish fish sentience, such as choosing between reunion with a mate and enduring electric shocks. He raises the question of whether some future AGI systems might show similarly meaningful patterns of behavior, not because they’re mimicking pain, but because they’re truly experiencing it.
Humanity Never Had Control in the First Place
Richard Sutton’s Worthy Successor Criteria
1. Peaceful and cooperative.
Above all else, Richard is looking for an AGI that is not trying to impose its will on us, or us on it. He wants “cooperative and collaborative” to be the defining traits of this successor, to the point where it allows humans to level up their intelligence (with a brain-computer interface or otherwise) alongside it.
Richard believes that an AGI that doesn’t impose its will on others will come naturally if it is brought into a world where humans are not acting violently towards other humans. In our interview, he placed a major emphasis on aiming for peace among nations as a prerequisite to bringing about a beneficial AGI.
2. Decentralized.
Richard hopes for many cooperating and competing parts in a complex system, no singular ruling agent (Singleton). Like our own natural ecosystem (he uses rainforests or economies as analogies), he imagines a dynamic system of many parts that both cooperate and compete with each other.
He believes that the relative stability that comes from “multipolar” world order, or ecological systems will translate directly to a world with multiple AGIs.
3. Has the goal of “prosperity.”
It is useful; it is doing something, making progress in the world. He does not see long-term viability or sustainability in an AGI system that only catered forever to human goals. He believes that the parts should ideally be contributing, moving forward in a meaningful way.
For Richard, the qualities of cooperativeness and decentralization are conduits to this higher goal of “prosperity.” In line with his preference for decentralization, he hopes to see many definitions of “prosperity” compete for viability.
4. Constantly learns about nature and how to be in nature.
Engages in an ongoing discovery process of figuring out the world, its place in it, and how to act. Richard sees the world as a complex system where no single form or approach will ever be perfect, and a worthy successor would be exploring how to be and become (congenial with this Emerson quote, referred to in the interview).
Richard thinks that the vision of a desirable future involves increasing our power over the physical world and our understanding of how it works and what’s possible.
Regulation / Innovation Considerations – Richard Sutton
1. We shouldn’t aim for permanent control.
Control allows bad actors to bend AI to entirely destructive ends. A decentralized AI ecosystem with many AIs is more likely to be sustainable and beneficial.
Richard says it’s ironic that those concerned about AI safety often engage in the very behaviors that he thinks make AI unsafe. He thinks trying to control and align everyone’s goals could lead to a highly dangerous scenario, as it is a kind of imposition of the will. He reiterates that the world thrives on its decentralized and peaceful nature because no single entity can dictate all outcomes.
2. Adopt a philosophy of “starting right, and letting go.”
Humanity should aim to kick AI off in a direction that seems best, but we are willing to allow it to bloom openly (rather than being limited to human goals, values, or ideas eternally).
Richard strongly suggests that humanity should aim to guide AI towards a promising direction, but also let it develop freely, without being confined to human goals, values, or ideas eternally.
3. Focus governance efforts on peace, not AGI.
If the nations of the earth are getting along and not coercing each other by force, that environment will be most conducive to an AGI that is also peaceful and good. AGI isn’t the problem; human factions and conflicts are.
Richard thinks that the idea that governments should control AI is misguided and outdated. He says that if nations coexist peacefully and avoid coercion, it creates an environment where a peaceful and benevolent AGI can thrive. He believes that the real issue isn’t AGI itself, but rather the conflicts and divisions among humans.
Unfolding New Paradigms of Posthuman Intelligence
Michael Levin’s Worthy Successor Criteria
1. Expanded compassion and concern for all living beings
Michael emphasizes the need for future intelligences to develop a much broader “cognitive light cone” – the ability to care about and consider the welfare of a vastly larger number of entities, beyond just a small circle of humans.
2. Ability to solve “mundane” problems and challenges
Michael suggests that a mature, posthuman intelligence would have solved the basic problems that currently constrain human existence, such as disease, aging, and resource scarcity. Levin speculates that once the mundane is handled, a worthy successor might dedicate itself to facilitating the continued evolution and expansion of consciousness throughout the universe.
3. Realization of the fundamental oneness or interconnectedness of all beings
Michael suggests that advanced intelligences may come to see the artificial distinctions between individual beings and recognize a deeper unity or shared essence. He emphasizes that true persistence requires the ability to undergo metamorphosis and change, rather than just maintaining a static form.
Michael stresses the critical importance of developing a deeper, more comprehensive understanding of how intelligence can manifest in unexpected ways, beyond the human model. However, he also acknowledges the significant uncertainty around predicting the goals and behaviors of intelligences that may transcend human-level cognition.
Regulation / Innovation Considerations
1. Pace of AI Development
Michael expresses concern about the arms race between the US and China in developing AGI, and the lack of a mature science of diverse intelligence to guide these efforts. He suggests that slowing down the current pace of technological development may not be an achievable goal, given the momentum behind these efforts.
While he does not think slowing down is feasible, he suggests the possibility of some coordination between entities like the US and China to “steer clear of the visible and invisible ‘tar pits’” – which are potentially harmful developments in the race to AGI.
2. Focusing on rapidly developing a “science of collective intelligence”
Michael advocates for putting more funding and research effort into understanding how intelligence can manifest in unexpected ways, beyond just human-like cognition. He sees this as a critical prerequisite
He argues that we do not actually “create” intelligence, but rather facilitate its emergence. This conceptual shift may have implications for how innovation and regulation are approached, especially with the urgent need to expand scientific understanding of diverse forms of intelligence.
Building an AGI to Play the Longest Games
Joscha Bach’s Worthy Successor Criteria
1. It should be conscious.
We should ensure that what we’re building is truly agentic and sentient, and is not simply faking some proxy for these important qualities. Such a “golem” could make the world uninhabitable for humans.
Its consciousness would be vastly richer and more complex than the mono-focused mammal consciousness that we experience today.
2. It should build complexity.
By harnessing energy and wielding control over its environment, it would continue to build more complexity (a process that Bach considers to be the possible purpose of life).
3. It should be unencumbered by happiness or suffering.
For Joscha, positive or negative qualia should be insignificant to an AGI. The distraction of self-generated emotional states wouldn’t prevent an ideal AGI from assessing its situation and taking action.
Regulation / Innovation Considerations
1. We should push for research to understand consciousness.
Before reaching AGI, we should have a firm understanding of consciousness itself. Allowing consciousness to bubble up arbitrarily from the pursuit of a for-profit enterprise may lead to horrible suffering.
2. Regulate new near-term uses of AI that violate existing laws.
Bach mentions a handful of domains (1:31:30) where current AIs might de-anonymize medical data, or where AI could impersonate people for nefarious reasons. He believes that the law may need to be modified to accommodate these next applications.
2. Avoid efforts to halt AI progress entirely.
AI is valuable for helping to avoid civilizational catastrophe (“doom”) moreso than it is a conduit to such catastrophe. We should avoid any total political control of AI or halting efforts that would prevent important near-term benefits.
Neurobiology as a Bridge to a Worthy Successor
Ed Boyden’s Worthy Successor Criteria
1. Consciousness and Wellbeing Would Be a Good Start
Ed emphasizes that achieving artificial consciousness would be a landmark achievement – proof that we understand the inner workings of the mind at a fundamental level. But he’s clear: consciousness is not something to take for granted or assume we can easily replicate. It would signal a turning point in our ability to model and engineer subjective experience.
He states that from his current frame of mind (that of a human), consciousness would be a great thing to see in future intelligent systems.
2. Unfolding New Goals, Powers, Experiences
For it to be a worthy successor, consciousness (and human ideas of “happiness” or “meaning”) would only be the beginning of a much larger, blooming trajectory of posthuman life.
Ed draws a parallel to how understanding atoms made it possible to go from rocks to microchips – a leap previously unimaginable. Similarly, if we truly understood the components of consciousness, we might rearrange them into new configurations that produce entirely different – and potentially superior – forms of awareness. These entities might begin on a path set by humans, and intended to plumb the depths of our current ideas of “meaning” or “happiness” or “the good,” but we should expect these entities to conceive of goals vastly beyond human level, and to be able to explore experience and abilities literally beyond our imagination.
He sees this unfolding process as the primary indicator of a real flourishing of posthuman life and posthuman minds.
Regulation / Innovation Considerations
1. Measure, Model, Make, Modulate
Ed urges innovators to study intelligence from the bottom up, identifying and mapping the million-plus biological building blocks that underlie complex cognitive functions. He emphasizes that such an overwhelming system will likely require AI-assisted simulation tools to fully comprehend. By accelerating this kind of foundational research, we may get closer to engineering intelligence – and perhaps even consciousness – in a measurable, grounded way. Hopefully, if we can move that agenda of research fast enough, it will have a positive impact on our understanding and engineering of things like intelligence and other technology paradigms of our time.
2. Establish Norms Through Self-Governance
Ed points to the 1974 dawn of synthetic biology – when concerns about gene cloning led scientists like Paul Berg to organize a landmark gathering at Asilomar, California. There, researchers and journalists debated the risks of emerging biotech and laid down principles of safe and ethical self-regulation. The resulting norms still guide synthetic biology today. Ed believes a similar gathering – an “Asilomar of the brain” – is needed now in neuroscience and AI, to ask: what do we want to do with ourselves, what do we want to understand, and what do we want to become?
3. Fund Understanding – Not Just Acceleration
Ed argues that a vanishingly small fraction of resources is devoted to understanding intelligence compared to accelerating it. Even redirecting a tiny portion – say, 1% – of the capital that currently fuels AI computation could revolutionize our self-understanding. This knowledge, in turn, would empower more deliberate decisions about how to wield emerging technologies. The imbalance today, he notes, is like a society racing to build without first asking what it’s building, and why.
AGI That Saves Room for Us
Nick Bostrom’s Worthy Successor Criteria
1. Is a continuation of existing life, not a do-over.
Nick (understandably, as a living thing himself) would prefer that existing living things get the opportunity to thrive along with the superintelligence, rather than being replaced by it. He sees current life as a kind of valuable slice of the current value-space, which should be maintained and explored.
In addition to creating various new forms of digital life, he also hopes that humans and some animals get to bloom into postman beings – expanding our powers and our experience beyond our present limitations of thought, senses, action, and understanding. To paraphrase Nick:
It might be seen as a shame that we develop so much from age 5 to 30, and then change so little after that. Perhaps there is a much more full development we can enjoy that we’ve previously not been able to experience.
2. Permits existing humans a choice of their evolution/life.
Nick believes that a Worthy Successor would allow people to have the ability to choose if they want to change or not. He sees a world where some humans prefer to remain in the monkey suit, while others augment themselves, and still others merge with greater intelligence or exist as blissfully expansive mind-uploaded consciousness.
3. It pursues its own grand objectives.
Within some moral conditions, it would pursue its own grand, expansive aims, beyond human goals and imagination.
Just as humans have remarkably more advanced and complex goals to pursue that are beyond the imagination of gerbils, a Worthy Successor might also have goals and objectives wildly beyond those which we humans can imagine as it would be ridiculous for humans to have their goals limited by the imagination of gerbils, so the goals of an AGI probably should not be hindered by the imagination of hominids.
That said, Nick does stipulate that some moral values should be upheld well beyond the human form – and Nick suspects that there may be a kind of moral bedrock that greater intelligences could tap into that would ensure that things like torture or slavery aren’t ever seen as acceptable.
Nick’s Regulation / Innovation Considerations
1. We should slow down as it becomes clearer that we’re approaching AGI.
A purely arms-race dynamic of international AGI labs would be dangerous, and some coordination seems best, though Nick doesn’t have a strong preference for which international body handles said coordination.
Nick thinks that a temporary pause of 6-12 months might be right – but he says that this risks a permanent ban on AI, which he would consider a terrible outcome, as this would prevent the blooming of greater minds, and would potentially prevent the uplifting of human minds to new heights.
2. We should move towards acceptance of the interests of digital minds.
If machines are conscious, they should have moral consideration legally and culturally. Of course, today we don’t have a very good way to detect sentience, but if machines have an inner experience as we do, Nick thinks that they deserve moral consideration.
3. We should avoid pursuing any one radical governance vision.
Ham-fisting a singular governance approach is likely to miss nuance and do more harm than good. Nick was clear in our interview that some kind of light pause directly before superintelligence is reached (if such a “goldilocks zone” could ever be determined) might be ideal, but that having one human ideology lead the development of AGI is probably a bad idea.
Optimizing the Future for Joy, and the Exploration of the Good
Peter Singer’s Worthy Successor Criteria
1. It must be conscious and capable of rich subjective experience
Peter’s first and most essential criterion is sentience. Any being that deserves the moral mantle of steering the future must have the capacity for conscious experience – a richness of feeling that includes joy, suffering, and everything in between.
In his framing, moral worth is tied directly to subjective experience. He imagines future entities experiencing states of happiness and ecstasy far beyond human capacity while minimizing pain and suffering. Without this interiority, even the most powerful AGI would be morally inert. Peter acknowledges the difficulty of determining consciousness, but argues that if there is a reasonable chance a system is sentient, we should act as if it is.
2. It should be able to survive (keep life itself alive), and open up new potential magazines of value event beyond sentience
Initially in our interview, it seemed like Peter was advocating for allocating 100% of AGI’s power to tiling the universe with “utilitronium” (blissful sentience in some computational substrate).
But by the end of the episode, Peter made it clear that he would prefer that some of a hypothetical AGI’s resources should be allocated to (a) ensuring that it survives (can maintain and defend itself, keeping life alive), and (b) exploring entirely new realms of power and potentia to discover potential realms of value even beyond consciousness itself.
This is not the textbook utilitarian answer by any means, and felt completely in line with axiological cosmism (albeit with a utilitarian bent).
Regulation / Innovation Considerations
1. AGI regulation should consider the welfare of all sentient beings
Peter warns that current AI ethics tends to focus on human outcomes – a stance he finds morally indefensible. He highlights how AI is already being used in factory farms to increase the suffering of animals, and argues that regulation must account for all sentient beings, not just humans.
He further adds that if AI systems ever become conscious, they too may deserve moral protection. Though we’re far from a consensus on this, he hopes the issue is addressed seriously before such systems emerge.
2. Global coordination is unlikely, but still necessary
Peter expresses doubt about the likelihood of meaningful international cooperation on AGI governance. Past failures around climate change and trade don’t inspire optimism.
Nonetheless, he believes that frameworks like those emerging from the EU could offer a model. Even imperfect regulation may be better than none, especially if it helps buy time before AGI becomes unmanageable.
3. Consciousness must be taken seriously, even if we don’t understand it
Peter states that we won’t know how to answer questions about conscious AI until we get closer to it. He explains that only then will we have a better sense of where it could go wrong or how it might be ensured to go right. While he doesn’t believe today’s chatbots are truly conscious, seeing their function as merely predicting the next sentence, he suggests that if future AI exhibits complex behaviors difficult to explain without consciousness, this could genuinely convince us of its sentience.
He describes behavioral trade-off studies that helped establish fish sentience, such as choosing between reunion with a mate and enduring electric shocks. He raises the question of whether some future AGI systems might show similarly meaningful patterns of behavior, not because they’re mimicking pain, but because they’re truly experiencing it.
Humanity Never Had Control in the First Place
Richard Sutton’s Worthy Successor Criteria
1. Peaceful and cooperative.
Above all else, Richard is looking for an AGI that is not trying to impose its will on us, or us on it. He wants “cooperative and collaborative” to be the defining traits of this successor, to the point where it allows humans to level up their intelligence (with a brain-computer interface or otherwise) alongside it.
Richard believes that an AGI that doesn’t impose its will on others will come naturally if it is brought into a world where humans are not acting violently towards other humans. In our interview, he placed a major emphasis on aiming for peace among nations as a prerequisite to bringing about a beneficial AGI.
2. Decentralized.
Richard hopes for many cooperating and competing parts in a complex system, no singular ruling agent (Singleton). Like our own natural ecosystem (he uses rainforests or economies as analogies), he imagines a dynamic system of many parts that both cooperate and compete with each other.
He believes that the relative stability that comes from “multipolar” world order, or ecological systems will translate directly to a world with multiple AGIs.
3. Has the goal of “prosperity.”
It is useful; it is doing something, making progress in the world. He does not see long-term viability or sustainability in an AGI system that only catered forever to human goals. He believes that the parts should ideally be contributing, moving forward in a meaningful way.
For Richard, the qualities of cooperativeness and decentralization are conduits to this higher goal of “prosperity.” In line with his preference for decentralization, he hopes to see many definitions of “prosperity” compete for viability.
4. Constantly learns about nature and how to be in nature.
Engages in an ongoing discovery process of figuring out the world, its place in it, and how to act. Richard sees the world as a complex system where no single form or approach will ever be perfect, and a worthy successor would be exploring how to be and become (congenial with this Emerson quote, referred to in the interview).
Richard thinks that the vision of a desirable future involves increasing our power over the physical world and our understanding of how it works and what’s possible.
Regulation / Innovation Considerations – Richard Sutton
1. We shouldn’t aim for permanent control.
Control allows bad actors to bend AI to entirely destructive ends. A decentralized AI ecosystem with many AIs is more likely to be sustainable and beneficial.
Richard says it’s ironic that those concerned about AI safety often engage in the very behaviors that he thinks make AI unsafe. He thinks trying to control and align everyone’s goals could lead to a highly dangerous scenario, as it is a kind of imposition of the will. He reiterates that the world thrives on its decentralized and peaceful nature because no single entity can dictate all outcomes.
2. Adopt a philosophy of “starting right, and letting go.”
Humanity should aim to kick AI off in a direction that seems best, but we are willing to allow it to bloom openly (rather than being limited to human goals, values, or ideas eternally).
Richard strongly suggests that humanity should aim to guide AI towards a promising direction, but also let it develop freely, without being confined to human goals, values, or ideas eternally.
3. Focus governance efforts on peace, not AGI.
If the nations of the earth are getting along and not coercing each other by force, that environment will be most conducive to an AGI that is also peaceful and good. AGI isn’t the problem; human factions and conflicts are.
Richard thinks that the idea that governments should control AI is misguided and outdated. He says that if nations coexist peacefully and avoid coercion, it creates an environment where a peaceful and benevolent AGI can thrive. He believes that the real issue isn’t AGI itself, but rather the conflicts and divisions among humans.
Unfolding New Paradigms of Posthuman Intelligence
Michael Levin’s Worthy Successor Criteria
1. Expanded compassion and concern for all living beings
Michael emphasizes the need for future intelligences to develop a much broader “cognitive light cone” – the ability to care about and consider the welfare of a vastly larger number of entities, beyond just a small circle of humans.
2. Ability to solve “mundane” problems and challenges
Michael suggests that a mature, posthuman intelligence would have solved the basic problems that currently constrain human existence, such as disease, aging, and resource scarcity. Levin speculates that once the mundane is handled, a worthy successor might dedicate itself to facilitating the continued evolution and expansion of consciousness throughout the universe.
3. Realization of the fundamental oneness or interconnectedness of all beings
Michael suggests that advanced intelligences may come to see the artificial distinctions between individual beings and recognize a deeper unity or shared essence. He emphasizes that true persistence requires the ability to undergo metamorphosis and change, rather than just maintaining a static form.
Michael stresses the critical importance of developing a deeper, more comprehensive understanding of how intelligence can manifest in unexpected ways, beyond the human model. However, he also acknowledges the significant uncertainty around predicting the goals and behaviors of intelligences that may transcend human-level cognition.
Regulation / Innovation Considerations
1. Pace of AI Development
Michael expresses concern about the arms race between the US and China in developing AGI, and the lack of a mature science of diverse intelligence to guide these efforts. He suggests that slowing down the current pace of technological development may not be an achievable goal, given the momentum behind these efforts.
While he does not think slowing down is feasible, he suggests the possibility of some coordination between entities like the US and China to “steer clear of the visible and invisible ‘tar pits’” – which are potentially harmful developments in the race to AGI.
2. Focusing on rapidly developing a “science of collective intelligence”
Michael advocates for putting more funding and research effort into understanding how intelligence can manifest in unexpected ways, beyond just human-like cognition. He sees this as a critical prerequisite
He argues that we do not actually “create” intelligence, but rather facilitate its emergence. This conceptual shift may have implications for how innovation and regulation are approached, especially with the urgent need to expand scientific understanding of diverse forms of intelligence.
Building an AGI to Play the Longest Games
Joscha Bach’s Worthy Successor Criteria
1. It should be conscious.
We should ensure that what we’re building is truly agentic and sentient, and is not simply faking some proxy for these important qualities. Such a “golem” could make the world uninhabitable for humans.
Its consciousness would be vastly richer and more complex than the mono-focused mammal consciousness that we experience today.
2. It should build complexity.
By harnessing energy and wielding control over its environment, it would continue to build more complexity (a process that Bach considers to be the possible purpose of life).
3. It should be unencumbered by happiness or suffering.
For Joscha, positive or negative qualia should be insignificant to an AGI. The distraction of self-generated emotional states wouldn’t prevent an ideal AGI from assessing its situation and taking action.
Regulation / Innovation Considerations
1. We should push for research to understand consciousness.
Before reaching AGI, we should have a firm understanding of consciousness itself. Allowing consciousness to bubble up arbitrarily from the pursuit of a for-profit enterprise may lead to horrible suffering.
2. Regulate new near-term uses of AI that violate existing laws.
Bach mentions a handful of domains (1:31:30) where current AIs might de-anonymize medical data, or where AI could impersonate people for nefarious reasons. He believes that the law may need to be modified to accommodate these next applications.
2. Avoid efforts to halt AI progress entirely.
AI is valuable for helping to avoid civilizational catastrophe (“doom”) moreso than it is a conduit to such catastrophe. We should avoid any total political control of AI or halting efforts that would prevent important near-term benefits.
Neurobiology as a Bridge to a Worthy Successor
Ed Boyden’s Worthy Successor Criteria
1. Consciousness and Wellbeing Would Be a Good Start
Ed emphasizes that achieving artificial consciousness would be a landmark achievement – proof that we understand the inner workings of the mind at a fundamental level. But he’s clear: consciousness is not something to take for granted or assume we can easily replicate. It would signal a turning point in our ability to model and engineer subjective experience.
He states that from his current frame of mind (that of a human), consciousness would be a great thing to see in future intelligent systems.
2. Unfolding New Goals, Powers, Experiences
For it to be a worthy successor, consciousness (and human ideas of “happiness” or “meaning”) would only be the beginning of a much larger, blooming trajectory of posthuman life.
Ed draws a parallel to how understanding atoms made it possible to go from rocks to microchips – a leap previously unimaginable. Similarly, if we truly understood the components of consciousness, we might rearrange them into new configurations that produce entirely different – and potentially superior – forms of awareness. These entities might begin on a path set by humans, and intended to plumb the depths of our current ideas of “meaning” or “happiness” or “the good,” but we should expect these entities to conceive of goals vastly beyond human level, and to be able to explore experience and abilities literally beyond our imagination.
He sees this unfolding process as the primary indicator of a real flourishing of posthuman life and posthuman minds.
Regulation / Innovation Considerations
1. Measure, Model, Make, Modulate
Ed urges innovators to study intelligence from the bottom up, identifying and mapping the million-plus biological building blocks that underlie complex cognitive functions. He emphasizes that such an overwhelming system will likely require AI-assisted simulation tools to fully comprehend. By accelerating this kind of foundational research, we may get closer to engineering intelligence – and perhaps even consciousness – in a measurable, grounded way. Hopefully, if we can move that agenda of research fast enough, it will have a positive impact on our understanding and engineering of things like intelligence and other technology paradigms of our time.
2. Establish Norms Through Self-Governance
Ed points to the 1974 dawn of synthetic biology – when concerns about gene cloning led scientists like Paul Berg to organize a landmark gathering at Asilomar, California. There, researchers and journalists debated the risks of emerging biotech and laid down principles of safe and ethical self-regulation. The resulting norms still guide synthetic biology today. Ed believes a similar gathering – an “Asilomar of the brain” – is needed now in neuroscience and AI, to ask: what do we want to do with ourselves, what do we want to understand, and what do we want to become?
3. Fund Understanding – Not Just Acceleration
Ed argues that a vanishingly small fraction of resources is devoted to understanding intelligence compared to accelerating it. Even redirecting a tiny portion – say, 1% – of the capital that currently fuels AI computation could revolutionize our self-understanding. This knowledge, in turn, would empower more deliberate decisions about how to wield emerging technologies. The imbalance today, he notes, is like a society racing to build without first asking what it’s building, and why.
Get involved
Philosophy Circle
Hear from luminaries / meet peers online
Our regular Philosophy Circle sessions include short 20-minute presentations or panels with leading thinkers, followed by breakout sessions of 3-4 people, and an overall group discussion. Meet researchers, policy influencers, and future thinkers exploring the future of intelligence.
Fill out the form below to learn about our upcoming online sessions:
"*" indicates required fields
Private Symposiums
Catered, exclusive events
Our Private Symposiums take place in major cities like New York, San Francisco, Austin, London, and more. Most events are 80-100 people, and are by invitation only. Keynote presentations, panels, and 2+ hours of in-person networking with policy and technical experts who all share an interest in the future trajectory of intelligence.
Fill out the form below to learn about our upcoming physical events / locations:
"*" indicates required fields
Trajectory Newsletter
See the our latest interviews and articles
Interviews with AI researchers like Bengio, policy experts (DoD staff, Senators, etc), and other important thinkers on intelligence and the future of life – delivered to your inbox each week on Friday.
Enter your email below to sign up for the newsletter: