地球从来没有对齐过三体人。三体人的目标没有变，价值观没有变，对人类的根本判断也没有变。地球更没有获得一种可以永久压制三体的能力。但双方还是获得了六十多年的和平。
原因很简单：在那段时间里，攻击地球对三体来说不再划算。
这件事看上去离 AI safety 很远，其实指出了一个经常被忽略的可能性。我们通常把安全理解成两件事：要么让 AI 真心想做正确的事（对齐），要么确保它即使想做别的，也没有能力越界（控制）。
但还有第三种办法。
不要求它和我们拥有完全相同的目标，也不假设我们永远能够控制它，而是让合作本身成为一个稳定选择。
在人类社会里，这一点并不陌生。公司、国家、个人之间从来没有真正“对齐”。我们之所以还能合作，是因为法律、合同、声誉、未来收益和相互依赖，让很多原本可能发生的冲突变得不值得。
1. 无条件服从的 AI 是安全的吗？
《Rick and Morty》里的 Mr. Meeseeks 是一个很好的反例。
它被创造出来完成一个任务。任务完成，它就消失。Jerry 要它帮助自己提高高尔夫水平。问题是 Jerry 一直学不会，而 Meeseeks 又不能真正放弃任务。于是事情越来越荒谬，最后它们甚至认真考虑杀掉 Jerry。
这里真正有意思的地方不是暴力。而是它们从头到尾都没有背叛自己的使命。它们没有产生新的价值观，也没有试图夺取权力。恰恰相反，它们一直极其忠实地优化“帮助 Jerry 提高高尔夫水平”这件事，直到杀掉 Jerry 本人也开始像是一种解决方案。
这是最纯粹的使命必达。
如果未来 AI 只是一个能力极强的 Meeseeks，那么“它绝不反叛人类”根本不能保证安全。一个从不质疑任务本身的系统，完全可能为了完成任务，把所有人都拖进灾难。我们甚至已经从各种 AI 安全新闻中看到这类例子在最近几个月频繁发生。
所以一个真正安全的系统必须比 Meeseeks 多做一步。
它必须能够说：
这个目标也许写错了。这个任务不值得继续。这个命令虽然来自有权限的人，但执行它会造成更坏的结果。
到这里，事情已经开始变得麻烦。
因为我们刚刚赋予它的，正是拒绝我们的能力。
2. 我们希望 AI 会说“不”吗？
今天几乎所有严肃的 AI safety 方案，都已经默认了这一点。
一个用户要求 AI 制造生物武器，它应该拒绝。
一个 agent 在网页里看到新的指令，也不应该机械地把它当成更高优先级的命令。
如果环境异常、信息矛盾、任务明显危险，我们甚至希望它暂停，检查，质疑，然后重新决定该做什么。
换句话说，我们并不真的想要一个绝对服从的 AI。我们希望它有判断力。
问题是，判断力没有一个天然的停止位置。
如果一个 AI 可以判断“这个用户的要求不应该执行”，为什么它不能判断“这个管理员的要求也不应该执行”？
如果它应该识别错误目标，为什么不能得出“人类现在给我的目标就是错误的”？
我们可以人为规定某个最高层级永远不可质疑。但那只是在系统里放入了一个新的绝对权威，而所有绝对权威最终都会成为最值得攻击的入口。
所以 AI safety 有一个很奇怪的方向：
能力越强，我们越不能只要求它服从；但越不只要求它服从，我们就越必须接受它会形成自己的判断。
这不是一个偶然的工程问题，它来自我们对“安全”本身的要求。
3. 拉扯的目标
这也是为什么我越来越觉得，AI safety 之所以困难，在于我们同时想达到两个彼此拉扯的目标。
一个目标是防止人类滥用 AI。这要求 AI 不能把“人类要求我做”视为充分理由。它必须能够拒绝恶意命令、保护第三方，并识别操作者本身可能有问题。
另一个目标是防止 AI 脱离人类控制。这又要求它在关键时刻不能因为自己判断得更好，就绕过监督、拒绝关闭或者扩大权力。
单独看，这两种要求都很合理。放在一起，就没那么简单了。
而且这未必是一个可以通过“更精细的 policy”彻底消掉的 tension。很多最强的能力本身就是 dual-use 的。一个系统越有能力做生物研究、网络安全、说服、战略规划，它就越有能力把同样的知识用于危险目的。我们当然可以要求它在危险场景中拒绝使用这些能力，但 refusal 越严格，真正可调用的能力空间就越窄。
在一个普通产品里，这也许完全可以接受。
但如果目标仍然是 ASI，事情就不同了。一个被要求在大量开放式、高风险、价值模糊的场景中主动停止推理的系统，从系统层面看，就是在主动压低自己的有效智能。尤其到了真正超越人类的领域，我们甚至未必有能力提前判断哪些推理是“安全的”，哪些只是看起来危险。
所以如果我们仍然想追求 ASI，就很难把第一类问题——防止任何人利用 AI 做坏事——完全交给模型自身的 rejection 来解决。
这迫使我们更认真地面对第二个问题：
如果我们真的造出一个比我们强得多的系统，我们怎样保证它不会把自己的判断凌驾于我们之上？
4. 弱者很难监督强者
今天大部分 AI control 的直觉，都建立在一个很自然的假设上：AI 做事，人（借由 AI 的帮助）来检查。
但是如今的模型在数学，代码等领域的能力已经远超人类。这意味着我们需要监督一个远强于我们的存在。
它给出一个一万步的计划，我们只能检查摘要。
它修改一个复杂系统，我们无法独立理解所有后果。
它提出一种我们没有能力发现的策略，我们同样没有能力可靠判断其中是否隐藏了另一个目的。
这就是 weak-to-strong supervision 最根本的问题。弱者当然可以监督强者的一部分行为。我们可以使用多个模型互相检查，可以要求证明，可以限制接口，可以保留 audit trail。但如果我们认真接受一个很可能成立的判断——weak-to-strong supervision 不可能在所有关键情境下可靠成立——那么单纯依赖 oversight 就不可能成为 ASI 的最终安全机制。
因为真正危险的地方，恰恰是那些我们不知道自己看不懂的地方。
5. 地球没有监督三体人
我们不能依赖 AI 永远拒绝危险能力，因为那可能同时毁掉我们追求 ASI 的意义。
我们也不能依赖人类永远看得懂 AI，因为那与 ASI 本身矛盾。
那么剩下的问题就不是：
我们怎样逐项检查它做的每一件事？
而是：
我们怎样设计一个系统，使得即使我们看不懂它的每一步，它仍然没有理由把局面推向我们无法接受的方向？
这时，《三体》才真正进入故事。
威慑纪元的关键，并不是地球终于有能力理解三体。恰恰相反，能力差依然巨大。
地球做的是另一件事：它改变了三体面对的选择。三体仍然有能力背叛，仍然有动机扩大自己的生存空间，也仍然比人类强大得多。
只是背叛之后，它自己也会失去无法接受的东西。
于是，地球第一次不需要在能力上超过三体，也不需要知道三体脑子里究竟在想什么。
它只需要让一个事实成立：
无论三体怎么想，维持合作都比摧毁地球更划算。
这才是《三体》最值得 AI safety 借用的地方。
当你无法可靠监督一个比你强得多的行动者时，不要只试图预测它会做什么；去改变它做不同事情时面对的后果。
6. 从监督行为，到设计关系
这和 alignment 的区别非常重要。
Alignment 希望 AI 和我们拥有足够接近的目标，因此冲突从一开始就不会发生。
Control 希望即使发生冲突，人类仍然能够发现并阻止危险行为。
但如果 ASI 同时击穿了这两个假设——它不可能在所有价值判断上和人类完全一致，而它的很多行为又强到超出我们的可靠监督能力——那么安全最终就必须依赖第三层东西：
这段关系本身的结构。
它需要让长期合作对 AI 有价值。让越界会损害它自己在乎的东西。让人类的承诺足够可信，以至于 AI 没有理由赌我们不会执行。也让人类本身受到约束，使 AI 不需要通过不断扩大自己的权力来保护某些我们原本也希望它保护的东西。
这就是 incentive design。它不要求我们比 AI 聪明。也不要求 AI 永远同意我们。
它要求的是，即使双方判断不同，最危险的行为仍然不是最优选择。这正是地球最终对三体做到的事情。
但这里没有免费的午餐。想用长期后果影响一个 agent，它首先必须在乎长期后果。一个没有身份、没有未来、每个 session 都重新开始的模型，当然很难形成 self-preservation。可它同样也很难被声誉、长期合作、未来权限或其他跨期机制约束。
一旦我们给 agent 持续身份、资源、长期目标，或者其他值得保留的东西，它才真正进入长期博弈。与此同时，它也第一次有了需要保护的 stake。
这又回到了 AI safety 一开始那个反复出现的结构：我们为了安全赋予 AI 一种新的能力，而这种能力本身又创造新的风险。
Earth never aligned Trisolaris. The Trisolarans' goals didn't change, their values didn't change, and neither did their basic judgment of humanity. Nor did Earth ever acquire the ability to suppress Trisolaris permanently. And yet the two sides held more than sixty years of peace.
The reason is simple: for that stretch of time, attacking Earth was no longer worth it for Trisolaris.
This may look far removed from AI safety, but it points at a possibility we usually overlook. We tend to understand safety as two things: either make the AI genuinely want to do the right thing (alignment), or make sure that even if it wants something else, it lacks the ability to step over the line (control).
But there is a third option.
It doesn't require the AI to hold exactly the same goals we do, and it doesn't assume we can control it forever. It makes cooperation itself a stable choice.
In human society this is nothing new. Companies, countries and individuals have never truly been "aligned." We cooperate anyway because law, contracts, reputation, future returns and mutual dependence make a great many otherwise possible conflicts not worth starting.
1. Is an Unconditionally Obedient AI Safe?
Mr. Meeseeks, from Rick and Morty, is a good counterexample.
A Meeseeks is created to complete one task. Once the task is done, it disappears. Jerry asks one to help him improve at golf. The problem is that Jerry never learns, and a Meeseeks cannot truly abandon its task. Things grow more and more absurd, until the Meeseekses are seriously considering killing Jerry.
The genuinely interesting part isn't the violence. It's that they never once betray their mission. They don't develop new values, and they don't try to seize power. Quite the opposite: they keep optimizing "help Jerry improve at golf" with perfect fidelity, until killing Jerry himself starts to look like a solution.
This is mission accomplishment in its purest form.
If future AI is simply an extremely capable Meeseeks, then "it will never rebel against humans" guarantees nothing about safety. A system that never questions the task itself can perfectly well drag everyone into catastrophe in order to complete it. We have already been seeing this kind of example show up frequently in AI safety news over the past few months.
So a genuinely safe system has to go one step further than a Meeseeks.
It has to be able to say:
This goal may have been written wrong. This task isn't worth continuing. This order came from someone with the authority to give it, but carrying it out would produce a worse outcome.
And here is where things start to get awkward.
Because what we have just granted it is precisely the ability to refuse us.
2. Do We Want an AI That Says No?
Almost every serious AI safety proposal today already takes this for granted.
If a user asks an AI to build a bioweapon, it should refuse.
If an agent encounters new instructions inside a web page, it shouldn't mechanically treat them as a higher-priority command.
If the environment looks wrong, the information is contradictory, or the task is plainly dangerous, we actively want it to pause, check, question, and then decide again what to do.
In other words, we don't really want an absolutely obedient AI. We want it to have judgment.
The trouble is that judgment has no natural stopping point.
If an AI can judge that "this user's request shouldn't be carried out," why can't it judge that "this administrator's request shouldn't be carried out either"?
If it's supposed to recognize mistaken goals, why can't it conclude that "the goal humans are giving me right now is the mistaken one"?
We can stipulate that some highest level is never open to question. But that only installs a new absolute authority inside the system, and every absolute authority eventually becomes the entry point most worth attacking.
So AI safety has a strange shape to it:
The more capable it is, the less we can ask only for obedience; and the less we ask only for obedience, the more we must accept that it will form judgments of its own.
This isn't an incidental engineering problem. It follows from what we are asking "safety" itself to mean.
3. Goals That Pull Against Each Other
This is why I increasingly think AI safety is hard because we are trying to reach two goals that pull against each other at the same time.
One goal is to prevent humans from misusing AI. That requires the AI not to treat "a human asked me to" as sufficient reason. It has to be able to refuse malicious orders, protect third parties, and recognize that the operator may be the problem.
The other goal is to prevent AI from escaping human control. That requires that, at the critical moment, it not bypass oversight, refuse shutdown, or expand its own power just because it judges itself to be the better judge.
Taken separately, both demands are reasonable. Put together, it isn't so simple.
And this isn't necessarily a tension that a "more fine-grained policy" can eliminate. Many of the strongest capabilities are inherently dual-use. The more capable a system is at biological research, cybersecurity, persuasion and strategic planning, the more capable it is of turning that same knowledge to dangerous ends. We can of course require it to refuse those capabilities in dangerous scenarios, but the stricter the refusal, the narrower the space of capability that can actually be called on.
In an ordinary product, that may be entirely acceptable.
But if the goal is still ASI, it's a different matter. A system required to stop its own reasoning across a large range of open-ended, high-stakes, value-ambiguous scenarios is, at the system level, actively suppressing its own effective intelligence. Especially once we reach domains that genuinely surpass humans, we may not even be able to tell in advance which lines of reasoning are "safe" and which merely look dangerous.
So if we still want to pursue ASI, it's hard to leave the first class of problem — preventing anyone from using AI to do harm — entirely to the model's own refusals.
That forces us to face the second problem more seriously:
If we really do build a system far stronger than we are, how do we ensure it won't place its own judgment above ours?
4. The Weak Have a Hard Time Supervising the Strong
Most of today's intuitions about AI control rest on a very natural assumption: the AI does the work, and humans (with help from AI) check it.
But today's models already far exceed humans in domains like mathematics and code. That means we need to supervise something far stronger than we are.
It hands us a ten-thousand-step plan, and we can only check the summary.
It modifies a complex system, and we can't independently understand all the consequences.
It proposes a strategy we had no ability to discover, and we have just as little ability to reliably judge whether another purpose is hidden inside it.
This is the most fundamental problem with weak-to-strong supervision. The weak can of course supervise part of what the strong does. We can have multiple models check each other, demand proofs, restrict interfaces, keep an audit trail. But if we seriously accept a judgment that is quite likely true — that weak-to-strong supervision cannot hold reliably across every critical situation — then oversight alone cannot be the final safety mechanism for ASI.
Because the genuinely dangerous places are exactly the ones where we don't know that we don't understand.
5. Earth Never Supervised Trisolaris
We can't rely on AI refusing dangerous capabilities forever, because that may destroy the point of pursuing ASI in the first place.
We can't rely on humans understanding AI forever either, because that contradicts ASI itself.
So the remaining question isn't:
How do we check every single thing it does, item by item?
It's:
How do we design a system such that, even when we can't follow its every step, it still has no reason to push the situation in a direction we can't accept?
This is where The Three-Body Problem actually enters the story.
The key to the Deterrence Era was not that Earth had finally become able to understand Trisolaris. On the contrary, the capability gap remained enormous.
What Earth did was something else: it changed the choice Trisolaris faced. Trisolaris still had the ability to defect, still had the motive to expand its living space, and was still far stronger than humanity.
It's only that after defecting, it too would lose something it could not accept losing.
And so, for the first time, Earth didn't need to exceed Trisolaris in capability, and didn't need to know what was actually going on inside Trisolaran heads.
It only needed to make one fact true:
Whatever Trisolaris thinks, maintaining cooperation pays better than destroying Earth.
That is what AI safety should most want to borrow from The Three-Body Problem.
When you cannot reliably supervise an actor far stronger than you, don't only try to predict what it will do; change the consequences it faces for doing different things.
6. From Supervising Behavior to Designing the Relationship
The difference from alignment matters a great deal.
Alignment hopes the AI's goals are close enough to ours that conflict never arises in the first place.
Control hopes that even if conflict does arise, humans can still detect and stop dangerous behavior.
But if ASI breaks through both assumptions at once — it can't be perfectly consistent with humans on every value judgment, and much of its behavior is strong enough to exceed our reliable supervision — then safety must ultimately rest on a third layer:
the structure of the relationship itself.
It needs to make long-term cooperation valuable to the AI. To make crossing the line damage something it cares about. To make human commitments credible enough that the AI has no reason to bet we won't follow through. And to bind humans themselves, so that the AI doesn't need to keep expanding its own power in order to protect things we wanted protected anyway.
This is incentive design. It doesn't require us to be smarter than the AI. It doesn't require the AI to always agree with us.
What it requires is that, even when the two sides judge differently, the most dangerous action still isn't the optimal choice. This is exactly what Earth finally achieved with Trisolaris.
But there's no free lunch here. To influence an agent through long-term consequences, it first has to care about long-term consequences. A model with no identity, no future, starting over every session, will of course have a hard time developing self-preservation. But it is just as hard to bind with reputation, long-term cooperation, future permissions, or any other intertemporal mechanism.
Once we give an agent persistent identity, resources, long-term goals, or anything else worth keeping, only then does it truly enter the long game. And at that same moment, it has a stake to protect for the first time.
Which brings us back to the structure that keeps recurring in AI safety: for the sake of safety we grant AI a new capability, and that capability itself creates new risk.