Open for submissions.
Research on the scientific foundations of understanding, predicting and steering AI systems.
Senior editorial board
Dylan Hadfield-Menell MIT
Vanessa Kosoy ALTER, CORAL
Jan Kulveit Charles University
Seth Lazar Johns Hopkins
Dan Murfet Timaeus
Tim Rudner University of Toronto
Andrew Saxe UCL
Benjamin Van Roy Stanford, Google DeepMind
Advisory board
Scott Aaronson UT Austin
Paul Christiano Alignment Research Center
Vince Conitzer Carnegie Mellon
Marcus Hutter Google DeepMind, ANU
Geoffrey Irving Resolution
Victoria Krakovna Google DeepMind
Jacob Tsimerman OpenAI, University of Toronto
- Reviewers are paid, with a bonus for early reviews.
- Every paper is published with a review summary.
- Submit your preprint any time, in any format.
Under review
Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control
Long-running AI agents create a control problem: each action they take changes the state, which in turn affects the trajectory of future actions. If the agent is not fully aligned, then guaranteeing safety requires approving consequential actions before allowing them to be executed. But requiring human approval at every step makes attention a bottleneck, and creates the risk of sophisticated agents persuading even diligent humans towards poor outcomes. Delegating review to other AI agents raises the same alignment problem: the reviewers may themselves be misaligned. We identify a condition on a reviewing panel that is weaker than individual alignment yet necessary and sufficient for a guarantee that the principal fares at least as well in expectation as under a designated baseline policy. Each reviewer agent reports whether an action proposal made by a proposer agent improves its own utility relative to the baseline. We show that a threshold rule tolerating k disapprovals is safe exactly when, after any k reviewers are removed, the principal's utility can be written as a nonnegative combination of the remaining reviewers' utilities, plus a term that is nonnegative on every feasible proposal. We call this property k-robust coalitional alignment. The characterization lifts to sequential control: in a discounted MDP with an arbitrary proposer agent, safety at every state is both necessary and sufficient for the induced policy to match or improve on the baseline. When reviewers vote strategically, full-panel coverage in reward-function space guarantees that every Nash equilibrium is safe under the unanimous approval rule; in contrast, more permissive thresholds can admit unsafe equilibria even when reviewers are individually aligned. Experiments with existing reviewer models show that collective review can remain sound without an aligned individual, even when some disapprovals are tolerated.
0/2 reviewers
Fragility of Value under Imperfect Alignment
As more responsibility is placed upon AI systems, it becomes increasingly important to guarantee that these systems are aligned with humanity. A common fear in AI safety is that human value is fragile – that is, optimizing too heavily for an imperfect proxy to human values will lead to a catastrophic outcome. In this paper, we present a model of the alignment problem where an agent undergoes idealized alignment training that guarantees its value function satisfies a proxy condition before optimizing the world. Our primary results identify conditions on the human value function and the accuracy of several proxy conditions under which an agent with an η-catastrophic value function, one that is guaranteed to take the expectation of human value below η in the limit of optimizing power, would be deployed. Our results highlight the danger of overoptimization and motivate AI designs that limit optimization pressure, such as quantilizers, rather than relying solely on pre-deployment training.
0/2 reviewers
General Agents Contain World Models, even under Partial Observability and Stochasticity
Deciding whether an agent possesses a model of its surrounding world is a fundamental step toward understanding its capabilities and limitations. In [10], it was shown that, within a particular framework, every almost optimal and general agent necessarily contains sufficient knowledge of its environment to allow an approximate reconstruction of it by querying the agent as a black box. This result relied on the assumptions that the agent is deterministic and that the environment is fully observable. In this work, we remove both assumptions by extending the theorem to stochastic agents operating in partially observable environments. Fundamentally, this shows that stochastic agents cannot avoid learning their environment through the usage of randomization. We also strengthen the result by weakening the notion of generality, proving that less powerful agents already contain a model of the world in which they operate.
1/2 reviewers
Superalignment Anti-Literature Review
AI capabilities have advanced significantly and may greatly surpass human abilities in an unknown timeframe. AI that is cognitively capable of escaping human control and commandeering all human infrastructure is sometimes called "superintelligent". Anticipating the possibility that we become able to create superintelligent AI, we review methods that might be used to retain control over it. We focus on the limitations of methods appearing in the literature, so we call this an "anti-literature review". Several methods we review appear to be promising approaches to creating controllable superintelligent AI, but we may have to accept a reduction in capability compared to uncontrolled AI. The specific approaches we discuss are reinforcement learning, red teaming, a shutdown button, doing what works for human-level AI, human-in-the-loop AI, recursive reward modeling, defensive AI, interpretability, pure imitation, constrained RL, myopic AI, narrow AI, AI sandbox, scientist AI, AI debate, the assistance game, pessimism, limited goal-information, steering vectors, a special shutdown button, current RF optimization, automated research, and provably safe AI.
2/3 reviewers