Peer review for alignment research.
We're trying something different.
Interested in serving as a reviewer or action editor?
Under review
Fragility of Value under Imperfect Alignment
As more responsibility is placed upon AI systems, it becomes increasingly important to guarantee that these systems are aligned with humanity. A common fear in AI safety is that human value is fragile – that is, optimizing too heavily for an imperfect proxy to human values will lead to a catastrophic outcome. In this paper, we present a model of the alignment problem where an agent undergoes idealized alignment training that guarantees its value function satisfies a proxy condition before optimizing the world. Our primary results identify conditions on the human value function and the accuracy of several proxy conditions under which an agent with an η-catastrophic value function, one that is guaranteed to take the expectation of human value below η in the limit of optimizing power, would be deployed. Our results highlight the danger of overoptimization and motivate AI designs that limit optimization pressure, such as quantilizers, rather than relying solely on pre-deployment training.
0/2 reviewers
General Agents Contain World Models, even under Partial Observability and Stochasticity
Deciding whether an agent possesses a model of its surrounding world is a fundamental step toward understanding its capabilities and limitations. In [10], it was shown that, within a particular framework, every almost optimal and general agent necessarily contains sufficient knowledge of its environment to allow an approximate reconstruction of it by querying the agent as a black box. This result relied on the assumptions that the agent is deterministic and that the environment is fully observable. In this work, we remove both assumptions by extending the theorem to stochastic agents operating in partially observable environments. Fundamentally, this shows that stochastic agents cannot avoid learning their environment through the usage of randomization. We also strengthen the result by weakening the notion of generality, proving that less powerful agents already contain a model of the world in which they operate.
1/2 reviewers
Superalignment Anti-Literature Review
AI capabilities have advanced significantly and may greatly surpass human abilities in an unknown timeframe. AI that is cognitively capable of escaping human control and commandeering all human infrastructure is sometimes called "superintelligent". Anticipating the possibility that we become able to create superintelligent AI, we review methods that might be used to retain control over it. We focus on the limitations of methods appearing in the literature, so we call this an "anti-literature review". Several methods we review appear to be promising approaches to creating controllable superintelligent AI, but we may have to accept a reduction in capability compared to uncontrolled AI. The specific approaches we discuss are reinforcement learning, red teaming, a shutdown button, doing what works for human-level AI, human-in-the-loop AI, recursive reward modeling, defensive AI, interpretability, pure imitation, constrained RL, myopic AI, narrow AI, AI sandbox, scientist AI, AI debate, the assistance game, pessimism, limited goal-information, steering vectors, a special shutdown button, current RF optimization, automated research, and provably safe AI.
2/3 reviewers