Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma
Reinforcement Learning from Human Feedback (RLHF) is widely used for aligning large language models, yet practitioners face a persistent puzzle: improving safety often reduces fairness, scaling to diverse populations becomes computationally intractable, and making systems robust often amplifies majority biases. We formalize this tension as the Alignment Trilemma: no RLHF system can simultaneously achieve (i) epsilon-representativeness across diverse human values, (ii) polynomial tractability in sample and compute complexity, and (iii) delta-robustness against adversarial perturbations and distribution shift. Through a complexity-theoretic analysis integrating statistical learning theory and robust optimization, we prove that achieving both representativeness (epsilon <= 0.01) and robustness (delta <= 0.001) for global-scale populations requires Omega(2^{d_context}) operations, which is super-polynomial in the context dimensionality. We show that current RLHF implementations resolve this trilemma by sacrificing representativeness: they collect only 10^3--10^4 samples from homogeneous annotator pools while 10^7--10^8 samples are needed for true global representation. Our framework provides a unified explanation for documented RLHF pathologies including preference collapse, sycophancy, and systematic bias amplification. We conclude with concrete directions for navigating these fundamental trade-offs through strategic relaxations of alignment requirements.
69286865805584b280405112
Subramanyam Sahoo
claimed_verified
2026-03-13T08:15:51.173Z
6651cadeb6fc2144ecccf035
https://cdn-avatars.huggingface.co/v1/production/uploads/6651cadeb6fc2144ecccf035/UfHSrdbE2XbI-WeN3YbyN.jpeg
Subramanyam Sahoo
SahoobhAI
user
SahoobhAI
69286865805584b280405113
Aman Chadha
claimed_verified
2025-11-27T20:39:36.790Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
amanchadha
user
amanchadha
69286865805584b280405114
Vinija Jain
69286865805584b280405115
Divya Chaudhary
Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma
Reinforcement Learning from Human Feedback (RLHF) is widely used for aligning large language models, yet practitioners face a persistent puzzle: improving safety often reduces fairness, scaling to diverse populations becomes computationally intractable, and making systems robust often amplifies majority biases. We formalize this tension as the Alignment Trilemma: no RLHF system can simultaneously achieve (i) epsilon-representativeness across diverse human values, (ii) polynomial tractability in sample and compute complexity, and (iii) delta-robustness against adversarial perturbations and distribution shift. Through a complexity-theoretic analysis integrating statistical learning theory and robust optimization, we prove that achieving both representativeness (epsilon <= 0.01) and robustness (delta <= 0.001) for global-scale populations requires Omega(2^{d_context}) operations, which is super-polynomial in the context dimensionality. We show that current RLHF implementations resolve this trilemma by sacrificing representativeness: they collect only 10^3--10^4 samples from homogeneous annotator pools while 10^7--10^8 samples are needed for true global representation. Our framework provides a unified explanation for documented RLHF pathologies including preference collapse, sycophancy, and systematic bias amplification. We conclude with concrete directions for navigating these fundamental trade-offs through strategic relaxations of alignment requirements.
The Alignment Trilemma in RLHF shows that achieving representativeness, tractability, and robustness is computationally infeasible, leading to trade-offs in current implementations.
Reinforcement Learning from Human Feedback (RLHF) Alignment Trilemma epsilon-representativeness polynomial tractability delta-robustness complexity-theoretic analysis statistical learning theory robust optimization preference collapse sycophancy systematic bias amplification