The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts .
693b82729874a2a5e4ffb261
Aileen Cheng
693b82729874a2a5e4ffb262
Alon Jacovi
693b82729874a2a5e4ffb263
Amir Globerson
693b82729874a2a5e4ffb264
Ben Golan
693b82729874a2a5e4ffb265
Charles Kwong
693b82729874a2a5e4ffb266
Chris Alberti
693b82729874a2a5e4ffb267
Connie Tao
693b82729874a2a5e4ffb268
Eyal Ben-David
693b82729874a2a5e4ffb269
Gaurav Singh Tomar
693b82729874a2a5e4ffb26a
Lukas Haas
693b82729874a2a5e4ffb26b
Yonatan Bitton
693b82729874a2a5e4ffb26c
Adam Bloniarz
693b82729874a2a5e4ffb26d
Aijun Bai
693b82729874a2a5e4ffb26e
Andrew Wang
693b82729874a2a5e4ffb26f
Anfal Siddiqui
693b82729874a2a5e4ffb270
Arturo Bajuelos Castillo
693b82729874a2a5e4ffb271
Aviel Atias
693b82729874a2a5e4ffb272
Chang Liu
693b82729874a2a5e4ffb273
Corey Fry
693b82729874a2a5e4ffb274
Daniel Balle
693b82729874a2a5e4ffb275
Deepanway Ghosal
693b82729874a2a5e4ffb276
Doron Kukliansky
693b82729874a2a5e4ffb277
Dror Marcus
693b82729874a2a5e4ffb278
Elena Gribovskaya
693b82729874a2a5e4ffb279
Eran Ofek
693b82729874a2a5e4ffb27a
Honglei Zhuang
693b82729874a2a5e4ffb27b
Itay Laish
693b82729874a2a5e4ffb27c
Jan Ackermann
693b82729874a2a5e4ffb27d
Lily Wang
693b82729874a2a5e4ffb27e
Meg Risdal
693b82729874a2a5e4ffb27f
Megan Barnes
693b82729874a2a5e4ffb280
Michael Fink
693b82729874a2a5e4ffb281
Mohamed Amin
693b82729874a2a5e4ffb282
Moran Ambar
693b82729874a2a5e4ffb283
Natan Potikha
693b82729874a2a5e4ffb284
Nikita Gupta
693b82729874a2a5e4ffb285
Nitzan Katz
693b82729874a2a5e4ffb286
Noam Velan
693b82729874a2a5e4ffb287
Ofir Roval
693b82729874a2a5e4ffb288
Ori Ram
693b82729874a2a5e4ffb289
Polina Zablotskaia
693b82729874a2a5e4ffb28a
Prathamesh Bang
693b82729874a2a5e4ffb28b
Priyanka Agrawal
693b82729874a2a5e4ffb28c
Rakesh Ghiya
693b82729874a2a5e4ffb28d
Sanjay Ganapathy
693b82729874a2a5e4ffb28e
Simon Baumgartner
693b82729874a2a5e4ffb28f
Sofia Erell
693b82729874a2a5e4ffb290
Sushant Prakash
693b82729874a2a5e4ffb291
Thibault Sellam
693b82729874a2a5e4ffb292
Vikram Rao
693b82729874a2a5e4ffb293
Xuanhui Wang
693b82729874a2a5e4ffb294
Yaroslav Akulov
693b82729874a2a5e4ffb295
Yulong Yang
693b82729874a2a5e4ffb296
Zhen Yang
693b82729874a2a5e4ffb297
Zhixin Lai
693b82729874a2a5e4ffb298
Zhongru Wu
693b82729874a2a5e4ffb299
Anca Dragan
693b82729874a2a5e4ffb29a
Avinatan Hassidim
693b82729874a2a5e4ffb29b
Fernando Pereira
693b82729874a2a5e4ffb29c
Slav Petrov
693b82729874a2a5e4ffb29d
Srinivasan Venkatachary
693b82729874a2a5e4ffb29e
Tulsee Doshi
693b82729874a2a5e4ffb29f
Yossi Matias
693b82729874a2a5e4ffb2a0
Sasha Goldshtein
693b82729874a2a5e4ffb2a1
Dipanjan Das
The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts .
The FACTS Leaderboard evaluates language models' factual accuracy across different scenarios using four sub-leaderboards: image-based questions, closed-book factoid questions, information-seeking with search API, and document-grounded long-form responses.
FACTS Leaderboard FACTS Multimodal FACTS Parametric FACTS Search FACTS Grounding automated judge models