The Quest for Reliable Metrics of Responsible AI
The development of Artificial Intelligence (AI), including AI in Science
(AIS), should be done following the principles of responsible AI. Progress in
responsible AI is often quantified through evaluation metrics, yet there has
been less work on assessing the robustness and reliability of the metrics
themselves. We reflect on prior work that examines the robustness of fairness
metrics for recommender systems as a type of AI application and summarise their
key takeaways into a set of non-exhaustive guidelines for developing reliable
metrics of responsible AI. Our guidelines apply to a broad spectrum of AI
applications, including AIS.
6903ff542c556835fa67ef4e
Theresia Veronika Rampisela
claimed_verified
2025-10-31T14:27:42.518Z
66fa381d94df3bc9dd23a991
/avatars/dbbf67f27593a510177dff06b00bf098.svg
TVR
theresiavr
user
theresiavr
6903ff542c556835fa67ef4f
Maria Maistro
6903ff542c556835fa67ef50
Tuukka Ruotsalo
6903ff542c556835fa67ef51
Christina Lioma
The Quest for Reliable Metrics of Responsible AI
The development of Artificial Intelligence (AI), including AI in Science
(AIS), should be done following the principles of responsible AI. Progress in
responsible AI is often quantified through evaluation metrics, yet there has
been less work on assessing the robustness and reliability of the metrics
themselves. We reflect on prior work that examines the robustness of fairness
metrics for recommender systems as a type of AI application and summarise their
key takeaways into a set of non-exhaustive guidelines for developing reliable
metrics of responsible AI. Our guidelines apply to a broad spectrum of AI
applications, including AIS.
Guidelines for developing reliable metrics of responsible AI are derived from assessing the robustness of fairness metrics in recommender systems.
fairness metrics recommender systems responsible AI