TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.
697c93736676f93322706042
Elena Bruches
claimed_verified
2026-02-02T16:59:23.851Z
66728993cc71f0dce43fb93a
https://cdn-avatars.huggingface.co/v1/production/uploads/66728993cc71f0dce43fb93a/3Lf8ltxKSDOhbQxOCt-cE.jpeg
Bruches Elena
brucheselena
user
brucheselena
697c93736676f93322706043
Vadim Alperovich
697c93736676f93322706044
Dari Baturova
claimed_verified
2026-02-02T16:59:21.364Z
63ea1ca8afddf881179de4fe
/avatars/2690766ee09eefff296bcdac46e87b3b.svg
Dari
doooori
user
doooori
697c93736676f93322706045
Roman Derunets
claimed_verified
2026-01-30T13:31:25.974Z
6415cb01486c7c9a5d1560f3
https://cdn-avatars.huggingface.co/v1/production/uploads/6415cb01486c7c9a5d1560f3/tKQPhr1o-1SLSvKg3un6J.jpeg
Roman Derunets
rmndrnts
user
rmndrnts
697c93736676f93322706046
Daniil Grebenkin
claimed_verified
2026-02-02T16:59:26.065Z
63cb976d80ba2ca4151b67a2
https://cdn-avatars.huggingface.co/v1/production/uploads/1675713278440-63cb976d80ba2ca4151b67a2.jpeg
Daniel Grebenkin
dangrebenkin
user
dangrebenkin
697c93736676f93322706047
Georgy Mkrtchyan
697c93736676f93322706048
Oleg Sedukhin
claimed_verified
2026-02-23T09:47:34.878Z
61dd9daedb45389905634d3f
/avatars/a0482317b6118d37da294b9dc2bf5a39.svg
Oleg Sedukhin
greyzyablik
user
greyzyablik
697c93736676f93322706049
Mikhail Klementev
697c93736676f9332270604a
Ivan Bondarenko
claimed_verified
2026-02-09T14:32:09.804Z
62b1e0f76a5435fd9a60a8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1655824626110-noauth.png
Ivan Bondarenko
bond005
user
bond005
697c93736676f9332270604b
Nikolay Bushkov
697c93736676f9332270604c
Stanislav Moiseev
TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.
TAM-Eval is a framework and benchmark for evaluating large language models on comprehensive test suite maintenance tasks including creation, repair, and updating across multiple programming languages.
Large Language Models unit testing test suite maintenance test automation test generation oracle prediction test file level evaluation repository context test suite pass rate code coverage mutation testing agentic workflows reference-free protocol