An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria

Autores
Zemin, Luciano Ricardo; Godio, Ariel; Cornejo, César Mauricio; Degiovanni, Renzo Gastón; Gutiérrez Brida, Simón Emmanuel; Regis, German Enrique; Aguirre, Nazareno Matias; Frias, Marcelo Fabian
Año de publicación
2025
Idioma
inglés
Tipo de recurso
artículo
Estado
versión publicada
Descripción
In this article, we empirically study the suitability of tests as acceptance criteria for automated program fixes, by checking patches produced by automated repair tools using a bug-finding tool, as opposed to previous works that used tests or manual inspections. We develop a number of experiments in which faulty programs from IntroClass, a known benchmark for program repair techniques, are fed to the program repair tools GenProg, Angelix, AutoFix, and Nopol, using test suites of varying quality, including those accompanying the benchmark. We then check the produced patches against formal specifications using a bug-finding tool. Our results show that, in the studied scenarios, automated program repair tools are significantly more likely to accept a spurious program fix than producing an actual one. Using bounded-exhaustive suites larger than the originally given ones (with about 100 and 1,000 tests) we verify that overfitting is reduced but (a) few new correct repairs are generated and (b) some tools see their performance reduced by the larger suites and fewer correct repairs are produced. Finally, by comparing with previous work, we show that overfitting is underestimated in semantics-based tools and that patches not discarded using held-out tests may be discarded using a bug-finding tool.
Fil: Zemin, Luciano Ricardo. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina
Fil: Godio, Ariel. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina
Fil: Cornejo, César Mauricio. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Degiovanni, Renzo Gastón. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina. Luxembourg Institute Of Science And Technology; Luxemburgo
Fil: Gutiérrez Brida, Simón Emmanuel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Regis, German Enrique. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Aguirre, Nazareno Matias. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Frias, Marcelo Fabian. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina. University of Texas at El Paso; Estados Unidos
Materia
Software and its engineering
Software testing and debugging
Formal software verification
Empirical software validation
Nivel de accesibilidad
acceso abierto
Condiciones de uso
https://creativecommons.org/licenses/by-nc-sa/2.5/ar/
Repositorio
CONICET Digital (CONICET)
Institución
Consejo Nacional de Investigaciones Científicas y Técnicas
OAI Identificador
oai:ri.conicet.gov.ar:11336/289776
Acceder
id CONICETDig_d94260380d764b3a8675f8d21e844a3b
oai_identifier_str oai:ri.conicet.gov.ar:11336/289776
network_acronym_str CONICETDig
repository_id_str 3498
network_name_str CONICET Digital (CONICET)
spelling An Empirical Study on the Suitability of Test-based Patch Acceptance CriteriaZemin, Luciano RicardoGodio, ArielCornejo, César MauricioDegiovanni, Renzo GastónGutiérrez Brida, Simón EmmanuelRegis, German EnriqueAguirre, Nazareno MatiasFrias, Marcelo FabianSoftware and its engineeringSoftware testing and debuggingFormal software verificationEmpirical software validationhttps://purl.org/becyt/ford/1.2https://purl.org/becyt/ford/1In this article, we empirically study the suitability of tests as acceptance criteria for automated program fixes, by checking patches produced by automated repair tools using a bug-finding tool, as opposed to previous works that used tests or manual inspections. We develop a number of experiments in which faulty programs from IntroClass, a known benchmark for program repair techniques, are fed to the program repair tools GenProg, Angelix, AutoFix, and Nopol, using test suites of varying quality, including those accompanying the benchmark. We then check the produced patches against formal specifications using a bug-finding tool. Our results show that, in the studied scenarios, automated program repair tools are significantly more likely to accept a spurious program fix than producing an actual one. Using bounded-exhaustive suites larger than the originally given ones (with about 100 and 1,000 tests) we verify that overfitting is reduced but (a) few new correct repairs are generated and (b) some tools see their performance reduced by the larger suites and fewer correct repairs are produced. Finally, by comparing with previous work, we show that overfitting is underestimated in semantics-based tools and that patches not discarded using held-out tests may be discarded using a bug-finding tool.Fil: Zemin, Luciano Ricardo. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; ArgentinaFil: Godio, Ariel. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; ArgentinaFil: Cornejo, César Mauricio. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; ArgentinaFil: Degiovanni, Renzo Gastón. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina. Luxembourg Institute Of Science And Technology; LuxemburgoFil: Gutiérrez Brida, Simón Emmanuel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; ArgentinaFil: Regis, German Enrique. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; ArgentinaFil: Aguirre, Nazareno Matias. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; ArgentinaFil: Frias, Marcelo Fabian. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina. University of Texas at El Paso; Estados UnidosAssociation for Computing Machinery2025-02info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionhttp://purl.org/coar/resource_type/c_6501info:ar-repo/semantics/articuloapplication/pdfapplication/pdfhttp://hdl.handle.net/11336/289776Zemin, Luciano Ricardo; Godio, Ariel; Cornejo, César Mauricio; Degiovanni, Renzo Gastón; Gutiérrez Brida, Simón Emmanuel; et al.; An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria; Association for Computing Machinery; ACM Transactions on Software Engineering and Methodology; 34; 3; 2-2025; 1-201049-331XCONICET DigitalCONICETenginfo:eu-repo/semantics/altIdentifier/url/https://dl.acm.org/doi/10.1145/3702971info:eu-repo/semantics/altIdentifier/doi/10.1145/3702971info:eu-repo/semantics/openAccesshttps://creativecommons.org/licenses/by-nc-sa/2.5/ar/reponame:CONICET Digital (CONICET)instname:Consejo Nacional de Investigaciones Científicas y Técnicas2026-08-25T15:27:21Zoai:ri.conicet.gov.ar:11336/289776instacron:CONICETInstitucionalhttp://ri.conicet.gov.ar/Organismo científico-tecnológicoNo correspondehttp://ri.conicet.gov.ar/oai/requestdasensio@conicet.gov.ar; lcarlino@conicet.gov.arArgentinaNo correspondeNo correspondeNo correspondeopendoar:34982026-08-25 15:27:22.202CONICET Digital (CONICET) - Consejo Nacional de Investigaciones Científicas y Técnicasfalse
dc.title.none.fl_str_mv An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
title An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
spellingShingle An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
Zemin, Luciano Ricardo
Software and its engineering
Software testing and debugging
Formal software verification
Empirical software validation
title_short An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
title_full An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
title_fullStr An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
title_full_unstemmed An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
title_sort An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria
dc.creator.none.fl_str_mv Zemin, Luciano Ricardo
Godio, Ariel
Cornejo, César Mauricio
Degiovanni, Renzo Gastón
Gutiérrez Brida, Simón Emmanuel
Regis, German Enrique
Aguirre, Nazareno Matias
Frias, Marcelo Fabian
author Zemin, Luciano Ricardo
author_facet Zemin, Luciano Ricardo
Godio, Ariel
Cornejo, César Mauricio
Degiovanni, Renzo Gastón
Gutiérrez Brida, Simón Emmanuel
Regis, German Enrique
Aguirre, Nazareno Matias
Frias, Marcelo Fabian
author_role author
author2 Godio, Ariel
Cornejo, César Mauricio
Degiovanni, Renzo Gastón
Gutiérrez Brida, Simón Emmanuel
Regis, German Enrique
Aguirre, Nazareno Matias
Frias, Marcelo Fabian
author2_role author
author
author
author
author
author
author
dc.subject.none.fl_str_mv Software and its engineering
Software testing and debugging
Formal software verification
Empirical software validation
topic Software and its engineering
Software testing and debugging
Formal software verification
Empirical software validation
purl_subject.fl_str_mv https://purl.org/becyt/ford/1.2
https://purl.org/becyt/ford/1
dc.description.none.fl_txt_mv In this article, we empirically study the suitability of tests as acceptance criteria for automated program fixes, by checking patches produced by automated repair tools using a bug-finding tool, as opposed to previous works that used tests or manual inspections. We develop a number of experiments in which faulty programs from IntroClass, a known benchmark for program repair techniques, are fed to the program repair tools GenProg, Angelix, AutoFix, and Nopol, using test suites of varying quality, including those accompanying the benchmark. We then check the produced patches against formal specifications using a bug-finding tool. Our results show that, in the studied scenarios, automated program repair tools are significantly more likely to accept a spurious program fix than producing an actual one. Using bounded-exhaustive suites larger than the originally given ones (with about 100 and 1,000 tests) we verify that overfitting is reduced but (a) few new correct repairs are generated and (b) some tools see their performance reduced by the larger suites and fewer correct repairs are produced. Finally, by comparing with previous work, we show that overfitting is underestimated in semantics-based tools and that patches not discarded using held-out tests may be discarded using a bug-finding tool.
Fil: Zemin, Luciano Ricardo. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina
Fil: Godio, Ariel. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina
Fil: Cornejo, César Mauricio. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Degiovanni, Renzo Gastón. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina. Luxembourg Institute Of Science And Technology; Luxemburgo
Fil: Gutiérrez Brida, Simón Emmanuel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Regis, German Enrique. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Aguirre, Nazareno Matias. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; Argentina. Universidad Nacional de Río Cuarto. Facultad de Ciencias Exactas Fisicoquímicas y Naturales. Departamento de Computación; Argentina
Fil: Frias, Marcelo Fabian. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Instituto Tecnológico de Buenos Aires; Argentina. University of Texas at El Paso; Estados Unidos
description In this article, we empirically study the suitability of tests as acceptance criteria for automated program fixes, by checking patches produced by automated repair tools using a bug-finding tool, as opposed to previous works that used tests or manual inspections. We develop a number of experiments in which faulty programs from IntroClass, a known benchmark for program repair techniques, are fed to the program repair tools GenProg, Angelix, AutoFix, and Nopol, using test suites of varying quality, including those accompanying the benchmark. We then check the produced patches against formal specifications using a bug-finding tool. Our results show that, in the studied scenarios, automated program repair tools are significantly more likely to accept a spurious program fix than producing an actual one. Using bounded-exhaustive suites larger than the originally given ones (with about 100 and 1,000 tests) we verify that overfitting is reduced but (a) few new correct repairs are generated and (b) some tools see their performance reduced by the larger suites and fewer correct repairs are produced. Finally, by comparing with previous work, we show that overfitting is underestimated in semantics-based tools and that patches not discarded using held-out tests may be discarded using a bug-finding tool.
publishDate 2025
dc.date.none.fl_str_mv 2025-02
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
http://purl.org/coar/resource_type/c_6501
info:ar-repo/semantics/articulo
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/11336/289776
Zemin, Luciano Ricardo; Godio, Ariel; Cornejo, César Mauricio; Degiovanni, Renzo Gastón; Gutiérrez Brida, Simón Emmanuel; et al.; An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria; Association for Computing Machinery; ACM Transactions on Software Engineering and Methodology; 34; 3; 2-2025; 1-20
1049-331X
CONICET Digital
CONICET
url http://hdl.handle.net/11336/289776
identifier_str_mv Zemin, Luciano Ricardo; Godio, Ariel; Cornejo, César Mauricio; Degiovanni, Renzo Gastón; Gutiérrez Brida, Simón Emmanuel; et al.; An Empirical Study on the Suitability of Test-based Patch Acceptance Criteria; Association for Computing Machinery; ACM Transactions on Software Engineering and Methodology; 34; 3; 2-2025; 1-20
1049-331X
CONICET Digital
CONICET
dc.language.none.fl_str_mv eng
language eng
dc.relation.none.fl_str_mv info:eu-repo/semantics/altIdentifier/url/https://dl.acm.org/doi/10.1145/3702971
info:eu-repo/semantics/altIdentifier/doi/10.1145/3702971
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
https://creativecommons.org/licenses/by-nc-sa/2.5/ar/
eu_rights_str_mv openAccess
rights_invalid_str_mv https://creativecommons.org/licenses/by-nc-sa/2.5/ar/
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Association for Computing Machinery
publisher.none.fl_str_mv Association for Computing Machinery
dc.source.none.fl_str_mv reponame:CONICET Digital (CONICET)
instname:Consejo Nacional de Investigaciones Científicas y Técnicas
reponame_str CONICET Digital (CONICET)
collection CONICET Digital (CONICET)
instname_str Consejo Nacional de Investigaciones Científicas y Técnicas
repository.name.fl_str_mv CONICET Digital (CONICET) - Consejo Nacional de Investigaciones Científicas y Técnicas
repository.mail.fl_str_mv dasensio@conicet.gov.ar; lcarlino@conicet.gov.ar
_version_ 1874775339323883520
score 13.265058

Publicaciones similares