Do better IR tools improve the accuracy of engineers’ traceability recovery?
Author
Summary, in English
Large-scale software development generates an ever-growing amount of information. Multiple research groups have proposed using approaches from the domain of information retrieval (IR) to recover traceability. Several enhancement strategies have been initially explored using the laboratory
model of IR evaluation for performance assessment. We conducted a pilot experiment using printed candidate lists from the tools RETRO and ReqSimile to investigate how different quality levels of tool output affect the tracing accuracy of engineers. Statistical testing of equivalence, commonly used in medicine, has been conducted to analyze the data. The low number of subjects in this pilot experiment resulted neither
in statistically significant equivalence nor difference. While our results are not conclusive, there are indications that it is worthwhile to investigate further into the actual value of improving tool support for semi-automatic traceability recovery. For example, our pilot experiment showed that the effect size of using RETRO versus ReqSimile is of practical
significance regarding precision and F-measure. The interpretation
of the effect size regarding recall is less clear. The experiment needs to be replicated with more subjects and on varying tasks to draw firm conclusions.
model of IR evaluation for performance assessment. We conducted a pilot experiment using printed candidate lists from the tools RETRO and ReqSimile to investigate how different quality levels of tool output affect the tracing accuracy of engineers. Statistical testing of equivalence, commonly used in medicine, has been conducted to analyze the data. The low number of subjects in this pilot experiment resulted neither
in statistically significant equivalence nor difference. While our results are not conclusive, there are indications that it is worthwhile to investigate further into the actual value of improving tool support for semi-automatic traceability recovery. For example, our pilot experiment showed that the effect size of using RETRO versus ReqSimile is of practical
significance regarding precision and F-measure. The interpretation
of the effect size regarding recall is less clear. The experiment needs to be replicated with more subjects and on varying tasks to draw firm conclusions.
Department/s
Publishing year
2011
Language
English
Pages
23-30
Publication/Series
[Host publication title missing]
Full text
- Available as PDF - 608 kB
- Download statistics
Document type
Conference paper
Publisher
Association for Computing Machinery (ACM)
Topic
- Computer Science
Keywords
- requirements traceability
- information retrieval
- controlled experiment
- equivalence testing
Conference name
MALETS 2011: International Workshop on Machine Learning Technologies in Software Engineering
Conference date
2011-11-12
Status
Published
Project
- Embedded Applications Software Engineering