A lot of processes and standards like Automotive SPICE, ISO 26262, IEC 61508, or DO 178b enforce traceability documentation. As a consequence, all relevant links between requirements and other artifacts in the software development process have to be maintained.
Once these trace links are available, they can be used for other purposes as well. Developers or testers can use them to navigate between related artifacts. QA-responsibles and project managers can use them to do an impact analysis for change requests and all kind of coverage analysis (implementations covered by tests, requirements covered by implementations, …).
“The ability to interrelate any uniquely identifiable software engineering artifact to any other, maintain required links over time, and use the resulting network to answer questions of both the software product and its development process. [CoEST definition of traceability]
But can the benefits be quantified? How much does a development team benefit from requirements traceability when maintaining software systems?
Prof. Dr. Patrick Mäder et. al. did an interesting empirical evaluation to answer this question. It was done in a controlled experiment with 71 subjects re-performing real maintenance tasks on two third-party development projects. In total the subjects solved 461 tasks. The findings show that subjects with traceability performed on average 24% faster on a given task and created on average 50% more correct solutions. That means that traceability does not only save effort but can profoundly improve the quality of software maintenance.
For the study, two existing software systems were chosen: GanttProject and iTrust. GanttProject is an open-source, cross-platform tool for project scheduling, implemented in Java. iTrust is a web-based medical application that provides patients and other medical stakeholders with a means to keep up with their medical records as well as to communicate between each other. iTrust is also realized in Java. Both are of siginificant size, as outlined in Table 1.
The experiment included eight distinct tasks, that had to be performed by the participants – four for each software system. For Gantt they selected bug reports from the existing issue tracking system and provided them unchanged to the participants. For iTrust they compared older versions of use case specifications with current ones and selected a single change request.
In addition to the time that was needed to complete the maintenance tasks, the correctness of the resulting solution was measured, as a factor of three levels: incorrect, partly correct and fully correct. Solutions were accepted if they removed the bug or led to the desired solution, even if this was not identical to the original solution of the Gantt and iTrust developers.
In order to have control of what participants could do during the experiment, they provided a specific tool with development and traceability support.
Traces were provided in a separate window and labeled with the names of artifacts they were connecting. A click on a trace opened the related source code file and highlighted the traced part. Traces were also highlighted in the file tree.
A comparison of all performed tasks showed that subjects working without traceability on average spent 889s working on a task. Subjects working with traceability on average spent only 678s. That means that on average traceability facilitated a 24% faster task completion.
Comparing the correctness of performed tasks, they found that participants working without traceability created 50% correct solutions. Participants working with traceability created on average 74% correct solutions. That means that traceability facilitates 50% more correct solutions to a task.
The team around Prof. Dr. Patrick Mäder also asked open questions how participants would improve the provided traceability and if they had comments on the experiment. Multiple participants positively referred to traceability in general:
Several participants requested additional documentation for the provided traces, for example:
Other improvement suggestions were:
Multiple participants suggested tool improvements, for example:
In the experiment traceability led to an efficiency gain of 24% and to 50% more correct solutions. All but two participants of the study replied after the experiment that they felt supported by traceability in performing.
It’s important to note, that this survey only covered a subset of usage scenarios in a development process supported by traceability.
In another survey about „Usage Scenarios for Requirements Traceability in Practice“ by Patrick Mäder et. al. 29 different usage scenarios in six groups were identified:
The team asked the participants of this study, why they applied traceability in their project. The following chart shows, that there is an obvious need for better support of development activities (80%).
As a conclusion, the available tool support has an important impact on the benefits and the answers of the participants suggest, that with improved and better tool support, even more benefits are realistic.
In the next blog article I will give a more detailed overview about the results of this survey.