Thus, what we as a field know about a software development technology is never based on a single study; rather, a body of knowledge must be accumulated slowly out of many individual studies. Multiple studies also help mitigate the weaknesses inherent in any empirical study by requiring the confirmation or refutation of the original findings by means of independent replications, which can address the original threats to validity although they will invariably suffer from threats of their own.
As one example of a software development technology, software reading techniques are a promising approach for improving the improving the effectiveness of software inspections, by providing procedural guidelines containing tested procedures for effective individual inspection \226 a step which is often ignored in state-of-the-practice inspection processes. Software reading techniques have been studied via a number of empirical studies since they were introduced in 1994. Because of the availability of experimental lab packages that describe the experimental design and encapsulate experimental materials, replications have been facilitated in classroom contexts. Several of these studies have explored slightly different hypotheses concerning the techniques; others have varied experimental materials or the background of subjects; still others have varied along different dimensions. This series of world-wide studies has helped the reading techniques evolve from an initial idea to industry insertion, refining the process to be of greater value to users.
In this chapter, we use examples from the set of experiments on reading technique to draw lessons learned about running effective replicated studies. We also examine the results of these replications and explain the analyses to describe the benefits and difficulties that can be associated with replications.
We conclude by using the analysis to suggest guidelines along which studies can be undertaken and reported, to facilitate analysis across studies in the future and to improve our capabilities for building up bodies of knowledge about software development technologies.