An empirical approach to software process improvement calls for guiding process development based on empirical study of software technologies in use. This approach helps direct the evolution of new technologies, by studying the problems developers have applying the technology in practice, and validates mature technologies, by providing indication of the expected benefit and the conditions under which they apply. So, a variety of different empirical studies are necessary for a given technology over time, with evolving goals and hypotheses.

Thus, what we as a field know about a software development technology is never based on a single study; rather, a body of knowledge must be accumulated slowly out of many individual studies. Multiple studies also help mitigate the weaknesses inherent in any empirical study by requiring the confirmation or refutation of the original findings by means of independent replications, which can address the original threats to validity although they will invariably suffer from threats of their own.

As one example of a software development technology, software reading techniques are a promising approach for improving the improving the effectiveness of software inspections, by providing procedural guidelines containing tested procedures for effective individual inspection \226 a step which is often ignored in state-of-the-practice inspection processes. Software reading techniques have been studied via a number of empirical studies since they were introduced in 1994. Because of the availability of experimental lab packages that describe the experimental design and encapsulate experimental materials, replications have been facilitated in classroom contexts. Several of these studies have explored slightly different hypotheses concerning the techniques; others have varied experimental materials or the background of subjects; still others have varied along different dimensions. This series of world-wide studies has helped the reading techniques evolve from an initial idea to industry insertion, refining the process to be of greater value to users.

In this chapter, we use examples from the set of experiments on reading technique to draw lessons learned about running effective replicated studies. We also examine the results of these replications and explain the analyses to describe the benefits and difficulties that can be associated with replications.

We conclude by using the analysis to suggest guidelines along which studies can be undertaken and reported, to facilitate analysis across studies in the future and to improve our capabilities for building up bodies of knowledge about software development technologies.