WEB-based SYStems -- Time-to-market vs. Reliability



Rationale and Approach

General Focus and Position

In most cases, the development approach used for Web-based systems has been ad hoc, and software development for the Web lacks rigour and a systematic approach. Hardly any attention is given to development methodologies, measurement and evaluation techniques, application quality, reliability, and project management. Further, most current applications development and management practices heavily rely on the knowledge and experience of individual developers and their own development practices. In addition, there is lack of proper testing of Web-based systems, and documentation which is needed for ‘maintenance and upgrade’ of the system among other needs.

The nature and rapid growth/evolution of the Web, the rapid boom in Web and Web-related technologies, the commercialisation of the Web, and the rush to "be on the Web" and to migrate legacy systems to Web environments can partly be attributed to problems of Web-based system development. Also the complexity of Web-based applications has grown significantly - from information dissemination (consisting of simple text and images to image maps, forms, CGI, applets, scripts and stylesheets) to online transactions, enterprise-wide planning and scheduling systems, Web-based collaborative work environments, etc.

While the focus in SPIQ was on more long-term measurement based software process improvement (SPI), PROFIT focuses more on SPI in turbulent environments. However, both SPIQ and PROFIT focused on more traditional software development. WEBSYS differs from SPIQ and PROFIT in the explicit focus on the development of Web-based systems and the extremely short life cycles and corresponding extreme requirements of reduced time to market. Furthermore, Web’s legacy as an information medium rather than an application medium is another cause of the problem. Many consider Web development primarily as an authoring problem rather than an application development problem to which some of the well-known software engineering and management principles and practices could apply – with some changes and tuning to suite to the Web environment.

Many attributes of quality Web-based Systems such as ease of navigation, accessibility, scalability, maintainability, usability, compatibility and interoperability, security, readability, and reliability are not given due consideration during development. Many developers seem to be unaware of the real issues and challenges facing major Web-based application development and its continual maintenance.

There is a need to engender an awareness of the need for more disciplined approaches to Web-based application development, and we need to move from the current, largely ad hoc (and personalised) approach to a better disciplined approach and process. Also we need to realise that Web-based system development is not just graphic design or content development any more, and there are growing number of complex applications – intranet-based applications, transactional systems, and other e-business applications. Hence, Web-based systems developments are becoming more like software projects, and less like work of art.

Web Engineering (WebE) is an emerging new discipline focusing on successful development of complex web-based systems and applications (Ref ICSE99 workshop). It is concerned with establishment and use of sound scientific, engineering and management principles and disciplined and systematic approaches to the successful development, deployment and maintenance of high quality Web-based systems and applications. Thus, we see a close relationship between Web Engineering, Software Reliability and Software Process Improvement.

Software reliability is defined as the probability that the software will fail in a certain interval under stated operational conditions. For a web site there are two factors that are important - robustness and availability.

Robustness is the system’s ability to keep on working even if it gets erroneous or unexpected input. For a web system, diverse, wrong or unexpected input is more the rule than the exception. Thus, robustness is of big importance here.

Availability is the probability that the system is available on demand. It depends on the system’s reliability and capability for recovery. For most users, availability will reflect the site’s quality of service. A high availability will be an important factor in a competitive environment.

Both these factors taken together are necessary for a stable application and good service. The diversity of operational platforms where the applications will run puts an extra load on the system, especially when it comes to robustness. It is important to bear in mind that if a web site is down, the owner most likely will loose business since a prospective user just will move on to the next site which offer the same or a comparable service.

There exist several techniques and methods that can be used to achieve reliability. See for instance [Stålhane98a, 98b]. Most of them are, however, mainly used in military systems and systems related to air and rail traffic. By modifying and carrying these techniques and methods over to web software we can obtain a considerable increase in system’s reliability. In addition, by identifying the impact of each technique and method and describe them in quantitative terms, we can offer a way of estimating the effect of including or leaving them out of the development process. See for instance [Stålhane94]. 

Further, the problems with changing the process, introducing new technology and in demonstrating their effects, are shared with SPI (Software Process Improvement). For instance, there may be no clear baseline to compare with, results may come late or are unconvincing, the current product delivery suddenly gets priority and all "experimentation" is postponed or forgotten, management is semi-committed, or too dependent on local champions. In addition comes the usual organisational and technological turbulence in a highly evolving business place. 

All WEBSYS partners are or have been involved in the industrial, NFR-supported SPIQ [Ulsund96] and PROFIT [Ulsund99] projects in, respectively, 1997-99 and 2000-02. In both SPIQ and PROFIT, about 10 companies have been or are participating – and with SINTEF, NTNU and UiO as research partners. In SPIQ, a downscaled and pragmatic SPI framework was defined as part of the SPIQ Method Handbook (in Norwegian) [Dybå00a]. The Handbook attempts to combine TQM ideas with QIP/EF/GQM. The partners have participated in more than 20 such SPI efforts in the software industrial during the last 3.5 years.  

Thus, the personnel that will work in the WEBSYS project have experience and knowledge related to the areas of Web-based systems development, software reliability, and software process improvement. The partners have more than 20 international publications so far in 2000 based on empirical work in Norwegian industry.

Method Platform

All three research partners have emphasised empirical studies in their former work on software process improvement and on software engineering in general, cf. the SPIQ Method Handbook [Dybå00a]. We have also been working on guidelines for empirical studies [Arisholm99a] – see also other references to own work in the previous section. In WEBSYS we will keep up our focus on performing solid empirical studies – including formal (controlled) experiments, case studies, interviews and observations (think-aloud protocols, etc.) and "software archaeology".

The two most relevant research methods for studies on Web-based systems development are variants of observation and experimentation, see e.g. [Yin94] and [Wohlin00]. Indeed, a new research field, Empirical Software Engineering, is emerging. This is especially due to the work of Victor Basili in Maryland and Hans-Dieter Rombach in Kaiserslautern, both with applied research institutes affiliated, and a long shared history for using the NASA-SEL as a "software laboratory".

Observation of real software development work has the advantage of being realistic regarding scale and environment, but there are difficulties due to lack of control of variables, i.e. cause-effect analyses are hard. A spectrum of empirical techniques are available, e.g. formal experiments, case studies, surveys, retrospective analysis, even literature studies etc. [Zelkowitz98]. 

Experiments may have control of variables, but have shortcomings with respect to realism, e.g. unrealistic small software development tasks. We therefore need a combination: To establish reasonable, validated results, we should both carry out research in a controlled environment (as realistically as possible) and in a realistic environment (as controlled as possible). For projects like WEBSYS, it is advisable to combine observation of real projects (often case studies) with surveys and "laboratory" experiments (often on own students).  

We have carried out a number of observational and experimental studies on software development where results on research method have been a major component (see references in Sections 5.1-5.3). UiO has published some papers on research methods [Arisholm99a] [Anda00] and [Jørgensen00], and been responsible for several seminars on research methods. In addition, UiO has implemented a measurement program in a large Norwegian software company to assess the impact of incremental development on productivity, time-to-market and delivery precision.

UiO has been working closely with prof. Welland and prof. Atkinson at Glasgow for many years on empirical studies of database schema evolution and the application of persistent language technology. Recently, UiO has also been working with Dr. Lionel Briand at Carleton University, Ottawa on empirical evaluation of metrics for object-oriented systems. 

NTNU has for five years used own students in running experiments in a software quality course [Jaccheri00]. It has run many case studies in co-operation with industry, see e.g. the one project evolution [Nguyen97] and on inspections in Ericsson [Conradi99] as part of the SPIQ project. There is also ongoing, joint work with University of Maryland on how to represent repeatable experiments, and in running repeatable experiments on own students [Conradi+00a]. Much empirical work, especially around testing and SPI, has also been done at SINTEF by future NTNU-colleague Tor Stålhane, see [Haugrud98], [Stålhane97], and by Tore Dybå, see Dybå[97, 99, 00b, 00c].

The planned guest researchers/postdocs may come from the collegial environments mentioned above, e.g. from Maryland, Kaiserslautern,Univ. of London, Politecnico di Milano, Univ. Lund. 

Collaboration between Academia and Industry

WEBSYS will be conducted in close co-operation between the academic partners and the industrial partners to give a strong focus on practical application of the methods and empirical feedback. Much of the academic work will be done as part of PhD degrees, where each candidate typically will work with one or two of the participating companies. The candidates will gather valuable practical experience and empirical data to test the research hypotheses of their theses in a sound way.


The companies will achieve better understanding and improvement of their own software development processes, and base the development of their Web-based tools on this increased competence and the results of WEBSYS and, hence, increase their competitiveness.


We will also continue our tradition in using MSc students to work together with researchers and PhD students in such empirical work. In addition, each partner has 3-4 courses where more controlled experiments have been carried out for more than 5 years.


WEBSYS intends to create and utilise a synergy with related projects. In addition to working together with software producers on concrete cases, technology transfer to Norwegian organisations will be ensured through the dissemination activities described in the work plan.