10.4    Reflections Upon the Research Approach

Confidence in Tema as a potentially effective approach to enterprise modeling also relies on the process by which it has been developed. A controversial research approach or a sloppy accomplishment of the research project may reduce the faith in the framework. Consequently, there is a need to discuss strengths and weaknesses of the research project in more detail as a part of evaluating the framework.

10.4.1    On the choice of research approach

In section 1.3.1, a research approach categorized as qualitative, observational and interpretive was argued for, based on the nature of the research questions and perceived need for empirical studies in the field of enterprise modeling. From a series of empirical studies, a number of lessons learned (referred to as principles) were outlined and a framework Tema designed to match the practice implicit to the principles.

A critical look on the status of the framework lead to the following observations: Firstly, the principles are based on subjective observations and interpretations from the four enterprise modeling projects. The subjectivity in observations and interpretations are due to an inherent bias of the researcher, being trained to notice some phenomena but not others.

Secondly, the principles are proposed as possible tendencies based on these interpretations, stripping away exceptions and perceived anomalies (Walsham, 1995:79). The lessons learned are local and specific to the studied projects, and may not necessarily be generalized outside the context of this research project.

Finally, Tema is comprised of a number of concepts and techniques that are assumed to be means to support the problem situations observed in practice and discussed as a part of the principles. Consequently, Tema may at best be seen as a more or less plausible approach to enterprise modeling.

Still, as Tema draws upon both observations from empirical studies and on theoretical contributions from other researchers, it is more plausible than approaches that do not have an empirical basis. The crux of Tema and any other proposed approach for enterprise modeling is its actual use in a practical setting.

10.4.2    On the accomplishment of the research project

By "accomplishment of the research project" is meant how the chosen research approach was made operational. Even if accepting that the chosen research approach in principle may be appropriate for development of Tema, a sloppy accomplishment may undermine the readers' confidence. Walsham (1995:78ff) explicitly comments upon this (italics added):
"Interpretive researchers are not saying to the reader that they are reporting facts; instead, they are reporting their interpretations of other people's interpretations. It is thus vital, in order to establish some credibility to the reader, that they describe in some detail how they have arrived at their 'results'. Reporting on 'soft' human issues is not an excuse for sloppiness."
As the main focus of chapter 10 is credibility, the current section is an attempt at describing in more detail how results have been reached. The discussion concerns four main issues:

1. Choice of projects for empirical studies

As pointed out in section 1.3.3, the choice of projects was rather pragmatic. The technology strategy project is clearly the most dominating project, with the three other projects as subsidiary sources of observations.

Still, both the number and variation in projects are considered satisfactory. Having four projects with slight variation can be seen as a means to ensure a certain robustness in Tema, as the framework must be applicable under various conditions. The Gazz project turned out to be somewhat peripheral to the other projects, and hence, only three of the projects are considered relevant to Tema on a broad basis.

A timely question concerns the fact that all four projects are from the same organization. How might this have influenced the observed behavior in the projects? There are at least two relevant issues to discuss: The degree of overlap in project participation and the degree of institutionalization of enterprise modeling practice.

Concerning overlap, one actor played a central role in both VPT and PA30, but was not involved in TEK-S at all. There was no other overlap between VPT and PA30. Three actors participated in both VPT and Gazz. One of the PG actors in TEK-S also participated in VPT, but he was not observed to be advocating neither why nor how to do enterprise modeling. To conclude, although there were considerable overlap between Gazz and VPT, the influence of overlap in project participation is not considered to influence significantly on the observed behavior.

Concerning institutionalization, there have been limited attempts within Statoil to summarize experiences with enterprise modeling (e.g., from PA30), but there are no widely available corporate guidelines on how to conduct enterprise modeling. The project participants did neither receive any official guidelines on enterprise modeling. To conclude, the studied projects are believed to be fairly independent of each other.

A positive effect of studying projects within the same organization is the improved understanding of the organization that is studied, providing a background for interpretation of observations, and becoming more of an insider than otherwise possible. A significant amount of time has been spent in Statoil, participating in seminars, presentations, workshops, and ad-hoc conversations in the corridors, in addition to the more formally held meetings as a part of enterprise modeling projects. A joint paper has also been written (Christensen et al., 1995).

2. The choice of research instruments and role

The dominating research instruments have been observation and document studies in VPT, Gazz and TEK-S, and interviews and document studies in PA30. Direct observation has been experienced as a fruitful approach to study the projects, as the research situations have been rich on contextual information in addition to the data directly relevant to enterprise modeling. With a loose research design, observation as the main source of data was found particularly suited, as the concepts used by the participating actors became central. An additional strength of the reliance on observation is that theories-in-use are more likely to be observed, not only theories espoused (Argyris, 1991:103).

Still, a timely question may be if not more focussed interviews would have provided even more confidence in the observations from the projects. In retrospect, more reliance on interviews in addition to observation and document studies is considered favorable. Through interviews, more information about how the actors themselves see enterprise modeling could have been gathered and compared to the observed practice.

Although planned interviews with prepared interview guides were not held in TEK-S, more informal dialogues took place during lunch, when having a break, waiting for meetings to start, etc. Still, interviews admittedly might have been used more widely in an attempt at triangulation of observations.

Recording of data from meetings was done taking notes using paper and pencil. In VPT, PA30 and Gazz, the notes were rough and coarse. In TEK-S, the notes were detailed to the level of statements from single actors (using a kind of shorthand). The notes were written out either the same or the following day, being fresh in memory (as advised by Walsham, 1995:78).

The role played in VPT, Gazz and TEK-S was as observer, taking notes, but not contributing in any significant way. The only exceptions were in the Gazz project, working on design principles for the user interface of the tool, and in the TEK-S project, designing the first few prototypes of the Web version of the strategy document and demonstrating them to the process group.

Mumford (1986:318) speaks of the ethical side of doing research: Informing all participants of the presence and intentions of the researcher and involving them in the research project. All the participants in VPT, Gazz, and the TEK-S process group and work groups were informed that the project was under study.

In TEK-S, one of the PG members (A) was approached by another actor (B), questioning the risk of allowing a student (me) to take part in the highly sensitive discussions of corporate strategy. Actor B admitted that he did not feel comfortable with the situation. Actor A answered that he trusted me and that if it would make actor B more secure, A would hire me. This proposal for solution seemed to remove B's hesitations concerning my participation (the hiring was not necessary to carry through). 

To conclude, although it is difficult to assess, the presence of a student in the meetings did not seem to influence the behavior of the actors significantly. It is unlikely that enterprise modeling would have been carried out differently without this presence.

3. Assurance of sensitivity to observations

According to Strauss and Corbin (1990:42ff), the researcher's sensitivity of observations in empirical studies is influenced by literature studies, professional experiences and personal experiences. Hence, a timely question is to what degree sufficient sensitivity was developed in order to make good observations (i.e., really understand what was going on in the project meetings).

Appropriate sensitivity was attempted developed in three ways:

Despite the efforts to develop sensitivity, observations pertaining to phenomena like actor power are scarce. A highly plausible explanation is lack of sensitivity to these issues. Hence, there is a possibility that significant events have been overlooked due to lack of sensitivity. Overall, the sensitivity is considered sufficiently developed within the areas covered by the research questions for the observations to be trustworthy (and most reliably in TEK-S).

4. Correctness of observations and interpretations

The observations and interpretations made in the course of the research project have to a large degree been confirmed by actors involved in the projects. A preliminary version of chapter 6 (with most of the observations, but without the interpretations) has been cleared with a central actor in the TEK-S PG. The description of the Gazz project has been cleared by three central actors in that project. The accounts given of PA30 and VPT have been cleared with the actor participating in both. Hence, observations from the projects have to a large degree been confirmed by actors taking part in the projects.

A common way to ensure that interpretations make sense to a wider audience than the researcher is to publish in refereed conferences or journals. This has been done only to a very limited degree: The joint paper on PA30 (Christiansen et al., 1995) discussed purposes of enterprise modeling in addition to essentials of the PA30 project. Chapter 4 on related areas has been developed from (Totland and Conradi, 1995). Still, the main results from the project have not been widely published (i.e., the main empirical study, the principles, and Tema).

To conclude, although the results reported in the thesis only have been published to a limited degree, the observations have been confirmed by actors in the studied projects.