10.2    Evaluation of TEMA

The purpose of the evaluation in section 10.2 is to discuss the strengths and weaknesses of Tema in itself how it matches the design criteria from chapter 8, and how elements of Tema were used in the projects that were studied.

10.2.1    Tema and the framework design criteria

The three criteria used in design of Tema were fit with principles, practical applicability and degree of presumptionism.

Fit with principles

As Tema can be considered an enactment of the principles developed as lessons learned from the empirical studies (and actually incorporates these principles), there is a natural fit between Tema and the principles. All assertions from chapter 7 have been brought over to the framework.

A timely question is if there exists any situation at all where one or more of the principles would not be incorporated in the framework. The most obvious situation would be if the inclusion of one of the principles had lead to violation of one of the other design criteria.
 

Tema is designed to be practically applicable. The amount of preparations before employing Tema in a modeling project (the team training stage of development) may be considered to counteract applicability. Hence, whether to include a preparatory stage or not depends upon the significance of the principles advocating thorough preparations and the relative importance of the two design criteria fit with principles and practical applicability

Recognizing the fit between the principles and the framework may increase the readers' confidence in Tema: If the principles underlying Tema are considered valid, the reader may also accept Tema as an effective approach to enterprise modeling.

Practical applicability

The second design criterion was practical applicability. Assessing the usability of an approach is difficult without experience from practical use, but a few reflections can be provided.

Firstly, the costs involved in ensuring that requirements to competence (team training, learning the techniques, becoming computer literate) may be significant, but when the competencies have been acquired, they can also be applied in other situations. Hence, the costs may be distributed over several projects.
 

Learning how to use a computer effectively to navigate on the Internet or to communicate with colleagues via email or bulletin boards may improve efficiency and effectiveness in the normal job as well. Hence, the basic competencies required in Tema are not only basic skills for Tema, but for other activities, too. 

Cost/benefit analysis of Tema may be requested and attempted, but the economic figures for the benefit are difficult to estimate. What is the benefit of improved understanding of the corporate business? Benefits may at best be assessed in retrospect.

Secondly, as Tema has focused on some aspects of enterprise modeling and neglected others, the framework only offers support for a limited number of aspects of a situation. Problem situations not foreseen by Tema may lead to breakdowns in the enterprise modeling process. Whether breakdowns will occur or not is not discussed within the framework, and leaving aspects out can be seen as jeopardizing the applicability. 
 

The actor power aspect is not treated in Tema. An underlying assumption in Tema is the motivated and cooperative actor that the sense-making process is harmonic in the sense that actors are allowed and able to put forward their views and have good intentions, working for the best of the company. This is clearly a simplified assumption. Conflicts over the act of representing realities may cause an enterprise modeling effort to break down.

Still, despite the above concerns over practical applicability, most of the framework components have been observed in the technology strategy project, carried out to more or less degree. The reports on how the actors worked in TEK-S and the other projects are the most persuading arguments for the claim that Tema is practically applicable (as will be outlined in section ).

Presumptionism

The principal stance on presumptionism in Tema was to make few assumptions, and when assumptions were made, they were to concern the Universe of Modeling (and not the Universe of Production). This is manifested in the prescription of a pool of techniques instead of a monolithic method and in the call for construction of simple modeling languages as a part of the modeling effort.

The most restricting assumptions of Tema are considered to be the requirements to team-based development of models and the preparatory team training stage. Team-based modeling is fundamental to Tema due to the underlying constructivistic worldview. Reality is constructed in the modeling process and cannot be left to one actor alone. This would lead to inferior representations unless the domain was widely agreed upon (and recall that Tema has been developed for modeling of domains that are not well known to the modelers).

To conclude, if one accepts the arguments posed in section 8.1.2 that a flexible approach (i.e., few assumptions) is required to handle the kind of enterprise modeling that Tema seeks to support (i.e., sense-making), one may also accept that Tema provides this required flexibility.

10.2.2    Tema and the empirical studies

Although Tema has not been applied in practice, elements of the framework have been observed as a part of the empirical studies. Confidence in Tema as an approach to enterprise modeling is hence sought in observations from the empirical studies.

A tabular form is used to provide an overview of each project, and only selected issues are discussed, since the full outline of the projects have been given in chapters 2 and 6. Tables are provided for both the development and dissemination phases (the aspects in each table were discussed in sections 8.3 and 8.4, respectively).

Of the projects, the strategy project (project D below) was by far the most influential on Tema, due to most focussed and prolonged studies. VPT and PA30 (projects A and B) are also discussed, despite lack of observations on some aspects of Tema. Gazz (project C) is considered to be a non-typical setting of Tema and is hence not assessed.

Project A: Elements of Tema in the VPT project

The main objective of the VPT project was to suggest new projects involving cooperation across disciplinary and organizational borders in order to reduce suboptimization. Enterprise modeling was a means to develop an overall view of the enterprise, and thereby make sense of the proposed projects. This coincides with the intent of Tema enterprise modeling not as an end in itself, but as a means to make sense of another product or service.
VPT development phase
Table 10.1 provides an overview of observations from the development phase of VPT supporting Tema. A particularly relevant observation is their explicit recognition of modeling as a creative process being in accordance with the argumentation for a pool of techniques in Tema. Still, the observed use of creative techniques was modest.
 
Aspects How the aspects were handled in the VPT project
Intermediate artifacts
  • Modeling language was developed as a part of modeling. 
  • Enterprise model evolving throughout the project. 
  • Subsidiary artifacts in terms of metaphors (enterprise model is a map) and stories (how holistic thinking had saved money on existing projects). 
  • Development activities
  • Ad-hoc approach to modeling (no predefined method). 
  • Modeling considered a creative process (brainstorming observed). 
  • Forums Face-to-face meetings.
    Media
  • Paper and whiteboard in meetings. 
  • Computerized drawing tool used to improve layout between meetings. 
  • Developers Knowledge about domain (physical flow was better understood than information flow).
    Actual use of models
  • Shared understanding and holistic thinking. 
  • Structuring device (structuring discussions, having something to point at when discussing elements of the enterprise). 
  • Table 10.1: The development phase of the VPT project

    An issue not compliant with Tema is the number of intermediate artifacts. The enterprise model evolved throughout the project, but did not exist in many different variants. One explanation may be that the project group meant they had sufficient knowledge about the enterprise to start modeling immediately.

    Another issue is the lack of training. However, as the use of creative techniques was not extensive (brainstorming was the only observed), preparatory training was not as required as if more extensive techniques had been used.

    VPT dissemination phase
    Table 10.2 summarizes the dissemination phase of the VPT project. Most of the focus was on the development phase. The enterprise model was to be used internally in the project group and had no intended audience outside the group. The main reason for disseminating the model was that rumors spread by word of mouth in the organization, people became curious about the project and requested the model.
     
    Aspects How the aspects were handled in the VPT project
    Exposed artifacts The overall enterprise model (no subsidiary artifacts).
    Dissemination activities Distribution of the model (no systematic presentation or feedback).
    Forums Not applicable (as the model was not presented, only distributed).
    Media Paper.
    Audience Knowledge and competence of the audience is unknown.
    Actual use of models Structuring of discussions had been observed by the project leader.
    Table 10.2: The dissemination phase of the VPT project

    The dissemination phase of VPT provides neither much support for nor many arguments against Tema, as the project did not focus on model dissemination.

    Project B: Elements of Tema in the PA30 project

    The overall purpose of the PA30 project was to change the work processes at the plant and use enterprise modeling to gain the required understanding of problems and opportunities before making changes. This is consistent with the intended use of Tema.
    PA30 development phase
    Table 10.3 summarizes the enterprise model development phase of PA30. An illustrative observation is the creation of their own modeling language incorporating bombs and clouds for representation of problems and opportunities.
     
    Aspects How the aspects were handled in the PA30 project
    Intermediate artifacts
  • Modeling language was developed as a part of modeling. 
  • Statement database and enterprise models. 
  • Few intermediate artifacts, more requested by the project team. 
  • Development activities
  • No team training, but pilots requested afterwards. 
  • Ad-hoc approach to modeling. 
  • Interviews to gather information and gain understanding. 
  • Group reflection upon development process was explicitly appreciated. 
  • Forums Face-to-face meetings (interviews and project meetings).
    Media
  • Paper and whiteboard in meetings. 
  • Computerized drawing tool to improve layout. 
  • Developers
  • General knowledge of the plant (four project group members). Knowledge of physical flow better than information flow. 
  • Detailed knowledge of the plant, problems and opportunities (interviewed workers). 
  • Competence in modeling (two facilitators). 
  • Actual use of models
  • Shared understanding and holistic thinking. 
  • Structuring device: Model as background for problems and opportunities
  • Table 10.3: The development phase of the PA30 project

    Observations that differ from Tema are the lack of training (although they admit afterwards that they would have liked to have run pilot interviews to increase their competence), and the exclusive reliance on interviews (and no creative techniques). They neither employed digital media or forums to any significant degree.

    PA30 dissemination phase
    Table 10.4 provides an overview of the dissemination phase of PA30.
     
    Aspects How the aspects were handled in the PA30 project
    Exposed artifacts
  • Enterprise model used in presentations. 
  • Oral explanation accompanying the enterprise model. 
  • Dissemination activities
  • Presentation in terms of model hanging on the wall (little feedback). 
  • Presentation as a part of the analysis phase (with positive feedback) 
  • Forums
  • Wall in the corridor. 
  • Face-to-face meetings in analysis phase. 
  • Media Paper.
    Audience The audience had good knowledge of the plant.
    Actual use of models Structuring device (in presentations of preliminary results from the analysis phase).
    Table 10.4: The dissemination phase of the PA30 project

    One observation supporting Tema is the lack of feedback when models were just hung on the walls. Without explicit stimulation, feedback is not to be expected. Another observation consistent with Tema is the use of models in the analysis phase of PA30 as structuring devices accompanied by explanations or additional information.

    PA30 does not provide any support for dissemination using digital media. Support for Tema's notion of subsidiary artifacts is neither found.

    Project C: Elements of Tema in the Gazz project

    The Gazz project differed substantially from the three other projects. The main difference was their focus on models intended for simulation. The modeling process was more dominated by representation than by sense-making, as the elements of the Gazz models generally were well known and agreed upon among the actors in the project team. Hence, the project is not considered typical to the ones supported by Tema and is not discussed further.

    Project D: Elements of Tema in the technology strategy project

    The technology strategy project was the most comprehensive empirical study in the research project and the main source of inspiration for Tema. Consequently, most of the ideas in Tema can be traced back to observations from TEK-S.
    TEK-S development phase
    An overview of the development phase of TEK-S is found in table 10.5. A particularly prominent observation influencing on Tema is the number of intermediate and subsidiary artifacts developed.
     
    Aspects How the aspects were handled in the TEK-S project
    Intermediate artifacts
  • Modeling language developed as part of modeling. 
  • Enterprise models (created in TEK-S or introduced from other projects). 
  • Subsidiary artifacts were explanations, definitions, metaphors, narratives, assessment matrices, concept categories. 
  • Development activities
  • Ad-hoc approach to modeling. 
  • Creative and analytical techniques used for sense-making (brainstorming, concept clustering, QFD analysis). 
  • Frequent summarizing. 
  • Creation of numerous intermediate artifacts. 
  • Preparation of intermediate artifacts before exposure. 
  • Forums
  • Face-to-face meetings. 
  • Project database (Lotus Notes). 
  • Media
  • Paper and whiteboard in meetings. 
  • World Wide Web for alternative version of the strategy. 
  • Developers
  • Intimate knowledge of the enterprise. 
  • Little knowledge of Web as medium. 
  • Little knowledge of techniques. 
  • Competence in use of Lotus Notes (access the project database, email). 
  • Actual use of models
  • Shared understanding and holistic thinking. 
  • Structuring device (structuring discussions, division of work, sequencing of presentations). 
  • Table 10.5: The development phase of the TEK-S project
    A preparatory stage like team training was not observed in the strategy project. Neither was the use of recommended techniques like dependency analysis, term characterization or OPPO assessment. The inclusion of these in Tema was argued for in chapter 8.
    TEK-S dissemination phase
    Table 10.6 summarizes observations from dissemination of enterprise models in the strategy project. Exposed artifacts, dissemination activities, forums, media, and use of models are all within the confines of Tema (although Tema advocates a more extensive use of subsidiary artifacts and improved competence in use of Web as a medium).
     
    Aspects How the aspects were handled in the TEK-S project
    Exposed artifacts
  • Enterprise models in the Web version (partially integrated with other information, and with recognizable elements for a computer). 
  • Enterprise models with explanations in oral presentations. 
  • Narratives and terminology in Web version. 
  • Dissemination activities
  • Involvement of the audience. 
  • Presentation of models. 
  • Questioning. 
  • Feedback (agreement, disagreement). 
  • Forums
  • Face-to-face meetings. 
  • Shared project database. 
  • Bulletin board in Web version of the strategy. 
  • Media
  • Paper and foils in meetings. 
  • World Wide Web (digital medium). 
  • Audience
  • Intimate knowledge of the enterprise. 
  • Competence in use of Lotus Notes (access database, email). 
  • Little competence in accessing Web version. 
  • Actual use of models Structuring device (structuring presentations).
    Table 10.6: The dissemination phase of the TEK-S project

    In general, what Tema proposes is a more conscious use of techniques for enterprise modeling as compared to the activities observed in TEK-S. By conscious is meant that the actors are trained in using the techniques and in reflection upon their use.

    Concluding remarks on Tema and the empirical studies

    One observation from the above presentation is that the empirical evidence supporting Tema is mostly associated with the development phase of enterprise modeling. Less support is found for the dissemination phase (but that has neither been studied as carefully as the development phase). However, this is also reflected in the current version of Tema, being more elaborate on activities in the development phase.

    A timely question is if there are some elements in Tema that have not been observed in any of the projects. The answer is yes the team training stage has not been observed. None of the project groups have been prepared for or trained in use of a framework or techniques for enterprise modeling. Still, an indication of the need for preparations in some way is the disagreements over method that occurred in TEK-S. Without having learned the techniques properly, they were difficult to apply effectively.

    Some of the techniques in the pool have neither been observed (term characterization, dependency analysis, proper QFD analysis, deployment flow charting, and OPPO assessment). Term characterization, dependency analysis and QFD analysis have been introduced to meet particular needs, deployment flow charting was proposed as an example of a diagramming technique, and OPPO assessment was included based on general applicability. Still, empirical evidence of the qualities of these techniques is not found in the projects studied.