10.1    Introduction to the Evaluation

Section 10.1 provides some reflections on approaches to validation of Tema, followed by an outline of the chapter.

10.1.1    Reflections on validation of TEMA

Validation is here seen as the act of convincing the readers of the conformance between predicted successful application of Tema and actual successful application in real world projects. Hence, the most persuasive argument in favor of Tema would probably have been independent practitioners reporting on successful application in a number of projects. As noted in section 1.3.1, this has not been practically feasible, and other approaches to building confidence have to be sought, as outlined in section 10.1.2.

Even if practical application is the preferred way to validation, it is not free of problems. Checkland and Scholes (1990:299), in their discussions on practical application of SSM, claim that a methodology like SSM is undecidable (italics added):

"It is an irony of work of the kind described here that the obvious question posed by the very existence of an approach to bringing about improvements in a problem situation cannot be answered! When people hear for the first time about SSM, or systems engineering, or cognitive mapping, or any such methodology, they naturally ask: Is it any good? Does it work? And they are very frustrated when they develop an understanding of the nature of methodology sufficient to realize that the question is unanswerable, that methodology is in fact undecidable."
The main argument they pose to support their claim is that SSM is a methodology a set of principles for method. Consequently, SSM as a prescriptive approach is inseparable from actual use by a particular user in a particular context, introducing actions that are not under the control of SSM. The same can be claimed on behalf of Tema it is a framework that has to be interpreted, adapted and enacted in a particular setting.
Tema advocates the active construction of metaphors as a means to improve sense-making. If a project group experience successful use of metaphors, they may interpret it as an indication of the effectiveness of the framework. If their use of metaphors is not successful, they may be accused of creating the "wrong" metaphors (and thereby blaming the failure on the project team and not on Tema). 

10.1.2    Outline of the presentation

Three different routes will be followed to convince the readers that Tema has the required qualities to be an effective framework for enterprise modeling: Although the focus is on increasing the confidence in Tema, perceived weaknesses are also discussed. The purpose of the evaluation is validation, not glorification.