Department of computer and Information Science
New Jersey Institute of Technology
Newark NJ, 07102
Short Paper prepared for the Workshop W9 Strategies for Collaborative Modeling and Simulation at CSCW 96, Boston, November 16.
In the early days of linear programming we developed input formats which allowed individuals with no mathematical understanding of linear programming to developed models. The results were that many non linear situations where modeled and often resulted in misleading decision support information.
One of the early breakthroughs on the use of computers was Forester's Systems Dynamics approach to the development of non liner feedback models. Few people are aware that Forester also developed simple procedures and front end formats to teach this modeling process to public school children and to facilitate the development of models by classes.
If one remembers that the “Club of Rome Model” was a collaborative effort to develop a predictive world model utilizing System Dynamics, one should realize that a modeling effort, for all its specificity and exactness can result in results that can be considered quite suspect and even controversial.
One of the earliest truly collaborative efforts was by John Warfield (1976) and his development of Interpretive Structural Modeling (ISM) as an approach to the group modeling of the relationships among goals, subgoals, and objectives. This relied on the ability of the contributors to estimate whether a relationship (0 or 1) existed between two goals. The resulting graph can then be analyzed to develop a linear hierarchy of clusters of nodes from the original graph. Warfield would bring a terminal to meetings and the computer analysis would be supplied as the group worked. Clearly individuals can answer the simple question of whether two items are related but when one puts together a resulting graph of hundreds of items it is impossible for the average human to make any sense of it. This is why various forms of modeling and network analysis tools (ISM, cluster analysis, multidimensional scaling, cluster analysis methods (Anderberg, 1973), cross impact, decision payoff matrices, etc.) can and do play such a key role in the collaborative process.
However, each method has its specific limitations and criteria that governs its appropriateness to a given situation. The ISM method can be demonstrated to have a great deal of sensitivity of the existence or non existence of a particular link in the graph. Therefore if there is any controversy among the group about the existence of this single relationship among the hundreds that can exist the result can have tremendous variation in changing the relative positions of most of the objects in the resulting clusters. This further points out the need for incorporating the ability to do sensitivity analysis on any result in order for the individual estimator or the group to test the range of validity for their results.
One area that has always emphasized various collaborative model building processes is the Delphi Method (Linstone and Turoff, 1975). Methods such as Cross Impact Analysis (the relationships among one time events utilizing subjective probability estimates) was evolved to function as an integral part of a Delphi process. It has also been extensively used as a collaborative model building face to face meetings. The Delphi Method books reviews three different analytical formulations that may be used support the analysis of inputs for this type of model (papers by Dalkey, Kane, and Turoff). Cross Impact Analysis is very relevant to such issues as relating decisions to consequences without having to make the linearity assumptions of conversion to dollars inherent in the more constrained decision payoff type matrix models.
Many of the modeling approaches may be viewed as special cases of the general area of “"structural modeling"” (Lendaris, 1980; Geoffrion, 1987). The concept of structural modeling is the formulation of model structures that users can utilize to build general purposes models without having to learn to program. The Lendaris tutorial oriented article should be required reading for those interested in understanding the model building process.
The second problem is consistency in the estimates made by the respondents. For example if too people say a certain even is "very likely".” Does one mean .7 probability and the other mean .9 probability. In the classical area of the analysis of subjective judgments this is the problem of "scaling" and the collaborative model builder has to be very cognizant of what types of scaling methods (Torgerson, 1957) maybe needed to insure the mutual understanding among the collaborators. The more multidisciplinary the group is the more likely this is a major concern. Unfortunately most scaling methods have been evolved for one time survey applications and not for iterative feedback in a group communication processes. There is still considerable adaptation and tailoring of the methods that are required for this application area.
Even when groups are very homogenous there are major consistency problems. In a Delphi on the future of the Steel Industry (paper by Goldstein, in the Delphi Method book) involving about 45 planners in the industry, the respondents were given a flow model developed by three experts. This was a model of about 45 flow links and only about 15 are regularly measures. The respondents were asked to fill in the missing 30 estimates for the flows in the industry the prior year. Instead of doing this as asked about 25 of the respondents redrew the model because they DID NOT AGREE with what the initial three experts had conceived. Too many computer people have an idealized impression that model building of a real world situation should be a straightforward process of collecting known knowledge in that domain.
This means further that the quantitative aspects of model building must be linked and justified by a foundation of qualitative observations that serve to justify and explain both consistencies and consensus. This is one of the reasons why the integration of Hypertext functionality (Rao & Turoff, 1990) into collaborative model building holds a great deal of promise for the integration of qualitative and quantitative aspects of model building.
Those of us that have worked with the Delphi method know that one of the consequences of its use is that one can make some fairly good judgments about the relative expertise of the respondents as a result of the process. Ethically this is one of the reasons why those that conduct Delphis need to have strong ethical values about the maintenance of the property of anonymity in the compilation of the results. One may utilize various methods to add confidence or scoring methods to the individual judgments into an appropriate group result.
When it comes to model building the potential for possible misuse is even more severe. There is rather convincing results in the literature that points out one can judge the expertise of an individual by the complexity of the model structure that they express (Hopkins, et. al., 1987). It has to be clear to the contributor just how his contributions will be used and the degree of confidentiality that will be available to the participants.
Work on collaborative model building should rest on a strong foundation in areas such as structural modeling and scaling. Furthermore it should take account of prior experience and design work with collaborative modeling in such areas of Delphi and Nominal Group Techniques.