HAROLD A. LINSTONE and MURRAY TUROFF
In this chapter we sample the rich menu of applications. The purposes of the Delphis are as varied as the users. Seven authors focus on specific planning tasks in the areas of government and business. Additional studies are also sketched in this introduction ("Comments on Other Studies").
The four articles covering this field address national, regional, and organizational planning problems. Turoff deals with the basic concept of the Policy Delphi and reviews several efforts of this type. Ludlow's concern is resource management in the Great Lakes region and a major aim is to establish improved communications between technical experts and interested citizens. Jillson focuses on the use of Delphi as an integral instrument in national drug abuse policy formulation, operating from three distinct perspectives: "topdown", "bottom-up", and "issue oriented". Jones explores priorities in System Concept Options for the U.S. Air Force Laboratories, with emphasis on comparing views of four in-house organizations competing for funds.
All four move beyond the use of Delphi as a forecasting tool and stress its value as a communications system for policy questions. A policy question is defined here as one involving vital aspects, such as goal formation, for which there are no overall experts, only advocates and referees. Its resolution must take into consideration the conflicting goals and values espoused by various interest groups as well as the facts and staff analyses. It should be clearly understood that Delphi does not substitute for the staff studies, the committee deliberations, or the decision-making. Rather, it organizes and clarifies views in an anonymous way, thereby facilitating and complementing the committee's work,
Whereas Turoff's panelists constitute a homogeneous group, Ludlow seeks to establish a communication process between the potential users of new knowledge and a team of interdisciplinary researchers. He raises a point which is of concern for Delphi studies generally. The probability used is of the personal or subjective type; it can be interpreted as a "degree of confidence". Scientists and engineers are brought up on a different kind of probability-frequency ofoccurrence, i.e., the limit of the ratio of the number of successes to the total number of trials as the latter approaches infinity. Thus the frequency type of probability assumes repeatability of the experiment (e.g., tossing a coin). But the subjective probability has meaning even if an event can occur only once. A boxing match is a one-time event; the odds usually associated with such a match indicate the "degree of confidence" in the outcome on the part of informed bettors. Both definitions are mathematically valid and have been used to develop distinct probability theories. Businessmen intuitively use the "degree of confidence" concept and therefore have no built-in resistance when faced with it in Delphi questionnaires.
Ludlow also presents an evaluation of Delphi by the three participating groups-technicians, behaviorists and decision-makers. Not surprisingly the latter prove to be the strongest proponents of the technique. They are, after all, the one group which must regularly seek a consensus and usually has to make decisions on complex issues without adequate information.
Miss Jillson's article is a progress report on a study designed to develop drug abuse policy options, explore the applicability of the Policy Delphi to questions of social policy generally, and determine the practicability of using it on both an "as-needed" and "on-going" basis (i.e., indefinite duration). The participants include researchers, administrators, and policymakers-both in the field and in impacted areas (e.g., police chiefs). A unique feature is the use of the three perspectives. In the "top-down" approach, the objectives for the next five years are emphasized; the "bottom-up" approach deals with factors which control transition between various states or levels of drug use and employs a matrix format; the "issue-oriented" approach crystallizes statements of policy issues in "should/should not" form.
Jones' Delphi reflects the typical consensus or Lockean oriented approach in its design to gain consensus among representatives or organizations subject to different pressures in their competition for limited financial resources. He uses 61 senior managerial and technical personnel (both military and civilian) representing most departments in the four organizations. Different organizational viewpoints are apparent although significant self-interest biases are not detectable. This effort contrasts very nicely with the Kantian nature of Ludlow, the Hegelian approach of Turoff, and the mixed Kantian-Hegelian work of Jillson.
Business and Industry
In a corporate environment Delphi fills several roles. Bell-Canada's Lawrence Day lists three: educational device for senior management, environmental trend background material for technological planners in research laboratories, and trading material for use with planner-counterparts in other organizations.
If the corporation is large and diversified it may have the analytical staff to run the study and the expertise to form the panels "in house". TRW used 140 of the "the most imaginative and creative members of TRW's technical staff of more than 7,000 graduate scientists and engineers". It should be noted that in the hierarchical environment of a business, the rate of response or participation tends to be higher than average. A university professor may feel no compunction about ignoring questionnaires or giving perfunctory or dilatory answers; an employee has stronger incentives to cooperate in a company exercise. Goldstein's experience in this regard is echoed in numerous other cases.
The ability to conduct a Delphi without bringing the respondents together physically is another advantage in the large corporation with units spread over a wide geographical area (e.g., multinational corporations). Overseas personnel can be drawn into a Delphi with relative ease and at minimal cost.
Day covers four Delphis performed by Bell-Canada's Business Planning Group: Educational Technology, Medical Technology, Business Information Processing Technology, and Home Communications Services.
In cases where the corporation does not have the expertise in either the subject of the forecast or the Delphi procedure, it may turn to a professional consulting organization. Enzer's article describes a Delphi on the subject of plastics undertaken for a client by the Institute for the Future. The field of materials for the future is a particularly difficult one for the forecaster. First, the number of possibilities is overwhelming (e.g., tailormade plastics). Second, in a hierarchy (e.g., relevance tree) which has metasystems at the top, followed by systems, subsystems and components, materials are close to the bottom. This means that a given material can branch upward in many ways, i.e., it can find use in a multitude of old and new systems by 1985, Third, each substitution of a new material for an old one may prove sensitive to slight variations in the relative prices, with major deviations resulting in the 1985 forecasts. These factors strongly suggest the desirability of using at least two entirely different methods of forecasting to provide a reasonable degree of confidence.
Nancy Goldstein gives us a hint of this in her article on a Delphi covering steel and ferroalloy materials. She compares the Delphi results with those of a conventional panel study conducted simultaneously. There is considerable agreement in the forecasts. Ideally one would seek a comparison between methods which are more radically different than a conventional panel and a Delphi. However, this particular comparison makes one startling methodological point: the conventional panel study brooks no uncertainty and no areas of disagreement in its forecasts; it presents a definitive view and a set of conclusions, The Delphi, on the other hand, reflects the uncertainties and highlights the differences among its participants. It is more concerned with exploring minds than setting down precise recommendations (cf. Scheele, Chapter II, C).
The Delphis by Day, Enzer, and Goldstein are largely Lockean in nature. However, the use of panels of differing backgrounds by Day has Kantian aspects, and Goldstein suggests Hegelian over-tones.
Another use of Delphi has recently evolved in business in connection with risk analysis.1 It concerns the uncertainties associated with new projects or investments. Normally decisions must be made in the absence of adequate information. The potential market for a new product is uncertain and the development costs may exceed the engineers' estimates. Marketing personnel in an operating unit of the corporation frequently exhibit a glowing optimism which neglects to credit the competition with high intelligence or quick reaction capability. Engineers tend to assume that the cost of a complex product is a linear function (i.e., sum) of the cost of the components which comprise it. They neglect the interactions which result in nonlinear behavior: the total cost is much greater than the sum of the parts. Thus the cost is grossly underestimated. One recent study of a large number of defense-oriented development projects indicated approximately a 50 percent chance of 50 percent cost overrun.2
Delphi may be used to advantage to provide input to the risk analysis. The most critical part of such analysis is the subjective probability distribution assumed for the uncertainties. Delphi can serve to probe the views of personnel connected with the project as well as outsiders (e.g., corporate offices or other units), senior executives as well as junior engineers and scientists. The anonymity is particularly valuable in a highly structured environment where individuals may feel constrained in expressing their own views.
Comments on Other Studies
Many applications of Delphi are carried on as integral parts of planning projects or as staff work of a proprietary nature. Therefore, a considerable amount of very good Delphi work has not been published in the open literature or in a form adequately explaining the process used. Before proceeding to the separate application papers it is worthwhile to note a number of Delphi studies of a unique nature which are not yet otherwise documented explicitly for those interested in the technique itself.
The Delphi method has been applied extensively in the medical area. The initial work by Bender et al.3 was largely of a straightforward forecasting variety. However, a number of Dr. Bender's Delphis did deal with estimating the necessity and desirability of potential medical research accomplishments. And it did not take very long before the application was broadened to include unique objectives other than future projections.
Dr. John W. Williamson, of John Hopkins University's School of Hygiene and Public Health, has utilized Delphi extensively for estimating historical data. Typical questions deal with determining the incidence of a given disease and the estimated rate of success in utilizing various treatment methods. This is, of course, an area where current reporting practices do not give reliable data owing to differing standards across the country and the effect of multiple complications, e.g., death due to pneumonia while ill with cancer. Usually Dr. Williamson would ask respondents for their best estimate of a number, then a low and high value which would shock them. Also requested would be an estimate of their confidence in the estimate and a statement whether their estimate was based upon a particular source, such as an article they had read. In a number of these exercises questions were asked which dealt with the results of unpublished new clinical studies. In this manner one could observe how well the Delphi panelists actually did on part of the exercise and utilize this insight to gain an impression of their capability for providing answers to the rest of the exercise.
An excellent and, perhaps, classic example of this is a study Williamson did at the Philips Electric Corporation Plant in Eindhoven, Holland, in 1970. Approximately 50 doctors who are involved with the company's medical program for the 36,000 employees participated in the Delphi. The first part of the Delphi asked the physicians to estimate the percent of male employees absent from work due to sickness during differing intervals of time. The population was further divided by young, old, blue collar, white collar. This required sixteen estimates from each doctor. When the real data was collected from the computer files three months later it was found that 12 of the 16 estimates were within 10 percent error and the other 4 within 30 percent error. Of course, this represented only a small portion of the exercise as the real objective was to determine what effect various potential changes to the health care program would have on the absenteeism rate. However, one could see that the physicians involved had a good feel for the situation as it existed. Also it was possible to examine how well various subgroups did, e.g., general practitioners vs. specialists. Dr. Williamson has conducted four major studies of the above type (involving validity checks) with a total of approximately 200 respondents over the past four years.
Professor Alan Sheldon of the Harvard Medical School, together with Professor Curtis McLaughlin of the University of North Carolina Business School, did a Delphi in 1970 on the Future of Medical Care. A unique feature of this Delphi was the process of combining the events evaluated by the respondents into scenarios in the form of typical newspaper articles. The respondents were then asked to propose additions or modifications to the scenarios and give their reaction to the scenario as a whole. This concept of utilizing the vote on individual items to group events into scenarios classed by such things as likelihood and/or desirability has become a standard technique.
Also with respect to scenarios it has become fairly common to provide the respondents on a forecasting Delphi with a scenario or alternative scenarios providing a reference point on considerations outside the scope of the Delphi but having impact on the subject of the inquiry. For example, in forecasting the future of a given industry the respondents might be given a "pessimistic," "optimistic," and "most likely" scenario on general economic conditions and asked that their estimates for any question be based on each alternative in turn.
While there have been a number of Delphis on the general future of medical care, a recent Delphi by Dr. Peter Goldschmidt of the Department of Hygiene and Public Health, Johns Hopkins University, dealt with the future of health care for a specific geographical entity, Ocean City, Maryland. The problem in health planning in this case is the tremendous influx of vacation people in the summer months. In order to examine the future growth of the Ocean City area and its resulting medical needs, it was felt the Delphi panel should include individuals who resided in the area and simultaneously worked in endeavors related to the mainstream of the local economy-recreation. Therefore, the Delphi involved long-time residents, hoteliers, bar owners, real estate dealers, and civic officials as well as the usual "experts" such as the regional planning people from local government and industry. This widening, or broadening, of the concept of "experts" to that of "informed" is becoming quite customary in the application of Delphi. In this particular Delphi, Dr. Goldschmidt was able to check the "intuition" of his respondents by comparing their estimates on vacation populations in Ocean City currently with estimates he could analytically infer from the processing load history of the sewage plant that serves the area. As in Williamson's case the results were quite good.
A superb example of the Delphi technique was carried out by Richard Longhurst as a master's thesis at Cornell University.4 The Delphi attempted to assess the impact of improved nutrition, family income, and prenatal care on pregnancy outcome in terms of birth weight and the resulting I.Q. and intellectual development of young children. The resulting output were of a form useful for incorporation into cost-benefit analyses of government programs to improve the nutrition of pregnant mothers and young children. This is, of course, an excellent example where data exist to indicate that malnutrition in the mother or young child has some degree of impact on the long-term intellectual capabilities of a child. However, this evidence is not of direct quantitative utility to the type of analyses an economist would like to perform in evaluating a government program, intellectual development comprised experts in child psychology and development; the other, in the area of pregnancy outcomes, was composed of experts in pregnancy, nutrition, and medical care. The group was given a specific group of low-income mothers in a depressed urban area as the population they were concerned with. This was a real group on which a good deal of data on socioeconomic status were available. In the first round the respondents were asked to sort out the relative importance of environmental components that might be manipulated by the introduction of a government program. The second round presented a set of feasible intervention programs which related to the factors brought out in the first round, They were then asked to estimate for each program the resulting incidence of low birth weight and the average LQ. score of 5-year-old children resulting from the pregnancies under the program. "They provide the baseline data on these parameters for the target group as it currently existed. Certain programs were estimated to reduce the incidence of low birth weight from 15% to 10% and to raise the five-year I.Q. scores from 85 to 100 points. The shift in I.Q. can then be used to shift the average education and earning power of the children when they are adults. This then is translatable into dollar benefits that can be used to compare the merits of alternative programs in this area. The Delphi itself involved respondents for three rounds and questionnaires were tightly designed to take about fifteen minutes to fill out.
The area of trying to translate scientific knowledge into an informed judgment on evaluating and analyzing decision options is clearly a potential one for the Delphi method.
Another effort in health care planning is the work of Professor David Gustafson at the University of Wisconsin. This work has been tied into the Governor's Health Planning and Policy Task Force effort. One of the Delphis Professor Gustafson conducted dealt with delineation of the current barriers to the performance of research and development in the health services area-the rather interesting topic of trying to clarify what the real "problem" is. The respondents were asked to delineate barriers of three types: (1) solution development barriers; (2) problem selection barriers; (3) evaluation barriers. For each barrier the group developed comments, implications, and possible reactions or corrective measures. A vote was taken on the significance of each barrier. This was an excellent example of utilizing Delphi to try and isolate the significant part of the problem. Very often, in planning areas, preconceptions by one individual lead to tremendous efforts on the wrong problem. The specification of a particular problem usually predetermines its method of investigation and at times its conclusions.
The use of Delphi for regional planning has probably become popular because of the feeling that there is a necessity to establish better communications among many individuals with diverse backgrounds.
In this area a significant number of Delphis have been conducted by various Canadian government agencies such as Health and Welfare, Department of Public Works, Department of the Environment, and the Postal Service, to name a few. Most of these are being done by internal staff and very often they tend to be short, focused on a very specific issue, and require a diverse backgroud of respondents. A good example is one done in 1974 by Madhu Agowal on "The Future of Citizen Participation in Planning Federal Health Policy." The Delphi sought to explore and delineate specific options for citizen participation and to determine the consequences of such programs.
There has been very active use of the Delphi in the educational establishment and a survey of that work may be found in an article by Judd.5 Curiously almost all educational Delphis have been confined to administrative matters and hardly considered as a teaching tool. It is not surprising that educationalists are enthusiastic about the method. There is a high degree of participatioe planning in higher education. Authoritarianism is eschewed to such an extent that anarchy sometimes results. There is also an entrenched bureaucracy which feeds on well-structured procedures and questionnaires of all kinds.
Delphi is used for several aspects of administrative planning: general goals, curricula, campus design, and development of teacher ratings and cost-benefit criteria. Judd describes many of the problems encountered in the use of Delphi in this environment.
However, to find a clue to what may prove to be the most serious difficulty, we must turn to the conclusion of a (non-Delphi) survey of school administrators conducted recently by R. Elboim-Dror6in Israel on the subject of education in the year 2000:
The lack of creative imagination as revealed in this study, the limited number of new alternatives and innovating ideas expressed by the subjects, and especially the students (of school administration), are a serious sign.
In such an atmosphere Delphi can be as barren as most of the paperwork which traditionally suffocates educational bureaucracies. When the educational field begins to see Delphi in the deeper context discussed in Chapter 11, when it starts to consider Delphi as an educational tool as well as a planning tool, then it may be able to escape this trap.
If there is a single message in the philosophy of Singer (see p. 33), it is that the past and the present are often as hard to interpret, or conjecture about, as the future. It is therefore not surprising to see the Delphi method applied to these areas as well as to the future. A recent Delphi devoted to examining the past is an exercise by Professor Russell Fenske of the University of Wisconsin. This involved about twenty-five leading researchers in the field of operations research who utilized the Delphi to review the "state of the art of industrial operations research": Each participant could propose significant contributions to the literature in the areas of theory, applications, and economic impact. The group then voted on each item for importance and impact. Also gathered were brief comments on the significance of an item and suggestions for important areas of future research.
Michael Marien has used the Delphi process to elicit from a panel of 14 futurists the most significant books (a "hot list") on the future. Considering the volume of material being produced in most professional areas, these particular applications of Delphi are likely to become quite popular in the future.
A similar concept has been applied in some corporate or organizational uses of I)elphi, where the study examines historical performance or factors that have affected the market place for a particular item. The objective is usually to focus on 50 to 100 key items out of hundreds of candidates so that a concise summary of the historical perspective can be prepared for management. This is largely the process of getting a group to filter out the signal of real information from the multitude of communications or noise that may exist on a particular complex topic. This concept is very similar to what Professor James Bright refers to as the monitoring function in technological forecasting, and an excellent example of a Delphi on "Events Leading to the Limitation or Elimination of the Internal Combustion Engine" forms the basis for an exercise in his recent book.7 The example, based upon a Delphi conducted in 1969 by a chemical company, was, to the best of our knowledge, the first which dealt exclusively with evaluating the past.
A much deeper systemic study of the past is envisioned in the "retrospective futurology" approach which applies dynamic programming to historic societies such as the city-state of Athens. The "hyper-sophisticated polling of experts" mentioned by Wilkinson in conjunction with this concept8 strongly suggests the Delphi method.
1 See D. B. Hcrtz, "Risk Analysis in Capital Investment," Harvard Business Review, Jan-Feb. 1964, p. 95, and 1). H. Woods, "Improving Estimates That Involve Uncertainty," Harvard Business Review, Jul-Aug. 1966, p. 91.
2 S. H. Browne, "Cost Growth in Military Development and Production Programs," unpublished, Dec. 1971.
3 Bender, Strack, Ebright & Von Haunalter, Delphic Study Examines Developments in Medicine, Futures, June 1969. George Teeling-Smith, Medicine in the 1990's, Office of Health Economics, England, October 1969.
4 Richard Longhurst, "An Economic Evaluation of Human Resources Program with Respect to Pregnancy Outcome and Intellectual Development," M.S. thesis, Cornell University, Ithaca, N.Y. December 1971.
5 R. C. Judd, "The Use of Delphi in Higher Education," Technological Forecasting and Social Change, 4, 173 (1973).
6 R. Elboim-Dror, "Educators' Image of the Future," Paper presented at the Third World Future Research Conference, Bucharest, September 1972.
7 James Bright, A Brief Inlroduction to Technology Forecasting: Concepts and Exercises, Pemaquid Press, Austin, Texas, 1972.
8 J. Wilkinson, R. Bellman, and R. Garaudy, "The Dynamic Programming of Human Systems," Occasional Paper, Center for the Study of Democratic Institutions, MSS Information Corp., New York, 1973, p. 29.