I. Introduction




General Remarks

It is common, in a book of this kind, to begin with a detailed and explicit definition of the subject- the Delphi technique. However, if we were to attempt this, the reader would no doubt encounter at least one contribution to this collection which would violate our definition. There is in addition a philosophical perspective that when something has attained a point at which it is explicitly definable, then progress has stopped; such is the view we hold with respect to Delphi.

In 1969 the number of Delphi studies that had been done could be counted in three digits; today, in 1974, tire figure may have already reached four digits. The technique and its application are in a period of evolution, both with respect to how it is applied and to what it is applied. It is the objective of this book to expose the richness of ,-hat may be viewed as an evolving field of human endeavor. The reader will encounter in these pages many different perspectives on the Delphi method and an exceedingly diverse range of applications.

For a technique that can be considered to be in its infancy, it would be presumptuous of us to present Delphi in the cloak of a neatly wrapped package, sitting on the shelf and ready to use, Rather, we have adopted the approach, through our selection of contributions, of exhibiting a number of different objects having the Delphi label and inviting you to sculpt from these examples your own view and assessment of the technique. For, if anything is "'.rue" about Delphi today, it is that in its design arid use Delphi is more of an art than a science.

However, as editors, we would be remiss if there were not some common thread underlying the articles brought together in this volume. As long as we restrict ourselves to a very general view, it is not difficult to present an acceptable definition of the Delphi technique which can be taken as underlying the contributions to this book:

Delphi may be characterized as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem.

To accomplish this "structured communication" there is provided: some feedback of individual contributions of information and knowledge; some assessment of the group judgment or' view; some opportunity for individuals to revise views; and some degree of anonymity for the individual responses, As the reader will discover, there are many different views on what are the "proper," "appropriate," "best," and/or "useful" procedures for accomplishing the various specific aspects of Delphi. We hope that the reader will find this book a rich menu of procedures from which he may select his own repast if lie should seek to employ the Delphi technique.

When viewed as communication processes, there are few areas of human endeavor which are not candidates for application of Delphi. While many people label

Delphi a forecasting procedure because of its significant use in that area, there is a surprising variety of other application areas. Among those already developed we find:

  • Gathering current and historical data not accurately known or available
  • Examining the significance of historical events
  • Evaluating possible budget allocations
  • Exploring urban and regional planning options
  • Planning university campus and curriculum development
  • Putting together the structure of a model
  • Delineating the pros and cons associated with potential policy options
  • Developing causal relationships in complex economic or social phenomena
  • Distinguishing and clarifying real and perceived human motivations
  • Exposing priorities of personal values, social goals

It is not, however, the explicit nature of the application which determines the appropriateness of utilizing Delphi; rather, it is the particular circumstances surrounding the necessarily associated group communication process: "Who is it that should communicate about the problem, what alternative mechanisms are available for that communication, and what can we expect to obtain with these alternatives?" When these questions are addressed, one can then decide if Delphi is the desirable choice. Usually, one or more of the following properties of the application leads to the need for employing Delphi:

  • The problem does not lend itself to precise analytical techniques but cart benefit from subjective judgments on a collective basis
  • The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds with respect to experience or expertise
  • More individuals are needed than can effectively interact in a face-to-face exchange
  • Time and cost make frequent group meetings infeasible
  • The efficiency of face-to-face meetings can be increased by a supplemental group communication process
  • Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured
  • The heterogeneity of the participants must be preserved to assure validity of the results, i.e., avoidance of domination by quantity or by strength of personality ("bandwagon effect")

Hence, for the application papers in this book tire emphasis is not on the results of a particular application but, rather, on discussion of why Delphi was used and how it was implemented. From this the reader may be able to transpose the considerations to his own area of endeavor and to evaluate the applicability of Delphi to his own problems.

Those who seek to utilize Delphi usually recognize a need to structure a group communication process in order to obtain a useful result for their objective. Underlying this is a deeper question: "Is it possible, via structured communications, to create any sort of collective human intelligence1 capability?" This is an issue associated with the utility of Delphi that. has not as vet received the attention it deserves and the reader will only find it addressed here indirectly. It will, therefore, be a subjective evaluation on his part to determine if the material in this book represents a small, but initial, first step in the long-term development of collective human intelligence processes.

Characteristics of the Delphi

The Delphi process today exists in two distinct forms. The most common is the paper-and-pencil version which is commonly referred to as a "Delphi Exercise." In this situation a small monitor team designs a questionnaire which is sent to a larger respondent group, After the questionnaire is returned the monitor team summarizes the results and, based upon the results, develops a new questionnaire for the respondent group. The respondent group is usually given at least one opportunity to reevaluate its original answers based upon examination of the group response. To a degree, this form of Delphi is a combination of a polling procedure and a conference procedure which attempts to shift a significant portion of the effort needed for individuals to communicate from the larger respondent group to the smaller monitor team).  We shall denote this form conventional Delphi.

A newer form, sometimes called a "Delphi Conference," replaces the monitor teat) to a large degree by a computer which has been programmed to carry out the compilation of the group results. This latter approach has the advantage of eliminating the delay caused in summarizing each round of Delphi, thereby turning the process into a real-time communications system. However, it does require that the characteristics of the communication be well defined before Delphi is undertaken, whereas in a paper-and-pencil Delphi exercise the monitor team can adjust these characteristics as a function of the group responses. This latter form shall be labeled real-lucre Delphi.

Usually Delphi, whether it be conventional or real-tune, undergoes four distinct phases. The first phase is characterized by exploration of the subject under discussion, wherein each individual contributes additional information he feels is pertinent to the issue. The second phase involves the process of reaching an understanding of how the group views the issue (i.e., where the members agree or disagree and what they mean by relative terms such as importance, desirability, or feasibility). if there is significant disagreement, then that disagreement is explored in the third phase to bring out the underlying reasons for the differences and possibly to evaluate them. The last phase, a final evaluation, occurs when all previously gathered information has been initially analyzed and the evaluations have been fed back for consideration.

On the surface, Delphi seems like a very simple concept that can easily be employed. Because of this, many individuals have jumped at trying the procedure without carefully considering the problems involved in carrying out such an exercise. There are perhaps as many individuals who have had disappointing experiences with a Delphi as there are users who have had successes. Some of the common reasons for the failure of a Delphi are:

  • Imposing monitor view's and preconceptions of a problem upon the respondent group by overspecifying the structure of the Delphi and not allowing for the contribution of other perspectives related to the problem
  • Assuming that Delphi can be a surrogate for all other human communications in a given situatio
  • Poor techniques of summarizing and presenting the group response and ensuring common interpretations of the evaluation scales utilized in the exercise
  • Ignoring and not exploring disagreements, so that discouraged dissenters drop out and an artificial consensus is generated
  • Underestimating the demanding nature of a Delphi and the fact that tire respondents should he recognized as consultants and properly compensated for their time if the Delphi is not an integral part of their job function

In addition to the latter problems associated with the Delphi technique another class of criticisms directed at Delphi is often raised in the literature. These are the "virtual" problems that do not in themselves affect the utility of the technique.2 Typical of these is the question of how to choose a "good" respondent group. This is, in fact, a problem for the formation of any group activity-committees, panels, study groups, etc. One has this problem no matter what communication mode is used; therefore, while it is a real arid significant problem, it is not a problem unique to Delphi. However, the nature of certain applications does, in fact, dictate special consideration of this problem and it is discussed in a number of articles. Another virtual problem frequently- arises when a particular Delphi design for a particular application is taken as representative of all Delphis, whereupon it is then observed that this design does not work for some other application. The problem here is that of making too explicit and restrictive a definition for Delphi. A third virtual problem is the honesty of the monitor team, and it is of the same concern as the honesty of any study or analysis group. In fact, there is probably more likelihood in most instances of exposure of misrepresentation in a Delphi summary than in a typical ,group study report. Finally, misunderstandings may arise from differences in language and logic if participants come from diverse cultural backgrounds. Since we consider these virtual issues to be somewhat irrelevant to Delphi per sc, we have made no attempt to give them special attention within this book. Other problems will be discussed in Chapter VIII.

It is quite clear that in any one application it is impossible to eliminate all problems associated with Delphi. There is, for example. a natural conflict in (he goal of allowing a wide latitude in the contribution of information and the goal of keeping the communication process efficient. It. is the task of the Delphi designer to minimize (here problems as much as possible and to balance the various communication "goals" within the context of the objective of the particular Delphi and the nature of the participants. Arriving at a balanced design for the communication structure is still very Much an art, even though there is considerable experience on how to ask and summarize various types of questions.

It can be expected that the use of Delphi will continue to grow. AS a result of this, one can observe that a body of knowledge is developing on how to structure the human communication process for particular classes of problems. The abuse, as well as the use, of the technique is contributing to the development of this design methodology.

Table 1 compares the properties of normal group communication modes and the Delphi conventional and real-time modes. The major differences lie in such areas as the ability of participants to interact with the group at their own convenience (i.e., random as opposed to coincident), the capacity to handle large groups, and the capability to structure the communication. With respect to time consideration there is a certain degree of similarity between a committee and a conventional Delphi process, since delays between committee meetings and Delphi rounds are unavoidable. Also, the real-time Delphi is conceptually somewhat analogous to a randomly occurring conference call with a written record automatically produced. It is interesting to observe that within the context of the normal operation of these communication modes in the typical organization--government, academic, or industrial the Delphi process appears to provide the individual with the greatest degree of individuality or freedom from restrictions on his expressions. The items highlighted in tire table will be discussed in more detail in many of the articles in this book,

While the written word allows fur- emotional content, the Delphi process does tend to minimize the feelings Mid information normally communicated in such manners as the tone of a voice, the gesture of a hand, or the look of an eve. In many instances these are a vital and highly informative part of a communication process. Our categorization of group communication processes is not meant to imply that the choice for a particular objective is limited, necessarily, to one communication mode. As the readers will see from some of the contributions to this book, there are instances where it is desirable to use a mix of these approaches.

Group Communication Techniques

Conference Telephone Call Committee Meeting Formal Conference or Seminar Conventional Delphi Real-Time Delphi
Effective Group Size Small Small to Medium Small to Large Small to Large Small to Large
Occurrence of Interaction by Individual Coincident with group Coincident with group Coincident with group Random Random
Length of Interaction Short Medium to Long Long Short to Medium Short
Number of Interactions Multiple, as required by group Multiple, necessary time delays between Single Multiple, necessary time delays between Multiple, as required by individual
Normal Mode Range Equality to chairman control (flexible) Equality to chairman control (flexible) Presentation (directed) Equality to monitor control (structured) Equality to monitor control or group control and no monitor (structured)
Conference Telephone Call Committee Meeting Formal Conference or Seminar Conventional Delphi Real-Time Delphi
Principal Costs Communications
  • Travel
  • Individual's Time
  • Travel
  • Individual's Time
  • Fees
  • Monitor Time
  • Clerical
  • Secretarial
  • Communications
  • Computer Usage
Time-urgent considerations Forced delays Forced delays Time-urgent considerations
Other Characteristics
  • Equal flow of information to and from all
  • Can maximize psychological effects
  • Efficient flow of information from few to many
  • Equal flow of information to and from all
  • Can minimize psychological effects
  • Can minimize time demanded of respondents or conferees

The Evolution of Delphi

The Delphi concept may be viewed as one of the spinoffs of defense research. "Project Delphi" was the name given to an Air Force-sponsored Rand Corporation study, starting in the early 1950's, concerning the use of expert opinion.3 The objective of the original study was to "obtain the most reliable consensus of opinion of a group of experts ... by a series of intensive questionnaires interspersed with controlled opinion feedback."

It may be a surprise to some that the subject of this first study was the application of "expert opinion to the selection, from the point of view of a Soviet strategic planner, of an optimal U. S. industrial target system and to the estimation of the number of A-bombs required to reduce the munitions output by a prescribed amount." It is interesting to note that the alternative method of handling this problem at that time would have involved a very extensive and costly data-collection process and the programming and execution of computer models of a size almost prohibitive on the computers available in the early fifties. Even if this alternative approach had been taken, a great many subjective estimates on Soviet intelligence and policies would still have dominated the results of the model. Therefore, the original justifications for this first Delphi study are still valid for many Delphi applications today, when accurate information is unavailable or expensive to obtain, or evaluation models require subjective inputs to the point where they become the dominating parameters. A good example of this is in the "health care" evaluation area, which currently has a number of Delphi practitioners.

However, because of the topic of this first notable Delphi study, it took a later effort to bring Delphi to the attention of individuals outside the defense community. This was the "Report on a Long-Range Forecasting Study," by T. J. Gordon and Olaf Helmer, published as a Rand paper in 1964.4 Its aim was to assess "the direction of long-range trends, with special emphasis on science and technology, and their probable effects on our society and our world." "Long-range" was defined as the span of ten to fifty years. The study was done to explore both the methodological aspects of the technique and to obtain substantive results. The authors found themselves in "a near-vacuum as far as tested techniques of long-range forecasting are concerned." The study covered six topics: scientific breakthroughs; population control; automation; space progress; war prevention; weapon systems. Individual respondents were asked to suggest future possible developments, and then the group was to estimate the year by which there would be a 50 percent chance of the development occurring. Many of the techniques utilized in that Delphi are still common to the pure forecasting Delphis being done today. That study, together with an excellent related philosophical paper providing a Lockean-type justification for the Delphi technique,5 formed the foundation in the early and mid-sixties for a number of individuals to begin experimentation with Delphi in non-defense areas.

At the same time that Delphi was beginning to appear in the open literature, further interest was generated in the defense area: aerospace corporations and the armed services. The rapid pace of aerospace and electronics technologies and the large expenditures devoted to research and development leading to new systems in these areas placed a great burden on industry and defense planners. Forecasts were vital to the preparation of plans 'as well as the allocation of R&D (research and development) resources, and trend extrapolations were clearly inadequate. As a result, the Delphi technique has become a fundamental tool for those in the area of technological forecasting and is used today in many technologically oriented corporations. Even in the area of "classical" management science and operations research there is a growing recognition of the need to incorporate subjective information (e.g., risk analysis) directly into evaluation models dealing with the more complex problems facing society: environment, health, transportation, etc. Because of this, Delphi is now finding application in these fields as well.

From America, Delphi has spread in the past nine years to Western Europe, Eastern Europe, and the Far East. With characteristic vigor the largest Delphi undertaken to date is a Japanese study. Starting in a nonprofit organization, Delphi has found its way into government, industry, and finally academe. This explosive rate of growth in utilization in recent years seems, on the surface, incompatible with the limited amount of controlled experimentation or academic investigation that has taken place. It is, however, responding to a demand for improved communications among larger and/or geographically dispersed groups which cannot be satisfied by other available techniques. As illustrated by the articles in this book, aside from some of the Rand studies by Dalkey, most "evaluations" of the technique have been secondary efforts associated with some application which was the primary interest. It is hoped that in coming years experimental psychologists and others in related academic areas will take a more active interest in exploring the Delphi technique.

While many of the early articles on Delphi are quite significant and liberally mentioned in references throughout this book, we have chosen to concentrate on work that has taken place in the past five years and which represents a cross section of diverse applications.

Although the majority of the Delphi efforts are still in the pure forecasting area, that application provides only a small part of the contents of this volume. Chapters II and III of this book consist of articles which provide an overview of the Delphi technique, its utility, the underlying philosophy, and broad classes of application.

Chapter IV takes up recent studies in the evaluation of the technique. Precision and accuracy are considered in this context. Between the reviews, articles, and associated references, the reader should obtain a good perspective on the state of the art with respect to experimentation.

Chapters V and VI describe some of the specialized techniques that have evolved for asking questions and evaluating responses. Foremost among them is cross-impact analysis (Chapter V). This concept reflects recognition of the complexity of the systems dealt with in most Delphi activities, systems where "everything interacts with everything." In essence, these sections explore the quantitative techniques available for deeper analysis of the subjective judgments gathered through the Delphi mechanism.

The effect computers can have on Delphi and speculations on the future of the technique itself are discussed in Chapter VII. The book concludes with a summary of pitfalls which can serve the practitioner as a continuing checklist (Chapter VIII).

We have striven to avoid making this volume a palimpsest of previously published papers: all but four of the articles have been especially prepared for this work, The four reprinted articles were selected from the journal Technological Forecasting and Social Change, a rich lode of material on Delphi. The extensive bibliography in the Appendix provides a guide to those who wish to probe the subject further. It is thus our hope that this volume will serve the reader as a useful reference work on Delphi for a number of years.

1 We refer to "intelligence" in this context as including attitudes and feelings which are part of the process of human motivation and action.

2 See, for example, Cordon Welty, "A Critique of the Delphi Technique," Proceedings of the American Statistical Association, 1971. Social Statistics Section.

3 N. Dalkey and O. Helmer, "An Experimental Application of the Delphi Method to the Use of Experts," Management Saence 9, No. 3 (April 1963), p, 458.

4 Rand Paper P-2982. Most of the study was later incorporated into Helmer's Social Technology, Basic Books, New York, 1966.

5 O. Helmer and N. Rescher, "On the Epistemology of the Inexact Sciences," Project Rand Report R-353, February 1960.







©2002 Murray Turoff and Harold A. Linstone