CHAPTER 3-4 ASSESSING CONSEQUENCES AND LIKELIHOODS How much would it cost to construct a building like that one over there? How many baseball gloves should your store plan to sell this spring if you price and advertise them as you did last year? What is the probability that a vaccine against AIDS will be discovered in the next five years at the current rate of government support of AIDS research? What are the chances that you will be a good enough violinist to make a living playing professionally? Estimating the likelihoods of the good and bad consequences that may follow a given course of action, and attaching values to them, are crucial activities in determining the overall "present value" of the option being considered. There usually are many possible ways of making estimates, based on some blend of abstract theoretical ideas, information already available, information you will develop from scratch, experience and ideas of other people, and other modes of operation of human brains in your and others' heads. Sometimes you can do most or even all of the job with systematic scientific procedures of the sort described in the previous chapter. For example, if you wish to know the probabilities of various quantities of baseball gloves being sold in your store in April, a systematic study of the April sales in the past four or five years, combined with the forecasting techniques described in Chapter 4-4, will probably provide a very reliable "probability distribution", as the result is called. But if this is the first year your store is open, you have no history to analyse scientifically, and hence you must use other methods of the sort described in this chapter. Or, how good a chance is there that a space launch will blow up before getting into orbit? Or that a vaccine against AIDS will be discovered before the year 2005? Or that the Roman Catholic Church will allow women to serve as priests as of the year 2020? These questions are of a different sort than are usually tackled with standard methods of empirical research. This chapter discusses a few selected issues in the practice of guesstimating answers to such questions. The estimation task is closely related to the discussions in Chapters 3-2 and 3-3 of experts, libraries, and scientific inquiry, and to the assessment of your basic values discussed in Chapter 2-1. It also overlaps the discussion of forecasting in Chapter 4-4. Sometimes an estimate is expressed as a single physical quantity, as for example the response to "How many people do you think will show up for the concert tonight?" And sometimes the estimate is expressed as the probability of a given "simple" event such as the discovery of an AIDS vaccine. (Calculating the probability of complex events from simple events is discussed in Chapter 1-4.) This chapter concentrates on the special problem of phrasing the estimate in the form of the probabilities associated with the event. The main modes of thought that enter into estimation may be characterized as theoretical (or engineering) and experiential (or scientific). The estimation may be any possible combination of the two modes. If the process is mainly experiential and if the experience is systematically gathered, the process is that described in Chapter 3-3. The remarks in this chapter refer to the theoretical mode, and its combination with unsystematically- gathered experience using various heuristics. It is common for theory and experience to contend for dominance, either within your own thinking or between two people working on the same estimate. In my experience, people tend to give too much weight to abstract thinking and too little to the available data. When a body of experience is available, I recommend that you lean on it to the greatest extent possible, and almost never push it aside in favor of theorizing. Of course there are situations when it is prudent to ignore the data completely, and rely wholly on the theory, but such situations are rare. SOME PRINCIPLES FOR ESTIMATING COSTS AND BENEFITS 1. Ensure that all the relevant costs and benefits are included and none are forgotten. It is amazingly easy to overlook a crucial factor -- for example, the raw materials needed, or the cost of taxes -- when estimating the cost of a building, or the benefit of the learning you will acquire in the course of a job. With respect to costs, the best technique for ensuring that all crucial factors are included is to refer to past experience, because it necessarily embodies all the relevant factors. Let's consider the issue in the context of estimating costs in a business situation. More specifically, imagine estimating the expenditures connected with a task with which the firm already has a great deal of experience - say a construction firm concerned with a bid on a small parking lot of the sort that the firm has done many times before. Another example might be a firm that has manufactured only one product - say, men's shirts - and wishes to estimate the cost of a batch. A third example is estimation of the cost of another hamburger shop to a chain franchising operation. A firm with considerable relevant experience may estimate cost directly from its own records. In the case of another hamburger shop, the estimate may simply be the average total cost of shops recently built by the chain, as already computed by them. In the case of the parking lot or excavation, the construction firm may be able to estimate with good accuracy the amounts required of the main components, labor and machine time, and their current prices. The shirt manufacturer may have to do a more detailed analysis to estimate the cost of a batch of shirts, if the garments have design specifications. That firm may consult its past purchase records for quantities and prices of yarn, buttons, wrapping, labor, and so on. Even if your firm has a great deal of experience with a process, however, cost estimation may run into many snags. For example, the firm may want information about costs at output levels different from those it has produced at before, and it may not have sufficient records about the inputs necessary to operate at those output levels. One must also check that there was no stockpiling and no depletion of stockpiles during past observation periods. And, of course, machines with long lives are a big problem; because of the change of technology over time, records alone may not show what the relevant expenditures on machinery will be during the period for which the decision will be made (which must almost always be a long time period if there are long-lived machines). Perhaps the only case in which all the necessary information on equipment costs could come from records would be where the firm does construction or other jobs on an ad hoc contractual basis -- rents all its equipment, and hires workers for the one job only. If the firm does not have direct experience with the process being costed, then it must make (or have made) an analysis of the process, together with estimates of the requirements for each aspect of the job. For example, after it has built 100 identical McDonald's hamburger franchises, the firm can refer directly to its records to learn the necessary inputs of wood, tile, paint, plumbing labor, carpentry labor, and so on; but before it has built the first such shop, the firm must get, from architects, engineers, and contractors, estimates of how much of each input the planned building would require. These construction experts analyze the various operations required, and measure the extent of each operation (such as the number of electrical outlets needed). Then they estimate the material and labor required for each aspect of the work on the basis of their general knowledge and experience with similar projects. Big construction projects such as dams or spacecraft for moon shots exemplify jobs whose cost estimation draws little from past records and requires careful analysis by engineers and designers. But even in already-operating assembly-line operations, one must often use engineering analysis to estimate the relevant expenditures, especially when dealing with new or altered items. For example, an auto manufacturer needs to know the outlays on material and labor, and the amount of assembly- line time, that will be required for various proposed new models of cars. The aspects of the car that continue the same as in the past years can be costed from past years' records; but the new aspects must be costed by analysis done by designers, engineers, and skilled craftsmen. Analysis is also necessary when a firm considers producing a quantity of output at a level of capacity different from the one it has been working at. For example, if the highest previous level of operation has been 10,000 units per week, and the firm wants to know the expenditures relevant to producing 15,000 units a week, the judgments of supervisors, plant engineers, maintenance men, and so on must be sought and used. Movie-making is an industry in which cost estimation is notoriously treacherous. Films that started out with budgets of $30 million have ended up costing $300 million (and then flopped!). Perhaps careful estimation can be done for such items as travel, set construction, and so on. But how can one predict in advance that the female star will get furious at the male star and walk off the set halfway through the filming? In costing new operations, experimentation is particularly useful. Consider the example of an engineer who wants to estimate the cost of digging a foundation for a building in half- frozen ground for the new pipeline, in an area where no building has been done before. They may put a man on a bulldozer to work for a day to see how fast the work goes. Or, if a shoe manufacturer wants to determine the cost of a new style, she often may have the people at the benches try out some of the necessary operations to see how long they take, and to find out how much material is required. Experimentation is also used as an adjunct to other estimation methods to fill gaps in records or knowledge. The two ways of estimating cost -- engineering and experiential -- are mirrored in the two modes of estimation in many other circumstances. In medicine, for example, a physician may attempt to estimate the likelihood that a patient who has had a stroke will have another stroke on the basis of the patients "risk factors" of weight, smoking behavior, blood pressure, race, and cholesterol, perhaps adding in the physician's specific knowledge of this patient's physical and mental condition. This is called the clinical approach. Or the physician may focus on the statistics that a person who has had one stroke will have another, an experiential approach. If there are no such statistics, the physician would have to rely on the clinical approach, but in the presence of such data, it would be poor thinking to ignore them. But raw statistics on the entire population that do not present separately the information by the categories of risk categories, are too broad, and the physician would be wise to adjust such aggregate data in light of the patient's particular risk factors. That is, it would be wise to make engineering-type adjustments to the experiential information in such a situation. An even better prediction for a given patient could be developed using formal methods of statistical analysis, but not even the best physicians have gone this far yet. And too many of them simply ignore the experiential data and proceed on clinical judgment alone -- whatever that means -- often because they are not comfortable with statistical data. Another example of the tension between theory and experience is found in estimating the availability of raw materials in the future based on geological and Malthusian theory, versus the history of raw-material scarcity based upon prices throughout human history; the two approaches give diametrically opposed predictions, with the latter being the approach that is validated by history, of course. The issue is not simply theory versus empirical knowledge in the abstract, but rather how good the particular theory and data are. A good theory will fit the known facts reasonably well, and is soundly constructed; a bad theory does not do so, and is worse than no theory. Yet abstractions have an amazing power to bewitch us, and perhaps the more so if we have more education. An example is the theory of economies of scale in manufacturing, the cornerstone of government monopoly operation under of the banner of greater efficiency. But this theory leaves out the stimulating forces of competition, and the deadening effect of their absence, which account for government monopoly doing worse than private enterprise in almost every case -- even the extreme (and amazing) case of competing electric utilities up and down the same streets; more details are in Chapter 00. Typically, bad theories leave out a crucial factor -- as, for example, the importance of energizing competition in the electrical-utility example above -- or include a wrong assumption -- such as the Marxian theory that people will work just as hard for the community when they do not feel exploited by the capitalist class as they do when they own the enterprise (such as the farm). This assumption is now (1990) in the process of being massively discredited by the happenings in Eastern Europe. 2. It is important to ensure that all the important impacts, upon all the people and groups that are affected, are brought into the estimation. The failure to do so is one of the great errors in thinking, discussed at length in Chapter 4-6(?). Sometimes the effect is merely offensive to one's tastes or values, as when people in (for example) the United States quantify the loss of life in the Vietnam War only with respect to American lives, omitting the loss of Vietnamese and others. Sometimes the effect is socially destructive, as when a chemical plant takes into account only its own direct costs and ignores the costs to the community of the pollutants that it dumps into the river. Other examples of focusing only on the "seen" and neglecting the "unseen" effects may be found in Chapter 4-6. Sometimes even obvious elements are neglected because the estimation seems difficult. A university is likely to ignore the value of a piece of land in its building plans simply because the accountants find it difficult to assign a value to the land. These are some of the factors often neglected when estimating the overall value of a course of conduct: a) Knowledge gained. A wise person or business will often take on a job that does not seem obviously profitable in order to learn skills that can be valuable later on. This includes information about the environment as well as individual knowhow. b) Credit and reputation. An activity that enhances those crucial elements leaves you better prepared for the future; a detrimental activity does the opposite. c) Attention. The number of activities that you can keep your eye on is limited. When you hire someone to paint your house, you must count as a cost the attention you will have to pay to make sure the job is getting done right. You may decide that a more expensive painter, whom you will not need to check on at all, would be a better buy, or you may even decide to forego having the job done because you cannot afford to divert attention to it. 3. Subjective benefits and costs are very slippery. What is the money cost to you of sitting and reading this hour? Costs other than money? How do you learn my costs of sitting here? How do I learn what my costs will be five years from now? One of the greatest difficulties in thinking about costs and benefits is that your assessment of them right now may be importantly different than your assessment of them later on, after the event is over. The shock of jumping into cold water seems insignificant after you jump, but beforehand it looms so large that you may shilly-shally about jumping for many minutes. And the value of making a trip to the deathbed of a dying friend may come to loom larger in your mind years after the friend dies. It often helps to assess these costs and benefits if you tryto imagine that now it is five years later. Ask yourself how you would assess the cost or benefit from that perspective. These later-on assessments tie into the subject of making binding commitments. A method of integrating this present- standpoint distortion into your decision-making apparatus is described in Chapter 00. 4. The tougher and more important the issue, the less information there is to go on, usually. It is in the nature of important issues that they come along infrequently and each one is different than others. Hence there is little information to draw on when a very important issue arises. If the situation becomes repetitive, information accumulates. Before the first prototype aircraft is built, there are great "unk-unks" -- industry slang for unknown unknowns. Afterwards, learning reduces the uncertainty greatly. ESTIMATING PROBABILITIES Every estimate is a matter of estimating probabilities. If you estimate that a building is 500 feet high, you may not actually add, "give or take 75 feet", but some such assessment of the possible error in your estimate is implicit. And even that sort of statement is a vague substitute for the "probability distribution" -- the probability of the building being between 400 and 425 feet high, the probability of being between 425 and 450 feet high, and so on. In that fashion, every estimate is implicitly a distribution of probabilities. Probabilities can be known with considerable reliability if a great deal of data exists. For example, probabilities of death by age and sex for randomly-selected individuals may be estimated on the basis of large amounts of experience, as may the probability of outcomes on a roulette wheel. In other cases one feels as if the probability is being picked out of the air almost without foundation. For example, our family once was in Chaco Canyon when drops of rain began to fall. The road was a dirt track. Would we be unable to leave because of mud if we did not leave immediately? Would we be trapped in mud if we did leave? We had nothing to go on, no knowledge of the likelihood of heavy rain in that desert area, no knowledge of the effect of rain on the road, and so on. Nor did we have any way of obtaining information. Yet we had to make some estimates in order to decide what to do. Mortality estimates are considered to be "objective", in contrast to our "subjective" estimates about being trapped in Chaco Canyon mud. There is much philosophical discussion relevant to these matters, but from an operational point of view we treat these probabilities in identical fashion. Some statisticians worry that people inevitably estimate subjective probabilities in a fashion that will fool themselves. Maybe. But there is no alternative to making such probability estimates. Decision-makers1 do not enjoy estimating probabilities. Physicians, for example, often say it cannot be done. But there is plenty of evidence that people will and can estimate probabilities with some accuracy. Sometimes it is necessary to push hard on oneself, or on someone else, to extract the probability estimates -- a process neatly called "executive psychoanalysis." The devices sketched out in this section should, however, help you to extract meaningful and useful probability estimates. The probability-estimation process discussed here is intended for use in all kinds of decision-making -- business, politics, war, or other policy or action situations. Estimating probabilities for use in "pure" science is a subcategory of the estimation procedures described here, and is subject to special limitations to which probability estimation for decision making is not subject. More later about probabilities in science. PROBABILITY ESTIMATION BY THE DECISION MAKER The operational estimate of a probability (or probability distribution) begins with an overall estimate by the decision maker or deputy. For example, consider the situation in which the firm is making a decision about whether to raise its price, and the likelihood of the main competitor's responding with a price raise is a crucial aspect of the price decision. The estimate of that likelihood will then probably begin as a rough horseback judgment by the executive in charge. If she is experienced and wise, the initial estimate may be sound; if not, not. The wise decision maker uses all knowledge and mental facilities, taking everything into account that she can think of. This sort of estimate is informal and follows no explicit rules. But if the executive forces herself to organize her thinking on paper or for presentation to other people, she may thereby improve her thinking very greatly. This suggestion to use pencil and paper (or computer) may be the most useful one in the chapter. If the decision is an important one, and if the decision maker is willing to do some even harder thinking, he may be able to improve his estimate by some mental gimmicks whose purpose is (1) to make sure that his thinking is consistent, and (2) to explore his mind for thoughts that have not yet come to light. We will first discuss such gimmicks for a yes-no probability estimate, such as that the competitor will raise his price in response to a price raise of ours. Then we shall move on to the job of estimating sets of probabilities (probability distributions) that contain more than two possibilities, such as the estimation of demand, which can have many possible sales quantities. In addition to using the gimmicks below, your estimates of probabilities can benefit from keeping in mind the kinds of pitfalls in thinking discussed in Chapters 4-5 and 4-6. Gimmick 1. Make sure that the mutually exclusive probability estimates add to unity. It is, of course, just a convention (an agreed-upon definition) that probabilities add to 1, but all our probabilistic reasoning is based upon this convention. Therefore you must arrange your estimates -- in the simple two-possibility case, the probabilities of "yes" and "no" -- to add to 1. It is often helpful to estimate separately the probabilities of "yes" and of "no." This check often reveals inconsistency. For example, the executive might estimate the probability of a competitor's price raise as .40. If his assistant then asks him the chance that the competitor will not raise his price, the executive may say, "50-50." This reveals an inconsistency, because .40 and .50 do not add to 1. Further thinking to resolve the inconsistency and make the probabilities add to 1 should improve the estimate. Gimmick 2. If you find it difficult to form a numerical probability estimate, it may help to proceed in stages. For example, first ask yourself whether the chance is more or less likely than 50-50. If less than .50, next ask yourself if you think the probability is closer to 0 or to .50. And so on. Gimmick 3. Sometimes it is useful to compare your situation to a clearcut gambling machine, such as a 32-slot roulette wheel. You might first ask yourself whether the likelihood in your situation is about the same as the probability of the ball falling into one of the 32 slots. If you feel that the probability is greater than that, then ask yourself whether the chance is about the same as the ball falling into one of two given slots of the wheel, or three, and so on. Then convert to probabilities. Gimmick 4. As a consistency check, try actually making a small wager with a friend on the matter to be estimated. See whether you would be willing to bet $3 to $1, say, that the competitor will raise his price if you raise yours. After you find the worst odds at which you would be willing to take a bet that he will raise his price, turn the bet around and figure the worst odds you would take that he will not raise his price. Together these two sets of odds give an estimate for your decision. (It is worth noting that many people who say it is impossible to make probability estimates of business events, because there is no "scientific" basis to go on, have no reluctance or difficulty in making a bet on a football game.) Gimmick 5. Use the "bet-yourself" technique, another consistency check.2 Imagine that you will receive a big prize -- say, a trip around the world -- if you are right about whether an event will occur. Then ask which side of a wager you would pick concerning the probability of that event's occurring. For example, assume that you are trying to estimate the distribution of breakdowns that you can expect in the factory next year, and you must begin with an estimate of the median. You first guess that the median number is ten. Now ask yourself if you would bet on more, or on less, than ten breakdowns, with the round-the-world trip as a prize if you are correct. You answer that with considerable confidence you would bet on more than ten breakdowns. If so, your best guess of the median is more than ten, because you should be undecided about which way you would place the bet when the median has been correctly chosen. So now move your estimate of the median to, say, eleven, and ask which way you would then bet to get the prize. If you are then undecided, you can stop. But if you are still pretty sure you would bet on more than eleven, you must next consider a number higher than eleven as the median, and so on, until you reach the point at which you really are undecided about which way you would bet to receive the prize. Gimmick 6. Sometimes it helps to ask yourself what proportion of such events would be "yes" if the same situation were to be rerun 1,000 times. This mental trick may produce an acceptable estimate even when it seems very difficult to attach an estimate to a single event in isolation. Gimmick 7. Sometimes it helps to ask yourself -- or the decision maker you are assisting -- for a range of probability estimates, rather than just a single point estimate. For example, you may find it easier to estimate the probability of the competitor's raising his price, if you raise yours, as somewhere between 30 and 50 percent, rather than a point estimate of 40 percent. If the range is thus fairly narrow, you can safely work with the midpoint in subsequent calculations.3 Gimmick 8. As is true of other approximations, it is often wise to break the estimation job into constituent parts if there are several identifiable important aspects to it. For example, you might wish to estimate the probability that a competitor will bring out a plastic contact lens if you do so. And you know that the likelihood of his doing so depends heavily on his being willing to invest money in research and development to develop the necessary new technology. Instead of estimating in one jump the probability of his bringing out the plastic lens, you might separately estimate the probabilities for the two stages. After you estimate these sets of probabilities separately, you can use the multiplication rule to get an unconditional probability estimate for the overall event. If there are more than a very few identifiable events that may have an influence on the outcome, you may be better off estimating the probability of the outcome in a single jump, rather than building a fairly complex tree composed of many probabilities, all of which are uncertain. But on the average breaking a complex sequence of events such as a plan into its constituent parts has been shown to lead to more accurate estimation.4 Estimates made directly about the probability of success for a plan tend to be more biased toward a successful outcome than are estimates that take into account the probabilities of the individual events that must occur if the plan is to succeed. Perhaps this is because when one makes an estimate of the probability of success directly, one tends to focus on the first event that must occur, and then lets that number influence the overall estimate. For example, assume we are estimating the probability that a building will be completed a year from now. The number of hitches that can occur, contingent on one another, is very large. But if we do not take each of them into account very explicitly, we may find that the high probability of the first event -- acquiring the land, which has a probability of, say, 70 percent influences our immediate view of the overall success of the project. In the previous example, the estimation task was broken into separate sequential stages. Another situation where it may be wise to break up an estimation job is where there may be several alternative events to the one you are interested in. Consider the example of the firm bidding on the contract to produce 2,500 Puritanian flags (Chapter 1-4). Assume there are five other possible competitors. One might simply estimate directly the likelihood of none of them bidding lower than some specified bid -- say, $20,000. Another procedure is to estimate the likelihoods for each of them individually, and then to combine the probabilities. Perhaps you estimate their probabilities of not bidding under $20,000 as follows: Firm Probability A .7 B .8 C .7 D .9 E .9 If so, the probability of none of them bidding under $20,000 is .7 x .8 x .7 x .9 x .9 = .32, and that is your probability of your making a successful bid at $20,000. It is interesting and useful to know that if you do not break up a sequential situation, such as a construction plan, into stages, you tend to overestimate the likelihood of a "success" occurring. But if you do not break up a simultaneous event, such as a bid, into the various other events, you tend to underestimate the probability of a "success."5 Gimmick 9. Ask some other qualified people inside or outside the firm to estimate the probabilities, and find a consensus by comparing the estimates. The best people to choose (and obviously, the hardest to find) are those who are well informed but have no emotional involvement in the situation. The Delphi technique is a systematic technique for developing the consensus of a group of people. First, people are asked for their individual estimates of the probabilities. Then they are presented with the estimates of all the other people and asked how they will change their own estimates, if at all. If a respondent's new estimates are still far away from the group's previous average, the respondent is required to offer an explanation. Then the process is repeated, a total of perhaps three or four times. There is some reason to believe that the last set of probabilities is better than the first set in many cases, if only because people have had to think harder about their estimates as time goes on. Of course, the possibility exists that some of the people who hold "far out" views are right and the majority wrong, in which case the Delphi process could lead to progressively worse estimates. So the process is imperfect -- but so is life. The use of experts will be referred to again toward the end of the chapter. We have been working with the yes-or-no type of probability estimate. In many cases one works instead with the estimation of a probability distribution with a wider variety of possibilities. Demand and cost functions are common examples. But this technique may be left to more technical works (e. g. Simon, 1980, Chapter ?). It may be instructive and useful to keep a record of your own probability estimates, and then examine how they compare with actual outcomes. This is especially useful if you are engaged in estimating some series of repeated events, such as the demand for various kinds of musical performers. If you find out that you systematically overestimate or underestimate, you can try to make corrections in advance for your propensity. Perhaps the most pervasive cause of bias is the estimator's emotional state -- his hopes and fears, his optimism and pessimism, the rewards and punishments that he expects contingent upon various possible outcomes. One device suggested earlier to get around this bias is to solicit judgments from people who are not involved actually or emotionally. But often this device is not feasible. Perhaps the best tactic in estimating probabilities when one is involved is to make explicit to oneself what one's feelings are. And the best way to do this is by writing them down, as honestly as one can. Once the feelings are out in the open and labeled, one can try to counteract them in the estimates. SUMMARY Estimating probabilities is difficult, but it must be done. The better one's information about the situation, the better the estimate is likely to be. Sometimes the probability estimate can be made on the basis of existing information. Sometimes experts should be consulted on scientific research undertaken. Various devices can help you extract reasonable probability estimates from yourself. You should separately estimate the probabilities of both "success" and "failure" to see that they add to 1. You may proceed in stages, beginning with the midpoint and then estimating midpoints between other points. You may make wagers with yourself or with others to make the matter more immediate and to check your feelings. And often it is useful to make a range of estimates -- or "high," "low," and "medium" estimates -- instead of just a single point estimate. And if the planned alternative has several events in sequence, or if there are several possible alternative outcomes to the one you are interested in, it is helpful to estimate the probabilities of the separate events and then to combine them. There are many possible sorts of biases, some of which may be mitigated by the various devices discussed in this chapter. But the greater danger comes from one's emotional situation -- one's hopes and fears, expected rewards and punishments, influencing one's judgment. The only antidote is to make one's feelings explicit in as honest a manner as possible. EXERCISES 1. Estimate the probabilities that you will find employment at various salaries when you next look for a job, and tell the bases for your estimation. 2. How would you go about estimating the probabilities of various quantities of this book being sold next year? After it is revised the next time? 3. A man and wife teach psychology and physics respectively at Bluewater State University. They want to move to the University of Hawaii. How should they estimate the probabilities of their both finding jobs there? How about if they both teach psychology? 4. If Ford puts an airbag into next year's cars as a standard item even though the law does not require it, what is the probability that General Motors also will in the following year at the latest? Chrysler? Both? How should Ford go about making its estimate? FOOTNOTES 1This material is more fully covered in Robert O. Schlaifer, Analysis of Decisions under Uncertainty (New York: McGraw-Hill, 1969), especially when read in connection with Howard Raiffa, Decision Analysis Reading, Mass.: Addision-Wesley, (1969). Many of the ideas in this chapter have been drawn from these two sources. 2William A. Spurr and Charles P. Bonono, Statistical Analysis for Business Decisions, rev. ed. (Homewood, Ill.: Irwin- Dorsey, 1973), pp. 117-20 discuss this technique at length. 3A similar comment has been made about all estimations, probabilistic and non-probabilistic, by Enthoven: We have found that in cases of uncertainty, it is often useful to carry three sets of factors through the calculations: an "Optimistic" and a "Pessimistic" estimate that bracket the range of uncertainty, and a "Best Estimate" that has the highest likelihood. These terms are not very rigorous. A subjective judgment is required. But it is surprising how often reasonable men studying the same evidence can agree on three numbers where they cannot agree on one. In fact, one of the great benefits of this approach has been to eliminate much senseless quibbling over minor variations in numerical estimates of very uncertain magnitudes. Alain C. Enthoven, "Economic Analysis in the Department of Defense," American Economic Review, LIII, May, 1963, 413-23. 4Amos Tverky and Daniel Kahneman, "Judgment under Uncertainty: Heuristics and Biases," paper written for the Fourth Conference on Subjective Probability, Utility, and Decision Making, Rome, 1973 discuss this matter. 5Ibid. 6Tverdky and Kahneman, "Judgment under Uncertainty." lookat April 1990 Page # thinking estim34# 3-3-4d