View source for Reference class forecasting

{{Short description|Method of predicting the future}}
{{Futures studies}}
'''Reference class forecasting''' or '''comparison class forecasting''' is a method of predicting the future by looking at similar past situations and their outcomes. The theories behind reference class forecasting were developed by [[Biography:Daniel Kahneman|Daniel Kahneman]] and [[Biography:Amos Tversky|Amos Tversky]]. The theoretical work helped Kahneman win the Nobel Prize in Economics.

Reference class [[Forecasting|forecasting]] is so named as it predicts the outcome of a planned action based on actual outcomes in a reference class of similar actions to that being forecast.

Discussion of which reference class to use when forecasting a given situation is known as the [[Reference class problem|reference class problem]].

==Overview==

Kahneman and Tversky<ref name="Kahneman and Tversky 1979">{{cite journal|last1=Kahneman|first1=Daniel|last2=Tversky|first2=Amos|title=Prospect Theory: An Analysis of Decision under Risk|journal=Econometrica|date=1979|volume=47|issue=2|pages=263–291|doi=10.2307/1914185|url=https://courses.washington.edu/pbafhall/514/514%20Readings/ProspectTheory.pdf|jstor=1914185|citeseerx=10.1.1.407.1910}}</ref><ref name="Kahneman and Tversky 1977">{{cite journal|title=Intuitive prediction: Biases and corrective procedures|url=http://apps.dtic.mil/dtic/tr/fulltext/u2/a047747.pdf|archive-url=https://web.archive.org/web/20130908065829/http://www.dtic.mil/dtic/tr/fulltext/u2/a047747.pdf|url-status=live|archive-date=September 8, 2013|date=1977|last1=Kahneman |first1=Daniel|last2=Tversky|first2=Amos}} Decision Research Technical Report PTR-1042-77-6. In
{{cite book|editor-last1=Kahneman|editor-first1=Daniel|editor-last2=Slovic|editor-first2=Paul|editor-last3=Tversky|editor-first3=Amos
|doi=10.1017/CBO9780511809477.031
|title=Judgment Under Uncertainty: Heuristics and Biases
|pages=414–421
|year=1982
|isbn=9780511809477
|chapter=Intuitive prediction: Biases and corrective procedures|last1=Kahneman|first1=Daniel|last2=Tversky|first2=Amos}}</ref> found that human judgment is generally [[Philosophy:Optimism bias|optimistic]] due to overconfidence and insufficient consideration of distributional information about outcomes.

People tend to underestimate the costs, completion times, and risks of planned actions, whereas they tend to overestimate the benefits of those same actions. Such error is caused by actors taking an "'''inside view'''", where focus is on the constituents of the specific planned action instead of on the actual outcomes of similar ventures that have already been completed.

Kahneman and Tversky concluded that disregard of distributional information, i.e. risk, is perhaps the major source of error in forecasting. On that basis they recommended that forecasters "should therefore make every effort to frame the forecasting problem so as to facilitate utilizing all the distributional information that is available".<ref name="Kahneman and Tversky 1977" />{{rp|416}}  Using distributional information from previous ventures similar to the one being forecast is called taking an "'''outside view'''".  Reference class forecasting is a method for taking an outside view on planned actions.

Reference class forecasting for a specific project involves the following three steps:
# Identify a reference class of past, similar projects.
# Establish a [[Probability distribution|probability distribution]] for the selected reference class for the [[Parameter|parameter]] that is being forecast.
# Compare the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

== Reference class tennis ==
The [[Reference class problem|reference class problem]], also known as reference class tennis, is the discussion of which reference class to use when forecasting a given situation.

Suppose someone were trying to predict how long it would take to write a psychology textbook. Reference class tennis would involve debating whether we should take the average of all books (closest to an outside view), just all textbooks, or just all psychology textbooks (closest to an inside view).<ref name="Conceptually">{{cite news|title=Week 10: Reference Class Forecasting|url=https://conceptually.org/concepts/reference-class-forecasting/|work=Conceptually|accessdate=20 April 2017}}</ref><ref name="lw">{{cite news|title=Outside view|url=https://wiki.lesswrong.com/wiki/Outside_view|work=LessWrong Wiki|accessdate=20 April 2017}}</ref>

==Practical use in policy and planning==

Whereas Kahneman and Tversky developed the theories of reference class forecasting, Flyvbjerg and COWI (2004) developed the method for its practical use in policy and planning, which was published as an official Guidance Document in June 2004 by the UK Department for Transport.<ref>{{Cite news|date=June 2004|title=Procedures for Dealing with Optimism Bias in Transport Planning|work=The British Department for Transport|url=https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/191523/Procedures_for_dealing_with_optimism_bias_in_transport_planning.pdf|access-date=23 July 2021}}</ref> In policy and planning environments, reference class forecasting is strengthened when project teams measure variables that were previously treated as intangible, since quantifying these uncertainties creates more accurate distributions for costs, schedules and expected benefits. This measurable approach reduces optimism bias and supports more accountable planning decisions.<ref>{{Cite book |last=Hubbard |first=Douglas W. |title=How to measure anything in project management |last2=Budzier |first2=Alexander |last3=Leed |first3=Andreas Bang |date=2026 |publisher=Wiley |isbn=978-1-394-23981-8 |location=Hoboken, New Jersey}}</ref>

The first instance of reference class forecasting in practice is described in Flyvbjerg (2006).<ref name="Flyvbjerg2006">{{cite journal|first1=Bent|last1= Flyvbjerg|date=2006|title=From Nobel Prize to Project Management: Getting Risks Right|journal=Project Management Journal|volume=37|issue=3|pages= 5–15|arxiv=1302.3642|ssrn=2238013|bibcode=2013arXiv1302.3642F|doi= 10.1177/875697280603700302|s2cid= 13203075}}</ref> This forecast was part of a review of the Edinburgh Tram Line 2 business case, which was carried out in October 2004 by Ove Arup and Partners Scotland. At the time, the project was forecast to cost a total of £320 million, of which £64 million – or 25% – was allocated for contingency. Using the newly implemented reference class forecasting guidelines, Ove Arup and Partners Scotland calculated the 80th percentile value (i.e., 80% likelihood of staying within budget) for total capital costs to be £400 million, which equaled 57% contingency. Similarly, they calculated the 50th percentile value (i.e., 50% likelihood of staying within budget) to be £357 million, which equaled 40% contingency. The review further acknowledged that the reference class forecasts were likely to be too low because the guidelines recommended that the uplifts should be applied at the time of decision to build, which the project had not yet reached, and that the risks therefore would be substantially higher at this early business case stage. On this basis, the review concluded that the forecasted costs could have been underestimated. The Edinburgh Tram Line 2 opened three years late in May 2014 with a final outturn cost of £776 million, which equals £628 million in 2004-prices.<ref>{{Cite news|date=February 2018|title=Report for the Edinburgh Tram Inquiry|url=https://static1.squarespace.com/static/5c312863266c07d084fcd39e/t/5e428984ff5d0d4b9479c774/1581418886198/OGP_Edinburgh-Tram-Report.pdf|access-date=23 July 2021}}</ref>

Since the Edinburgh forecast, reference class forecasting has been applied to numerous other projects in the UK, including the £15 (US$29) billion Crossrail project in London. After 2004, The Netherlands, Denmark, and Switzerland have also implemented various types of reference class forecasting.

Before this, in 2001 (updated in 2011), AACE International (the Association for the Advancement of Cost Engineering) included Estimate Validation as a distinct step in the recommended practice of Cost Estimating (Estimate Validation is equivalent to Reference class forecasting in that it calls for separate empirical-based evaluations to benchmark the base estimate):

<blockquote>
The estimate should be benchmarked or validated against or compared to historical experience and/or past estimates of the enterprise and of competitive enterprises to check its appropriateness, competitiveness, and to identify improvement opportunities...Validation examines the estimate from a different perspective and using different metrics than are used in estimate preparation.<ref>[https://web.archive.org/web/20160530211731/http://www.aacei.org/tcmfree/tcmframework_webedition.pdf AACE International, "Total Cost Management Framework, Section 7.3, Cost Estimating and Budgeting", 2011. p. 147]</ref>
</blockquote>

In the process industries (e.g., oil and gas, chemicals, mining, energy, etc. which tend to dominate AACE's membership), benchmarking (i.e., "outside view") of project cost estimates against the historical costs of completed projects of similar types, including probabilistic information, has a long history.<ref>Merrow, E. and Yarossi, M., "[https://www.proquest.com/openview/42daeef9f8c368ef28f3434d206eeb93/1.pdf Assessing Project Cost and Schedule Risk]", 1990 AACE Transactions. pp H.6.1-7</ref> A method combining reference class forecasting and competitive crowdsourcing, Human Forest, has also been used in the life sciences, to estimate the likelihood that vaccines and treatments will successfully progress through clinical trial phases.<ref>{{Cite journal |last1=Atanasov |first1=Pavel D. |last2=Joseph |first2=Regina |last3=Feijoo |first3=Felipe |last4=Marshall |first4=Max |last5=Conway |first5=Amanda |last6=Siddiqui |first6=Sauleh |date=2022-07-27 |title=Human Forest vs. Random Forest in Time-Sensitive COVID-19 Clinical Trial Prediction |url=https://papers.ssrn.com/abstract=3981732 |language=en |location=Rochester, NY|doi=10.2139/ssrn.3981732 |ssrn=3981732 |s2cid=245174381 |url-access=subscription }}</ref><ref>{{Cite web |title=NSF Award Search: Award # 1919333 - Human Forests versus Random Forest Models in Prediction |url=https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919333 |access-date=2022-09-20 |website=www.nsf.gov}}</ref>

In the book ''[[Astronomy:Anthropic Bias|Anthropic Bias]]'', [[Philosophy:Philosophy|philosopher]] [[Biography:Nick Bostrom|Nick Bostrom]] described ways in which statistical reasoning about reference classes can be applied to answering scientific questions related to our existence in the universe, such as with the [[Astronomy:Anthropic principle|anthropic principle]], the [[Astronomy:Fine-tuned universe|fine-tuned universe]] hypothesis and its possible explanations, the [[Doomsday argument|doomsday argument]], the size of the universe and possibility of a [[Religion:Multiverse|multiverse]], and thought experiments such as the sleeping beauty problem. Bostrom investigates how to reason when one suspects that [[Philosophy:Evidence|evidence]] is biased by "observation selection effects", in other words, when the evidence presented has been pre-filtered by the condition that there was some appropriately positioned observer to "receive" the evidence.<ref>{{Cite web|title = Anthropic Bias {{!}} anthropic-principle.com|url = http://www.anthropic-principle.com/?q=anthropic_bias|website = www.anthropic-principle.com|accessdate = 2015-11-03}}</ref><ref name=":0">{{Cite book |last=Bostrom |first=Nick |title=Anthropic bias: observation selection effects in science and philosophy |date=2002 |publisher=Routledge |isbn=978-0-415-88394-8 |series=Studies in philosophy |location=New York}}</ref>

Bostrom argues against the self-indication assumption (SIA), a term he uses to characterize some existing views, and introduces the self-sampling assumption (SSA): ''that you should think of yourself as if you were a random observer from a suitable reference class''. He later refines SSA into using observer-moments instead of observers to address certain paradoxes in anthropic reasoning, formalized as the strong self-sampling assumption (SSSA): ''Each observer-moment should reason as if it were randomly selected from the class of all observer-moments in its reference class.''<ref>{{Cite web |last=Bostrom |first=Nick |date=2005 |title=Self-Location and Observation Selection Theory |url=https://anthropic-principle.com/preprints/self-location |access-date=2023-07-02 |website=anthropic-principle.com}}</ref> These different assumptions are affected differently based on the choice of reference class. An application of the principle underlying SSSA (though this application is nowhere expressly articulated by Bostrom), is: If the minute in which you read this article is randomly selected from every minute in every human's lifespan, then (with 95% confidence) this event has occurred after the first 5% of human observer-moments. If the mean lifespan in the future is twice the historic mean lifespan, this implies 95% confidence that N < 10n (the average future human will account for twice the observer-moments of the average historic human). Therefore, the 95th percentile extinction-time estimate in this version is 4560 years.

== See also ==

{{div col|colwidth=30em}}
*{{annotated link|Base rate fallacy}}
*{{annotated link|Consensus forecast}}
*{{annotated link|Finance:Cost overrun}}
*{{annotated link|Event chain methodology}}
*{{annotated link|Finance:Financial risk}}
*{{annotated link|Forecasting}}
*{{annotated link|Philosophy:Optimism bias}}
*{{annotated link|Philosophy:Planning fallacy}}
*{{annotated link|Reference class problem}}
{{div col end}}

== References ==
{{Reflist}}

==Bibliography==
*{{cite journal|pmid=12858711 |year=2003 |last1=Lovallo |first1=D |title=Delusions of success. How optimism undermines executives' decisions |journal=Harvard Business Review |volume=81 |issue=7 |pages=56–63 |last2=Kahneman |first2=D  |url=https://hbr.org/2003/07/delusions-of-success-how-optimism-undermines-executives-decisions}}
*{{cite book|chapter-url=https://books.google.com/books?id=tenILJ-MowQC&pg=PA133|title=Decision-making on Mega-Projects|isbn=9781848440173|editor-last1=Priemus|editor-first1=Hugo|editor-last2=Flyvbjerg|editor-first2=Bent|editor-last3=van Wee|editor-first3=Bert|date=2008|doi=10.4337/9781848440173.00014|chapter=Public Planning of Mega-Projects: Overestimation of Demand and Underestimation of Costs|last1=Flyvbjerg|first1=Bent}}
*{{cite book|doi=10.1093/oxfordhb/9780199563142.003.0014|year=2011|last1=Flyvbjerg|first1=Bent|chapter=Over Budget, Over Time, Over and Over Again: Managing Major Projects|chapter-url=https://books.google.com/books?id=FI7l8O1tlkkC&pg=PT321|title=The Oxford Handbook of Project Management|isbn=9780199563142|editor-last1=Morris|editor-first1=Peter W. G|editor-last2=Pinto|editor-first2=Jeffrey K|editor-last3=Söderlund|editor-first3=Jonas|}}

[[Category:Evaluation methods]]
[[Category:Economic forecasting]]

{{Sourceattribution|Reference class forecasting}}