Rank reversals in decision-making

From HandWiki

In decision-making, a rank reversal is a change in the rank ordering of the preferability of alternative possible decisions when, for example, the method of choosing changes or the set of other available alternatives changes. The issue of rank reversals lies at the heart of many debates in decision-making and multi-criteria decision-making, in particular.

Unlike most other computational procedures, it is hard to tell if a particular decision-making method has derived the correct answer or not. Such methods analyze a set of alternatives described in terms of some criteria. They determine which alternative is the best one, or they provide relative weights of how the alternatives perform, or just how the alternatives should be ranked when all the criteria are considered simultaneously. This is exactly where the challenge with decision making exists. Often it is hard, if not practically impossible, to determine whether a correct answer has been reached or not. With other computational methods, for instance with a job scheduling method, one can examine a set of different answers and then categorize the answers according to some metric of performance (for instance, a project's completion time). But this may not be possible to do with the answers derived by most decision making methods. After all, determining the best decision making method leads to a decision making paradox.

Thus the following question emerges: How can one evaluate decision-making methods? This is a very difficult issue and may not be answered in a globally accepted manner.

A critical part in answering this fundamental question is played by what is known as rank reversals.

Rank reversal

One way to test the validity of decision-making methods is to construct special test problems and then study the solutions they derive. If the solutions exhibit some logic contradictions (in the form of undesirable rank reversals of the alternatives), then one may argue that something is wrong with the method that derived them.

To see the above point more clearly, suppose that three candidates are evaluated for some job opening. Let us designate these candidates as A, B, and C. Suppose that some decision making method has determined that the best candidate for that job is person A, followed by B, who is followed by C. This is the first ranking and it is indicated as follows: A > B > C (where > means better than). Next, suppose that candidate B (who is not the best one) is replaced by an even worse candidate, say person D. That is, now we have B > D, and candidate B is replaced by D while candidates A and C remain in the pool of candidates with exactly the same characteristics as before. When the new set of alternatives (i.e., candidates A, D and C) are ranked together and by assuming that the criteria have exactly the same weights as before, then should not candidate A still be the best one? It turns out that under some decision making methods the best alternative may be different now.[1] This is known as a rank reversal and it is one of the types of rank reversals.

The first type of rank reversal in the above context was observed by Belton and Gear in 1983 as part of a study [2] of the analytic hierarchy process (AHP).[3] They first considered a simple decision problem comprised by 3 alternatives and 2 criteria. Next a copy of a non-optimal alternative was introduced. When the 4 alternatives (i.e., the previous 3 plus the copy) were evaluated, and under the assumption that the criteria weights are exactly the same as before, it was observed that now the indication of the best alternative can change. That is, a rank reversal may occur with the AHP. A few years later it was observed that the AHP, as well as a new variant to it that was introduced by Professor Thomas Saaty (the inventor of the AHP) in response to the previous observation by Belton and Gear, may exhibit rank reversals when a non-optimal alternative is replaced by a worse one (and not a copy of an alternative as in Belton and Gear's experiment).[4]

The issue of rank reversals has captured the interest of many researchers and practitioners in the field of decision-making. It is something that continues to be considered controversial by many and is frequently debated.[5][6][7][8][9][10]

Different types of rank reversals

There are many different types of rank reversals, depending on how the alternatives in a problem are defined and evaluated. These types are described next as Type 1, Type 2, Type 3, Type 4, and Type 5.

Rank reversals of Type 1

As stated earlier, one may introduce identical or near-identical copies of non-optimal alternatives and then check to see if the indication of the best alternative changes or not.[2]

Rank reversals of Type 2

Another way is to replace a non-optimal alternative with a worse one and then see if the indication of the best alternative changes or not.[4]

Rank reversals of Type 3

First consider a problem with all the alternatives together and get a ranking. Next, decompose the original problem into a set of smaller problems defined on two alternatives at a time and the same criteria (and their weights) as before. Get the rankings of these smaller problems and check to see if they are in conflict with the ranking of the alternatives of the original (larger) problem.[11]

Rank reversals of Type 4

Type 4 is like Type 3, but ignores the ranking of the original (larger) problem. Instead, check to see if the rankings of the smaller problems are in conflict with each other. For instance, suppose that the following 3 alternatives A, B, and C are considered. Next, suppose that some 2-alternative problems are solved and the rankings A > B, B > C, and C > A, are derived from these 2-alternative problems. Obviously, the above situation indicates a case of non-transitivity (or contradiction) as we get A > B > C > A.

Rank reversals of Type 5

All previous types of rank reversals are known to occur with the analytic hierarchy process (AHP) and its additive variants, the TOPSIS and ELECTRE methods and their variants.[1][11][12]

The weighted product model (WPM) does not exhibit the previous types of rank reversals, due to the multiplication formula it uses.[1][11] However, the WPM does cause rank reversals when it is compared with the weighted sum model (WSM) and under the condition that all the criteria of a given decision problem can be measured in exactly the same unit.[4] The same is true with all the previous methods as well. This is the Type 5 ranking reversal.

It is quite possible to define more types of rank reversals. One only needs to determine ways to alter a test problem and see how the ranking of the alternatives of the new problem differs from the original ranking of the alternatives of the original problem. Furthermore, the difference in rankings, somehow, should indicate the presence of undesirable effects.

Are rank reversals always undesirable?

Decision-making methods are used to make decisions in many aspects of human activity. This is especially true with decisions that involve large amounts of money or decisions that may have huge impact on large numbers of people. Given the well-established fact that different methods may yield different answers when they are fed with exactly the same problem, the question is how to evaluate them. Rank reversals are at the very heart of assessing the merits of such methods. At the same time, they are at the center of many heated debates in this area. Many authors use them as means to criticize decision making methods or to better explain rational behavior. [5][6][7][8][9][10]

Consider a simple example of buying a car. Suppose that there are two cars available to the decision maker: Car A and Car B. Car A is much cheaper than Car B but its overall quality is much less when compared to that for Car B. On the other hand, Car B is more expensive than Car A but it is also of better quality. A decision maker who is concerned of the high price issue, may choose Car A over the better quality and more expensive Car B. Next suppose that the car dealer presents to the decision maker a third car, say Car C, which is way more expensive than Car B but now the overall quality of Car C is marginally higher than that of Car B. Under such a scenario, it is quite possible for a decision maker to alter his/her opinion and purchase Car B instead of Car A, even if he/she has not actually seen Car C.

Such events may take place with many rational decision makers.[dubious – discuss] In other words, rank reversals may actually be possible in rational decision making. The issue of having rank reversals by rational decision makers has been studied extensively by Amos Tversky.[13] In other words, having rank reversals in certain occasions and of certain types may not be indicative to faulty decision making. However, the key question is how to be able to distinguish when rank reversals indicate that something is wrong or when they do not conflict rational decision making. This is a highly debated issue in the decision making community.

Methods that have been verified to exhibit rank reversals

The following is just a partial list of multi-criteria decision making methods which have been confirmed to exhibit various types of rank reversals:[1] [4][5][6][7][8][9][10][14][15][16][17]

  • The analytic hierarchy process (AHP) and some of its variants.
  • The ELECTRE (outranking) method and its variants.
  • The TOPSIS method.
  • The PROMETHEE (outranking) method.
  • Multi-attribute utility theory (MAUT).
  • Choice between odds ratios and risk measures and between risk differences and risk ratios. Odds ratios inflate estimates when prevalence is not nominal even if in only some subgroups.[18]

References

  1. ↑ 1.0 1.1 1.2 1.3 Triantaphyllou, E. (2000). Multi-Criteria Decision Making: A Comparative Study. Dordrecht, The Netherlands: Kluwer Academic Publishers (now Springer). pp. 320. ISBN 0-7923-6607-7. http://www.csc.lsu.edu/trianta/Books/DecisionMaking1/Book1.htm. 
  2. ↑ 2.0 2.1 Belton, V.; A.E. Gear (1983). "On a Short-Coming of Saaty's Method of Analytic Hierarchies". Omega 11 (3): 228–230. doi:10.1016/0305-0483(83)90047-6. 
  3. ↑ Saaty, T.L. (1990). "How to make a decision: The analytic hierarchy process". European Journal of Operational Research 48 (1): 9–26. doi:10.1016/0377-2217(90)90057-I. 
  4. ↑ 4.0 4.1 4.2 4.3 Triantaphyllou, E.; S.H. Mann (1989). "An Examination of the Effectiveness of Multi-Dimensional Decision-Making Methods: A Decision-Making Paradox". International Journal of Decision Support Systems 5 (3): 303–312. doi:10.1016/0167-9236(89)90037-7. http://www.csc.lsu.edu/trianta/Journal_PAPERS1/PARADX1.htm. Retrieved 2010-06-25. 
  5. ↑ 5.0 5.1 5.2 Leskinen, P.; J. Kangas (2005). "Rank reversals in multi-criteria decision analysis with statistical modeling of ratio-scale pairwise comparisons". Journal of the Operational Research Society 56 (7): 855–861. doi:10.1057/palgrave.jors.2601925. 
  6. ↑ 6.0 6.1 6.2 Ishizaka, A.; M. Lusti (2006). "How to derive priorities in AHP: a comparative study". Central European Journal of Operations Research 14 (4): 387–400. doi:10.1007/s10100-006-0012-9. https://researchportal.port.ac.uk/portal/en/publications/how-to-derive-priorities-in-ahp(9f4f5d6a-36b1-4728-a8a1-2c0d3ed1fef0).html. 
  7. ↑ 7.0 7.1 7.2 Kujawski, E. (2005). "A reference-dependent regret model for deterministic tradeoff studies". Systems Engineering 8 (2): 119–137. doi:10.1002/sys.20027. https://digital.library.unt.edu/ark:/67531/metadc782055/. 
  8. ↑ 8.0 8.1 8.2 Saaty, T.L. (2005). "Making and validating complex decisions with the AHP/ANP". Journal of Systems Science and Systems Engineering 14 (1): 1–36. doi:10.1007/s11518-006-0179-6. 
  9. ↑ 9.0 9.1 9.2 Bevilacquaa, M.; M. Braglia (2000). "The analytic hierarchy process applied to maintenance strategy selection". Reliability Engineering & System Safety 70 (1): 71–83. doi:10.1016/S0951-8320(00)00047-8. 
  10. ↑ 10.0 10.1 10.2 Zahir, S. (2009). "Normalisation and rank reversals in the additive analytic hierarchy process: a new analysis". International Journal of Operational Research 4 (4): 446–467. doi:10.1504/IJOR.2009.023538. 
  11. ↑ 11.0 11.1 11.2 Triantaphyllou, E. (2001). "Two new cases of rank reversals when the AHP and some of its additive variants are used that do not occur with the multiplicative AHP". Multi-Criteria Decision Analysis 10: 11–25. doi:10.1002/mcda.284. 
  12. ↑ Wang, X.; E. Triantaphyllou (2008). "Ranking irregularities when evaluating alternatives by using some ELECTRE methods". Omega 36: 45–63. doi:10.1016/j.omega.2005.12.003. 
  13. ↑ Tversky, A. (1969). "Intransitivity of Preferences". Psychological Review 76: 31–48. doi:10.1037/h0026750. 
  14. ↑ Zanakis, S.H.; A. Solomon; N. Wisharta; S. Dublish (1998). "Multi-attribute decision making: A simulation comparison of select methods". European Journal of Operational Research 107 (3): 507–529. doi:10.1016/S0377-2217(97)00147-1. 
  15. ↑ Ertugrul, I.; N. Karakasoglu (2008). "Comparison of fuzzy AHP and fuzzy TOPSIS methods for facility location selection". The International Journal of Advanced Manufacturing Technology 39 (7–8): 783–795. doi:10.1007/s00170-007-1249-8. 
  16. ↑ Zhang, K.; C. Kluck; G. Achari (2009). "A Comparative Approach for Ranking Contaminated Sites Based on the Risk Assessment Paradigm Using Fuzzy PROMETHEE". Environmental Management 44 (5): 952–967. doi:10.1007/s00267-009-9368-7. PMID 19763684. Bibcode: 2009EnMan..44..952Z. 
  17. ↑ Olson, D.; H.M. Moshkovich; R. Schellenberger; A.I. Mechitov (1995). "Consistency and Accuracy in Decision Aids: Experiments with Four Multiattribute Systems". Decision Sciences 26 (6): 723–748. doi:10.1111/j.1540-5915.1995.tb01573.x. 
  18. ↑ Norton EC, Miller MM, Wang JJ, Coyne K, Kleinman LC. Rank reversal in indirect comparisons. Value Health. 2012 Dec;15(8):1137-40. doi: 10.1016/j.jval.2012.06.001. Epub 2012 Sep 7. PMID: 23244817; PMCID: PMC3527821



Retrieved from "https://handwiki.org/wiki/index.php?title=Rank_reversals_in_decision-making&oldid=3357696"