The trouble with risk matrices, KD Wall

Tags: likelihood, decision maker, risk score, SME, risk matrix, risk management, probability, risk scores, risky decision problems, Oscar Morgenstern, scoring function, moderate risk, Amos Tversky, Daniel Bernoulli, quantitative values, risk averse, Availability Heuristic, initial value, subjective assessment, Defense Resources Management Institute, Anchoring and Adjustment, estimate, ect, Individuals, regression toward the mean, utility function
Content: Calhoun: The NPS Institutional Archive DSpace Repository DRMI (Defense Resources Management Institute)
Faculty and Researchers Collection
2011 The Trouble With Risk Matrices
Wall, Kent D. Downloaded from NPS Archive: Calhoun
THE TROUBLE WITH RISK MATRICES Kent D. Wall Naval Postgraduate School (DRMI) August 18, 2011 Abstract Risk matrices are used in many organizations as a basis for risk management decisions. They are easy-to-use and have intuitive appeal. They are often defended as the only avenue of approach to risk management because quantitative information is "scarce or nonexistent". Unfortunately their theoretical basis is super...cial and the validity of the qualitative information they employ is highly suspect. Assessments of the likelihood of occurrence and their impacts su¤er all the shortcomings associated with subjective assessment. These all combine to produce a less than useful portayal of risk management information. The true value of subject matter experts (SME) is in aiding the devlopemnt of quantitative models representing the inter-relations between variables that ultimately determine the outcome of interest for the decision maker. 1 INTRODUCTION Many organizations in public and private sector confront decision making in an environment fraught with signi...cant uncertainty. As a result risk becomes an overriding concern and many decisions become ones of risk management. Risk management requires that: (1) risk is unambiguously de...ned; (2) the de...nition supports the measurement (quanti...cation) of the risk; (3) the risk reduction produced by each prospective risk management alternative is; and (4) the cost of each risk management alternative is known. Given this information the manager can then choose the "best" alternative (where best means that alternative that reduces the risk the most subject to the budget constraint). Often the information supporting the quanti...cation or measurement of risk appears lacking or is hard to obtain. In such circumstances a "qualitative" approach is sought. Predominant among such approaches is the use of the "risk matrix".
The risk matrix is like a simple spreadsheet model composed of rows and columns. There usually are four or rows and columns. The rows and columns categories of likelihood (probability) and "impact"(severity). We assume throughout that the rows the categories of probability (likelihood) while the columns the categories of the outcome, often called the "impact", "severity" or "consequence". The cells within the matrix are assigned numbers called risk scores that purport to represent a quantitative assessment of the risk ­the higher the score the higher the risk. Figure 1 displays a generic risk matrix.
almost certain
highly likely
almost never
serious catastrophic
Figure 1: Example Risk Matrix
These matrices have intuitive appeal. Individuals easily can imagine risk as a function of the two dimensions portrayed by the matrix. For example, it seems self-evident that a highly likely event associated with catastrophic loss is riskier than an unlikely event associated with negligible loss. The cells of the matrix often are color-coded to communicate this (i.e., the cells in the upper right corner are red, the cells in the lower left corner are green, while the cell in between range from yellow to orange as we proceed along the diagonal from lower right to upper left). Green is "good" (none or very low risk), red is "bad" (very high risk), yellow and orange indicate increasing risk. As a management tool the matrix o¤ers a way to rank things like R&D projects in terms of their potential for cost overruns, schedule delays or substandard performance. Other examples of ranking objects by perceived risk are: military operations alternatives, national defense budgets, possible terrorist threats and de...ciencies in accounting systems. 2
Risk matrices purport to serve risk management decisions. They are seen as useful tools by many practitioners. In reality their de...ciencies render them less than useful. To understand why, we must ...rst look behind the curtain and see how a risk matrix produces a ranking of risks. This is followed by a section reviewing the relevant theory, both prescriptive and descriptive, that forms the foundation of decision making under risk and risk management. Finally, we contrast the risk matrix assessment process with theoretical foundations using a example. This highlights the de...ciency in the rationale behind the risk matrix. Our ...ndings are summarized in the last section. 2 RISK ANALYSIS USING RISK MATRICES The use of risk matrices to rank risks proceeds as follows. Each of the alternatives (projects, operational alternatives, budget allocations, etc.) become the objects of evaluation by subject matter experts (SME). The SME assign to each object a probability category and an impact category. This locates objects in the matrix using the two labels as row and column addresses. The result is a matrix "populated"with the alternatives. Numerical scales are assigned to the row and column categories and used to compute a risk score for each cell. Often the cells are color-coded in an attempt to highlight the meaning and interpretation of the scores. Figure 2 is an example from the U.S. Army (TRADOC 2009, Appendix I). Here the categories of likelihood are assigned "scores" from 1 (= almost never) through 5 (= almost certain), while the categories of consequence are assigned "scores" from 1 (= negligible) through 4 (= catastrophic). The reasons for this coding is not given. Sometimes the cell scores are obtained by "weighting" (e.g., consequence is weighted by 0.6 and likelihood by 0.4 before) multiplying the two. The justi...cation for this is not explained. We ...nd cells of the same color are not scored the same. For example, the red cells in the "Northeast" corner are not all equal. It appears that the risk corresponding to "almost certain and serious" is less than the risk associated with "highly likely and catastrophic". Evidently there is more to risk than what is communicated by colorcoding. The risk score calculation is described formally as follows. Let Si denote the ithprobability (row) category score and Sj denote the jth outcome (column) category score. Denote the subjectively assessed probability by pb and the subjectively assessed consequence by yb: If the assessed probability is such that the SME declares it to fall within the ith row category, then this probability is denoted pbi and is represented in subsequent calculations by its score value Si(pbi): If the assessed outcome is such that the SME declares it a member of the jth column category, this outcome is denoted ybj and is represented in subsequent 3
almost certain 2.6 [ Y ]
3.2 [ O ]
3.8 [ R ]
4.4 [ R ]
highly likely
2.2 [ G ]
2.8 [ Y ]
3.4 [ O ]
4.0 [ R ]
1.8 [ G ]
2.4 [ Y ]
3.0 [ O ]
3.6 [ O ]
1.4 [ G ]
2.0 [ G ]
2.6 [ Y ]
3.2 [ O ]
almost never 1.0 [ G ]
1.6 [ G ]
2.2 [ G ]
2.8 [ Y ]
serious catastrophic
Figure 2: Example Risk Matrix with Color Code and Risk Score
calculations by its score value Sj(ybj): The risk score assigned to the (i; j)th cell within the risk matrix is now represented by:
Rij = R(Si(pbi); Sj(ybj)):
Rij exhibits monotonic behavior in its arguments:
Rij Ri+1;j and Rij Ri;j+1:
There is no guidance as to the form of R(Si(pbi); Sj(ybj)): Its de...nition appears to depend upon who is using the risk matrix. Most often we ...nd Rij computed as the product of its two arguments:
Rij = Si(pbi) Sj(ybj)
or a modi...cation of this as in the U.S. Army TRADOC example in Figure 2: Rij = [0:4 Si(pbi)] [0:6 Sj(ybj)]:
The rationale for risk matrices apparently derives from the quantitative de...nition of risk proposed by Kaplan and Garrick (1981) They state that ".... risk consists of the answer to three questions: (1) What can go wrong?; (2) How likely is it to go wrong?; and (3) If it does go wrong, what is the outcome?" These authors represent the answers symbolically as a triple:
fsi; pi; yig
where si is the name of the scenario (event) representing "what can go wrong", pi is the probability of occurrence (likelihood) of si and yi is the measure (magnitude) of the outcome (damage) associated with the occurrence of si: Thus it appears as self-evident that risk is a function of two quantities: pi and yi: It is determined by the probability and the associated outcome of a scenario. A matrix representation is a natural way to describe this: let the rows represent the pi and the columns represent the yi: Thus Figure 1 appears to capture all that is relevant. Right? Wrong? There is more to risk than just likelihood and outcome. In fact, this is implicitly acknowledged by those who use risk matrices, for they feel compelled to add more information. First, there is universal use of color-coding of the cells to convey something about the amount of risk to associate with a cell: red cells represent more risk than orange cells, orange cells represent more risk than yellow cells and green cells represent less risk than yellow cells. Second, there is an apparent need for more detailed information than the colors provide. Indeed, we ...nd the cells contain a number representing either the risk score or the risk rank. These numbers are used to order the risks, even among cells with the same color. Evidently, all cells of the same color are not created equal! The use of colors and risk scoring are attempts to provide missing information ­Risk assessment requires more than just probability, pi; and outcome, yi: The information omitted from the Kaplan and Garrick triple is decision maker preference1. To make a decision (e.g., NO, there is no signi...cant risk; or YES, there is signi...cant risk and something must be done to manage it), the decision maker must be able to express preference over the set of probable outcomes. That is, the decision maker must be able to express how he/she feels about the outcomes, fyi; 1 i ng: Is there a range, or speci...c values, of yi that the decision maker prefers to avoid? Is there a range, or particular values, of yi that the decision maker prefers to have happen? Decision theory describes decision maker preference by a payo¤ function (sometimes called a value function). We denote this function by v(yi): The Kaplan and Garrick "trio"needs to be expanded to a "quartet". Risk is really a combination of the answers to four questions: (1) "What can go wrong?"; (2) "How likely is it to go wrong?"; (3) "If it does go wrong, what is the outcome?"; and "How do you feel about it (the likely outcomes)?" The Kaplan and Garrick "trio" must be 1 Kaplan and Garrick omit any reference to decision maker preferences because their primary focus is provivision of information necessary for public evaluation of risk. They leave it to the public through the democratic political process to establish preferences. 5
replaced by a "quartet":
fsi; pi; yi; v(yi)g:
The importance of decision maker preferences requires a brief review of a well-developed and widely accepted model of decision maker preference. This model will serve to highlight a major source of error in risk matrices: If the color-coding and scoring are not determined by the decision maker, then they do not represent risk and are meaningless!
3.1 Models of Preference
Decision maker preference in risky decision problems is represented by a payo¤ function, v(Y ): Its development has a long history beginning with Daniel Bernoulli (1738, published in translated form in 1954). His model of preference was later called a utility function by John von Neuman and Oscar Morgenstern (1944). Their work established the utility function as the gold-standard for the payo¤ function until the mid-1970's. Utility theory always had troubling de...ciencies when it came to descriptive power ­it is unable to explain observed decision behavior in important situations. This led Daniel Kahneman and Amos Tversky (1979) to develop the prospect function as a better payo¤ function for representing preferences in risky decisions. This is now the new gold-standard for decision theory (although its embrace by the economics community proceeds slowly).
The prospect function version of the decision maker's payo¤ has three salient characteristics: (i) a reference point for the outcome that separates gains from losses; (ii) loss aversion in which losses loom larger than gains; and (iii) risk aversion for gains and risk taking for losses. Much empirical research ...nds a power function form adequate to represent the payo¤ function:
v(Y ) = v+(Y ) + v (Y )
v+(Y ) = [Y Y R] if Y Y R > 0 v (Y ) = jY Y Rj if Y Y R < 0
when more Y is preferred to less Y: If preferences are reversed so less Y is preferred to more Y then we have v+(Y ) = [Y Y R] if Y Y R > 0 v (Y ) = jY Y Rj if Y Y R < 0: Y R denotes the reference point that distinguishes the Y representing gain from the Y representing loss. Loss aversion is represented by
the constant, > 1: Decreasing marginal returns for both losses and
gains are captured by the powers 0
< 1 and 0
< 1:The
prospect functions is convex for losses an concave for gains. Empirical
estimates, in terms of median values, are:
2< <5
0:5 and 0:8
for a large number of studies with a wide range of subjects. Figure 3 depicts a payo¤ prospect function in the case where more Y is preferred to less when Y R = 3:0; = = 0:70:
Figure 3: Payo¤ Prospect when "More is Better"
3.2 Models of Choice under Risk
Normative decision theory represents the choices made by a decision
maker using a mathematical optimization construct: The alternative
chosen is that which maximizes the decision maker's payo¤ function,
v(Y ): In decisions involving uncertainty the theory replaces the payo¤
function with the expected payo¤, Efv(Y )g where the expectation is
taken with respect to the distribution function of Y; PY : If fAm; 1
m M g denotes the set of alternatives, and PY jAm denotes the distribution function of Y conditioned on the alternative, then the
decision problem is
max Efv(Y )jAmg = max
v(Y )dFY jAm 1
if Y is a continous-valued outcome and
kX =N
max Efv(Y )jAmg = max v(yk) p(Y = ykjAm)
Am k=1
if Y is discrete-valued with N distinct values 2. The solution is denoted as A : Since v(Y ) is negative for outcomes the decision maker prefers to avoid, the maximization operation means that A will be the risk minimizing alternative (i.e., the least negative among the set of alternatives).
This model is useful both normatively and descriptively. In the normative mode it informs the decision maker by inducing an ordering over the alternatives, from "best" to "worst". Descriptively it provides an explanation of why and how decisions are made. No matter what the mode of use, the model tells us what information the decision maker requires to make a rational informed decision: (i) the outcomes as a function of the alternatives, Y (Am); (ii) the conditional likelihood over the outcomes, PY jAm; and (iii) the preferences (payo¤ function) of the decision maker over the outcomes, v(Y ):
3.3 Theoretical Implications for Risk Matrices in Practice Risk matrices cannot integrate all that matters using only two variables: one de...ning the rows (probability or likelihood) and one de...ning the columns (outcome, consequence, impact or severity). A third quantity is required: payo¤ (preference). This describes how the decision maker values the probable outcomes. Risk cannot be quanti...ed/measured/described until we know how much the decision maker desires to avoid the various outcomes representing loss. Thus, the need to add a third dimension to what is in reality a two-dimensional object. Risk matrices in practice attempt this by introducing colorcoding and/or numerical-coding (scoring) to the cells. Risk matrices facilitate decision making only if they faithfully depict the information the theory demonstrates is necessary for informed decisions. Risk matrices must accomplish this in situations where quantitative data appears unavailable, nonexistent or too diў cult and costly to obtain. This means that risk matrix construction relies on subjective assessment. Subjective assessment is, however, fraught with problems.
4 AN ILUSTRATIVE EXAMPLE We now contrast the qualitative risk matrix approach with the quantitative approach based on the theory of decision making under risk. 2 Eq. (9) also applies when Y is continuous-valued but we use a relative frequency histogram approximation to the Conditional Probability ditribution of Y:
An illustrative example serves our purposes. Suppose you want to know the risk inherent in doing nothing (the status quo) to mitigate infrastructure damage due to earthquakes. This is the ...rst step in risk management: assessing the risk inherent in the "do nothing" alternative. If the answer is "NO", there is only insigni...cant risk, then it can be ignored. If the answer is "YES", there is signi...cant risk, then you will be interested in assessing the risk associated with risk management alternatives. The process of deciding which of the proposals is "best" involves the same process of assessment as the original assessment, so we only consider this. Suppose further that the outcome of concern is the total program cost (implementation plus damage repair) and the time horizon is the next 20 years. 4.1 The Quantitative Approach Given access to cost analysts, actuaries, earthquake engineers, simulation modelers and data you develop a simulation model that represents this knowledge. The uncertainty derives from the number, and strength, of earthquakes occurring over the next 20 years. Geologists say the number of annual earthquakes is modeled by a Poisson process with mean given by geological data. They say the size of the next earthquake (Richter scale) is modeled by an exponential distribution with mean given by geological data. The earthquake engineers and insurance industry provide a model that explains damages as a function of Richter scale value. You develop a spreadsheet model, use "add-in" simulation software, and produce a relative frequency histogram of total program cost (approximating the distribution, PY jA0). This is depicted in Figure 4. This provides a complete picture of the Figure 4: Example Total Cost Relative Frequency 9
uncertain situation confronting the decision maker. All that remains is to determine how the decision maker "feels about it". We know from theory that this is captured by the computations in Eq. (9). The decision maker evaluates PY in the light of his/her preferences, v(Y ): The decision made depends on the value of Efv(Y )g : if Efv(Y )g > 0 there is an expected gain and insigni...cant risk; if Efv(Y )g < 0 then there is an expected loss and signi...cant risk. To illustrate, assume the decision maker has preferences over Y that give rise to the payo¤ function of Fig. 5 (here = 5; = 0:8; = 0:7 and Y R = 20:): A decision maker with this payo¤ function ...nds there Figure 5: Decision Maker v(y) for Earthquakes is expectation of a loss; i.e., Efv(Y )g = 8:431: This person will conclude there is signi...cant risk and seek management alternatives to reduce the risk. The dependence on Y R is important. If Y R > 27:772 then Efv(Y )g > 0 and the decision maker will conclude there is no signi...cant risk. Of course, the actual decision is a function of all three parameters of the payo¤ function model. The quantitative approach provides the decision maker the information required to make an informed decision. There is no more complete a characterization of the uncertainty in the outcome of interest, Y; than that provided by PY : Given this information (cf., Fig.4) the decision maker can determine its implication in terms of his/her payo¤ function for Y ; i.e., v(Y ): Without knowledge of PY there is no hard evidence upon which to base a decision. 10
4.2 The Qualitative Approach If you have no access to data and simulation modeling you rely on a qualitative approach using a risk matrix. You seek direct subjective assessment of likelihood and outcome by subject matter experts (SME), some or all of whom would be used by the quantitative approach in model construction. Let's assume you decide on four categories for this outcome: negligible (N), marginal (M), serious (S) and catastrophic (C). You also decide to use categories for likelihood: almost never (AN); unlikely (U), likely (L), highly likely (HL) and almost certain (AC). Together, these a risk matrix with four columns and rows as in Figure 1 reproduced here.
almost certain
highly likely
almost never
serious catastrophic
Figure 6: Earthquake Risk Matrix Once the risk matrix is de...ned the qualitative approach requires subjective assessment of the situation by the SME. This involves three steps. 4.2.1 Subjective assessment of probability and outcome are unreliable First, and foremost, the SME must subjectively assess the existing risk in terms of the two dimensions of the matrix: the estimate of the likely outcome, ybj and the estimate of its probability of occurrence, pbi: Unfortunately, all humans su¤er substantial limitations in judgment under uncertainty (Kahneman, Slovic and Tversky, 1982). Three factors signi...cantly a¤ect the accuracy of subjective assessment: the representativeness heuristic; the availability heuristic; and anchoring with insuў cient adjustment. Each lead to subjective assessments that exhibit bias and error. These limitations guarantee systematic errors in pbi and ybi: Appendix A provides a brief summary of the details.
4.2.2 Interpretation of category labels is unreliable Second, there are problems with interpretation of the subjective assessments in terms of the categories of the risk matrix. To illustrate, let the outcome categories be Yj = fYjl y < Yjug and probability categories be, Pi = fPil p < Piug: Clearly these are incomplete without specifying the upper and lower limits (signi...ed by the superscripts l and u). Unfortunately there is no guidance how to do this. It is sometimes left to the SME for their de...nition. Other times de...nitions are provided by organizational doctrine, policy, or an analyst other than the SME. 4.2.3 Combining unreliable information compounds the problem The last step requires the SME to reconcile the outputs of the ...rst two steps. The SME must ...nd a "home" for the assessed risk by placing it in one of the cells of the risk matrix. There are at least two possibilities here. If no explicit (quantitative) de...nitions exist for the categories, then it is left to the SME to interpret the words de...ning them. There are signi...cant diў culties in the interpretation, especially with respect to probabilities. Research ...nds that individuals attach very di¤erent meanings to the words de...ning the probability categories. Individuals interpret probabilities in the approximate range 0.0 - 0.05 as if they are indistinguishable from P = 0; while probabilities in the approximate range 0.92 - 0.99 as if they are indistinguishable from P = 1: Individuals over-weight the importance of probability when 0:05 < P < 0:4 while they under-weight probabilities when 0:4 < P < 0:9: This can result in an unlikely or highly unlikely risk receiving a likely classi...cation, or a likely or highly likely risk receiving an almost certain classi...cation. These diў culties also arise in the inverse. Given a numerical probability, di¤erent individuals will assign it to di¤erent rows in the matrix without explicit knowledge of the probability category de...nitions. On the other hand, given a probability identi...ed by a verbal label, such as "highly likely", di¤erent individuals will identify it with di¤erent numerical probabilities and insert it into di¤erent rows. Again there is no guidance on how to do this. The same can be said for the outcome columns. Even when categories are explicitly de...ned in quantitative terms, individuals interpret these ranges using di¤erent words. Appendix B presents research ...ndings that illustrate these problems. Even if explicit category de...nitions exist for the SME, problems still arise. We illustrate using three di¤erent partitions for each outcome category: 12
N: Y1 = f0 y < 5g; f0 y < 10g; f0 y < 5g M: Y2 = f5 y < 20g; f10 y < 15g; f5 y < 10g S: Y3 = f20 y < 40g; f15 y < 25g; f10 y < 20g C: Y4 = f40 yg; f25 yg; f20 yg and three di¤erent partitions for each probability category: AN: P1 = fp :10g; fp :05g; fp :01g U: P2 = f:10 < p :33g; f:05 < p :25g; f:01 < p :20g L: P3 = f:33 < p :90g; f:25 p :80g; f:20 p :70g HL: P4 = f:90 p :95g; f:80 p :95g; f:70 p :90g AC: P5 = f:99 pg; f:95 pg; f:90 pg Let us assume the SME are in possession of the probability information given in Figure 4. Even with all this information a single cell location is not possible. For example, denote by "X"the risk classi...cation using the ...rst partition of the fPig and fYjg. Likewise, denote the classi...cation corresponding to the second partition by "Y". Finally, denote the classi...cation associated with the third partition "Z". None of the three category groups produce unambiguous risk classi...cations. as pictured in Figure 7. There is no unambiguous position
almost certain highly likely
almost never
Figure 7: Risk Matrix Alternative Locations
for "X"because P fY2g = 0:33 while P fY3g = 0:66: There is no unambiguous location for "Y" because P fY4g = 0:66 while P fY5g = 0:30: There is no unambiguous location for "Z" because P fY4g = 0:33
while P fY5g = 0:66: Even when the complete probabilistic picture is available to the SME subjectively judge must still be applied to force the situation into the matrix framework. The introduces error and may depend upon a proclivity to be pessimistic or optimistic when classifying the risk. 4.2.4 Risk scores are not informative The ...nal piece of information presented in the risk matrix is the risk score of Eq. (1). This score supposedly provides the decision maker a way to order the risks in the matrix from highest to lowest score. Unfortunately, the calculated score is based on subjective judgment and su¤ers from the same errors in judgment we ...nd in the assessment of pbi and ybi: Developing a risk score requires specifying two components. First, there is the de...nition of the category scales that convert categories into numerical values. Second, there is the form of the function Sij: Again there is no guidance provided to help decide what to do. The current example requires numerical values for the probability categories fSi; 1 i 5g and four numbers for the outcome categories, fSj; 1 j 4g: The most common scaling uses a direct linear mapping: Si = i and Sj = j: This appears self-evident to most users of the qualitative approach. There are however, important implications for the eventual risk score calculation that may be unintended. For example, the outcome "serious"is three times more important than "negligible"and "catastrophic" is 1.25 more important than "serious". There is no guarantee that the implied trade-o¤ s correspond to the preferences of the decision maker. In reality, the decision maker may consider the "catastrophic" outcome to be 5 times more important than the "serious"outcome. Similar observations apply to the probability category scaling. Appendix B presents more detail on the errors introduced by all category scaling. The next step in computing a risk score is speci...cation of Sij: The only requirement is that it be monotonically increasing in each factor. If no other information is provided, then the simple product of the scales de...ned in Eq. (2) is used: Sij = Si Sj = i j: In particular we have Sij = Si j which implies risk neutrality with respect to the outcome. This contradicts the fact that decision makers are not risk neutral. Using what appears to be an intuitively 14
attractive risk scoring function requires one to ignore the contradiction. The risk score computed this way has little to do with decision maker preferences and cannot be used to rank risks. If the more general case of Eq.(1) is employed, the qualitative approach o¤ers no guidance as to the preferred form of this function. The theory of decision making under risk o¤ers guidance in terms of pi and yj but not in terms of Si and Sj: 4.2.5 Summary of the qualitative approach Every step of the qualitative approach involves subjective judgment. The need to force all the information contained in Figure 5 into a single cell of the matrix requires this. Unfortunately, all subjective judgment is imbued with bias and heuristics that limit its eў cacy. Subjective judgment is required to extract from Figure 5 one representative ybj and its corresponding pbi:We ...nd this leads to non-unique classi...cation. Subjective judgment is required in deciding the de...nition (partitioning) of each category. Di¤erent individuals use di¤erent partitions, even when explicit de...nitions exist. Subjective judgment is required in deciding the scaling of the categories, Siand Sj: Scaling is arbitrary and ignores decision maker preferences. Subjective judgment is required in deciding on the form of the risk scoring function, Sij:The most popular form, multiplication of Siand Sj; combined with the most popular form of scaling, the direct linear map from categories to the positive integers, produces an inappropriate risk score. 5 COMPUTED SCORES DO NOT REPRESENT RISK Users of risk matrices claim risk scores provide the information needed to rank risks. This claim has little or no foundation. Both the theory of decision making, and research results describing actual decisions, yield models that do not support the risk scoring in risk matrices. This, above all else, makes the information value of risk scores highly suspect. Let us see why. Normative decision theory de...nes risk as the expected value of the 15
decision maker's payo¤ 3:
kX =N
Efv(Y )g = p(Y = yk) v(yk) = R:
(cf., Eq.9 with reference to the decision alternative, Am; suppressed for notational convenience)4. Comparison of Eq.2 and Eq.10 shows some similarity: (i) both involve probability values; (ii) both involve the outcome values; and (iii) both involve combining this information via multiplication. Indeed, it may be argued that Eq.10 motivates the form of Eq.2. The di¤erences are, however, signi...cant. Both fac- tors in Eq.2 are of dubious quality and content. The scales used to represent the pbi category and the ybjcategory of are arbitrary. Multiplying the two only compounds the errors. Unless Si(pbi) = pbi and Sj(ybj) = v(ybj) we obtain meaningless results. The multiplication bears no resemblance to the expected pay-o¤ of the decision maker. The Sj(ybj) factor does not represent decision maker preferences and does not serve as a proxy for the decision maker pay-o¤ function.
Risk matrices have intuitive appeal and seem easy-to-use. They o¤er an approach to risk management that appears to avoid the need for quantitative information. As a result they have found widespread use. Unfortunately their true value is little more than a "placebo e¤ect" (cf. Chpt. 7 in Hubbard, 2009). First and foremost, they rely on a awed process of subjective assessment. Any information obtained this way is fraught with bias and systematic errors. The use of SME does not eliminate this problem. SME are humans and are no less insulated from these problems than any other individual. Second, the construction of the matrix itself is arbitrary. The number of rows and columns is arbitrary. Risk matrices can exhibit anywhere between three and six categories of outcome and probability. Increasing the number of categories does not improve accuracy or precision. The de...nition of the categories is arbitrary. Category labels for probability and outcome use terms that vary from one application to the next. More importantly, the interpretation of the meaning of these labels is left entirely to the individual, whether SME or the decision 3 Only the discrete form (c.f., Eq.9) of the normative theory is used to provide a straight forward comparison with Eq.10 4 If the decision maker is strictly loss averse then v(yk) reduces to a simple switch where v(yk) = 1Xif yk corresponds to a "loss" and is zero otherwise. In this case Efv(Y )g = p(Y = yk) = R where the sum is only over those yk that represent a loss. 16
maker. In fact, there is no assurance that the SME and decision maker employ identical interpretations of the categories. Experimental results show this creates errors understanding the information being communicated by the matrix. The term "high risk" that is given one combination of probability and outcome by an individual may be interpreted as "moderate risk" by another individual. These e¤ects are exacerbated when SME attempt to force a veritable continuum of possibilities into a ...nite set of categories. Figure 7 illustrates this problem. Third, category scaling arbitrary. Scales are necessary to convert categories to quantitative values for computing risk scores. No consideration, however, is given to the consequences of employing ordinal, interval or a ratio scales. Yet the choice of scale can have a substantial impact on resulting risk score and the decision as to how to allocate scare resources in an e¤ort to manage risk. In reality we ...nd scaling leads to range compression errors. Fourth, and most important, the entire exercise of risk scoring is arbitrary. The theoretical foundations that should be employed in quantifying risk are ignored. The risk scoring functions that appear in the risk matrix literature are o¤ered as a way to combine the numerically scaled values for probability and outcome but no basis is provided for their de...nition. They appear to be an e¤ort to capture the belief that risk is a monotonic function of the severity (impact, magnitude or size) of the outcome of concern and the probability of its occurrence. Unfortunately, the most popular scoring function, the product of probability and outcome scales, is just plain wrong. Decision makers are risk averse and not risk neutral. More important is a total neglect of decision maker preferences in de...ning the risk scoring function. There is a veritable mountain of experimental results that shows that (decision maker) risk is determined by decision maker preferences over the likely values of the outcome. Moreover, the function that best expresses this risk is a prospect function. This function does not use the product of probability and outcome. It's form is dictated by four fundamental components: (a) a critical value by which the decision maker distinguishes those outcomes representing loss from those representing gain; (b) loss aversion by which losses are weighted more than gains (of the same absolute magnitude); (c) risk aversion for gains; and (d) risk taking behavior for losses. None of these components is contained in a function formed by the product of probability and outcome. All four components are determined by decision maker preferences, yet nowhere in the development of the risk matrix is there any attempt to solicit and incorporate decision maker preferences. 17
The principal reason for using the risk matrix is a perceived lack of quantitative information. Users claim it either does not exist or cannot be obtained without requiring signi...cant time and resources. Apparently the time and resources expended to employ SME is viewed di¤erently than those expended on quantitative analysis. SME have valuable insights and experience that can and should be employed. Their expertise, however, is misapplied. It would be a far better use of SME experience and judgment if they are applied to the development of models describing the process that generates the outcome of concern. This includes the speci...cation of variables, their interrelationships, ranges of likely values for parameters (con...dence intervals), and information on possible probabilistic models of the uncertainties. Simulation models and the generation of data for a thorough quantitative risk analysis would follow in a matter of hours. It is hard to imagine the time required for this to be any more than the time required for SME to develop and complete a risk matrix!
8 EXAMPLES OF RISK MATRICES The followiing section provides a small sample of risk matrices found in the public sector.
8.1 State of California This risk management matrix uses risk ranking instead of risk scoring to order the risks in the matrix. A risk score (i.e., larger score = greater risk) is easily obtained. For example, replace the rank, rij, by Sij = 13 rij: The categories are explicitly de...ned, but no explanation or justi...cation is given the cell ranking. One may ask what is the necessity of including the bottom row.
8.2 U.S. Navy
This risk matrix provides no explicit category de...nitions. The category scale is linear for both the row and the column. There is no explanation or justi...cation for the risk scoring. The patterns of risk scores suggests a linear relationship over a subregion of the matrix.
8.3 Australia/New Zealand This matrix does not have explicit de...nitions for categories and no risk scores. Risks can be ordered only by color (wording). The diў culties in using this matrix is illustrated by noting that an insignificant consequence is rated di¤erently when it is almost certain but
0.7 ­ 1.0 0.4 - 0.7 0.0 ­ 0.4 0
Neg. $0 - $20 9 11 12 0
Marg. $20-$40 5 8 10 0
$40-$80 $80-$100
Figure 8: Californina Department of Transportation Risk Management Matrix
Low Impact 1
Very High Likelihood
4 Risk uRnRisikist2ks2D=D4
Mod. Impact High Impact Critical Impact
5 RisRkRisiksuk2n2EiEts = 5
6 Risk units = 6
6 Risk units = 6 Zone A
High Likelihood 3
3 Risk units = 3
4 5 Risk units = 4 Risk units= 5 ZRRisiksok22CnC e B
6 Risk units = 6
Moderate Likelihood
1 2 Risk units = 1 Risk units = 2 Zone C
4 Risk units = 4
5 Risk units = 5
Low Likelihood 1
1 Risk units = 1
1 Risk units = 1 RRisiksk22AA
3 Risk units = 3
4 Risk RuRisniksikts2=2BB4
Figure 9: U.S. Navy Region SW Risk Management Process Matrix
still insigni...cant. If such a risk is insigni...cant then it must be so no matter how likely it is.
1. Bernoulli, Daniel (1954), "Exposition of a New Theory on the Measurement of Risk" (original 1738), Econometrica, Vol.22, pp. 23-36. 2. Hubbard, D. W., The Failure of Risk Management, Hoboken, NJ: John Wiley & Sons, 2009. 3. Hubbard, D. W., How to Measure Anything (2 nd), Hoboken, NJ: John Wiley & Sons, 2010. 4. Kahneman, D. and Amos Tversky (1979), "Prospect Theory: An Analysis of Decision Making under Risk", Econometrica, 19
LIKELIHOOD Insignificant
Almost certain Medium
Possible Low
Unlikely Low
Minor Medium Medium Medium Low Low
Medium Medium Low
High Medium Low
Catastrophic Extreme Extreme High Medium Medium
Figure 10: Australia - New Zealand Standard for Risk Management
Vol.47, no. 2, pp. 263-291. 5. Kahneman, D., Paul Slovic and Amos Tversky (1982) Judgment under uncertainty: Heuristics and biases, Cambridge, UK: Cambridge University Press. 6. Kaplan, S. and B. John Garrick, "On the Quantitiative De...nition of Risk", Journal of Risk Analysis, Vol.1, no.1, pp. 11-27. 7. Neumann John von and Oscar Morgenstern (1947), The Theory of Games and Economic Behavior, Princeton, NJ: Princeton University Press. 8. Savage, Sam, The Flaw of Averages, Hoboken, NJ: John Wiley & Sons, 2009. 9. U. S. Army Training and Doctrine Command (TRADOC), "Capabilities Based Assessment (CBA) Guide Version 3.0, Sept. 2009. A APPENDIX: SUBJECTIVE ASSESSMENT ERROR Subjective assessment is subject to cognitive limitations that a¤ect human ability to interpret and evaluate information. These produce errors and bias in the qualitative inputs to risk matrices. The most important of these, for our purposes, are those that a¤ect the subjective assessment of: (1) the impact/severity/consequence; and (2) the likelihood/probability/frequency. Subject matter experts (SMEs) are no less insulated from these shortcomings than the man in the street ­SME are human beings ­enough said! A.1 Representativeness Heuristic Humans are susceptible to the "Representativeness" heuristic. Individuals are more apt to classify an object based on how representative the object is of a certain class or group of objects. This has serious implications for probability assessment. Questions like: "What is the
probability that object A belongs to class B?""What is the probability that event A originates from process B?" "What is the probability that process B will generate event A?"are examples of where this heuristic leads to serious errors in assessment. Research has found six factors that give rise to the representativeness heuristic. Ignoring the prior probabilities. Individuals ignore facts related to the "base rate". In terms of conditional probability we know that P (A and B) = P (AjB) P (B): Research ...ndings show that humans erroneously omit the prior probability or "base rate", P (B); from this calculation. Failure to account for sample size. The variability in a sample statistic is a function of the sample size. The sample statistics (e.g., the mean value) computed from a small sample (e.g., 10 observations) will have more variability than the same sample statistic computed from a larger sample (e.g., 100 observations). Research shows that humans consistently fail to incorporate this fundamental result in probability statements. Misconception of chance (e¤ects of random behavior). The classic example of this failure is the sequence of coin tosses: HTHTTH, HHHTTT, and HHHTH. Individuals consistently rate the ...rst sequence as more indicative of a fair coin than the second sequence. Likewise the third sequence is rated as more indicative of a fair coin than the fourth sequence. Individuals infer the characteristics of a long sequence (global result) when judging the characteristics of a very short sequences. This erroneous inference is a demonstration of an e¤ect called the "law of small numbers". Statistically sophisticated individuals commit this error more than one would think. Insensitivity to predictability of the outcome. Normative statistical theory maintains predictions are a function of prior information and the expected predictive accuracy of the available evidence. When predictability is nonexistent, prior probabilities dominate the process so all predictions should be very close to the base rate. If predictability is perfect, then the evidence dominates the process so predictions should be very close to actual outcomes. In reality, however, the ordering of the perceived likelihood of outcomes coincides with the ordering of outcomes by their representativeness. Illusions of validity. Subjects develop an unwarranted con...dence in their predictions when there is a good ...t between the predicted outcome and the input information. This con...dence is the product of the representativeness of the input information and nothing else. 21
Failure to recognize the regression e¤ect. Humans do not incorporate the regression e¤ect or "regression toward the mean"when making subjective assessment . First, they do not expect this e¤ect in many situations where it is bound to occur. Second, when they experience this e¤ect, they invent spurious causal explanations for it. Failure to appreciate the regression e¤ect leads to subjective assessment based on representativeness. This can produce pernicious e¤ects. For example, experienced ight instructors have concluded that harsh criticism of students rather than verbal rewards is better in training, contrary to accepted psychological doctrine. A.2 Availability Heuristic Humans are susceptible to the "Availability" heuristic. Individuals assess the likelihood of an event by the ease with which instances or occurrences come to mind. This heuristic is useful in assessing likelihood by frequency but it is inuenced by three factors that produce serious biases. Retrievability. An event whose instances are easily retrieved from memory will appear more numerous than an event of equal frequency. Salience also e¤ects retrievability. For example, the impact on frequency of actually witnessing an event will be greater than that from merely reading about it in the newspaper. Imaginability. Sometimes one must evaluate the likelihood of an event not stored in memory. In such situations one usually constructs mental images of the event and evaluates the likelihood of the event by the ease with which the event can be constructed. Unfortunately, the ease with which an event can be constructed does not reect their actual frequency. This bias is observed in many real-life situations. Illusory correlation. Individuals signi...cantly over-estimate the co-occurrence natural associates. Here the ease with which one associates the occurrence of two events seriously biases the likelihood estimate that these two events will co-occur in the future. A.3 Anchoring and Adjustment Many times individuals make estimates by starting from some initial value and then adjust it to obtain the ...nal estimate. This is the "Anchoring and Adjustment" heuristic. The initial value may come from the problem formulation or it may be from some other partial computation. In either case, the adjustment is typically insuў cient. There are three ways it fails. 22
Insuў cient adjustment. Experiments reveal how signi...cant is the anchoring e¤ect on the ...nal estimate of a quantity. Individuals given a low initial value are anchored so that their ...nal estimate is consistently lower than the true value. Individuals given a high initial value produce ...nal estimates consistently above the true value. Humans do not adjust suў ciently to o¤set the bias in the initial information. Biases in the evaluation of conjunctive and disjunctive events. Studies show that individuals consistently underestimate the probabilities of disjunctive events and overestimate the probabilities of conjunctive events. This bias is the result of insuў cient adjustment of prior probability estimation. Anchoring in the assessment of subjective probability distributions. Both naive and sophisticated individuals fail to adequately adjust probability estimates from initially given values. This is revealed in many experiments where individuals consistently state overly narrow con...dence intervals for the likelihood of some event. This is but one example of a trait exhibited by individuals when assessing random outcomes: overcon...dence. Figure 11 presents an example of overcon...dence in expert judgment. Seven experts (geotechnical engineers) were asked to estimate the failure height of an earthen wall. Each expert provided their estimate (blue square) along with its 25% and 50% percentiles (red triangle and red horizontal bar, respectively). The heavy black line depicts the actual failure value. Even SMEs are not well calibrated! Figure 11: Expample of Expert Overcon...dence 23
B APPENDIX: CATEGORY AND SCALE ERRORS The use of categories for the row dimension and the column dimension introduce ambiguities that exacerbate errors. For example, individuals attach varying meanings to words like "high", "medium" and "low" in relation to assessing outcomes and probabilities. The use of more categories to produce a ...ner grid does nothing to alleviate problems and may even make them worse. The problems with the use of categories is very similar to the ambiguities of the 2 statistic in deciding goodness-of-...t: choosing di¤erent categories gives gives di¤erent values for the statistic. This can lead to the wrong decision. Errors introduced by range compression. Forcing the conversion of an otherwise meaningful unambiguous quantity into a score with few (usually 3 to 5) values introduces error. Consider the Intergovernmental Panel on climate change (IPCC) example given below in Figure 12. The de...nition of the likelihood categories require both an 11% and 30% likelihood to be grouped together as "unlikely". Yet 11% corresponds to 1-in-9 odds while 30% represents almost 1-in-3 odds. Clearly this is a case of introducing "round-o¤" error. The same problems arise in relation to the outcome/impact axis of the risk matrix. For example, Hubbard (2009, pg. 131) describes an IT project risk matrix in which a return-on-investment (ROI) of between 1% and 299% is assigned a score of 1, while 300%cording to the guidelines of the Intergovernmental Panel on Climate Change (IPCC). The third and fourth columns summarize subjects responses and the ...fth column presents the errors of the responses. It is most interesting that, when provided with the explicit oў cial guidelines, subjects still gave answers at variance with the instructions! Figure 12: Example of Interpretations of Probability Problems with assigning numerical values (scales) to represent and quantify the categories. The arbitrary assignment of numerical scales is based on the presumption of regular or uniformly spaced intervals. Hubbard (2009, pp.133-134) presents the following example illustrating the errors introduced by this presumption. Figure 13 shows the relative value of the mitigating e¤ect against IT project failure risk of the project sponsor rank in an organization. The scale assignment means that the inuence of a CEO, CFO or CIO is three times as important as a VP. The inuence of a VP is twice as important as that of a line manager. In reality, after acquiring historical data on the actual impact of sponsor rank on project success, we ...nd the actual di¤erences do not support the presumption of regular (uniformly spaced) intervals. While the rank ordering is preserved, the di¤erence between C-level, senior VP and VP is quite small and that the inuence of a manager is very much less that even the "ordinary" VP. Problems with lack of independence between events. Scoring methods presume independence among factors and risks. They do not account for correlation between random variables or the e¤ect of deterministic interdependencies. Two "medium-impact", "mediumlikelihood" outcomes located in a risk matrix may represent a very high risk if they happen together (are correlated or interdependent). The only way such a situation can be taken into account is through the de...nition of a new outcome that represents the joint occurrence 25
Figure 13: Sponsor Rank and IT Sucess of the two constituent outcomes. There is no evidence that this is rede...nition of outcomes is practiced in the application of risk matrices. 26

KD Wall

File: the-trouble-with-risk-matrices.pdf
Title: The Trouble With Risk Matrices
Author: KD Wall
Author: Wall, Kent D.
Published: Sat Jan 1 00:00:00 2011
Pages: 27
File size: 0.45 Mb

, pages, 0 Mb


, pages, 0 Mb

The Mission, 1 pages, 0.87 Mb
Copyright © 2018