POLICY EXC is a template set for use with an Excel spreadsheet to facilitate analyzing and aiding human judgment by using statistical capacity for an Ordinary Least Squares (OLS) multiple regression model. This page in no way attempts to provide an introduction to the techniques or underlying theory of judgment analysis; for this purpose users should refer to the bibliography. The information provided here is intended to provide a brief overview for easy first-time use of the POLICY EXC template set.
The POLICY EXC template set contains four .xls files:
for
two-cue tasks
pexc2.xls
three-cue tasks
pexc3.xls
four-cue tasks
pexc4.xls
five-cue tasks
pexc5.xls
These four files should be saved in a safe, permanent folder and then copied into a second folder--and additional folders--for actual use. The introduction on this page makes use of the pexc2.xls template for a two-cue task: the evaluation of job candidates from test scores and interview ratings. All four .xls files in the template set are used identically, so one example should suffice.
The following instructions apply specifically to the two-cue task file, pexc2.xls, although the procedure for the other files is similar.
Task Definition and Case Presentation - pexc2.xks (Sheet 1)
Column A (shaded light blue) should contain the sequence of judgments for the two-cue task.
Only a single set of judgments can be analyzed in one .xls file, so, as soon as judgments are entered in Column A, the file should be saved immediately with a useful and unique identification (e.g., SALLY01A.XLS). Judgments can be entered for up to 100 cases. Judgments must be numerical.
Columns B and C (shaded light yellow) should contain the sequence of cue values for the two-cue task. These cue values can be entered before or after the sequence of judgments are entered in Column A. The file should be saved immediately following cue value input. Cue values can be entered for up to 100 cases. Cue values must be numerical. Each pair of cue values in Columns B and C must correspond exactly to the judgment in Column A entered immediately to the left. If there are 25 pairs of cue values, there must be 25 judgments.
Note: A judge can review the numerical cue values displayed in Columns B and C of Sheet 1 and directly enter a sequence of numerical judgments in Column A. Alternatively, you can devise your own paper or spreadsheet forms to present cue values and collect judgments in a manner more appropriate for your study, then subsequently enter the sequence of judgments and cue values in Sheet 1. Text for cue values and/or judgments CANNOT be entered in Sheet 1. If you are using text to differentiate cue values and/or judgments, you must code this text numerically prior to entering the cue values and/or judgments in Sheet 1.
Boxes in the first row (at the top) of Columns B and C provide a space to enter a brief name for each cue. Boxes in Column H are used to provide additional information about the judgment analysis: the judge’s name, the number of cases, the lowest/minimum judgment value, and the highest/maximum judgment value. Ideally, the judge should use the full range of the judgment scale. The default in POLICY EXC is to estimate a nonlinear model of the judgment policy whenever feasible. If you prefer that the contribution of a cue be estimated in a strictly linear manner, enter LFO (“Linear Form Only”) in the yellow box for either or both cues.
Note: Using the capabilities of Excel, you can generate random values for a cue by using the formula = X + Y * RAND ( ) where X is equal to the lowest/minimum cue value; Y is equal to the difference between X and the highest/maximum cue value. You then must replace the cell formulas with fixed numerical values by highlighting all randomly generated cells and, finally, on the Edit menu clicking the “copy” function followed by clicking the “paste special” function with “Values.”
Statistical Analysis and Results - pexc2.xls (Sheet 2)
By clicking on the tab “Sheet 2” immediately below the spreadsheet, you will see a second spreadsheet that provides the statistical analysis and results. At the top-left corner of this spreadsheet is a five-row by five-column box (shaded red) containing 25 #####s. Immediately below the red box are the four-step instructions for statistically analyzing the judgment policy. Following these four steps, the statistical results will be displayed to the right in Columns AB through AE (shaded green).
An upper right triangular matrix displays the bivariate correlations between the judgments (y) and the two cues (x1 and x2). Arithmetic means and standard deviations are shown below for y, x1, and x2, respectively. Finally, the unstandardized regression coefficients (i.e., b-coefficients, not ?-coefficients) for the pair of linear (e.g., x1:b) and nonlinear (e.g., x1sq:b) components of each cue are listed, as well as the regression constant. At the bottom of the green box is the multiple coefficient of determination (R2).
Judgment Policy - pexc2.xls (Sheet 3)
By clicking on the tab “Sheet 3” immediately below the spreadsheet, you will see a third spreadsheet that displays the judgment policy with the relative weight (calculated as a percentage of the total) and function form for each cue.
Note: Using the capabilities of Excel, you can design Sheet 3 in whatever style of presentation that you prefer. For example, you can move and enlarge the function form displays, alter the definition of the horizontal and or vertical axes, change colors, provide supplemental text, or superimpose other lines or curves on the plots for comparative purposes.
Columns AA through AD display—for the set of cue values used in Sheet 1—the sequence of original judgments (y) and the predicted judgments (y') that are generated by the OLS multiple regression model (i.e., from a completely consistent application of the judgment policy shown in Sheet 3). In particular, Column AD contains the residuals, that is, the extent of difference between the original judgments and the predicted judgments (y – y'). Positive residuals indicate that the original judgment is higher than the judgment policy would predict; negative residuals indicate that the original judgment is lower than the judgment policy would predict.
Note: Larger negative or positive residuals can indicate specific cases where the original judgments might be reconsidered. By clicking on the tab “Sheet 1,” you can return to the original sequence of judgments and cue values to focus on specific cases where the discrepancy between the original judgments and the predicted judgments appear greatest.
