THE RANK-ORDER METHOD FOR APPELLATE SUBSET SELECTION

ESSAY THE RANK-ORDER METHOD FOR APPELLATE SUBSET SELECTION Michael J. Hasday* Appellate courts in many countries will often use a subset of the entire appellate body to decide cases. The United States courts of appeals, 1 the European Court of Justice, 2 and the highest courts in Canada, Israel, South Africa, New Zealand, and the United Kingdom all use subsets. 3 Utilizing such subsets has the advantage of increasing judicial efficiency, but creates the possibility that there will be an attempt to manipulate the composition of the panel for strategic purposes. 4 In general, there have been two methods that appellate courts have used to choose their subsets: direct selection and random assignment. In direct selection, the chief judge or a designated court administrator (hereinafter, the Selector ) simply selects the members and size of the panel for that particular case. 5 In random assignment, the size of the panel is preset and the composition of the panel is randomly assigned from the full set of judges. 6 Both of these subset selection methods likewise involve a trade-off. Direct selection allows the Selector to choose panels that reflect the views of the entire set of judges. At the same time, direct 2017 Michael J. Hasday. Individuals and nonprofit institutions may reproduce and distribute copies of this Essay in any format at or below cost, for educational purposes, so long as each copy identifies the author, provides a citation to Notre Dame Law Review Online, and includes this provision in the copyright notice. * Associate, Anderson & Ochs, LLP. Juris Doctor, Northwestern Pritzker School of Law, 2001. Thanks to Matt Ginsberg, Carol M. Hasday, and Joni Hasday for their helpful comments. Of course, all views expressed, and any mistakes made, are my own. The author welcomes correspondence on his proposal, which may be sent to: mikehasday@gmail.com. 1 See J. Robert Brown, Jr. & Allison Herren Lee, Neutral Assignment of Judges at the Court of Appeals, 78 TEX. L. REV. 1037, 1069 78 (2000). 2 See Jens Frankenreiter, Informal Judicial Hierarchies: Case Assignment and Chamber Composition at the European Court of Justice (May 11, 2016) (unpublished paper), https://ssrn.com/abstract=2778807. 3 See Benjamin Alarie, Andrew Green & Edward M. Iacobucci, Panel Selection on High Courts, 65 U. TORONTO L.J. 335 (2015). 4 See id. at 335 36. 5 See id. at 336 (describing the power the chief justice of Canada is given to decide the size and composition of the panel hearing an appeal ). 6 See Brown & Lee, supra note 1, at 1069 78 (describing the various methods by which United States courts of appeals randomly assign judges to panels). 17

18 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 selection also permits the Selector to game the outcome in particular cases. 7 Random assignment prevents such purposeful gaming, but allows for unrepresentative outlier panels to form simply by the luck of the draw. 8 This Essay introduces a new method for selecting subsets that combines the best elements of both the direct selection method and random assignment, while avoiding their pitfalls. 9 This new method which I call the rank-order method creates subsets that are judicially efficient and representative of the appellate body as a whole. The rank-order method is specifically designed for use in politically charged cases where the reactions of specific judges to the particular case can be predicted with a fair degree of accuracy. 10 Importantly, the rank-order method also mitigates against possible judicial gaming. 11 This Essay proceeds as follows: Part I discusses the fatal flaws of random assignment and direct selection: outlier panels and judicial gaming, respectively. Part II introduces the rank-order method and explains how this method is superior to either random assignment or direct selection. Part III provides detailed examples of how the rank-order method works in practice. Part IV concludes. I. THE FATAL FLAWS OF RANDOM ASSIGNMENT AND DIRECT SELECTION: OUTLIER PANELS AND JUDICIAL GAMING A. The Random Assignment Method Random assignment is the subset selection method that is (mostly) used in the United States federal appellate courts. 12 Essentially, in each of the federal appellate circuits, panels of three are randomly selected from the entire appellate body to hear each oral argument. 13 Random assignment has several advantages over other methods. First, it prevents any manipulation of the panels. 14 Second, it allows for 7 See Alarie, Green & Iacobucci, supra note 3, at 336. 8 Alex Kozinski & James Burnham, I Say Dissental, You Say Concurral, 121 YALE L.J. ONLINE 601, 607, 609 (2012). 9 The author of this Essay previously developed a method that was designed specifically for the United States courts of appeals. See Michael Hasday, Ending the Reign of Slot Machine Justice, 57 N.Y.U. ANN. SURV. AM. L. 291 (2000). The rank-order method, while using elements of the earlier method, has broader applicability because it can be used by any appellate court around the world, regardless of the subset size. Additionally, while the earlier method relied upon litigant preferences, the rank-order method is more likely to be actually employed by courts because it can be used covertly. Finally, the rank-order method contains an algorithm to equalize the chances of any particular judge on the appellate body being selected for a particular case, a mechanism that can be applied to either the earlier method or the rank-order method. 10 RICHARD A. POSNER, HOW JUDGES THINK 23 (2008). 11 See infra notes 29 31 and accompanying text. 12 See Brown & Lee, supra note 1, at 1069 78. A recent study demonstrates that panel assignment in the United States federal appellate courts is not completely random. See Adam S. Chilton & Marin K. Levy, Challenging the Randomness of Panel Assignment in the Federal Courts of Appeals, 101 CORNELL L. REV. 1 (2015). 13 See Brown & Lee, supra note 1, at 1069 78. 14 See Adam M. Samaha, Randomization in Adjudication, 51 WM. & MARY L. REV. 1, 15 17 (2009).

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 19 cases to be equally distributed among the judges. 15 Third, it is arguably the easiest to defend to the public. 16 The fatal flaw of random assignment, however, is outlier panels that is, panels that are not representative of the appellate body as a whole. Representative panels are beneficial because they contain judges who have a diverse range of perspectives and thus are more likely to identify the correct outcome in cases where one outcome is clearly preferable, and are more likely to reach a moderate outcome in cases where no clearly preferable outcome exists. 17 In contrast, outlier panels cause appellate decisionmaking to be inconsistent, unpredictable, and ideologically extreme. 18 Moreover, the Condorcet Jury Theorem 19 suggests that if each judge on the appellate body has a better than fifty/fifty chance to reach the right answer for any given case, then an outlier decision is going to be the wrong one. 20 It is easy to see how random assignment would create outlier panels as a matter of simple probability. For example, if someone randomly selected a three-judge panel from a thirteen-member appellate body that would (hypothetically) vote eight to five in a given case, there would be a nearly one in three chance for an outlier panel to result. 21 Indeed, a recent study using sophisticated mathematic techniques found that the problem of outlier panels is severe in a random assignment system. 22 This study, which focused on the Ninth Circuit Court of Appeals, determined that at least nine percent of all decisions would be decided differently if randomly reassigned and at 15 See id. 16 See United States v. Mavroules, 798 F. Supp. 61, 61 (D. Mass. 1992) (noting that random assignment prevents judge shopping by any party, thereby enhancing public confidence in the assignment process ). 17 Samuel P. Jordan, Early Panel Announcement, Settlement, and Adjudication, 2007 BYU L. REV. 55, 95 (2007). Jordan concludes that, [i]n short, diversity of viewpoints improves the accuracy and consistency of the court s decisionmaking when viewed as a whole. Id. at 96. 18 See Hasday, supra note 9, at 295 98. 19 For background on the Condorcet Jury Theorem, see Maxwell L. Stearns, The Condorcet Jury Theorem and Judicial Decisionmaking: A Reply to Saul Levmore, 3 THEORETICAL INQ. L. 125, 130 (2002) ( Under specified conditions, the [Condorcet Jury] Theorem provides that simple majority rule increases the likelihood that a group, or a single decisionmaker relying upon group output, will select the correct outcome.... The Jury Theorem posits that if each decisionmaker has a greater than 50% chance of selecting the correct answer and if none of the members is an expert (or if experts cannot be identified in advance), then the probability of selecting the correct answer increases along with the size of the jury. ). 20 See Michael Abramowicz, Essay, En Banc Revisited, 100 COLUM. L. REV. 1600, 1630 36 (2000). 21 I am assuming that if a panel splits two to one in favor of the appellant, it will come down in favor of the appellant, and vice versa. To determine the percentage of randomly assigned threejudge subsets matching en banc rulings of the entire appellate body, where an x-judge appellate body would split y z in an en banc ruling, the formula is: (A + B) / C, where A=y! / [3!(y 3)!], B = z{y! / [2!(y 2)!]}, C=x! / [3!(x 3)!]. 22 See Ryan Copus & Ryan Hübert, Targeting Inconsistency: The Why and How of Studying Disagreement in Adjudication (July 21, 2016) (unpublished draft), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2812914.

20 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 least forty percent of all decisions could be decided differently depending on the panel. 23 Random assignment methods try to account for the problem of outlier panels through the use of en banc panels that is, the entire appellate body can decide to hear and possibly reverse any case after the randomly selected subset decides it. In this way, the majority of the appellate body theoretically has control of every case. For example, in the United States courts of appeals, each of the appellate circuits allows for the use of en banc panels to review cases decided by randomly selected, three-member subsets. 24 Requests for en banc review are frequent, moreover, because of the large number of outlier panels in the appellate circuits. As Ninth Circuit Court of Appeals Judge Alex Kozinski remarked: In the days when federal courts of appeals were much smaller, en banc activity was relativity rare. This is because most judges participated in a significant number of key decisions, and this usually kept circuit law in line with the views of a majority of the court s active judges. But as courts have grown, outlier panels happen more frequently, commensurately increasing the number of en banc calls. 25 However, the cost of having every judge on a circuit hear a case inevitably results in a situation where en banc panels hear only a small fraction of outlier decisions. Indeed, statistics show that en banc panels review considerably less than one percent of all federal appellate court decisions. 26 B. The Direct Selection Method In contrast to random assignment, direct selection theoretically allows for majority control of all subset panels. In the Supreme Court of Canada, for example, the chief justice can unilaterally determine whether the high court will decide a particular case with all nine of its judges or will use a subset of either seven or five judges. 27 If opting for a subset, the chief justice simply decides which judges are on the panel. 28 If a particular case would be decided 5 4 by the full set of nine judges, the chief justice can assure majority control by placing at least three members of the 23 Id. at 32. Numerous other studies have shown that the outcome of three-judge panels is influenced by how many Republican-appointed (R) or Democratic-appointed (D) judges are on them. See, e.g., Deborah Beim & Jonathan P. Kastellec, The Interplay of Ideological Diversity, Dissents, and Discretionary Review in the Judicial Hierarchy: Evidence from Death Penalty Cases, 76 J. POL. 1074, 1080 (2014); Cass R. Sunstein et al., Ideological Voting on Federal Courts of Appeals: A Preliminary Investigation, 90 VA. L. REV. 301, 305 (2004). This also provides evidence that random assignment systems in appellate courts produce outlier panels, i.e., most likely when a majority R appellate circuit has a majority D three-judge panel, and vice versa. 24 See LAURAL HOOPER ET AL., FED. JUDICIAL CTR., CASE MANAGEMENT PROCEDURES IN THE FEDERAL COURTS OF APPEALS 35 (2d ed. 2011). The Ninth Circuit has what is called a limited en banc, which is just an eleven-member subset that is randomly selected, except for the chief judge who is guaranteed a spot. Id. at 36. The rank-order method could be utilized to create limited en banc panels as well. 25 Kozinski & Burnham, supra note 8, at 609. 26 See HOOPER ET AL., supra note 24, at 36. 27 See Alarie, Green & Iacobucci, supra note 3, at 336. 28 See id.

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 21 majority coalition in the five-member subset, or four members of the majority coalition in the seven-member subset (assuming, of course, that the chief justice can accurately predict the members of the majority coalition). The fatal flaw of direct selection is judicial gaming that the Selector will deliberately choose a panel to promote his or her policy views rather than the collective policy views of the entire circuit. Alarie, Green, and Iacobucci found that the chief justice could have potentially manipulated about seventeen percent of all cases heard by the Supreme Court of Canada from 1954 to 2013. 29 Alarie, Green, and Iacobucci did not present much evidence that the respective Canadian chief justices over that period actually employed judicial gaming in the sense of arranging subset panels to reflect the policy views of the chief justice instead of the court as a whole. 30 But the potential is there. As they write, both the discretionary nature of the appointment process and the nature of norms imply a risk that, while manipulation of the panels did not materialize in Canada in the recent past, it could happen in the future. 31 Even without judicial gaming per se, direct selection can produce outlier panels. For example, the Selector may employ a seemingly neutral qualification to pick the judges for a particular case such as experience or perceived subjectmatter expertise but that neutral qualification may result in a panel whose collective views are unrepresentative of the entire appellate body. Even if a majority of the judges choose the Selector who would accordingly be expected to select subset panels that reflect majority views in any important case the Selector may be motivated to center the panel at the median of the majority, rather than the median of the appellate body as a whole. Consequently, while the subset panel may vote like the majority of the entire appellate body would vote on a given case, the panel would likely issue a more extreme decision than a truly representative panel would issue. Imagine, for example, the 2017 United States Supreme Court as a nine-member appellate body. It is easy to see that a three-judge panel of Neil Gorsuch, Clarence Thomas, and Samuel Alito might vote the same as a panel of Stephen Breyer, Anthony Kennedy, and John Roberts on a given case, but the underlying decision would likely be much more extreme. Finally, even if the Selector tries in good faith to form representative panels, direct selection creates many other problems. As Adam Samaha has stated, direct selection may create distrust and disagreement among judges as the Selector may use its assignment power to steer cases away from a disfavored class of judges or, at the very least, to maintain an existing pecking order or division of expertise that incoming judges might prefer to disrupt. 32 In short, direct selection has many problems even apart from judicial gaming. 29 See id. at 344, 346. These scholars report that if the United States Supreme Court had the same discretionary assignment system as Canada, slightly more than one third of its decisions over a similar time frame would have been manipulable. See id. at 344 n.32. 30 See id. at 381. 31 Id. 32 Samaha, supra note 14, at 71.

22 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 II. THE RANK-ORDER METHOD IS SUPERIOR TO EITHER RANDOM ASSIGNMENT OR DIRECT SELECTION Consider this assessment from then-seventh Circuit Court of Appeals Judge Richard Posner: Every lawyer knows that the accident of which judges of a court of appeals are randomly drawn to constitute the panel that will hear his case may determine the outcome if the case is controversial. Every judge is aware of having liberal and conservative colleagues whose reactions to politically charged cases can be predicted with a fair degree of accuracy even if the judge who affixes these labels to his colleagues would not like to be labeled politically himself. 33 Posner was right, but he may not have realized that appellate courts can use the very predictability of how judges react to cases to prevent cases from being decided based upon the accident of panel assignment. This is where the rank-order method comes in. The rank-order method is designed to create panels that are representative of the entire appellate body. It is superior to both random assignment and direct selection because it avoids their respective fatal flaws: outlier panels and judicial gaming. Here is how the method works. First, the chief judge or designated court administrator (hereinafter, the Ranker ) 34 decides based on considerations of judicial efficiency how many judges should hear a particular case. Second, the Ranker rank-orders all of the judges on the appellate body from the judge that is perceived to be most favorable to the appellant to the judge that is perceived to be least favorable to the appellant. Third, matching panels are created in the following manner: 1. Let x = the number of judges in the appellate body. 2. Let n = the number of judges desired on the subset panel. 3. Let y = n(x+1)/2. 4. Form all panels where the sum of the ranks of the n judges equals y. 5. Eliminate certain panels in order to have the judges equally (or close to equally) distributed among the panels. (Note that the precise method for this step will be explained in detail in Part III.) 6. Randomly select the panel from the panels remaining after Step 5. Accordingly, under the rank-order method, the fatal flaw of random selection outlier panels is eliminated. Indeed, the rank-order method is 33 POSNER, supra note 10, at 23. 34 The Ranker can also be the collective views of a committee, which may consist of judges on the court, court personnel, and/or outside experts. Under this method, courts can opt to have the entire process be a black box with only the resulting panel made public.

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 23 specifically designed to create representative or average panels. To wit: under this method, since an average judge has a ranking of (x+1)/2, the sum of the rankings of the n average judges is n(x+1)/2, represented by y in Step 3 above. We can see how the rank-order method is better at eliminating outlier panels than random selection by comparing how both methods produce panels that match en banc rulings. Imagine an eleven-judge appellate body that would decide a case seven judges for the appellant and four judges for the appellee. Under a randomly assigned three-judge subset, about seven out of ten of the decisions match the en banc ruling (i.e., three out of ten would be outlier panels). Under the rank-order method, assuming an accurate ranking, nine out of ten subsets match en banc rulings (i.e., just one out of ten would be outlier panels). The chart below shows a fuller statistical comparison between the two methods for three-judge subsets: 7-Judge Circuit Split 4 3 5 2 6 1 7-0 Random Assignment 63% 86% 100% 100% Rank-Order Method 80% 100% 100% 100% 9-Judge Circuit Split 5 4 6 3 7 2 8 1 9 0 Random Assignment 60% 77% 92% 100% 100% Rank-Order Method 67% 100% 100% 100% 100% 11-Judge Circuit Split 6 5 7 4 8 3 9 2 10 1 11 0 Random Assignment 58% 72% 85% 95% 100% 100% Rank-Order Method 64% 91% 100% 100% 100% 100% It is important to note that, if anything, this understates the effectiveness of the rank-order method in eliminating outlier panels. Indeed, the outlier panels that the rank-order method produces are actually just near misses. For example, for a fifteenjudge appellate body that splits ten to five, the rank-order method may produce the following final panels, A T, with the numbers representing the rank of each judge 35 : 35 For this example, I am assuming that the following five panels are eliminated per Step 5: (3, 8, 13), (4, 8, 12), (5, 9, 10), (6, 7, 11), and (7, 8, 9). Under Step 5, a different set of panels could be eliminated but in no event would the number of outlier panels among the final panels in a 10 5 circuit split be greater than one out of twenty. By way of comparison, random assignment will produce outlier panels about one out of four times in a 10 5 circuit split.

24 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 15-Judge Circuit A: 1, 8, 15 B: 1, 9, 14 C: 1, 10, 13 D: 1, 11, 12 E: 2, 7, 15 F: 2, 8, 14 G: 2, 9, 13 H: 2, 10, 12 I: 3, 6, 15 J: 3, 7, 14 K: 3, 9, 12 L: 3, 10, 11 M: 4, 5, 15 N: 4, 6, 14 O: 4, 7, 13 P: 4, 9, 11 Q: 5, 6, 13 R: 5, 7, 12 S: 5, 8, 11 T: 6, 8, 10 Accordingly, nineteen of the twenty panels under the rank-order method would match the hypothetical en banc panel under this example, which assumes that if two out of three judges would vote a certain way, the three-judge panel would decide

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 25 that way. (Except for Panel D, all of the panels have at least two judges with ranks 1 10, thus certifying an en banc match if the rank-order was accurate.) However, even the one outlier panel Panel D is a near miss. Panel D has the No. 1 ranked judge, and then two judges that just miss being in the majority coalition, the No. 11 and No. 12 ranked judges. There is a panel effects literature that teaches us that in certain instances, a minority judge who feels particularly strong about a case can actually control the outcome of the case, if the majority judges only tepidly support their decision. 36 Even the rare outlier panel that the rankorder method produces may not be much of an outlier panel after all. Additionally, the rank-order method mitigates against the problem of judicial gaming the fatal flaw of direct selection. While it is relatively easy to manipulate the outcome under a direct selection system, it is much harder under the rank-order method. To see why, consider the scenario in which the Selector (under direct selection) or Ranker (under the rank-order method) is trying to game the selection process by producing a subset panel in which the minority coalition prevails. The Selector can easily game the process by stacking the subset panel with minority judges. Although there might be a political cost to this maneuver it should be readily apparent to the judges what the Selector is doing such a maneuver will undoubtedly be effective in producing a minority panel. Under the rank-order method, the ability of the Ranker to game the system depends on the number of judges in the appellate body, the number of judges in the subset panel, the number of judges in the majority, the number of final panels after rank-order methodology is applied, and whether each judge appears the same number of times in the final panels. However, while the Ranker can somewhat increase the odds of a minority panel by submitting a strategic/inaccurate ranking, there is no situation in which the Ranker can come close to guaranteeing that a minority panel will result. Consider the simple example of a three-judge subset from a nine-judge appellate body. Under the rank-order method, here are the panels that result (A F, with the numbers representing the rank of each judge). 37 9-Judge Circuit A: 1, 5, 9 B: 1, 6, 8 C: 2, 4, 9 D: 2, 6, 7 36 See, e.g., Jonathan P. Kastellec, Race, Context and Judging on the Courts of Appeals: Race-Based Panel Effects in Death Penalty Cases 2 (July 26, 2016), (unpublished paper) https://ssrn.com/abstract=2594946 ( [I]n cases that implicate the interests of women or minorities, the addition of a single woman or minority judge, respectively, to an otherwise all-white or allmale three-judge panel significantly increases the likelihood of a liberal decision by the panel. ). 37 Two panels are eliminated per Step 5: (2, 5, 8) and (4, 5, 6). Note that the remaining judges are each represented among the six acceptable panels two times each.

26 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 E: 3, 4, 8 F: 3, 5, 7 Assume that judges with the ranks one to six are the majority judges, while judges with the ranks seven to nine are the minority judges. Under the rank-order method, with an accurate ranking, all the panels will result in a majority decision because each of the panels, A to F, contain at least two judges with the numbers one to six. However, let s assume that the Ranker does not want a majority outcome and is trying to game the system to produce a minority outcome. Under this circumstance, the Ranker could rank the minority judges as one, six, and nine, which would give the Ranker a one in three chance of producing a minority panel (A and B now each produces a minority panel). However, no matter how the Ranker ranks the judges, the Ranker cannot increase the odds of a minority panel to greater than one in three. Thus, the Ranker will face the same political cost as the Selector, as it will be obvious to the entire appellate body that the Ranker is submitting an incorrect ranking given that similar judges are being placed at opposite ends of the ranking spectrum. But the payoff will be much less. Finally, the rank-order method only requires a reasonably accurate ranking to be effective not total accuracy so an appellate court need not be concerned that minor errors in forming the ranking will somehow bias one party or the other. In the example above, for instance, the accuracy rate will not be affected by erroneously having the true No. 5 judge in the No. 4 or No. 6 spot in the ranking order. III. THE PROPOSAL IN PRACTICE This Part will describe in detail how the rank-order method works in practice, using the following two examples. The first example is applicable to the United States courts of appeals, where subsets of three are selected from appellate bodies of varying sizes. The second example is applicable to the Supreme Court of Canada, where subsets of five or seven may be selected from the full set of nine judges. A. Example 1 (American) Imagine an appellate body with eleven judges, in which it is determined for judicial efficiency reasons that three is the optimal number of judges to decide the case. Judge A is perceived to be the most favorable to the appellant, Judge B is the next most favorable, and so on until Judge K who is least favorable. Thus, the Ranker would then rank the judges as follows: 1. Judge A 2. Judge B 3. Judge C

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 27 4. Judge D 5. Judge E 6. Judge F 7. Judge G 8. Judge H 9. Judge I 10. Judge J 11. Judge K Here, x = 11 and n = 3. Thus, y = 18. Per Step 4, panels would then be formed where the sum of the ranks of the judges equals 18. For example, a panel with Judge A, Judge F, and Judge K would equal 18 (1 + 6 + 11). Accordingly, here are all the acceptable panels under this example: Panel 1: Judge A, Judge F, Judge K (ranks 1, 6, 11) Panel 2: Judge A, Judge G, Judge J (ranks 1, 7, 10) Panel 3: Judge A, Judge H, Judge I (ranks 1, 8, 9) Panel 4: Judge B, Judge E, Judge K (ranks 2, 5, 11) Panel 5: Judge B, Judge F, Judge J (ranks 2, 6, 10) Panel 6: Judge B, Judge G, Judge I (ranks 2, 7, 9) Panel 7: Judge C, Judge D, Judge K (ranks 3, 4, 11) Panel 8: Judge C, Judge E, Judge J (ranks 3, 5, 10) Panel 9: Judge C, Judge F, Judge I (ranks 3, 6, 9) Panel 10: Judge C, Judge G, Judge H (ranks 3, 7, 8) Panel 11: Judge D, Judge E, Judge I (ranks 4, 5, 9) Panel 12: Judge D, Judge F, Judge H (ranks 4, 6, 8) Panel 13: Judge E, Judge F, Judge G (ranks 5, 6, 7)

28 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 However, per Step 5, we eliminate certain of the above panels so that the judges appear equally (or close to equally) among the panels. Here is how this step works: We count the number of times each judge appears on the Step 4 panels and then eliminate the panels one-by-one using these rules: (a) eliminate the panel that contains none of the judges that least commonly appear; (b) if more than one panel satisfies the condition of Step 5(a), then use the following as tiebreakers: (i) the panel with the greatest number of collective appearances of all panel members (first tiebreaker); (ii) random selection (second tiebreaker); (c) repeat this panel elimination process until all the judges are on the panels in equal number or no panel satisfies the condition of Step 5(a). Accordingly, under this example, we count the number of times each judge appears on the acceptable panels and get the following: Judge A: 3 times Judge B: 3 times Judge C: 4 times Judge D: 3 times Judge E: 4 times Judge F: 5 times Judge G: 4 times Judge H: 3 times Judge I: 4 times Judge J: 3 times Judge K: 3 times Then per Step 5(a), we eliminate the panel that contains none of the judges that least commonly appear. In this case, that would be a panel that does not contain Judge A, Judge B, Judge D, Judge H, Judge J, or Judge K. Here, both Panel 9 and

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 29 Panel 13 satisfy this condition. Thus, we then use the first tiebreaker to see which panel is eliminated, which is the collective appearances of all panel members. Panel 9 has Judge C (4 appearances), Judge F (5 appearances), and Judge I (4 appearances), so the collective appearances is 13 (4 + 5 + 4). Panel 13 has Judge E (4 appearances), Judge F (5 appearances), and Judge G (4 appearances), so the collective appearances is also 13 (4 + 5 + 4). Thus, we go to the second tiebreaker, which is random selection. Here, we randomly select Panel 9 for elimination. 38 Accordingly, we then start the process again (with Panel 9 eliminated) and get the following counts: Judge A: 3 times Judge B: 3 times Judge C: 3 times Judge D: 3 times Judge E: 4 times Judge F: 4 times Judge G: 4 times Judge H: 3 times Judge I: 3 times Judge J: 3 times Judge K: 3 times Then per Step 5(a), we eliminate the panel that contains none of the judges that least commonly appear. In this case, that would be a panel that does not contain Judge A, Judge B, Judge C, Judge D, Judge H, Judge I, Judge J, or Judge K. Only one panel satisfies this criterion: Panel 13. After Panel 13 is eliminated, we get the following count: Judge A: 3 times Judge B: 3 times Judge C: 3 times 38 In this example, the end result will be the same whether we randomly select Panel 9 or Panel 13 for elimination, as the unselected panel will be eliminated in the next round.

30 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 Judge D: 3 times Judge E: 3 times Judge F: 3 times Judge G: 3 times Judge H: 3 times Judge I: 3 times Judge J: 3 times Judge K: 3 times Since all the judges are equally distributed, we stop per Step 5(c). We then randomly select a panel from the remaining eleven panels (note that the judges are equally distributed among the panels): Panel 1: Judge A, Judge F, Judge K (ranks 1, 6, 11) Panel 2: Judge A, Judge G, Judge J (ranks 1, 7, 10) Panel 3: Judge A, Judge H, Judge I (ranks 1, 8, 9) Panel 4: Judge B, Judge E, Judge K (ranks 2, 5, 11) Panel 5: Judge B, Judge F, Judge J (ranks 2, 6, 10) Panel 6: Judge B, Judge G, Judge I (ranks 2, 7, 9) Panel 7: Judge C, Judge D, Judge K (ranks 3, 4, 11) Panel 8: Judge C, Judge E, Judge J (ranks 3, 5, 10) Panel 10: Judge C, Judge G, Judge H (ranks 3, 7, 8) Panel 11: Judge D, Judge E, Judge I (ranks 4, 5, 9) Panel 12: Judge D, Judge F, Judge H (ranks 4, 6, 8) B. Example 2 (Canadian) Imagine an appellate body with nine judges, in which it is determined for judicial efficiency reasons that five is the optimal number of judges to decide the case. Judge A is perceived to be the most favorable to the appellant, Judge B is the

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 31 next most favorable, and so on until Judge I who is least favorable. Thus, the Ranker would then rank the judges as follows: 1. Judge A 2. Judge B 3. Judge C 4. Judge D 5. Judge E 6. Judge F 7. Judge G 8. Judge H 9. Judge I Here, x = 9 and n = 5. Thus, y = 25. Per Step 4, panels would then be formed where the sum of the ranks of the judges equals 25. For example, a panel with Judge A, Judge B, Judge E, Judge H, and Judge I would equal 25 (1 + 2 + 5 + 8 + 9). Accordingly, here are all the acceptable panels under this example: Panel 1: Judge A, Judge B, Judge E, Judge H, Judge I (ranks 1, 2, 5, 8, 9) Panel 2: Judge A, Judge B, Judge F, Judge G, Judge I (ranks 1, 2, 6, 7, 9) Panel 3: Judge A, Judge C, Judge D, Judge H, Judge I (ranks 1, 3, 4, 8, 9) Panel 4: Judge A, Judge C, Judge E, Judge G, Judge I (ranks 1, 3, 5, 7, 9) Panel 5: Judge A, Judge C, Judge F, Judge G, Judge H (ranks 1, 3, 6, 7, 8) Panel 6: Judge A, Judge D, Judge E, Judge F, Judge I (ranks 1, 4, 5, 6, 9) Panel 7: Judge A, Judge D, Judge E, Judge G, Judge H (ranks 1, 4, 5, 7, 8) Panel 8: Judge B, Judge C, Judge D, Judge G, Judge I (ranks 2, 3, 4, 7, 9) Panel 9: Judge B, Judge C, Judge E, Judge F, Judge I (ranks 2, 3, 5, 6, 9) Panel 10: Judge B, Judge C, Judge E, Judge G, Judge H (ranks 2, 3, 5, 7, 8) Panel 11: Judge B, Judge D, Judge E, Judge F, Judge H (ranks 2, 4, 5, 6, 8)

32 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 Panel 12: Judge C, Judge D, Judge E, Judge F, Judge G (ranks 3, 4, 5, 6, 7) However, per Step 5, we eliminate certain of the above panels so that the judges appear equally (or close to equally) among the panels. Accordingly, under this example, we count the number of times each judge appears on the acceptable panels and get the following: Judge A: 7 times Judge B: 6 times Judge C: 7 times Judge D: 6 times Judge E: 8 times Judge F: 6 times Judge G: 7 times Judge H: 6 times Judge I: 7 times Then per Step 5(a), we eliminate the panel that contains none of the judges that least commonly appear. In this case, that would be a panel that does not contain Judge B, Judge D, Judge F, or Judge H. Here, only Panel 4 satisfies this condition. Accordingly, we then start the process again (with Panel 4 eliminated) and get the following counts: Judge A: 6 times Judge B: 6 times Judge C: 6 times Judge D: 6 times Judge E: 7 times Judge F: 6 times Judge G: 6 times Judge H: 6 times

2017] THE RANK- ORDER METHOD FOR APPELLATE SUBSET SELECTION 33 Judge I: 6 times Since none of the remaining panels satisfies the condition of Step 5(a), we stop per Step 5(c). We then randomly select a panel from the remaining eleven panels. Panel 1: Judge A, Judge B, Judge E, Judge H, Judge I (ranks 1, 2, 5, 8, 9) Panel 2: Judge A, Judge B, Judge F, Judge G, Judge I (ranks 1, 2, 6, 7, 9) Panel 3: Judge A, Judge C, Judge D, Judge H, Judge I (ranks 1, 3, 4, 8, 9) Panel 5: Judge A, Judge C, Judge F, Judge G, Judge H (ranks 1, 3, 6, 7, 8) Panel 6: Judge A, Judge D, Judge E, Judge F, Judge I (ranks 1, 4, 5, 6, 9) Panel 7: Judge A, Judge D, Judge E, Judge G, Judge H (ranks 1, 4, 5, 7, 8) Panel 8: Judge B, Judge C, Judge D, Judge G, Judge I (ranks 2, 3, 4, 7, 9) Panel 9: Judge B, Judge C, Judge E, Judge F, Judge I (ranks 2, 3, 5, 6, 9) Panel 10: Judge B, Judge C, Judge E, Judge G, Judge H (ranks 2, 3, 5, 7, 8) Panel 11: Judge B, Judge D, Judge E, Judge F, Judge H (ranks 2, 4, 5, 6, 8) Panel 12: Judge C, Judge D, Judge E, Judge F, Judge G (ranks 3, 4, 5, 6, 7) CONCLUSION Random assignment and direct selection are the two most straightforward methods to select appellate subsets. But they are simple not optimal. Alarie, Green, and Iacobucci have remarked that while these two methods may have once had their role, a more nuanced assignment method for the United States and Canada is needed in the future. 39 Indeed, it has been suggested that certain circuits of the United States courts of appeals are already moving away from random assignment, perhaps because of the concern over outlier panels. 40 Direct selection will also likely face increasing pressure in Canada, as the selection process for 39 See Alarie, Green & Iacobucci, supra note 3, at 381. 40 See Chilton & Levy, supra note 12, at 50 n.164 (noting that [t]he Second Circuit... appears to have fewer panels with no judges appointed by a Republican president and fewer panels with all three judges appointed by a Republican president than expected by a purely random assignment system). Since the Second Circuit was roughly evenly divided between Democraticappointed and Republican-appointed judges during the period studied (2008 2012), see id. at 31 tbl.3, and given that the party of the appointing president is a rough proxy for voting patterns across certain issues, see Sunstein et al., supra note 23, at 305, a logical inference is that having mixed partisan panels will make outlier decisions less likely.

34 NOTRE DAME LAW REVIEW ONLINE [VOL. 93 becoming a judge on that court is becoming more politicized. 41 To be sure, for the ordinary case, the flaws of random assignment and direct selection may not be that apparent. But for politically charged cases, the flaws become fatal. The rank-order method is both a middle ground between the extremes of random assignment and direct selection, and a marked advancement. Appellate courts should use this method going forward. 41 See Alarie, Green & Iacobucci, supra note 3, at 381 n.67.