PARLIAMENTARY STUDIES PAPER 11

Transcription

1 PARLIAMENTARY STUDIES CENTRE CRAWFORD SCHOOL OF ECONOMICS AND GOVERNMENT OF ECONOMICS AND GOVERN- A Statistical Analysis of Government Responses to Committee Reports: Reports Tabled between the 2001 and 2004 Elections David Monk ANU COLLEGE OF ASIA & THE PACIFIC PARLIAMENTARY STUDIES PAPER 11

2 About the Parliamentary Studies Centre The Parliamentary Studies Centre is devoted to furthering academic research into parliamentary institutions, structures and processes, with a particular emphasis on comparative studies. The Centre operates as a research broker or facilitator, as distinct from a stand-alone research entity. Funding is sought for researchers who are already well placed to carry out relevant research, thereby minimising organisational overheads. The Centre was established by the Policy and Governance Program in the Crawford School of Economics and Government and the Political Science Program, Research School of Social Sciences at the Australian National University. In 2007, the Centre began a three-year project entitled Strengthening Parliamentary Institutions, funded through an Australian Research Council linkage grant and co-sponsored by the Department of the Senate and the House of Representatives, Commonwealth Parliament of Australia. The research consists of case studies of aspects of the Australian Parliament and comparative studies of institutional strengthening in legislatures elsewhere in Australia and overseas. The Centre welcomes expressions of interest from researchers seeking advice or assistance with academic research in parliamentary studies, including those interested in participating in the current project on strengthening parliamentary institutions. About the Author David Monk is Director, Executive and Committee Support, at the Department of the House of Representatives in Canberra. He has worked for a number of parliamentary committees and central government agencies in New South Wales and Canberra. He is currently undertaking a doctorate in political science at the Australian National University. This paper may be read in conjunction with Parliamentary Studies Paper 10 by David Monk, which describes the rationale for the approach adopted here. Readers with a particular interest in the topic are also encouraged to review the dataset on which this paper is based. Both are available on the Parliamentary Studies Centre website; see publications.php. Crawford School of Economics and Government, The Australian National University ISSN [online] The views expressed are those of the individual authors and are not necessarily those of the Crawford School of Economics and Government, The Australian National University. This paper can be cited as: Monk, David (2009), A statistical analysis of government responses to committee reports:reports tabled between the 2001 and 2004 elections, Parliamentary Studies Paper 11, Crawford School of Economics and Government, Australian National University, Canberra.

3 A Statistical Analysis of Government Responses to Committee Reports: Reports Tabled between the 2001 and 2004 Elections David Monk This paper measures government acceptance of parliamentary committee recommendations. This is one way in which a committee can demonstrate its effectiveness. Of the reports in our sample recommending new action by the government, 60.0 per cent demonstrated a minimum level of effectiveness in that at least one of their recommendations was accepted. The most important factors correlated with a report s effectiveness were the type of committee, the type of inquiry, the level of bipartisanship, the level of press coverage and whether a majority report contained recommendations. INTRODUCTION The requirement for governments to respond to committee reports has existed since the 1970s. The Senate set up its extended committee system in 1970, and in 1973 passed a resolution requiring the government to respond to Senate committee recommendations within three months. 1 In 1978, the government declared that it would respond to all committee reports within six months, effectively extending the Senate system to the House of Representatives and joint committees. This period was reduced to three months in In most cases, the government response takes the form of a document tabled in the relevant chamber(s). For bill inquiries, the practice is for the relevant minister to comment on committee reports during debate on the bill. These comments tend to be briefer than the tabled written responses. A government response is a significant milestone in the life of an inquiry. In it, the government usually lists what action it plans to take, or has taken, with respect to each recommendation. Where the government rejects a recommendation, it will often state the reasons. The House of Representatives Standing Committee on Procedure has noted the importance of a government response, stating that, without some indication that the government has at least seriously considered the recommendations in a report, then The value of the activity becomes questionable. 3 As I suggest in the companion paper to this one, 4 because committees are political entities operating in a political environment, it seems fair to evaluate them according to how various groups react subjectively to their reports. One way of measuring how the government perceives a report is to calculate the number of recommendations it agrees to implement. This is not a perfect measure, because in practice the government may not meet its commitments. As an approximation or proxy of the government s perception, however, it can be considered a reasonable measure of a report s effectiveness. Committee reports have various aims and there are a number of diverse groups in the political system. Therefore, it seems fair to view the government response as just one way among many in which committees can demonstrate their effectiveness. If other groups, such as stakeholders, voters or the legislature, respond favourably to a committee report, then the lack of a government response should not prevent that report from being considered effective. In other words, a positive government response is a sufficient, but not necessary, way for committees to demonstrate their effectiveness. COLLECTING THE DATA Notwithstanding the three-month requirement, governments have sometimes taken over three years to draft and table a response to a committee report. parliamentary studies PAPER 11 Parliamentary Studies Centre, Crawford School of Economics and Government, ANU College of Asia & the Pacific, The Australian National University

4 Therefore, the start date for this study needed to be at least three years before the project commenced (in September 2007). This led to the selection of the period between the 2001 and 2004 elections (the 40th Parliament) as the focus of the study. The Senate and House registers of committee reports were copied and combined into one master list of 512 reports. A list of random numbers was generated and paired with this master list, giving a random ordering of reports. Because some joint committees were included in both chambers registers, House listings of such reports were omitted and only the Senate listings were used. The master list then comprised 496 reports. The reports were assessed in the order in which they appeared on the random list. To be included in the sample, a majority or minority report needed to make at least one recommendation to the government for new action. Previous studies have noted that committees are sometimes able to word recommendations in such a way that the government finds it easy to accept them, thus inflating the acceptance rate. 5 This study s requirement for new action was to prevent the distorting effect of such recommendations. Reports that did not make recommendations were excluded from the sample. This meant that Senate estimates reports and the regular Senate inquiries into annual reports were not included. Nor were the activities of the Senate Standing Committee on Regulations and Ordinances, which rarely tables reports. It also meant that reports containing single recommendations supporting proposed government action, as do many of the reports by the Parliamentary Standing Committee on Public Works, were excluded. In total, 196 reports from the master list of 496 were examined to generate a sample of 95. The sample comprised 34 joint committee reports; 21 Senate references and select committee reports; 30 Senate legislation committee reports; and 10 House committee reports. All the reports and government responses were collected from the parliamentary website ( DESCRIBING AND DEFINING THE DATA Attaching numbers to a flexible, subjective process such as committee reporting and the government response requires a number of decisions to be made on rules and definitions, to ensure that the data are prepared in a systematic, uniform way. These rules and definitions are listed below, along with some summary data for the sample. Further information is given in Appendix 1. Government response Generally, this was simple to resolve. For most inquiries, the government tables in one or both chambers a document entitled the government response. In the case of bills, however, the government s response is made orally in chamber debate. For this study, the government was considered to have responded to a report where either a minister or a parliamentary secretary had made a statement, recorded in Hansard, about the government s attitude towards a recommendation. Of the 95 reports in the sample, 63 (66.3 per cent) received a government response. Recommendations Identifying majority recommendations was straightforward. Majority reports generally use automated formatting for recommendations that results in them being listed under a special heading in the text as well as at the front of the report. This made it easy to compile a list of majority recommendations. The recommendations of minority reports were harder to identify, because they do not conform to any set style. The definition of a minority recommendation was therefore widened to include any statement that the government should take some reasonably defined new action, or that a bill should be amended. Terms like should or must generally indicated a recommendation, whereas terms like could or might did not. The next step was to exclude recommendations that were either too easy or too hard for the government to accept or implement. The remainder were termed eligible recommendations. To be deemed eligible, a recommendation had to fall within the government s area of responsibility and require new action. Therefore, recommendations directed at the private sector or at independent entities such as the Australian Securities and Investment Commission were not included. Recommendations for the government to maintain the current arrangements, usually indicated by the word continue, were also excluded. 6 The presence of either majority or minority recommendations in a report depended on the political dynamics of the inquiry. Where there was no dissent, then the report contained majority recommendations only. If there was dissent, then the report usually contained both majority and minority recommendations. It was also possible for a report to contain only minority recommendations. This occurred where a majority on the committee (usually government members) declared that the government should continue with a planned course of action, and the minority raised its own recommendations. Almost 2 PARLIAMENTARY STUDIES CENTRE, CRAWFORD SCHOOL of economics and government

5 Figure 1 Recommendations (majority, minority or both) by type of committee, (%) Joint Senate references Senate legislation House Weighted average Source: Sample of 95 committee reports. Majority only Majority and minority Minority only all such reports were produced by Senate legislation committees. Figure 1 demonstrates the pattern of recommendations made by each type of committee. One of the aims of this paper is to determine whether the type of recommendations in a committee report (minority or majority) affects the government response to a report. Acceptance of a recommendation It was generally straightforward to judge whether or not the government had accepted a recommendation, but there were some situations where this could be less clear. For instance, the government might say that it agreed with a recommendation in principle but did not have the resources to implement it. For this study, the government response to individual recommendations was graded according to its degree of conformity with the proposal. The categories are shown in Figure 2. The first three categories in Figure 2 agreed, partially agreed and under examination are counted as acceptance of a recommendation. Agreed is self-explanatory. Partially agreed occurred where a recommendation had several parts and the government agreed to implement at least one, but not all, of them. It also occurred where the government agreed with the idea behind a recommendation but used a different method to achieve a similar result. Under examination occurred where the government was still considering a recommendation. Given that the government could take several years to respond to a report, it seemed fair to count this as acceptance. After all, if the government had wished to reject such a recommendation, it could have done so at the outset. The principle Figure 2 Grading of possible government responses to individual recommendations Acceptance Agreed Partially agreed Under examination Already being done Agreed in principle Rejection Rejected parliamentary studies PAPER 11 3

6 Table 1 Committees sources of referral and composition, Committee type External referral Self-referral Majority Joint Senate references and select House, Senate and ministers Senate Self-referral sometimes within terms of reference Self-referral sometimes within terms of reference for select committees Government Opposition Senate legislation Senate Estimates and annual reports Government House House and ministers Annual reports and Auditor- General reports Government Source: Harry Evans (ed.), Odgers Australian Senate Practice, 11th edition, Department of the Senate, 2004, p. 382; I.C Harris, B.C. Wright and P.E. Fowler, House of Representatives Practice, 5th edition, Department of the House of Representatives, 2005, pp. 624, 628. underpinning the decision to count such cases as acceptance was that the government had promised new action or at least left open the possibility of it. 7 In a small number of cases, the government made a general comment in support of some or all of a report s recommendations, without specifying which ones it would accept. This was treated as the government accepting one majority and one minority recommendation, where these types of recommendations existed. Such cases were therefore placed in the agreed category. The other three categories are considered rejections because they preclude the government taking new action. Sometimes, a government response will state that programs and procedures are already in place, so no new action is required. Where these programs and procedures were implemented before a report was tabled, this was counted as rejection. If this occurred afterwards, it was counted as acceptance. The disadvantage of this decision is that it is arbitrary. It does not allow for instances of bureaucratic anticipation, where the government and the bureaucracy begin to address an issue before a report is tabled due to committee pressure. The advantage of the rule is that it is clear and allows for consistent decision making within the current limits of information. As noted earlier, in-principle agreement implies that the government will not take new action in relation to a recommendation. Lack of a government response is treated as rejection of all recommendations. In making judgments about government responses, the study was careful to look at the content of the response to each recommendation, not just the heading, which could be misleading. 8 After deciding which recommendations had been accepted, the study calculated two acceptance rates, one for majority recommendations and the other for minority recommendations. Across all reports, the average acceptance rate was 39.5 per cent for majority recommendations and 5.9 per cent for minority recommendations. The government accepted at least one recommendation (either majority or minority) for 57 reports (60.0 per cent). Committee type Over time, the chambers have made changes to their committee systems to suit their needs. During the sample period ( ), there were four main types of committee: (1) joint; (2) Senate references and select; 9 (3) Senate legislation; and (4) House. The Senate introduced the arrangement of having two types of committee in By convention, the government has a majority on the legislation committees and the opposition a majority on the references committees. Table 1 shows the sources of referral to committee and the composition of the various committee types during the sample period. The main point of difference between the committees is that Senate references committees are the only ones on which the opposition has a majority. Within Australia s institutions of state, there are few bodies that an opposition can control. Achieving a majority on the floor of the House delivers that chamber and the whole of the executive to the dominant political party. On the other hand, since 1972 governments have rarely held the balance of power in the Senate. 10 This makes it the one institution where oppositions are able to exert some measure of control. Over time, oppositions have used the prestige and resources of the Senate to promote their values and challenge the government. Thus, the Senate gives parties that have lost the last election, or at least do not have a majority of seats in the House, a second bite of the policy cherry. 11 Due the invisible hand 4 PARLIAMENTARY STUDIES CENTRE, CRAWFORD SCHOOL of economics and government

7 Figure 3 Average acceptance rate of recommendations, (%) a Joint Senate references Senate legislation House Weighted average Majority Minority a Error bars show 95% confidence intervals where sample size and distribution permit. No House committee report in the sample made a minority recommendation. Source: Sample of 95 committee reports. of competition, the opposition s use of the Senate to challenge the government has brought about more democratic outcomes, greater transparency and wider debate, a role that the electorate generally seems to support. 12 As a result, it can be expected that, on average, Senate references committees will have lower acceptance rates than other types of committees due to their traditional lack of cooperation with the government. Figure 3 demonstrates this. As the figure shows, the average acceptance rate for the majority recommendations of Senate references committees was 17.0 per cent in , the lowest rate for any type of committee. This suggests that governments pay the least attention to Senate references committee reports. It also supports the theory that oppositions tend to use Senate references committees for overtly political ends. Or, as one opposition staffer said in March 2008 in relation to the work programs of Senate committees, you use whatever mechanisms you can to get your issues up. 13 Governments responded most favourably to Senate legislation and joint committee reports, which had average acceptance rates for majority recommendations of 54.5 and 52.0 per cent respectively. The most likely reason in the case of the Senate legislation committees is that the scrutiny of bills is a traditional and appreciated aspect of their work. Their high success rate also corroborates John Uhr s comment that committees are better at refining government proposals than conducting larger-scale policy work. 14 In the case of joint committees, the reasons probably lie in the committees high degree of consensus and authority. Joint committees are established by both chambers, giving them an innate consensus that no other type of committee would have. They also have a great deal of authority, because their membership comprises both senators and members. Figure 3 shows that the average acceptance rate for minority recommendations is very low across all committee types. One reason for this may be that majority reports attract the vast majority of the administrative resources expended during the drafting process, and therefore tend to be better researched and argued. The second possible reason is that majority reports enjoy the support of the majority of the committee and therefore carry more weight. The acceptance rate for minority recommendations does not appear to vary greatly across committee type. In a related paper (see note 4), I suggest that a benchmark for committees to demonstrate their effectiveness is for the government to accept at least one of their recommendations. Figure 4 presents the statistics for such reports by committee type. From this perspective, joint committees again perform well, with a high proportion of reports having at least one recommendation accepted. House committees have a high level of reports with at least one recommendation accepted (70 per cent; see Figure 4) but a low overall acceptance rate (22.4 per cent; see Figure 3). In other words, many of their reports pass the effectiveness test in that at least one recom- parliamentary studies PAPER 11 5

8 Figure 4 Proportion of reports where at least one recommendation is accepted, (%) a Joint Senate references Senate legislation House Weighted average a Error bars show 95% confidence intervals. Clopper and Pearson values are used for House committees. Source: Sample of 95 committee reports. mendation is accepted, but overall a high proportion of the recommendations of House committee reports are not accepted. Senate legislation committees have the highest acceptance rate of majority recommendations of all committees (54.5 per cent; see Figure 3) but the lowest proportion of reports with at least one recommendation accepted (38.5 per cent; see Figure 4). One possible explanation for this is that a high proportion of Senate legislation committee reports contain only minority recommendations for new action. Some joint committees are established by an act of parliament rather than a resolution of both chambers. Examples are the Joint Committee of Public Accounts and Audit and the Joint Committee on Corporations and Financial Services. This extra prestige may increase the acceptance rate for those committees reports. Of the 34 joint committee reports in the sample, 21 were produced by committees established by legislation. The regressions at the end of the paper test whether a legislated function is correlated with the acceptance rate (see Appendix 1). It is also possible to check whether the way in which the government accepts or rejects a recommendation differs across committee type. Figure 5 presents the data for majority recommendations; presenting the data for minority recommendations would not be instructive, because almost all minority recommendations are rejected. The figure provides a breakdown of the data by category of acceptance or rejection. The first three categories agreed, partially agreed and under examination therefore add up to the totals for majority reports shown in Figure 3. The chart supports a number of observations. First, House committees have the highest proportion of already taking action responses, where the government lists a number of actions that it is already taking in response to a recommendation. House committees are the only type of committee where the government has exclusive control of the committee referral process as well as a majority on the committee. This supports the view that House committees tend to conduct low-risk inquiries whose recommendations simply reflect current government thinking on an issue. This may explain why House committees have a low acceptance rate of recommendations but a large number of reports where the government has accepted at least one recommendation. The short deadline for bill inquiries and the fact that ministers respond to them during debate in the chamber leaves little scope for the government to respond that it is already taking action on a recommendation or that it is under examination. Therefore, these two categories are absent for Senate legislation committees. Also, the total number of times that the government has responded that a matter is under examination is low, suggesting that this is not a routine method of avoiding responding to a report. This supports the decision in this study to interpret under examination responses as acceptance of a report s recommendations. Type of inquiry In an analysis of committee effectiveness, Derek Hawes suggests that the subject matter of a report may affect how the government responds to it. 15 We tested this proposition in two ways. The first was to select reports that had a contentious subject mat- PARLIAMENTARY STUDIES CENTRE, CRAWFORD SCHOOL of economics and government

9 Figure 5 Breakdown of government responses to majority recommendations, (%) Joint Senate references Senate legislation House Weighted average Agreed Partially agreed Source: Subsample of 77 committee reports. Under examination Already taking action Agreed in principle Rejected ter. During the 40th Parliament ( ), probably the two most contentious issues were terrorism and immigration. Our sample contained 10 terrorism reports and five immigration reports. The regressions at the end of the paper test whether these categories are correlated with the government response (see Appendix 1). The expectation would be that their contentious nature would be associated with a reduced acceptance rate. The second was to classify inquiries according to the object of scrutiny: ministerial conduct (three reports), 16 administration (13 reports), bills (31 reports) and policy (48 reports). One theory to be tested was Hawes observation that administrative inquiries are less contentious and therefore have higher acceptance rates. 17 In distinguishing between administrative and policy reports, we looked at whether a report focused on how agencies managed themselves, in which case it was classified as an administrative report, or rather on new programs, legislation or large funding initiatives, in which case it was classified as a policy report. Where a report contained a blend of administrative and policy recommendations, the classification was based on which sort was most numerous. A similar approach was taken to differentiating bill inquiries from policy inquiries. On occasion, a report that ostensibly discussed a bill contained mainly policy recommendations. In such cases, the report was classified as a policy rather than bill inquiry. Bipartisanship The value or otherwise of bipartisanship in committee work is a major topic in the literature. Bipartisanship is a matter of balance. If committees conduct safe inquiries that are sure to result in bipartisan reports, there may be doubt about their relevance. But if they conduct very contentious inquiries, members may not be able to reach agreement on the report, giving it less authority. Some commentators suggest that committees should concentrate on areas where political parties are yet to form a position. This would give committee members more flexibility in negotiating (because they would not be locked into a party political position) and increase the chances of a bipartisan report. 18 Bipartisan reports are attractive to government. One way of viewing government is as a seeker of ideas to develop new policy and satisfy the simultaneous demands of those who fund and support the party and those who allocate power between the political parties (voters). Governments have close links to their power bases and are well informed about their demands and interests. However, they are less well informed about what policies have support across the electorate. One source of mainstream policies is bipartisan committee reports. This study uses an index to measure rates of bipartisanship. The basis for the calculation is the percentage of committee members who support a majority report and do not attach additional comments to it. Where there is no majority report, parliamentary studies PAPER 11

10 Figure 6 Average bipartisanship index for committee reports tabled in (%) a Joint Senate references Senate legislation House Weighted average a Error bars show 95% confidence intervals where sample size and distribution permit. Source: Sample of 95 committee reports. the index is the largest percentage of committee members who support an individual report. If some committee members support a majority report but attach additional comments, this is still categorised as dissent because the majority report clearly did not meet their needs. If the entire committee supports a majority report and no member makes additional comments, then the bipartisanship index for that report is 100. If two members of a six-person committee dissent, then the index will be Figure 6 shows the level of bipartisanship in the sampled reports. The first observation one can make from the chart is that House committee reports display a very high level of bipartisanship. Only one member dissented, once, in the 10 House reports in the sample. With 10 members on each committee, the average bipartisanship index for these committees was 99. While it may be correct to value bipartisanship, the very high levels in House committees, combined with the low acceptance rate for House recommendations, further supports the argument that they tend to conduct low-risk inquiries. Joint committees also displayed high levels of bipartisanship, with an average score of 92, but had a much higher acceptance rate of recommendations. One interpretation would be that joint committees conduct inquiries of greater relevance to the government. Finally, Senate committees often split along party lines. During the period covered by this study, most Senate committees had six members, including two minority members. A split along party lines would indicate a bipartisanship index of 66.7, which is close to the average for Senate references and Senate legislation committees. Of the 51 Senate committee reports in the sample, only seven had a bipartisanship index of 100. These comments have clear implications for committee effectiveness. Governments will probably be less likely to accept recommendations where committee members are unable to produce a unanimous report. From the perspective of the government, dissent reduces committee effectiveness. This may explain the low acceptance rate for the majority reports of Senate references committees, which also had the lowest rate of bipartisanship in the sample. Media coverage Committees work in a political environment. Therefore, one way of assessing the effectiveness of reports is to consider the amount of media coverage they receive. To measure this, we prepared an index based on the coverage of reports in five major metropolitan newspapers: the Sydney Morning Herald, the Age, the Australian, the Australian Financial Review and the Canberra Times. These are the newspapers for which the parliamentary library maintains comprehensive coverage. 19 It would have been difficult to track publicity across the life of an inquiry, which might last over a year, so we focused instead on the period shortly after tabling. The papers were searched electronically for any mention of a report during the two days after it was tabled. A period of two days was chosen because some Senate reports tabled late in the day would have missed the deadline for publication in the next day s paper. The index had two components. The first was whether a report was mentioned in a particular newspaper. For each paper in which it was 8 PARLIAMENTARY STUDIES CENTRE, CRAWFORD SCHOOL of economics and government

11 Figure 7 5 Average press coverage rate for committee reports tabled in a Joint Senate references Senate legislation House Weighted average a Error bars show 95% confidence intervals where sample size and distribution permit.permit. Source: Sample of 95 committee reports. mentioned, the report was awarded one point. The second component was the location of the article: the closer to the front page, the more newsworthy the report. The inverse of the page number was calculated for this: if an article was located on page 1, the report received one point plus one extra point; if on page 2, one point plus half a point; if on page 3, one point plus one-third of a point; and so on. The highest score that a report could receive per newspaper was two, giving each report a potential score of between zero (no mention in any newspaper) and 10 (page 1 article in all five newspapers). Figure 7 shows the results for each committee type. From the graph it is clear, first, that committee reports overall do not receive a great deal of media coverage. The average for the whole sample was 1.16, which equates roughly to an article on page 6 in one of the five sampled newspapers. Second, it is apparent that Senate references committees received the most coverage by a clear margin. Their score was 2.94, which equates to an article towards the end of the news section in three of the five newspapers. Of the eight reports in the sample that received scores greater than five, all but one was a Senate references inquiry report. The three highest-scoring reports in the sample concerned the intelligence on Iraq s weapons of mass destruction (10), the Bali bombing (7.98) and a certain maritime incident (7.83). 20 Predictably, all three were also reports inquiring into ministerial conduct. One might expect that more intense press coverage would make a government more likely to respond positively to a committee report. In John Uhr s words, just think of which types of parliamentary committees generate the most media publicity and you probably have a committee that has already got the attention of the Government. 21 This proposition will be tested later in the paper. Number of recommendations The final factor examined here is the number of eligible recommendations in a report. In 2004 Bob Charles, the retiring chair of the Joint Committee of Public Accounts and Audit, said in an interview that he believed that committees should try to produce bipartisan reports that did not contain too many recommendations: If you have 50 recommendations, the three or four that are really important tend to get lost in the flood. 22 Staff who have worked for Charles say that he would express concern about any draft committee report that contained more than 12 recommendations. In the sample, the number of eligible majority recommendations ranged from zero (because some reports contained no majority recommendations for new action) to 89. Three reports, all by Senate references committees, contained 50 or more recommendations; one of these contained 89. The average number of majority recommendations (in reports that contained such recommendations) was The number of eligible minority recommendations ranged from zero to nine. The average number of recommendations (in reports that contained such recommendations) was 3.2. Although this paper makes the simplifying assumption that all recommendations are equally important, Bob Charles s economical philosophy can be tested. The statistical study that follows tests parliamentary studies PAPER 11 9

12 Table 2 Association between report characteristics and the probability of the government accepting at least one recommendation, reports tabled , logit model Characteristic p-value Base case New value Change (%) Senate references Joint Senate references 47.7 House Joint House 41.7 Terrorism inquiry No Yes 32.8 Bill inquiry No Yes Inquiry into ministerial conduct No Yes 74.7 Any majority recommendations? Yes No 57.3 No. of majority recommendations a Bipartisanship Press coverage a The characteristics for the number of majority recommendations appear more than once in the mathematical model. The lower p-value has been reported for this characteristic. Source: Sample of 95 committee reports. whether the number of recommendations is correlated with the government response. MODELLING THE GOVERNMENT RESPONSE 23 Assessing minimum effectiveness This part of the study involved running a regression on a number of report characteristics against government acceptance of at least one of a report s recommendations. 24 The aim was to determine whether there was a statistical association between each characteristic and the government response. This was then used to formulate an equation that could be used to predict the probability that the government would accept at least one of a report s recommendations and therefore that the report would achieve a minimum level of effectiveness from the government s perspective. For example, the model might be used to obtain an estimate of the change in effectiveness of Senate references committees if their (currently very low) levels of bipartisanship increased but all else remained the same. The model was also used to weed out those characteristics that were not statistically significant, that is, those where the association between a characteristic and the government response varied so much that it was not possible to predict with any reliability what the association was. In such cases, the characteristic was removed from the model. After removing characteristics with low reliability, the model was left with nine characteristics. The results are given in Table 2. A more thorough discussion of these processes is in the appendix. The p-value provides a measure of how reliable (or statistically significant) a characteristic is in predicting whether the government will find a committee report effective. A p-value of 0.05 (or 5 per cent) means that there is a 5 per cent chance of being incorrect if one concludes that there is an actual correlation. This is the most commonly used p-value for determining significance, although many studies report the 1 and 10 per cent levels of significance as well. The other factor that should be taken into account is effect size. If a characteristic is correlated with both a large change in effect across the variations typically found in the sample, as well as being statistically reliable, then it is very important to the model. The table adopts a hypothetical base case of a joint committee tabling a report with four majority recommendations on a topic other than ministerial conduct, terrorism or a bill. The bipartisanship score for the base case is 80.0 per cent, equivalent to eight of 10 members agreeing on a report. This is close to the sample average of 78.5 per cent. With these characteristics, the model predicts a 77.3 per cent chance that the government will accept at least one of a report s recommendations. The change column shows how the figure of 77.3 per cent would increase or decrease in response to a change in characteristic (shown in the new value column). For example, if a report was tabled by a 10 PARLIAMENTARY STUDIES CENTRE, CRAWFORD SCHOOL of economics and government

13 Senate references committee rather than a joint committee (row 1), then the probability of effectiveness would drop by 47.7 per cent to 29.6 per cent. 25 This does not mean that exactly the same report would have a reduced chance of its recommendations being accepted simply because a different type of committee wrote it, although something like this may be involved. What it more likely means is that the approach and tone adopted by a Senate references committee would make its recommendations less acceptable to the government than those of a joint committee. The type of committee running an inquiry tends to have large-scale effects. For House committees, for example, the probability of effectiveness (compared with joint committees) drops by 41.7 per cent. The type of committee that most resembles a joint committee is a Senate legislation committee. The difference between these two committee types was not sufficiently large to warrant including Senate legislation committees as an additional characteristic in the model. Reports that investigate ministerial conduct also have large-scale effects. All else being equal, this type of report is correlated with a reduced chance of effectiveness of 74.7 per cent. These were the most partisan reports during the period, so it is unsurprising that the government was least inclined to respond publicly to them. Also important were reports that contained majority recommendations. Reducing the number of majority recommendations from four to zero was associated with a decrease in the probability of effectiveness of 57.3 per cent. Under the scenario presented in the table, press coverage is not associated with a large effect. However, in certain circumstances press coverage can have a major impact on the probability of effectiveness. For example, if we assume a different base case of a Senate references inquiry tabling a report with a 66.7 per cent bipartisanship measure, four majority recommendations and no press coverage, the chance of the government finding this report effective would be 8.4 per cent. But increasing the press coverage index to 5 would be associated with a new chance of effectiveness of 70.7 per cent, and increasing it to 10 would be associated with a new chance of effectiveness of 98.5 per cent. That is, it is possible to use the media to push the government into accepting some of a report s recommendations, but only if the media coverage is intense. Tested against the 95 reports in the sample, the model predicts the correct result 85.3 per cent of the time, compared with 50 per cent for flipping a coin or 60 per cent for automatically predicting a positive result in all cases. Therefore, the model has reasonable predictive power. These results confirm some of the observations made earlier in the paper. The exception is Bob Charles s comment about the desirability of limiting the number of recommendations. Rather, a larger number of majority recommendations in a report appears to increase the chance that the government will find something useful in it. This is similar to the lottery effect, where the more tickets one buys, the greater the chance of winning a prize. The impact of this characteristic is a result of setting a non-zero acceptance rate as the benchmark, rather than a particular proportion, such as 50 per cent. However, there is a limit to the lottery effect. This is because the number of majority recommendations appears in the model as both linear and squared terms. The coefficient for recommendations squared has a negative sign, which means the second derivative will also have a negative sign. Differentiating the regression equation by majority recommendations and solving gives a global maximum of 46 majority recommendations. That is, if the government does not find anything worth accepting in a report with 46 recommendations, then adding more majority recommendations will not, on average, be associated with an increased chance of a report being deemed effective by the government. A criticism of this model is that committees can maximise their chance of effectiveness by tabling larger reports with more recommendations. A number of observations can be made in response to this concern. First, government acceptance of at least one recommendation is not the only measure of effectiveness included in the framework. The acceptance rate of recommendations, for instance, can be considered a supplementary performance indicator for this initial measure of effectiveness. The views of the legislature, stakeholders and the public must also be considered in assessing overall effectiveness. Inflating one effectiveness measure is likely to result in lower performance on these other three measures. Second, it is advisable to compare any effectiveness measure with an efficiency measure, such as cost. It should be possible to publish the cost of a committee report as Canadian committees have done in the past, and as the Audit Office of New South Wales does for its performance audits. 26 Including the cost of an inquiry in a report should moderate any tendency to overemphasise the importance of a particular effectiveness measure. Acceptance rate of majority recommendations This part of the study involved running a regression to determine what factors were correlated with the acceptance rate for majority reports. 27 The regression was based on the subsample of 77 committee reports that contained majority recommendations. parliamentary studies PAPER 11 11

14 Table 3 Association between report characteristics and the government acceptance rate for majority recommendations, reports tabled , OLS model Characteristic p-value Base case New value Change (%) Senate references Joint or Senate legislation Senate references 32.3 House Joint or Senate legislation House 34.2 Terrorism inquiry No Yes 25.9 Bill inquiry Policy or admin. inquiry Bill Inquiry into ministerial conduct Policy or admin. inquiry Ministerial 36.2 No. of majority recommendations a Bipartisanship Press coverage a The characteristics for the number of majority recommendations appear more than once in the mathematical model. The lowest p-value has been reported for this characteristic. Source: Subsample of 77 committee reports. Our interest was in determining which characteristics made an effective report more so from the perspective of the government. The results are given in Table 3. The model adopts a base case of a joint or Senate legislation committee publishing a policy or administrative report with four majority recommendations, a bipartisanship score of 80 per cent and a press coverage score of 1.1. The model predicts that, on average, the government will agree to implement 37.8 per cent of the recommendations in such a report. In practical terms, this amounts to the government accepting one or two of its four recommendations. The sample is similar to that in Table 2 and a related issue is being investigated. Therefore, there are a number of similarities in the results. All the characteristics in this model appear in the previous one and have the same sign over typical ranges in the sample. However, the removal of reports with minority recommendations made the presence or otherwise of majority recommendations irrelevant, so it does not appear in Table 3. Once again, Senate legislation committees are statistically no different to joint committees. All else being equal, the government found the reports of these two committee types to be the most valuable according to this measure. The effects in this model are reasonably large: between 25 and 50 per cent for the values typically found in the sample. All except terrorism are statistically significant, which means that the effects are reliable as well. However, the model would probably benefit from the inclusion of additional characteristics. Currently, it explains 38.3 per cent of the variation in the regression, which is similar to stating that it explains 38.3 per cent of the variation in the acceptance rate for reports with majority recommendations. 28 One rule of thumb is that a model should explain at least 50 per cent of the variation in a regression. Nevertheless, the results again confirm some of the initial discussion in relation to the effect of committee type, inquiry type, bipartisanship and press coverage. Acceptance of at least one minority recommendation This part of the study involved running a regression to determine what factors were correlated with the government accepting at least one minority recommendation. 29 The regression was based on the subsample of 44 committee reports containing at least one minority recommendation. Although it might have been preferable to model the acceptance rate for these reports, the skewed distribution of the data did not permit this and it was necessary to construct a more limited model. 30 The results are given in Table 4. The model adopts a base case of any committee type conducting a bill inquiry, with the report having three minority recommendations and 10 majority recommendations. Under these circumstances, the model predicts that there is a 15.7 per cent chance 12 PARLIAMENTARY STUDIES CENTRE, CRAWFORD SCHOOL of economics and government

15 Table 4 Association between report characteristics and the government accepting at least one minority recommendation, reports tabled , logit model Characteristic p-value Base case New value Change (%) Bill inquiry Bill inquiry No 15.4 No. of minority recommendations No. of majority recommendations Source: Subsample of 44 committee reports. that the government will accept at least one recommendation. This low level of effectiveness from the perspective of government is consistent with the pattern for minority recommendations shown in Figure 3. It is also consistent with the previous finding that committee type is not relevant to how the government responds to minority recommendations. On average, it appears that minority recommendations have such a low profile and are so poorly resourced during development that they have a uniformly low acceptance rate, regardless of committee type. Once again, the lottery effect is relevant to whether the government accepts at least one minority recommendation. In this case, however, the number of majority as well as minority recommendations in a report becomes relevant. The low level of attention given minority recommendations generally means that they are not a significant characteristic in determining whether the government accepts at least one of a report s recommendations. But when one focuses on reports that contain only minority recommendations, then the number of minority recommendations in the report does become relevant. Also interesting is that the number of majority recommendations affects whether the government accepts minority recommendations. This suggests that the overall scale of a report is important. If a report is sufficiently large, then it is possible that the government may be induced to consider its minority recommendations seriously, as a by-product of examining a large number of majority recommendations. For the hypothetical case of a committee conducting a non-bill inquiry with 60 majority and three minority recommendations, then the chance of the government accepting at least one minority recommendation rises to 44.3 per cent. A number of characteristics could not be included in the model due to the nature of the sample. In particular, the government did not accept any of the minority recommendations in the reports inquiring into immigration, terrorism or ministerial conduct. Therefore, the model could not attach effect sizes or statistical probabilities to these characteristics. The best conclusion that can be drawn for these characteristics is that minority recommendations in these types of reports have a very low chance of being accepted by the government. APPLYING THE MODEL This study focuses on explaining the government response to reports tabled between the 2001 and 2004 elections. Groups and individuals who use it to assist them in their decision making about future reports should be aware of the assumptions underlying the research. The analysis assumes, for example, that all recommendations in a report are equally important, but in the real world this is rarely the case. Also, there has been a change of government since the sample period. To apply the results of the regression to a different government implies that there are negligible differences between governments in relation to committee reports, but this may not be the case. We also know that the models leave a large proportion of the government response unexplained. Therefore, the regression results should be considered indicative of the trade-offs that committees face, rather than an authoritative predictor of the likely government response. Maintaining bipartisanship Let us assume that a joint committee is deliberating on a report with nine recommendations on a nonterrorism-related policy matter. The expected press coverage is 1.1 and all 20 committee members are expected to agree on the report. But let us assume that, due to external factors, the eight opposition parliamentary studies PAPER 11 13