MST Understanding Your INSPIRE Report: Definitions and Measurements

MST Understanding Your INSPIRE Report: Definitions and Measurements This document explains how outcomes presented in the INSPIRE Data Highlights Report are defined and calculated. Calculations use data entered into INSPIRE as well as data entered into the MST Enhanced Website (a web-based data collection system utilized by all MST teams world-wide and managed by the MST Institute). Variables derived from INSPIRE data are indicated in red, while those from the MST EWS are indicated in blue. Contents Population Served... 2 Discharged Youth... 4 Clinical Discharges... 5 Treatment Outcomes at Discharge... 8 6-Month Follow-up Sample... 12 6-month Follow-up Outcomes... 13 Placement Types at 6-month Follow-up... 15 Sources of 6-month Follow-up Data... 15 Implementation Quality Assurance... 16 1

Outcome What it means How it is measured Population Served 1 Number of youth served This measure captures the total number of youth enrolled in the MST program at any point during the reporting period. 2 Number of new youth enrolled Number of youth who were enrolled in the MST program (i.e., had a first session) during the reporting period. This is a count of the number of youth meeting the following criteria: 1. First session date is before or during the reporting period. 2. Discharge date, if it exists, falls on the first day of the reporting period or later. This is a count of the number of youth with a first session date in the current reporting period. 3 Number of new parents/caregivers served 4 Referral source for new enrollments For new youth enrolled in MST during the reporting period, the number of parents/caregivers participating in MST. It is based on the number of parents/caregivers involved at the start of treatment. This pie chart reflects the referral sources of youth enrolled in MST during the reporting period. For all youth with a first session date in the current reporting period, the total number of parent/caregiver relationships indicated in the INSPIRE enrollment screen is counted. The percents are based on total number of youth with a first session date in this reporting period, for whom a referral source is provided. Where more than one referral source is indicated, only the primary referral source is counted for each youth. The MST EWS offers 10 choices for referral source. Those 10 choices are categorized as follows for the INSPIRE report: social services (i.e., child protection) CYS probation dept + parole dept + court-related service Juvenile Justice mental health + substance abuse agency Mental Health Services education Schools omnibus agency or court + other + not specified Other 2

5 Placement risk for new enrollments 6 Demographics for new enrollments This pie chart reflects placement risk for youth enrolled in MST during the reporting period. It indicates the percent of you who were not at immediate risk of placement out-of-home; the percent at immediate risk and for whom MST can be considered a diversion from placement; and the percent who are stepping down from out-of-home placement to MST. While it can be considered one indicator of the severity of the population served, it is also impacted by county placement practices. The number of youth for whom this data was available is indicated below the pie chart. If the number of youth included in the pie chart is significantly lower than the number of youth enrolled, the data may not be representative of all youth served. For youth enrolled before May 2013, the immediate risk category includes youth stepping down from placement. This data reflects demographic information for youth who were newly enrolled during the reporting period. Gender, age, and race/ethnicity are reflected. 3 Providers indicate placement risk for each youth by choosing one of three options: not at risk, at immediate risk/diversion, stepping down from placement. Providers are instructed to respond based on information available at the point of enrollment. While the preferred data source is the referral source and/or agency with the authority to make this type of placement decision, the MST supervisor or intake staff makes the final determination regarding whether the youth is at imminent risk of placement. Out-ofhome placement may include placement in a dependency system, foster care system, juvenile justice system, drug and alcohol system, or mental health system. The percent is calculated based on youth with a first session date in the reporting period, for whom the question is answered. For gender and ethnicity, percents are based on youth for whom data are available. Age is determined by first session date date of birth, and an average is calculated based on all youth for whom this data is available. The MST EWS offers 8 mutually exclusive options for race/ethnicity. (If multiracial is selected, primary and secondary race can be selected, but for the purpose of INSPIRE only multiracial would be counted.) The options and how they are categorized for the INSPIRE report are as follows: White/Non-Hispanic White/Non-Hispanic Black/African-American Black/African-American Hispanic/Latino Hispanic/Latino American Indian / Alaskan Native / Indigenous Peoples American Indian

7 Presenting problems for new enrollments This bar graph depicts the presenting problems for youth newly enrolled during the reporting period, based on information available at the time of enrollment. Asian Asian Native Hawaiian Pacific Islander Multiracial Multiracial Other Other For each of 12 presenting problems, the percent is calculated based on the following sample formula: (# of youth with problem 1) (# of youth with any problem 1-12) Thus, youth for whom no presenting problem is identified are excluded from the calculation. Discharged Youth 8 Number of youth discharged This is the total number of youth discharged from the MST program during this reporting period. It includes both clinical and administrative discharges. 9 Administrative discharges (number and percent) Administrative discharges are cases closed for reasons unrelated to treatment progress. This includes placement for an event that occurred prior to MST enrollment, withdrawal from MST for administrative reasons, youth removed by the funding or referral source for administrative reasons, and youth/family moved out of the service area. A count of the number of youth with a discharge date in the reporting period. When a case is closed, therapists choose one of 7 reasons for closure (case progress). A youth is counted as an administrative discharge if the reason for closure is any of the following reasons, which result in closure for reasons unrelated to case progress: MST program administrative withdrawal (e.g., inappropriate referral) Administrative withdrawal by funder/referral source Youth was placed for an event that occurred prior to MST involvement Youth/family moved Percent is based on total number of discharges. Formula is: (# of administratively discharged youth) (total # of discharged youth) 4

10 Clinical discharges (number and percent) 11 Reasons for administrative discharges Clinical discharges are those cases that were closed for clinical reasons (i.e., reasons with the therapist s influence). This pie chart presents the reasons for case closure, among youth who were administrative discharges during the reporting period. When a case is closed, therapists choose one of 7 reasons for closure (case progress). A youth is counted as a clinical discharge if the reasons for closure are not administrative (as described under Administrative Discharges). Clinical discharges include the following three reasons for closure: Case completed (this does not necessarily mean that all goals were met, but therapist and family agreed to end treatment) Lack of engagement Youth was placed for an event that occurred while in MST Percent is based on total number of discharges. As noted in Row 9, there are 4 reasons for closure (case progress) that are counted as administrative discharges. The percent of youth with each reason is calculated, based on the total number of administrative discharges. Clinical Discharges (see Row 10 for definition of clinical discharge) 12 Number and percent at immediate risk of placement and stepping down This data reflects placement risk at the time of enrollment for youth who were clinically discharged during the reporting period. For youth who were clinically discharged during the reporting period, the number who were at immediate risk of placement and the number stepping down from placement at enrollment are counted. See Row 5 for more information about placement risk. Percent is calculated based on the number of youth for whom this data is available. 5

13 Average length of stay for completed case 14 Average length of stay for incomplete cases 15 Youth satisfied with MST (number and percent) 16 Youth whose parents/caregivers were satisfied with MST (number and percent) The average length of treatment, in months, for youth who completed MST during the reporting period. The average length of treatment, in months, for youth who were clinically discharged during the reporting period but did not complete treatment. These numbers reflect how many youth who were clinically discharged during this reporting period were satisfied with the MST program. These numbers reflect how many youth who were clinically discharged during the reporting period had had a parent/caregiver who was satisfied with the MST program. 6 See Row 17 for explanation of completed cases. Length of treatment is measured as the number of days from first session date to discharge date. Youth who completed MST during the reporting period are included in the calculation. See Row 17 for explanation of incomplete cases. Length of treatment is measured as the number of days from first session date to discharge date. Youth who were clinically discharged but did not complete MST are included in the calculation. In INSPIRE, the provider indicates Yes or No for the question Was the youth satisfied with his/her experience in the MST program? Providers are instructed that the data source can be surveys, exit interviews, or another approved tool completed at the time of discharge from the MST program, and to leave the question blank if they do not know the answer. Calculation includes clinically discharged youth for whom the question is answered. Formula is: (# of youth satisfied) _ (# of youth satisfied + # of youth not satisfied) In INSPIRE, the provider indicates yes or no for the question Were the parent(s)/caregiver(s) satisfied with their experience in the MST program? Providers are instructed that the data source can be surveys, exit interviews, or another approved tool completed at the time of discharge from the MST program, and to leave the question blank if they do not know the answer. Calculation includes clinically discharged youth for whom the question is answered. Formula is:

(# of youth with satisfied caregiver) (# of youth with satisfied caregiver + # of youth with unsatisfied caregiver) 17 Treatment completion rate This pie chart reflects the percent of clinical discharges who completed treatment (i.e., case was closed at mutual agreement of therapist and family) vs. those who did not (i.e., youth was placed out of home or team was unable to engage family in treatment). Completion does not necessarily mean that goals were met and includes cases where therapist and family agree that further progress is unlikely even though all goals have not been met. 18 Discharge type This pie chart reflects the percent of clinical discharges who were successful vs. those who were not. Success is defined as meeting all four of the following criteria: 1) treatment was completed, 2) youth is living at home, 3) youth is engaged productively in work/school, and 4) youth has had no new arrests since treatment began. Youth who were clinically discharged but do not meet these criteria are counted as unsuccessful. At discharge, therapists choose from 7 reasons for closure (case progress), three of which are considered clinical discharges. These three reasons and how they are categorized by INSPIRE to determine completion rate are as follows: Completion Case Completed Lack of engagement + placement for an event that occurred while in MST Case Not Completed When a case is closed, therapists indicate in the MST Enhanced Website the reason for closure (case progress). For each of the three Ultimate Outcomes of MST youth is living at home, engaged productively in work/school, and has no new arrests - the therapist indicates Yes or No. A youth is counted as successful if the following criteria are met: 1. Youth completed MST 2. Youth met all three Ultimate Outcomes A youth is counted as unsuccessful if either: 1. The youth completed MST but failed to achieve all three Ultimate Outcomes, or 2. The reason for closure is lack of engagement or placement for an event that occurred while in MST. Percents are based on the total number of clinical discharges. 7

Treatment Outcomes at Discharge Bar graph and table on p. 3 19 Number of youth included in analysis Outcomes at discharge are based on the number of youth who were clinically discharged during the reporting period. Administrative discharges are not included. For each outcome, the number of youth in the analysis depends on the number of clinically discharged youth for whom information was provided. The table below the graph provides the number improving and number reporting for each outcome. 20 No new criminal offenses The number and percent of clinically discharged youth who had no new criminal offenses during treatment. No new offenses is defined as not charged with a misdemeanor or felony, or adjudicated delinquent or criminally convicted, for a new offense that occurred while in the MST program. Charges, adjudications, and convictions for offenses that occurred prior to MST and non-criminal violations of the terms of probation are not counted. Who is included in the calculation? All clinically discharged youth. At discharge, therapists answer the following statement in the MST Enhanced Website: Youth has not been arrested since the beginning of MST treatment, for an offense committed during MST treatment. Programs are instructed to answer No (thereby indicating a new offense) if the youth was charged with a misdemeanor or felony, or adjudicated delinquent or criminally convicted, for a new offense (other than non-criminal violation of the terms of probation) that occurred while in treatment. The EWS requires that this question be answered for all closed cases. Therefore, the calculation of percent includes all clinically discharged youth. Formula is: (#with no new offenses) (# of clinical discharges) 21 Living in the community The number and percent of clinically discharged youth who were living in the community at discharge (i.e., were not placed out of home while receiving MST). Youth placed solely for an event that occurred prior to enrollment in MST are not included in this analysis, since they are considered administrative discharges. Who is included in the calculation? All clinically discharged youth. When a case is closed, therapists indicate the reason for closure in the MST Enhanced Website. Reasons include: completion, lack of engagement, placement for an event that occurred prior to MST, placement for an event that occurred while in MST, administrative withdrawals, and family moved. The EWS requires that this question be answered for all closed cases. Therefore, the calculation of percent includes all clinically discharged youth. Formula is: 8

# living in community = (# clinical discharges) (# placed for an event that occurred while in MST) % in community = (# living in community) (# of clinical discharges) 22 Achieved treatment goals The number and percent of clinically discharged youth who achieved their treatment goals (i.e., referral behaviors were satisfactorily addressed). Who is included in the calculation? All clinically discharged youth for whom information about treatment goal attainment is provided. 23 Negative drug screens The number and percent of clinically discharged youth who had negative drug screens during the last 2 months of treatment. Who is included in the calculation? All clinically discharged youth who 1) were identified with a drug or alcohol problem at enrollment and 2) had drug screens during the final 2 months of treatment. Youth who were not identified at intake as having issues with substance use or who did not have drug screens during the final two months of MST are not included in the analysis. Upon discharge, the therapist answers Yes or No to the following question in INSPIRE: Did the youth achieve the goals on his/her treatment plan? The formula includes all clinically discharged youth for whom the question is answered: (# who met goals). (# for whom question about goals was answered) The calculation of percent includes clinically discharged youth who: 1) had drug or alcohol use marked as a presenting problem at enrollment and 2) had at least one drug screen with a date in the last two months of treatment and 3) the results of those drug screens are entered into INSPIRE. A youth is counted is having negative drug screens if he/she meets those criteria and the results of all drug screens during the last 2 months of treatment were negative. Formula is: (# of youth with no positive drug screens) _ (# of youth with positive drug screens + # of youth with no positive drug screens) N/A indicates that no youth met the conditions for inclusion in the calculation (e.g., no youth had substance use concerns at referral, had drug screens in the last 2 months of treatment, and were clinically discharged). 9

24 Reduced/eliminated substance use The number and percent of clinically discharged youth who showed evidence of reduced or eliminated substance use. Who is included in the calculation? All clinically discharged youth 1) who were identified with a drug or alcohol problem at intake and 2) for whom a discharge question about substance use is answered. At enrollment, the provider indicates whether drug or alcohol use is a presenting problem (tobacco use is excluded). At discharge, the provider is instructed to indicate the youth s substance use during the final 2 months of treatment using a variety of sources (e.g., clinical judgment, participant reports, behavioral/emotional indicators of use, drug screens). Five response options are provided: Use was eliminated; Use was reduced in frequency or intensity; Use continued with no change from intake; Use increased; No history of substance use. A youth is counted as having a positive outcome if use is reduced or eliminated. Calculation of percent includes all clinically discharged youth 1) who were identified with drug or alcohol use at intake and 2) for whom the status of use at discharge is provided: (# with reduced or eliminated use) (# reduced + # eliminated + # no change + # increased) If substance use is indicated as a presenting problem but no history is marked at discharge, the youth is excluded. 25 Improved school attendance Number and percent of clinically discharged youth who demonstrated improvement in school attendance. Who is included in the calculation? Clinically discharged youth for whom school attendance was identified as a presenting problem at enrollment and the outcome question regarding attendance is answered. At enrollment, the provider indicates whether school attendance is a presenting problem. At discharge, the provider indicates for all youth whether school attendance at discharge is improved, worse, or no change, based on therapist s clinical judgment. The calculation includes all clinically discharged youth for whom attendance is a presenting problem and attendance outcome is provided. Formula is: (# with improved attendance). (# improved + worse + no change) 10

26 Improved academic performance Number and percent of clinically discharged youth who demonstrated improvement in academic performance. Who is included in the calculation? Clinically discharged youth for whom academic performance was identified as a presenting problem at enrollment and the outcome question regarding academics is answered. At enrollment, the provider indicates whether academic performance is a presenting problem. At discharge, the provider indicates for all youth whether academic performance at discharge is improved, worse, or no change, based on therapist s clinical judgment. The calculation includes all clinically discharged youth for whom academic performance is a presenting problem and academic outcome is provided. Formula is: (# with improved attendance). (# improved + worse + no change) 27 Improved family functioning Number and percent of clinically discharged youth discharged whose families showed improvement in family functioning, as indicated by therapist report. Improved family functioning includes improved parenting skills and/or improved family relations. Who is included in the calculation? All clinically discharged youth. At discharge, the therapist indicates in the MST Enhanced Website whether the youth met 6 Instrumental Outcomes (response options are Yes or No). Two of these outcomes are whether there is evidence the caregiver has improved parenting skills needed to handle problems in the future and whether there is evidence of improved family relations specific to the dynamics that were contributing to the youth s referral behavior. Youth for whom improvement is reported in either area is counted as having improved family functioning. The EWS requires that this question be answered for all closed cases. Therefore, the calculation of percent includes all clinically discharged youth. Formula is: (# of youth with caregivers that improved parenting and/or improved family relations) (# of clinical discharges) 11

6-Month Follow-up Sample 28 Number of youth included in follow-up sample 29 Placement risk for youth in follow-up sample 30 Treatment completion rate for follow-up sample This is a count of how many youth are included in the 6- month follow-up sample. Only youth who were clinically discharged and who have a 6-month follow-up interview are included. This is the number and percent of youth in the followup sample who 1) were at risk of placement when they were enrolled in MST and 2) who stepped down to MST from an out-of-home placement. This can be compared to the risk profile of youth enrolled in the program as one indicator of how well the follow-up sample represents the youth served by the program. This indicates what percent of the follow-up sample are youth who completed MST vs. youth who did not complete MST. This can be compared to the completion rate for the program as a whole as one indicator of how well the follow-up sample represents the youth served by the program. Youth are counted if 1) they met the criteria for clinical discharge (see Row 10 for definition) and 2) there is one or more follow-up interview date that is 149-211 days from the discharge date. Follow-up interviews may include phone calls, surveys, or written reports. The respondent may be the youth, a family member, the referral source, or any person/agency knowledgeable about the youth at follow-up. Youth in the follow-up sample are included in this calculation if the item regarding placement risk at enrollment was answered. See Row 5 for more information about how placement risk is determined. See Row 17 for an explanation of how completion rate is determined. All youth in the follow-up sample are included in this calculation. 12

6-month Follow-up Outcomes Bar graph on p. 4 and corresponding table on p. 5 31 No new criminal charges The number and percent of youth who had no new criminal offenses during the 6-month follow-up period. No new offenses is defined as not charged with a misdemeanor or felony, or adjudicated delinquent or criminally convicted, for a new offense. Charges, adjudications, and convictions for offenses that occurred prior to discharge from MST and non-criminal violations of the terms of probation are not counted. 32 No out-of-home placements The number and percent of youth who had no out-ofhome placements during the 6-month follow-up period. The follow-up interview includes a Yes/No question regarding whether the youth has had any new offenses since discharge. Calculation of percent includes any youth in the follow-up sample for whom this question is answered: (# with no new offenses) (# answering question about offenses) *If there are multiple 6-month follow-ups for a youth, the youth is considered to have offended if any of the follow-ups indicates a new offense. The follow-up interview includes a Yes/No question regarding whether the youth has had any out-of-home placements since discharge. Calculation of percent includes any youth in the follow-up sample for whom this question is answered: (# with no placements) (# answering question about placement) *If there are multiple 6-month follow-ups for a youth, the youth is counted as having been placed if any of the followups indicates a placement. 33 In school/graduated/ged The number and percent of youth who, at 6-month follow-up, were either still in school, had graduated, or had earned a GED. This is an indicator of youth who are continuing or have completed their high school education (i.e., have not dropped out). The follow-up interview includes one question about the youth s current educational status and includes 5 response options: Quit school; Work training/vocational program; Still in school; GED; High school diploma. Youth are counted as having a positive outcome if any option besides Quit School is selected. (continued) 13

Calculation of percent includes all youth in the follow-up sample for whom this question is answered: (# in work training + # in school + # GED + # h.s. diploma) (# answering question about education) 34 No known substance use The number and percent of youth who had no known substance use during the 6-month follow-up period. The follow-up interview asks a Yes/No question: To the best of your knowledge, since discharge has the youth used drugs or alcohol? (Do not include tobacco use) Calculation of percent includes all youth in the follow-up sample for whom this question is answered: (# with no known substance use) (# answering question about substance use) *If there are multiple 6-month follow-ups for a youth, the youth is counted as having used if any of the follow-ups indicates substance use. **If the follow-up was completed prior to May 2013, using an earlier version of the follow-up interview, a slightly different formula is used. 35 Maintained behavior change The number and percent of youth whose behavior at 6- month follow-up is better or about the same as their behavior at discharge from MST. This indicates to what extent youth are maintaining behavior changes made during treatment. The follow-up interview asks how the youth is currently behaving, compared to how he/she was behaving at discharge: Better, About the same, Worse. Calculation of percent includes all youth in the follow-up sample for whom this question is answered: (# behaving better or the same) (# for whom question about behavior is answered) 14

Placement Types at 6-month Follow-up 36 Placement types This bar graph indicates, for the youth who were placed out-of-home during the 6-month follow-up period, the type(s) of placements they had. More than one placement type can be selected for a youth and therefore placement types are not mutually exclusive. The number of youth included in the sample is indicated below the bar graph. At follow-up, respondents are asked whether the youth has been placed out of home since discharge from MST (Yes or No) and if Yes, they are asked to indicate the type(s) of placement: Child welfare, Juvenile justice, Mental health / MA funded. Calculation of percent includes all youth who were placed and for whom at least one placement type is selected. This number (i.e., the sample size) is indicated below the bar graph. For each placement type, percent is calculated as: (# of youth with placement type X) (# of youth with any placement type reported) Sources of 6-month Follow-up Data 37 Sources of follow-up data The percent of youth with follow-up data from each data source are presented in this bar graph. A youth may have more than one source of data for his/her follow-up. Therefore, data sources are not mutually exclusive and may total more than 100%. This bar graph indicates who has provided the 6-month follow-up data presented. The follow-up interview includes a question regarding the source(s) of follow-up data, which may include: Youth, Adult Caregiver, Adult Non-caregiver, County Staff, County database, or Other. Calculation of percent is based on the number of youth for whom at least one follow-up source is reported: (# of youth with source X) (# of youth with any source reported) 15

Implementation Quality Assurance 38 Percent of families reporting therapist adherence within the desired range 39 Number of MST teams that met all 18 required MST program practices at Program Implementation Review This indicates the percent of youth that received model-adherent MST during the reporting period. The number of teams that met all 18 required program practices and characteristics at semi-annual review. A Program Review Form is completed by the MST Expert in collaboration with the MST Supervisor and is used alongside other program data to guide the team s Continuous Improvement Plan. The specific items that are not met are listed on the report. A team may not meet a program practice for reasons outside of its immediate control. This section of the report is intended to help identify barriers to quality implementation in Pennsylvania. The Therapist Adherence Measure-Revised (TAM-R) is a 28 item questionnaire completed by an adult caregiver periodically throughout MST treatment and indicates the extent to which the therapist is providing MST in a manner that adheres to the model. TAM-R scores can range from 0 1.0. A score of >=.61 is considered by MST Services to be satisfactory. If a youth has more than one TAM-R score during the reporting period, TAM-R scores for that youth are averaged. Calculation of percent is based on all youth with at least one TAM-R completed during the reporting period: (# of youth with TAM-R average >=.61) (# of youth with at least one TAM-R) The Program Review Form consists of 39 items, 18 of which are required program practices and characteristics necessary for MST fidelity. The number reported is based on Program Review Forms that covered a 6-month period that ended during the reporting period. When the value for an outcome is n/a, it indicates that either 1) the outcomes is not applicable (e.g., the number of youth successfully discharged is not applicable if no youth were discharged during the reporting period) or 2) data needed to calculate the outcome was missing or not available. 16

This document has been prepared by the EPISCenter. The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University, and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices. The EPISCenter can be contacted at 814-863-2568 or via email at EPISCenter@psu.edu. 17