MULTISYSTEMIC THERAPY (MST) DEFINITIONS, MEASUREMENTS, & CALCULATIONS FOR INSPIRE DATA HIGHLIGHTS REPORT This glossary explains how outcomes presented in the INSPIRE Data Highlights Report are defined and calculated. Calculations use data entered into INSPIRE as well as data entered into the MST Enhanced Website (a web-based data collection system utilized by all MST teams world-wide and managed by the MST Institute). Variables derived from INSPIRE data are indicated in red, while those from the CSS are indicated in blue. Outcome Definition Measurement & Calculation Population Served Number of youth served Number of new youth enrolled this reporting period Percent of enrolled youth who were imminent at risk of placement This measure captures the total number of youth that received any MST within the reporting period. Number of referred youth that entered the MST program (had a first session) during this reporting period. Percent of youth that began treatment during this reporting period that are at imminent risk of out-of-home placement if they do not participate in or successfully complete the MST program. Includes youth enrolled in the MST program in lieu of out-of-home placement, as a last chance at avoiding placement, stepping down from a placement, or expected to be placed out-of-home if they do not make progress in the MST program. Does not include youth for whom there is not a clear and immediate placement risk (e.g., youth who could be placed if their behavior problems continue worsen over time or who would likely be referred to another home/community 1 The number of youth meeting the following criteria: 1. Date of first session is before or during the reporting period. 2. Discharge date, if it exists, falls on the first day of the reporting period or later. Number of youth for whom the first session date falls within the current reporting period. The goal of this measure is to capture the number of new youth who are enrolled in MST within the reporting period. In INSPIRE, provider indicates yes or no to the question Would this youth have been placed out of home if they were not accepted to participate in the MST program? Provider is instructed to respond based on information available at the point of enrollment and that the preferred data source is the referral source and/or agency with the authority to make this type of placement decision, although the MST supervisor or intake staff makes the final determination regarding whether the youth is at imminent risk of placement. (cont. on next page) Calculation of percent includes only newly enrolled youth for whom the question is answered. Formula is: (# of youth at risk for placement) (# of youth at risk + # of youth not at risk)
Number of new parents/caregivers served Percent of youth from each referral source treatment program if they do not succeed in MST). Out-of-home placement refers to placement in a dependency system, foster care system, juvenile justice system, drug and alcohol system, or mental health system. For new youth enrolled in MST this reporting period, the number of parents/caregivers participating in MST. Based on the number of parents/caregivers involved at the start of treatment. Of youth referred to MST during the current reporting period, the percent that were referred by each source. Only youth who were subsequently enrolled in MST are included. Number of parent/caregiver relationships entered in the INSPIRE enrollment screen, for youth with a first session date that falls within the current reporting period. Percent is based on total number of youth with a referral date in this reporting period, for whom a referral source is indicated in the MST Enhanced Website. Where more than one referral source is indicated, only the primary referral source is counted for each youth. Because Pennsylvania providers generally do not enter youth into the MST Services system unless the youth is enrolled, the percents reported are based open enrolled youth and do not include youth who were referred but not enrolled. The MST Website choices for referral source and their categorization for the INSPIRE report is as follows: Mental Health Services = mental health + substance abuse agency JPO = probation dept + parole dept + court-related service CYS = social services (i.e., child protection) Schools = education Other = omnibus agency or court + other + not specified Formula example: (# of youth from referral source A) (total # of youth from referral sources A + B + C + D + E) 2
Implementation Quality Assurance Percent of youth with therapist adherence scores within the desired range Number of MST teams that met all 18 required MST program practices at Program Implementation Review Discharged Youth Number of youth discharged The percent of youth for whom MST is being implemented with adherence to the clinical model. The number of teams complying with all 18 required MST program practices and characteristics at the time of the teams semiannual reviews. Examples of required program practices include acceptable team size and caseload, participation in weekly supervision, excluding inappropriate referrals, providing 24/7 on-call coverage, etc. Many factors may affect adherence (e.g., a team would not meet all 18 required practices if low referrals have resulted in below-minimum caseloads). Total number of youth discharged from the MST program during this reporting period. Includes successful, unsuccessful, and administrative discharges. 3 Throughout the course of MST, caregivers periodically complete a 28-item Therapist Adherence Measure-Revised which is then entered into the MST Enhanced Website. The TAM-R score indicates whether the therapist is implementing MST with adherence to the model, for that youth. MST Institute has identified an average score of.61 as the minimum for satisfactory adherence. If a youth has more than one TAM-R completed during the reporting period, the TAM-R scores for that youth are averaged. Formula is: (# with an average TAM-R score above.61) ( # with at least one TAM-R completed during the reporting period) Twice a year the MST Expert completes a Program Implementation Review that assesses whether each team is following MST program practices, therapist and supervisor adherence to the clinical model, and program outcomes. As part of the review, a Program Review Form is completed collaboratively by the MST Supervisor and MST Expert. The respondents answer Yes or No to 26 items to indicate whether required and recommended program practices are being met. This outcome measures looks solely at whether teams that had a Program Implementation Review during the reporting period were complying with the 18 required program practices. Sites enter into INSPIRE the team name, date of review, and whether they met all 18 practices (Yes/No). Formula is: (# of teams meeting all 18 required program practices this reporting period) (# of teams with Program Implementation Reviews this reporting period) Number of youth with a discharge date that falls within the given reporting period.
Successful discharges (number and percent) Unsuccessful discharges (number and percent) Successful discharge is defined as meeting all four of the following criteria: 1) living at home, 2) engaged productively in work/school, 3) no new arrests, and 4) treatment was completed. Unsuccessful discharges are those that do not meet the criteria for successful discharge and were not closed for administrative reasons (see below definition of Administrative Discharges). When a case is closed, therapists indicate in the MST Enhanced Website the reason for closure. Reasons include: completion, lack of engagement, placement for an event that occurred prior to MST or for an event that occurred while in MST, administrative withdrawals, and family moved. The therapist also indicates on a Yes/No basis whether the youth met the three Ultimate Outcomes of MST living at home, engaged productively in work/school, and no new arrests. Youth is counted as successful if discharge date falls within the reporting period and the following criteria are met: 1. Youth completed MST (i.e., ended as planned by agreement of the stakeholders youth, family, and therapist) 2. Youth met all three Ultimate Outcomes Percent is based on total number of discharges (i.e., successful, unsuccessful, and administrative discharges). Formula is: (# of successfully discharged youth) (total # of discharged youth) When a case is closed, therapists indicate in the MST Enhanced Website the reason for closure. Reasons include: completion, lack of engagement, placement for an event that occurred prior to MST or for an event that occurred while in MST, administrative withdrawals, and family moved. The therapist also indicates on a Yes/No basis whether the youth met the three Ultimate Outcomes of MST living at home, engaged productively in work/school, and no new arrests. Youth is counted as unsuccessful if discharge date falls within the reporting period and either: 1. The youth completed MST but failed to achieve all three Ultimate Outcomes, or 2. The reason for closure is lack of engagement or placement for an event that occurred while in MST. Percent is based on total number of discharges (i.e., successful, unsuccessful, and administrative discharges). Formula is: (# of unsuccessfully discharged youth) (total # of discharged youth) 4
Administrative discharges (number and percent) Average length of stay for successful discharges Average length of stay for unsuccessful discharges Youth satisfied with MST (number and percent) Number of youth whose cases were closed for administrative reasons including: youth placed for an event that occurred prior to MST enrollment, MST administrative withdrawal (e.g., inappropriate referral, team recommended discharge but family disagrees), removal by funding/referral source, and the youth/family moved out of the service area. The average length of treatment, in months, for youth who were successfully discharged from MST during the reporting period. The average length of treatment, in months, for youth who were unsuccessfully discharged from MST during the reporting period. Number and percent of youth discharged (successfully or unsuccessfully) during this reporting period that reported satisfaction with their experience in the MST program. When a case is closed, therapists indicate in the MST Enhanced Website the reason for closure. Reasons include: completion, lack of engagement, placement for an event that occurred prior to MST or for an event that occurred while in MST, administrative withdrawals, and family moved. Youth is counted as an administrative discharge if the reason for closure is an administrative withdrawal (e.g., youth was removed by funding source), placement for an event that occurred prior to MST involvement, or the family moved. Percent is based on total number of discharges (i.e., successful, unsuccessful, and administrative discharges). Formula is: (# of administratively discharged youth) (total # of discharged youth) Length of treatment is measured as the number of days from first session date to discharge date. Youth who were successfully discharged during the reporting period are included in the calculation. Length of treatment is measured as the number of days from first session date to discharge date. Youth who were unsuccessfully discharged during the reporting period are included in the calculation. In INSPIRE, the provider indicates yes or no for the question Was the youth satisfied with his/her experience in the MST program? Program is instructed that the data source can be surveys, exit interviews, or another approved tool completed at the time of discharge from the MST program, and to leave the question blank if they do not know the answer. If a survey is used, sites are encouraged to establish a consistent threshold/score for determining whether the respondent is satisfied. Calculation includes successfully and unsuccessfully discharged youth for whom the question is answered. Formula is: (# of youth satisfied) _ (# of youth satisfied + # of youth not satisfied) 5
Youth whose parents/caregivers were satisfied with MST (number and percent) Outcomes at Discharge Number of youth included in analysis Percent who remained drug-free Number and percent of youth discharged (successful and unsuccessful) during this reporting period who had a parent/caregiver reporting satisfaction with their experience in the MST program. Outcomes at discharge are based on the number of youth who were successfully or unsuccessfully discharged during the reporting period. Administrative discharges are not included. For each outcome, the number of youth in the analysis depends on the number of youth for whom information was provided. The table below the graph, showing the number improving and number reporting, highlights the number of youth in the analysis for each outcome. The percent of youth discharged this reporting period (successfully or unsuccessfully) who were identified with a drug or alcohol problem at enrollment and were drug-free at discharge, as indicated by negative drug screens. Youth who were not referred due to concerns about substance use or who did not have drug screens during the final three months of MST are not included in the analysis. 6 In INSPIRE, the provider indicates yes or no for the question Were the parent(s)/caregiver(s) satisfied with their experience in the MST program? Program is instructed that the data source can be surveys, exit interviews, or another approved tool completed at the time of discharge from the MST program, and to leave the question blank if they do not know the answer. If a survey is used, sites are encouraged to establish a consistent threshold/score for determining whether the respondent is satisfied. Calculation of percent includes successfully and unsuccessfully discharged youth for whom the question is answered. Formula is: (# of youth with satisfied caregiver) (# of youth with satisfied caregiver + # of youth with unsatisfied caregiver) Youth meets the following criteria: 1. In INSPIRE, the provider answers yes to the enrollment question Was youth referred due to concerns about his/her drug and alcohol abuse? Therapist is instructed that tobacco use not be included and that the preferred source of information is the referral source or a member of the youth s interagency service planning team. 2. The youth had at least one drug screen with a date that falls within the last 3 months of treatment, for which the results were available to the therapist and were entered into INSPIRE
Percent with no new criminal offenses Percent of youth discharged this reporting period (successfully or unsuccessfully) who were not charged with a misdemeanor or felony, or adjudicated delinquent or criminally convicted, for a new offense that occurred while in the MST program. Charges, adjudications, and convictions for offenses that occurred prior to MST and noncriminal violations of the terms of probation are not counted. 3. The results of all drug screens during the last three months of treatment were negative. Calculation of percent includes successfully or unsuccessfully discharged youth that were referred due to concerns about substance use and had at least one drug screen in the last three months of treatment. Formula is: (# of youth with no positive drug screens) _ (# of youth with positive drug screens + # of youth with no positive drug screens) N/A indicates that no youth met the conditions for inclusion in the calculation (e.g., no youth had substance use concerns at referral, had drug screens in the last 3 months of treatment, and were successfully or unsuccessfully discharged). At discharge, the therapist answered Yes to the following statement in the MST Enhanced Website: Youth has not been arrested since the beginning of MST treatment, for an offense committed during MST treatment. Programs were instructed to answer No (thereby indicating a new offense) if the youth was charged with a misdemeanor or felony, or adjudicated delinquent or criminally convicted, for a new offense (other than noncriminal violation of the terms of probation) that occurred while in treatment. discharged youth for whom the question is answered. Formula is: (# of youth with no new charges) (# of youth with no new charges + # of youth with new charges) 7
Percent living in the community Percent of families with improved functioning Percent of youth discharged this reporting period (successfully or unsuccessfully) who were living in the community at discharge (i.e., were not placed out of home while receiving MST). Youth placed solely for an event that occurred prior to enrollment in MST are not included in this analysis, since they are considered administrative discharges. Number of families of youth discharged this reporting period (successfully or unsuccessfully) that showed improvement in family functioning, as indicated by therapist report. Improved family functioning includes improved parenting skills and/or improved family relations. When a case is closed, therapists indicate in the MST Enhanced Website the reason for closure. Reasons include: completion, lack of engagement, placement for an event that occurred prior to MST, placement for an event that occurred while in MST, administrative withdrawals, and family moved. discharged youth. Formula is: 1.00 - (# of youth placed for an event that occurred while in MST) (# of successful + unsuccessful discharges) At discharge, the therapist indicates in the MST Enhanced Website whether the youth met 6 Instrumental Outcomes (response options are Yes or No). Two of these outcomes are whether there is evidence the caregiver has improved parenting skills needed to handle problems in the future and whether there is evidence of improved family relations specific to the dynamics that were contributing to the youth s referral behavior. Youth for whom improvement is reported in either area is counted as having improved family functioning. discharged youth. Formula is: (# of youth with caregivers that improved parenting OR improved family relations) (# of successful + unsuccessful discharges) 8
Percent with improved school attendance Of youth discharged this reporting period (successfully or unsuccessfully) who were referred due to concerns about school attendance, the percent that demonstrated improvement while in the MST program. Youth meets the following criteria: 1. In INSPIRE, therapist indicated at enrollment that the youth was referred due to concerns about school attendance. Preferred source of information is referral source or a member of the youth s interagency service planning team. 2. In INSPIRE, therapist responds Improved to the question, Did the youth show improvement in school attendance during his/her time in MST? Therapist answers based on his/her clinical judgment. Calculation of percent includes all successfully or unsuccessfully discharged youth that were referred due to concerns about school attendance, for whom the question about improvement is answered. Formula is: (# of youth with improved school attendance) (# of youth with improved + no change + worse) Percent with improved school performance Of youth discharged this reporting period (successfully or unsuccessfully) who referred due to concerns about academic performance, the percent that demonstrated improvement while in the MST program. Youth meets the following criteria: 1. In INSPIRE, therapist indicated at enrollment that the youth was referred due to concerns about school performance (academic). Preferred source of information is referral source or a member of the youth s interagency service planning team. 2. In INPSIRE, therapist responds Improved to the question, Did the youth show improvement in school attendance during his/her time in MST? Therapist answers based on his/her clinical judgment. Calculation of percent includes all successfully or unsuccessfully discharged youth that were referred due to concerns about school performance, for whom the question about improvement is answered. Formula is: (# of youth with improved school performance) (# of youth with improved + no change + worse) 9
Outcomes 6 Months Post-Discharge Number of youth for whom data was available The number of youth who were successfully or unsuccessfully discharge, for whom 6-month follow-up data was collected during this reporting period. For each follow-up outcome, the number of youth in the analysis depends on the number of youth for whom information was provided. The table below the graph, showing the number improving and number reporting, highlights the number of youth in the analysis for each outcome. The following criteria are met: 1. Youth was successfully or unsuccessfully discharged (see above) 2. Follow-up was completed and data entered into INSPIRE. Source(s) of data may include interviews, surveys, official records, etc. Information may be provided by the youth, a caregiver, a non-parent relative, or a community agency such as a probation officer or case worker. 3. Date of follow-up interview is within 5-7 months (149-211 days) from the discharge date. No new charges or out-ofhome placements Percent of youth who did not have any of the following reported at 6-months post-discharge: 1) new misdemeanor or felony charges, 2) adjudicated delinquent or criminally convicted for a new offense, or 3) placement out of the home. Includes both delinquency placements and other placements for youth behavior or mental health. A youth is counted as having no new charges or placements if the following criteria are met: 1. A NO is indicated for Has the youth been charged with a misdemeanor or felony, or adjudicated delinquent or criminally convicted, for a new offense since discharge/last interview? (non-criminal probation violations are excluded) AND 2. Either a. NO is indicated for Has the youth experienced any behavior issues since discharge/last interview? b. YES is indicated for Has the youth experienced any behavior issues but the respondent reports that the youth has not been placed for any behavior problems. discharged youth with a 6-month post-discharge interview this reporting period for whom either 1) all relevant information is available or 2) not all relevant questions are answered, but a new offense or placement is indicated. Formula is: (# of youth with no new charges or placement) _ (# of youth with no new charges or placement + # of youth with a new charge or placement) 10
No drug use No school problems Percent of youth with no drug or alcohol use reported at 6-months post-discharge. Tobacco use is not considered when determining whether the youth is using substances. Percent of youth with no school problems reported at 6-months post-discharge. Youth is counted as drug-free if: 1. The respondent answers YES to the question Has the youth experienced any behavior issues since discharge/last interview? but does not report the presence of drug or alcohol use OR 2. The respondent indicates that the youth has not had any behavior issues since discharge. discharged youth with a 6-month post-discharge interview this reporting period for whom the question about behavior issues since discharge is answered. Formula is: (# of youth with no drug use indicated) _ (# youth with behavior issues + # youth without behavior issues) Youth is counted if the respondent answers NO when asked Has the youth experienced any issues in school since discharge/last interview? discharged youth with a 6-month post-discharge interview this reporting period for whom the question is answered. Formula is: (# of youth with no school problems) (# of youth with school problems + # of youth without school problems) When the value for an outcome is n/a, it indicates that either 1) the outcomes is not applicable (e.g., the number of youth successfully discharged is not applicable if no youth were discharged during the reporting period) or 2) data needed to calculate the outcome was missing or not available. This document has been prepared by the EPISCenter. The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University, and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices. The EPISCenter can be contacted at 814-863-2568 or via email at EPISCenter@psu.edu. 11