Performance Based Contracting Guide

This packet contains print-outs of charts specific to your program(s) that provide the following information:

1) How identified risk factors that impact Treatment Opportunity Days apply to youth served in your program.

2) How identified risk factors that impact Sustained Favorable Discharge Rates apply to youth in your program.

3) Your FY 06 and FY 07 actual and “risk adjusted” performance on these two indicators.

4) Your FY 09 performance benchmarks on the two indicators as adjusted by the risk factors.

The following are definitions of terms used in this packet:

Identified Risk Factor: A set of factors – mostly related to child characteristics - associated with positive and negative outcomes for children on the performance indicators of Treatment Opportunity Days Rate and Sustained Favorable Discharge Rate.

Performance Based Contracting: Contractually establishing measurable outcome goals upon which financial penalties and/or incentive payments are based.

Risk Adjustment: Using statistical analysis to determine the significance and relative weights of the identified risk factors on achieving outcomes.

Spell: A residential stay, from admission to discharge. When a child’s residential stay is “Interrupted” by runaway, for example, and the child returns to the same program, they remain in the same residential spell. Children discharged from one program to another or to a different living arrangement, who are then re-admitted to their former program, are initiating a new spell. Children may have more than one residential spell at the same agency, or more than one agency, in any given year.

Sustained Favorable Discharge Rate (SFDR): Out of all residential spells that occur at a program during a fiscal year, the percentage of residents who have been favorably discharged and are in the same placement after 180 days.

Treatment Opportunity Days Rate (TODR): The percentage of total days of care in a fiscal year that residents are not on run, hospitalized, or incarcerated.

What is Performance Based Contracting?

Performance Based Contracting is a method of evaluating a program based upon a set of predefined metrics which are measurable, achievable, and relevant to established goals. Another way of looking at it is that in performance based contracting the contractor is typically required to achieve certain specific objectives, and monetary payment is typically related to the degree to which those contracted objectives are achieved.

In this PBC initiative, the Department and provider representatives to CWAC work committees defined the goals for the residential treatment system in Illinois as the following:

► To improve the safety and stability of children during residential treatment.

► To effectively and efficiently reduce symptoms and improve the functional skills of children through residential treatment.

► To improve outcomes for children at and following discharge from residential treatment.

The two performance indicators incorporated into residential contracts, Treatment Opportunity Days Rate, and Sustained Favorable Discharge Rate, were derived from those goals.

A Note About Risk Adjustment

Given that each residential provider serves a population of children with a different “mix” of characteristics, to measure performance fairly across providers that “case mix” has to be taken into account. Risk adjustment is an effort to equalize the relevant differences between client populations. In risk adjustment, we have identified certain factors that statistically affect what we are attempting to measure – for this fiscal year, SFDR and TODR. Certain clients carry a greater risk for poor outcomes on these indicators because of their history or complexity of presentation. These clients are weighted so as to equalize their overall effect on SFDR and TODR, so that programs with proportionally more children at risk of poor outcomes will not be unfairly measured on these performance indicators. The risk adjustment strategy developed for this initiative works as follows:

A statistical analysis was performed on a large population of children placed in residential treatment during a 3 year period in Illinois. This analysis was performed on certain characteristics of those children (demographic, placement history, etc.), to determine which of those characteristics, or potential risk factors, were associated with positive or negative outcomes on the performance indicators, and, to determine the relative importance of each risk factor in predicting outcomes compared to the others. Those weighted risk factors were then applied to children in each program during FY06 and FY07 to arrive at predictions as to how each program should have performed on SFDR and TODR, given the characteristics of the children they served. Performance benchmarks for FY09 are based on the average predicted, or risk adjusted, performance of each agency in FY06 and 07.

Why is Illinois doing this?

Illinois embarked on this PBC initiative in order to improve outcomes for children served in the residential treatment system. Residential treatment is a powerful intervention that can reduce and stabilize the most maladaptive manifestations of emotional disturbance/mental illness and create the opportunity for children to improve their overall functioning. However, the ability to transfer gains achieved by children while in residential treatment to other, community-based placements remains a significant shortcoming of residential treatment and the child welfare system generally. The PBC initiative, along with other complementary initiatives (e.g. the Transition/Discharge Protocol) is intended to help address this shortcoming. The Department and its partners, the Child Care Association of Illinois, and Child Welfare Training Institute, received a federal grant to help organize the residential PBC initiative and evaluate this collaborative effort between the Department and the provider community.

How has the process worked so far?

The CWAC Data Test Workgroup has been meeting since February, 2007 with representation from providers, UIC, Northwestern, Chapin Hall and DCFS, to develop the PBC performance indicators and risk adjustment strategy. Although a great deal of thought, input, and effort has gone into this endeavor so far, much further work needs to be done. The Data Test Workgroup laid the groundwork for this initiative through its previous work developing system-wide quality improvement measures (e.g. the Dashboard), and expects to continue its work, particularly to further develop and refine the risk adjustment process. Judge Kate Kearney, who is evaluating the performance contracting process as part of a multi-state Child Welfare Institute evaluation project, has attended many DCFS-provider meetings over the past 14 months and indicates that the level of public-private collaboration in Illinois stands out compared with the other states.

For each program, the base for analysis is treatment spells which may or may not be the same as the number of youth served in your program in a given year. The treatment spells represent the youth in your program; keep this in mind while looking at these results. For example ~ if youth were in and out of the program to the extent that they were discharged and readmitted, there will be a higher number of treatment spells than the number of youth actually living in your program.

Description of columns:

Column A, Risk Factors – Each item can affect the number of days a youth will be present in a program to receive treatment services (as opposed to being in detention, psychiatrically hospitalized, or on runaway status).

Columns B, C, G, and H, Actual Number and Actual % or Mean of Your Youth for FY – These columns show, per fiscal year, the number and percentage of treatment spells (youth) in your program that either have or didn’t have the factors listed in Column A. The values in Columns C and H are all %s, except for Age, which is the average age of the treatment spells (youth) in your program.

Columns D and I, Statewide % or Mean for FY – These columns show, per fiscal year, the percentage of treatment spells (youth) in the overall comparison group of DCFS youth in residential care who have or don’t have the factors listed in Column A. The values in Columns D and I are all %s, except for Age, which is the average age of the treatment spells (youth) in the overall comparison group of DCFS youth in residential care.

Columns E and J, Risk Adjustment Weight for FY– This column describes how heavily a particular risk factor will impact the treatment spells (youth) described in Columns B and G, and the ability of the youth represented to remain in placement to receive services; intuitively, it makes sense that some factors will have more impact than others.

Columns F and K, FY Impact on Expected Performance -- This column shows the major risk factors that affect the ability of your youth to be available for treatment and are taken into account through risk adjustment. The total in this column is an approximation of the magnitude that risk factors increase or decrease your expected rates on the performance indicators.

(continued)

(Treatment Opportunity Days Chart Key, continued)

Meaning of Numbers in Columns

· If a number in Column E, F, J, or K is 0, that means the factor in Column A has NO IMPACT on the youth’s ability to remain in the program for treatment opportunity days

· If a number in Column E, F, J, or K is LESS THAN 0, this shows that the factor in Column A has impact, and DECREASES the expected Treatment Opportunity Days for your youth. The farther down from zero, the greater the impact of that factor and the lower the expectation for Treatment Opportunity Days.

· If a number in Column E, F, J, or K is GREATER THAN 0, this shows that the factor in Column A is impacting your youth in a way that INCREASES their ability to be present for treatment.

What the combination means for you:

Because any given program’s client population is likely to vary to some extent from year to year, performance benchmarks for FY09 will be based on the weighted average risk adjustment for 2 years, FY06 and FY07. The averages are weighted both by the number of spells and by the proportion of children possessing the risk factors that can impact outcomes. If your program served more youth with a higher proportion of significant risk factors during one of these two years, this will impact the overall average risk adjusted benchmark for FY09.

Looking at column B/G, you can see the commonality of certain risk factors in your specific program. Comparing the percentages of that column to the next column (Statewide Average), you can see how your client mix at a particular time compares with the statewide mix. Glancing at the next column, you can see the risk factors that are expected to have the most and least impact on treatment participation within your program.

From a broader perspective, comparing the information across the two fiscal years can reveal changes in clients served, change of placement history, etc. As providers, this information can be reviewed in terms of examining additional factors within providers’ control that might impact youths’ ability to remain in treatment, such as the provider’s approach to runaway behaviors and use of hospitalization and detention. There may be ways to influence the TODR via additional training, adjusted expectations for staff, varied programming for clients served based on prevalence of factors, etc.

Importance of this Outcome:

TODR is penalty based (however, the beds are guaranteed), so there is no consequence if you outperform (ie, if you achieve a higher percentage of

days in the program than your projected TODR). The ceiling for TOD is 98%. If you underperform, there is a financial penalty determined as follows at the end of the fiscal year:

The total number of days that all youth are present in the program through the end of the fiscal year is divided by the total number of days accrued from all residential spells during the fiscal year. If this percentage is below the benchmark percentage set for your program, you will be assessed 25% of the per diem cost for each bed day represented in the difference between the benchmark and actual rates.

For example, if a program has 10 residential spells and each spell lasts 365 days, 3650 days represents a 100% TODR. If that program’s benchmark TODR is set at 95%, all youth would need to be present for 3468 days (in other words, their absences could total 182 days, or on average approx. 18 days per youth per year). If the program achieves a 90% TODR, this means youth were present for 3285 days (for total absences of 365 days, or 36.5 days on average per child). The 25% penalty would be applied to the difference between the 95% benchmark TODR and 90% actual TODR, as follows:

It would be important for agencies to have their own reconciliation process on at least a quarterly basis, to compare with the final TODR when it is figured.

(Treatment Opportunity Days Chart Key, continued)

TODR Example

Analysis: The most significant child factors are the history of aggression/meds and history of residential care placement, both of which are associated with a lower expected TODR. Being a moderate level provider means Kids Hope United Northern is expected to have a lower expected TODR (compared to if they were a moderate group home). Being in an exurban location means Kids Hope United Northern is expected to have a higher TODR (compared to if they were in Chicago).

For each program, the base for analysis is the percentage of all spells from which youth that were favorably discharged were able to sustain their discharge placement for 180 days. This outcome inherently guides length of stay in addition to discharge stability, as the SFDR is describing the number of youth expected to be discharged AND remain stable post-discharge for at least six months. The treatment spells represent the youth in your program; keep this in mind while looking at these results.

Description of columns:

Column A, Risk Factors – Each item can affect the number of days youth could be expected to sustain stable placement past discharge from the residential program.

Columns B, C, G, and H, Actual Number and Actual % or Mean of Your Youth for FY – These columns show, per fiscal year, the percentage and number of treatment spells (youth) in your program that either have or didn’t have the factors listed in Column A. The values in Columns C and H are all %s, except for Age, which is the average age of the treatment spells (youth) in your program.

Columns D and I, Statewide % or Mean for FY – These columns show, per fiscal year, the percentage and number of treatment spells (youth) in the overall comparison group of DCFS youth in residential care who have or don’t have the factors listed in Column A. The values in Columns D and I are all %s, except for Age, which is the average age of the treatment spells (youth) in the overall comparison group of DCFS youth in residential care.

Columns E and J, Risk Adjustment Weight for FY – This column describes how heavily a particular risk factor will impact the treatment spells (youth) described in Columns B and G, and the ability of the youth represented to be discharged within a timely manner and remain in stable placement post-discharge.

Columns F and K, FY Impact on Expected Performance -- This column shows the major risk factors that affect the ability of your youth to be discharged within a timely manner and maintain stability in the next placement. The total in this column is an approximation of the magnitude that risk factors increase or decrease your expected rates on the performance indicators.

(Sustained Favorable Discharge Chart Key, continued)

Meaning of Numbers in Columns

· If a number in Column E, F, J, or K is 0, that means the factor in Column A has NO IMPACT on the youth’s ability to be discharged in a reasonable time frame and maintain stability for 180 days thereafter.

· If a number in Column E, F, J, or K is LESS THAN 0, this shows that the factor in Column A has impact, and DECREASES the expected timeframe for discharge as well as the expected stability in the post-residential placement. The farther down from zero, the greater the impact of that factor and the lower the expectation for SFDR.

· If a number in Column E, F, J, or K is GREATER THAN 0, this shows that the factor in Column A is impacting your youth in a way that INCREASES their ability to be discharged and maintain stability post-discharge.

What the combination means for you:

Importance of this Outcome:

If you underperform on SFDR by having a lower than expected rate of discharges that sustain for the 180 days, there is no penalty. SFDR is incentive based, so you will get a bonus if you outperform your target, as follows:

If a program has 10 residential spells through the year, and has a benchmark SFDR of 20%, the agency is expected to favorably discharge 2 children whose post-discharge placement must remain intact for 180 days. If that agency achieves an SFDR of 40%, or 4 favorable discharges sustained for at least 180 days, the agency will receive a bonus for 2 children for the average number of days that the 4 SFD children remained in their discharge placements, up to a maximum of 270 days per child.

The SFDR bonus amount is based on the difference between the statewide weighted average of residential per diems and the statewide weighted average per diems of different types of step down placements. These per diem averages are yet to be announced, but the bonus payment should essentially be determined as follows for the above example:

SFDR Example:

Analysis:

The most significant child factors are average age and spell < 365 days duration (i.e., average length of stay of Kids Hope Northern’s casemix). The older the average child in the caseload is, the higher we would expect Kids Hope Northern’s SFDR to be. The more “new” kids in the caseload, the more we would expect Kids Hope Northern to have a lower expected TODR.

TREATMENT OPPORTUNITY DAYS RATE KEY

Column Headings	Explanation
FY 06 and FY 07 # Spells	This number represents the number of documented spells that occurred in the fiscal year. Any client discharged and readmitted during the year will be counted as more than one spell.
Actual TOD Rate (%)	This is the percentage of treatment opportunity days that your program achieved; ie, of the spells represented, the percentage of days clients were present for treatment.
Risk Adjusted TOD Rate (%)	Given the Risk Adjustment Factors, this is the rate you would have been expected to achieve.
Actual Minus RA rate	If a negative number, your program underperformed. If a positive number, your program achieved a higher than expected rate of TOD.
FY 09 Benchmarks ~ Average # of Spells	This is an average number of spells based on the number of spells that occurred in FY 06/07.
Avg TOD rate (%)	This is the average TODR based on FY 06/07.
Weighted avg risk adjusted TOD rate (%)	This is the average of the risk adjusted expectations for FY08 and 07. It’s weighted by the number of spells, so the year with the greater number of spells will have a relatively larger impact on the FY09 benchmark.
Avg TOD minus avg RA rate	If in FY09 your agency has the number of spells that you had, on average, during FY06 and 07, and your performance in FY09 remains the same as your average performance from FY06/07, this is the difference between that level of performance and the risk adjusted benchmark for FY09.

SUSTAINED FAVORABLE DISCHARGE RATE KEY

Column Headings	Explanation
FY 06 and FY 07 # Spells	This number represents the number of documented spells that occurred in the fiscal year, which may coincide with # of clients served.
Actual SFD Rate (%)	This is the percentage of sustained favorable discharges that your program achieved; ie, of the spells represented, the percentage of clients who with SFD.
# SFDs	Translation from percentage ~ how many actual spells there were.
Risk Adjusted SFD Rate (%)	Given the Risk Adjusted Factors, this is the rate you would have expected to achieve.
Actual Minus RA rate	If a negative number, your program underperformed. If a positive number, your program achieved a higher than expected rate of SFD.
FY 09 Benchmarks ~ Average # of Spells	This is the number of spells anticipated for FY 09 based on FY 06/07.
Avg SFD rate (%)	This is the percentage expected for FY 09 based on averaging FY 06/07.b
#SFDs	How many SFD spells should occur in FY 09.
Weighted avg risk adjusted SFD rate (%)	This is the average of the risk adjusted expectations for FY08 and 07. It’s weighted by the number of spells, so the year with the greater number of spells will have a relatively larger impact on the FY09 benchmark.
Avg SFD minus avg RA rate	If in FY09 your agency has the number of spells that you had, on average, during FY06 and 07, and your performance in FY09 remains the same as your average performance from FY06/07, this is the difference between that level of performance and the risk adjusted benchmark for FY09.

Issues related to treatment efficiency and efficacy, especially post-discharge, are increasingly relevant across the country and calls for reinvigorating (or as AACRC states, reinventing) residential services are growing louder. This is not, of course, just a residential treatment issue; efficiency and efficacy also relates to the overall system of care, wherein the entire service system is not designed to adequately address sustaining progress of highly context dependent youth; something we are beginning to focus on and address through the current varied initiatives.

Although quite a bit has been completed in preparation for this first year of performance-based contracting, the list of ‘next steps” remains long. For example, the following items need further research and development:

· We need to establish a clinical measure to be able to incorporate this aspect into the risk factors

· We will continue to examine the data for additional significant risk factors

· We will advance the process for identifying exemptions (severe/profound DD youth and youth that are considered performance exceptions)

· A collaborative residential provider quality improvement initiative is recommended to encourage learning and sharing best practices to improve outcomes

· We recommend continued assessment and reflection of systems and processes that impact the "front-end," as well as discharge/transitional service delivery that may affect performance

The risk adjustment profile will be run again at the end of FY 09 and may result in benchmarks being altered if an agency’s population changes significantly; this will impact smaller agencies where 1 or 2 youth can significantly impact the rates.

It is important to note that mistakes may have been made with the current data; please contact DCFS per cover letter guidelines, as re-examination could result in readjustment of benchmarks. The final benchmarks will be sent out with contracts in May.

Final Note:

This guide incorporates the work of many members of the Data Test Committee. If, as a consumer, you identify an area that is either notably missing or could be explained more clearly, please send your comments in so this guide can remain a useful document.