Indicator-specific guidance is provided separately for results indicators where target setting is required, including C3 and B7-Early Childhood and Preschool Outcomes. There are eight sections that support target setting for this indicator. This indicator specific guidance is intended to be used as a companion to the general guidance.
| Topic |
Guidance |
| I. Indicator Description |
Percent of infants and toddlers/preschoolers with IFSPs/IEPs who demonstrate improved
A. Positive social-emotional skills (including social relationships)
B. Acquisition and use of knowledge and skills (including early language/ communication)
C. Use of appropriate behaviors to meet their needs.
(20 U.S.C. 1416(a)(3)(A) and 1442)
Targets are required for Summary Statement 1 and Summary Statement 2 for each of the three outcomes, yielding a total of six targets. Unless sampling, targets are based on all children with IFSPs/IEPs who exited the Part C/619 program within the reporting year and received services for at least six months.
For Part C: If the State’s Part C eligibility criteria include infants and toddlers who are at risk of having substantial developmental delays, targets should be based on all children excluding those at-risk. |
| II. Federal Indicator Changes |
No changes effective with the release of the FFY 2020-25 Part C Measurement Table. |
| III. State Indicator Specific Changes |
- Has the state made any changes to the data collection methods or data source?
- Has the state made any changes to
- The measurement approach, e.g., changing from the use of one tool and publisher algorithms to the Child Outcomes Summary (COS) process?
- Assessment tools?
- Implementation of the COS process or other data collection methods, including adjustments made during the COVID-19 pandemic?
- Calculations due to changes in publisher algorithm conversions?
|
| IV. State Initiatives Related to this Indicator |
- What state initiatives (e.g., SSIP, targeted training, or other improvement activities) are in place that may impact the outcome results? (Data quality initiatives are discussed in line VI.)
- How and when are these initiatives predicted to impact the results?
- Consider the implementation status of the activities and plans for scaling up in determining when the impact would be expected.
- Consider whether the improvement activities are being implemented statewide or in a limited subset of programs/districts. If focused on a subset, consider the proportion of children in that subset and how that subset will impact the overall state summary statements.
- Are the improvement activities intended to impact a specific child outcome? If so, consider how much more progress is expected for that outcome over the other outcomes.
- It can take several years to have entry and exit data on a full cohort of children after an improvement activity is implemented and to see the full impact on one or more of the child outcomes.
|
| V. Data to Consider |
The state will want to have the following data available for at least the last five years:
- Performance data relative to targets for Summary Statements 1 and 2 for each of the three outcomes
- Progress category data for each of the three outcomes
- The number/percent of children receiving services for at least six months
- Completeness of data
- Baseline data for Summary Statements 1 and 2 for each of the three outcomes.
|
| VI. Indicator Specific Data Quality Issues |
Has the state identified any data quality issues?
- Completeness of the data
- If completeness of the data is a concern, the state first needs to consider how many more children they expect to report child outcomes data for a given year.
- The state needs to consider if the children not currently being reported on in the child outcomes data are different from the group of children for whom child outcomes data are currently reported. If so, how might they be different and what type of progress might be expected from those children (e.g., are most of the children without outcomes data medically fragile or have established conditions)? Are there differences in completeness based on geographic region or race/ethnicity?
- The state can use the Summary Statement Calculator to project the impact on the state’s summary statement values by entering their current year’s data and increase children in the progress categories based on their expectations. For instance, if a particular program/agency that serves children with an established condition is reporting very few children, the state might expect more children to be in progress categories b, c, or d rather than e. The state can simulate various hypothesized combinations to project the impact on the summary statement(s) and use that to guide their target setting.
- Outliers in progress category data
- Sometimes data quality issues can occur because of overrating/overestimating children’s functioning, resulting in a higher than expected percentage of children in progress category e; conversely, data quality issues can occur because of underrating/underestimating functioning, resulting in a higher than expected percentage of children in progress category a.
Improvements in data quality would result in fewer children in progress category e in the first case, and fewer children in progress category a in the second.
For either of these issues, the state would want to analyze the data to determine the magnitude or scope of the issue, e.g., is it occurring statewide or just in some local programs/districts and is it a major or minor issue? The state would also have to consider what the children’s actual progress has been (i.e., if the percentage of children in progress category a appears too high, is their functioning greatly underestimated, meaning one would see more children in progress categories c or d, or is it minor and result in more children in progress category b?)
In either of these two scenarios (under- or overrating), the state can use the Summary Statement Calculator to project the impact on the state’s summary statement values. The summary statement calculator allows the state to move a portion of the children from one progress category to another in various combinations to determine the impact on each summary statement.
- Consider the scope and timing of any strategies to improve data quality.
- How have the results trended over time (upward, downward, stable, or fluctuating)?
- Has new training been added or has the data collection method changed? If so, look at the trends since that change.
- Statewide improvement efforts could take several years to implement and realize results. It is not unreasonable for the targets to be stable (flat?) for the first few years before increasing.
- Data quality issues may not be present across all three outcomes. They may occur with one or two outcomes but not all three.
|
| VII. Indicator Specific Analyses |
- Examine trends in the data.
- How have the summary statement percentages compared to the targets over time? Were targets met? If not, what were some possible reasons?
- Were there changes to the targets? If so, what were the changes? When did they occur and why?
- How have your progress categories percentages changed over time? Were these trends expected?
- Has data completeness remained stable over time or has it varied?
- How do your trends compare to the national average and to similar states? Are there characteristics of your state that explain your position?
- How much do results vary across local programs/districts?
- Compare the summary statement data by local program/district to identify variation and outliers. Consider the low-performing programs/districts and determine how the data would change if those programs/districts moved closer to the mean of the state. Use the Summary Statement Calculator to determine reasonable targets.
- What other factors, such as program improvement efforts) or child characteristics, (disability/eligibility, socioeconomic status, or race/ethnicity,) could be impacting results? Do any of these factors help explain the differences by program/district?
- If the state has experienced changes in the types of children served, consider disaggregating by those characteristics.
- How much of an increase from the baseline will be needed for a meaningful increase?
- The state can enter the baseline year data into the Meaningful Differences Calculator for States to determine how much of an increase in the summary statement percentages is needed for a meaningful, statistically significant increase. If the number of children in the targeted year is expected to increase or decrease by 100 or more, that adjustment should be made to the N size for the year(s) of the future summary statement values.
- The state should consider whether it is reasonable to expect a meaningful increase each year or just toward the end of the 5-year SPP/APR period. See the section on Target Setting Methods for additional methods, including how to set incremental targets.
|
| VIII. Indicator Specific Resources |
State Child Outcomes Data Profile (disseminated annually to states by ECTA/DaSy).
Provides current year and historical data for summary statements, progress categories, and completeness. Displays unexpected patterns in progress category data and comparisons to national averages.
Summary Statement Calculator. Converts progress category data to summary statements for each of the three outcomes.
Child Outcomes Year-to-Year Meaningful Differences Calculator for States (2017). Can be used to look at the statistical significance of change in a state’s child outcomes summary statements from year-to-year.
|
