Tip Sheet 3: Establishing a Fidelity Threshold

Introduction

A fidelity threshold score is important for understanding whether practitioners are implementing practices sufficiently to achieve intended child and family outcomes. A threshold score is a predetermined level on a fidelity tool that distinguishes between practitioners not demonstrating and those demonstrating adequate practice implementation. This enables you to identify practitioners who may need more support and practice to reach fidelity and those whose support could be reduced. Using a fidelity tool repeatedly can also identify practitioners whose practice implementation has drifted from acceptable to unacceptable levels over time.

Some practices have associated fidelity tools with thresholds that the developer established while conducting research to form the evidence base for the practice. Unfortunately, many evidence-based practices do not have such a research-based fidelity threshold score. If you are using a tool that does have an associated fidelity threshold score, use that threshold; if you are using an existing fidelity tool without a threshold score, work with the tool developer to determine an appropriate threshold score.

This tip sheet offers considerations for selecting a fidelity threshold when one has not been provided. Use the resources listed or consult with evaluation or research experts for additional guidance.

Where to Start?

Before you begin:

  • Start with a high-quality fidelity tool. A fidelity threshold score will be useful only if the associated tool provides accurate and reliable information on practice implementation.
  • Assemble a team of experts. When there is no established fidelity threshold, deciding where to set the threshold requires professional judgment. The best decision-makers have deep knowledge of the intended implementation of the practice and how the practice contributes to intended outcomes. They can be practice developers and other content experts or practice experts from your organization who understand the practice and have experience supporting practitioners in their implementation. For example, state or local program directors, training providers, coaches, and experienced practitioners can provide valuable input on selecting a meaningful threshold score.

Guiding Questions

There are many approaches to establishing threshold scores. Answering the following questions will help you select the best approach for your state or organization:

  • How many fidelity levels do you need? A common approach is to use a single threshold to indicate two results: The practitioner reached the fidelity threshold or did not reach it. Another option is to set multiple threshold levels, such as “does not meet fidelity,” “meets fidelity,” and “expert,” to identify practitioners needing more support and those who might serve as coaches or mentors. This decision depends on your goals for the fidelity assessment and how you plan to use the results.
  • Are some practice components more important than others? If so, consider giving more weight in the scoring to items related to the more important practice components. Or if your expert team believes that certain practice components are essential to producing the desired outcomes, designate particular items or sections of the tool as “must pass” in order for practitioners to reach fidelity. These approaches account for differences in the importance of various program components for achieving targeted outcomes, but they require more complex calculations.
  • How will you address variation in a practitioner’s scores across items on the fidelity tool? It may not be reasonable to expect a competent practitioner to score in an acceptable range on every item. A typical approach is to use a threshold score that represents an average acceptable score across all items. This means that a practitioner can receive a high score on some items and a low score on others, but the scores overall indicate acceptable fidelity of implementation. If your team believes that a low score on particular items would not be acceptable, consider the greater weighting or “must pass” options above.
  • Does your fidelity tool have subsections with multiple items for different practice components or domains? If so, you may want to identify threshold scores for each component or domain. This will help you determine if practitioners are reaching fidelity on all or only some practice components.

A Process for Identifying a Threshold Score

Once your team has preliminary answers to the guiding questions above, use a continuous improvement process to develop, refine, and finalize a fidelity threshold score (or scores).

  • Identify an initial fidelity threshold score. Have your experts review the items on the fidelity tool and envision how a practitioner with the minimum level of acceptable practice implementation would score. With this hypothetical practitioner and your answers to the guiding questions in mind, have the experts identify a minimally acceptable total score (e.g., 78 out of 100).
  • Make adjustments. Review the initial threshold score. Set the threshold high enough that the acceptable level of fidelity is likely to have a meaningful impact on outcomes but low enough that all competent practitioners (not just the most expert) can achieve fidelity.
  • Conduct a pilot test. Have experts use the fidelity tool and the proposed threshold to score implementation of several practitioners. Then determine if the fidelity scores match the experts’ professional opinion about whether practitioners are implementing the practice in a way that will produce intended outcomes. Revise, retest, and reflect until the experts agree the threshold can distinguish between sufficient and insufficient levels of implementation.

Resources

The process of setting a fidelity threshold score is called “standard setting” in the assessment field. Two resources are:

The contents of this tool and guidance were developed under grants from the U.S. Department of Education, #H326P120002 and #H373Z120002. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. Project Officers: Meredith Miceli, Richelle Davis, and Julia Martin Eile.