Comparison of Fidelity Assessment Methods

Research Type

Program/Policy Level Research


Roudebush VAMC, HSRD, Center of Excellence:

  • Co-Principal Investigator, Angela L. Rollins, PhD
  • Co-Principal Investigator, Michelle P. Salyers, PhD
  • Alan McGuire, PhD, Co-Investigator
  • Linda Collins, BS, Research Assistant
  • Collaborating Faculty:
  • John McGrew, PhD, Co-Investigator
  • Doug Leslie, PhD, Co-Investigator
  • Sandy Resnick, PhD, Co-Investigator
  • Lia Hicks, MBA, Fidelity Assessor
  • Jennifer Wright, MSW, Fidelity Assessor


National policy has dramatically increased the emphasis on implementing evidence-based mental health services to meet the needs of people with severe mental illness, and the VHA has made great strides at providing effective, community-based services. One of the cornerstones of the VHA approach is Mental Health Intensive Case Management (MHICM), a model that is based on one of the most well-defined and empirically supported approaches: assertive community treatment. Most recently, VHA policy shifts have resulted in a proposed set of uniform mental health services to ensure access to a standard set of high quality mental health services, such as MHICM, across the entire VHA. However, successful implementation of evidence-based practices on a broad scale requires psychometrically valid, yet practical, ways to assess and monitor degree of implementation (i.e., fidelity). Currently, the only rigorous method to monitor implementation is an on-site fidelity visit, which is a very time-intensive, expensive, and burdensome approach for both the assessor and the program.

The Research Questions:
The primary objective of the proposed study is to examine the effectiveness of innovative and potentially cost-effective methods to ensure the quality of mental health services for disabled veterans with mental illness. Specifically, we will examine the reliability, concurrent validity, and predictive validity of three methods of fidelity assessment: self-report, phone-based remote assessment, and a “gold-standard” on-site fidelity assessment. We will also explore costs associated with each method.

Systems-Research Tools:
This study will examine the accuracy of three approaches to fidelity measurement:  self-report (the approach currently used in the national evaluation of MHICM programs), a new phone-based fidelity assessment, and the gold standard on-site fidelity assessment using the well-validated DACTS (Dartmouth Assertive Community Treatment Scale).

Project Overview:
Currently all MHICM teams (n=111) provide self-reported fidelity annually to the Northeast Program Evaluation Center (NEPEC). VHA tracks hospitalization outcomes over time. We will recruit 32 teams to participate in a phone-based assessment and an on-site fidelity visit with experienced fidelity assessors. We will stratify teams on type of VHA facility and previous year’s performance on the self-assessment. The order of phone and on-site assessments will be counter-balanced, with separate assessors, to reduce potential bias. We will examine level of agreement between fidelity approaches with intraclass-correlations. We will examine predictive validity using regression modeling to examine the association between fidelity method and improved hospitalization outcomes. For our cost identification analysis, we will compare costs across the three methods of assessment, using a mixed (repeated measures) model approach. We will also include a formative evaluation to inform future dissemination of fidelity assessment methods in the VHA and elsewhere.


The proposed study addresses a critical policy issue for VHA and other large health systems: identifying the best ways to monitor program implementation.  The current self-report method for monitoring MHICM programs is of unknown validity.  Establishment of valid yet practical ways to monitor implementation is critical to continue to ensure high-quality care as proposed in the VHA’s strategic mental health plan and the uniform services plan.