Assessment of the Therapeutic Alliance Scales: A Reliability and Validity Meta-Analytic Evaluation

Abstract

Abstract Extensive research has been conducted out on the construct of therapeutic alliance. With the growing emphasis on evidence-based practice in psychology it is vital that measures used in both clinical and research settings are empirically well-suited for the population under investigation. However, many measurement issues related to the reliability and validity of the alliance construct remain unaddressed or unresolved. Two studies were designed to add to the scientific evidence on the therapeutic alliance by establishing empirical evidence of the psychometric properties of this construct’s most commonly used measures, with the intention of identifying the most psychometrically sound alliance measures. This was first done by systematically reviewing the literature to identify studies that used the most commonly used alliance measures. Next, key psychometric properties of each measure (internal reliability and predictive validity) were reviewed to determine if the alliance was assessed in the context of individual adult psychotherapy. In the first study, I conducted a reliability generalization analysis to (a) estimate the average reliability coefficient of each alliance measure identified in the systematic review and (b) examine the potential influence that study characteristics may have had on the reliability estimates. Six different alliance measures were included (Agnew Relationship Measure, California Psychotherapy Alliance Scales, Counselor Rating Form, Penn Alliance Scales, Therapeutic Bond Scale, Working Alliance Inventory), in various formats and rater versions, resulting in a total of 17 alliance measure variants for this first analysis. In the second study, I conducted a validity generalization analysis using only those studies from the first study that were identified as containing outcome data. The purpose of this study was to synthesize the alliance-outcome effect sizes that have been reported for the most commonly used therapeutic alliance measures and to assess the potential impact study characteristics may have on those effect sizes. Five different alliance measures (California Psychotherapy Alliance Scales, Counselor Rating Form, Penn Alliance Scales, Therapeutic Bond Scale, Working Alliance Inventory, Vanderbilt Therapeutic Alliance Scale) in various formats and rater versions, resulting in a total of 15 alliance measure variants were included in this analysis. This second study was different from previous alliance-outcome meta-analyses as I only included studies that (a) could be identified as providing psychotherapy, as opposed to other mental health services, (b) assessed the alliance from individual adult psychotherapy, (c) were identified as using the most commonly used alliance measures, and (d) measured the alliance at the midpoint of treatment, or earlier. This second study also differed from previous meta-analyses as I conducted separate analyses for correlational data and partial correlational data. The reliability generalization study found that majority of the alliance measures were good choices for assessing the alliance based on their mean reliability coefficients. The validity generalization study found relatively no difference in the early alliance’s ability to predict treatment outcomes in individual adult psychotherapy between full correlation data (r = .24) and partial correlation data (r = .23). There was also no difference found among the different alliance measures, or their variants, in their ability to predict treatment outcomes, suggesting that no one alliance measure is statistically better at predicting outcomes. The results from both studies suggest that, based on their overall level of reliability as well as their ability to predict treatment outcomes, both researcher and clinicians should consider these measures, with few exceptions, as comparably good choices for assessing the alliance in adult individual psychotherapy.