Measuring what matters: Effectiveness of implementing evidence-based supported employment for adults with severe mental illness

Abstract

Background/Aims: This investigation compared the utility of two approaches to measuring the effectiveness of a supported employment programme. Both were multi-site trials of evidence-based supported employment which differed by research design. The first approach, a non-randomised multisite intervention, used a parallel group design that did not involve a control group at the site level. The second approach was conducted simultaneously in different locations. This was a controlled trial with randomisation to an evidence-based supported employment programme and a control condition consisting of existing local employment services. The results of the four-site randomised controlled trial (RCT) have recently been reported and provide benchmarks for this comparison. Methods: A four-site parallel group design (n=160) evaluated the implementation of the individual placement and support approach in Brisbane (two sites), Sunshine Coast and Bellerive (Tasmania). The employment intervention involved a full-time employment specialist employed by an existing employment service, to be hosted by a community mental health team. Results: There was no significant difference in commencing employment outcomes between the foursite non-randomised approach to implementation and the four-site RCT (35.6% vs 42.5%). The RCT provided two cohorts whose performance benchmarked the implementation in the non-controlled sites. The promising results of both evaluation approaches show that randomisation did not inflate results, and both methods show utility for programme evaluation in different settings. Conclusions: For the purpose of evaluating new programme implementations, both RCT and non-RCT parallel group designs have potential utility. If the focus is on determining which programme type or which programme ingredient is the most effective, then an RCT design with high internal validity is recommended. In this case, the non-randomised parallel group design had utility through its use of more natural implementation conditions with greater ecological validity. If suitable benchmarks are available and strong potential confounders, such as client age, diagnostic mix and service delivery attributes, are managed, then non-randomised evaluations can be informative, particularly when randomisation is neither practical nor feasible.