Jason Lowther

In my last blog I looked at how designing a more experimental approach into this and future programmes could yield lots of insight into what works where. This week I would like to extend this thinking to look at how “theory-based” approaches could provide further intelligence, and then draw some overall conclusions from this series.

As well as rigorous analysis of quantitative impacts, theory-based approaches to evaluation can help to test ideas of how innovative interventions work in practice – the “how?” question as well as the “what works?” question[1].

For example the Troubled Families practitioners might have developed theories such as:

• Having consistent engagement with a key worker, and working through a clear action plan, will increase families’ perception of their own agency and progress.
• Having regular and close engagement with a key worker will enable informal supervision of parenting and reduce risk around child safeguarding concerns.
• Having support from a key worker and, where needed, specialist health and employment support, will increase entry to employment for people currently on incapacity benefit.

Interestingly each of these appears to be supported by the evaluation evidence, which showed much higher levels of families feeling in control; lower levels of children in need or care; and reduced benefits and employment (compared to controls).

• Having consistent engagement with a key worker, and working through a clear action plan, will increase families’ perception of their own agency and progress. The evaluation showed almost 70% of TFP families said they felt “in control” and their worst problems were behind them, much higher than in the “control” group of families.
• Having regular and close engagement with a key worker will enable informal supervision of parenting and reduce risk around child safeguarding concerns. The TFP “final synthesis report”[2] shows the number of children taken into care was a third lower for the TFP families than for the “control” group (p.64).

• Having support from a key worker and, where needed, specialist health and employment support, will increase entry to employment for people currently on incapacity benefit. Again, the final synthesis report suggest that the weeks on incapacity benefit for TFP families was 8% lower than the controls, and the entry into employment 7% higher (pp.56-57).

 

The TFP evaluation probably rightly writes off these last few examples of apparent positive impacts because there is no consistent pattern of positive results across all those tested. Given that the evaluation didn’t attempt to test particular theoretical hypotheses like this, it is possible that they have occurred through natural random variation. But if a much more targeted search for evidence built on theory delivered these results consistently, that would be worth celebrating.

Next week I will conclude the series by reflecting on the four key lessons we can learn from the TFP evaluation experience.

[1] See Sanderson, I. (2002) ‘Evaluation, policy learning and evidence‐based policy making’, Public administration, 80(1), pp. 1-22. And White, M. (1999) ‘Evaluating the effectiveness of welfare-to-work: learning from cross-national evidence’, Evaluating Welfare to Work. Report, 67.

[2] https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/560499/Troubled_Families_Evaluation_Synthesis_Report.pdf

 

 

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther