Jason Lowther

Over the last five blogs I have looked in some detail at the Troubled Families Programme and in particular its independent evaluation. I’ve argued that the evaluation shows some important impacts from the programme, but has so far missed valuable learning by failing to capture the local angle, covering too short a time horizon, and not designing in a theory-informed experimental approach. This week I want to reflect on four lessons from the experience.

The TFP has delivered real impacts. We know that the TFP has changed how services for these families are delivered. The independent evaluation finds it has mainstreamed “whole-family” approaches, stimulated local multi-agency working, opened up previously impossible data sharing and made employment support more responsive. Families on the programme feel (and told the researchers) that it’s made a big difference to their lives. And the figures local authorities submitted about the changes in families who were classified as “troubled” (out of school, out of work, committing crime, etc) are audited and truthful – they do represent actual changes in people’s circumstances.

The TFP evaluation questions whether these impacts would have occurred in any case, without the TFP itself. But the evaluation was hamstrung by being undertaken too early and for insufficient time, by limited data (for example because academy schools are not required to co-operate on sharing vital information), and by the lack of an experimental and theory-based approach.

So what can we learn from the TFP experience?

First, the TFP isn’t the panacea ministers claimed – trumpeting an incredible 99% success rate whilst delaying publication of the independent evaluation set up the department to face a storm of media criticism. But it has made a big difference: the TFP changed how these services are delivered, the families noticed a significant improvement, and councils have rightly claimed for progress made.

Secondly, the department and evaluators have done a good job at trying to rigorously assess whether the TFP worked better than “business as usual”. Next time, it would be best to build a rigorous experimental approach into the programme design up front – and to develop some testable theories of how the programme is supposed to effect change.

Thirdly, national summaries can only take us so far. The real diamonds of learning are at local level. Departments should fund and support local areas to learn quickly from the natural experiments that happen when different councils adopt and adapt national policy which is based on limited prior knowledge and evidence.

Fourthly, although challenging for politicians with an eye on their ministerial career, pilots need to be given chance to bed-in before being pulled up for evaluation, and evaluation needs to run long enough to know whether we are getting results. Evaluators can learn from past experience and “new” approaches such as theory-based evaluation.

As TFP and other government programmes roll out in future, these four lessons can make sure that we learn and improve outcomes as quickly as possible.

 

 

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther