Unlocking research for local government

Jason Lowther

 

This post originally appeared on the Solace website. You can find it here

Local government needs evidence, from the apparently mundane but nonetheless critical (‘What choice of cladding will minimise the risk of fire spreading?’) to extraordinary insights (‘How do people choose what to eat and whether to be active?’, ‘What skills will today’s youngsters need in the jobs market of 2050?’)

In 2014, Solace commissioned an initial Local Government Knowledge Navigator (LGKN) report, From Analysis to Action: Connecting Research and Local Government in an Age of Austerity, which demonstrated that:

1. Councils have a wide range of evidence needs;
2. There is relevant research and expertise in academia but local government doesn’t make the most of this;
3. There are some impressive examples of collaboration between academia and local authorities but engagement is inconsistent, and often depends on existing links between individual researchers and local government officers or politicians; and
4. There is a need for a change of culture in both communities, and the development of more systematic approaches to achieving connectivity between them.

The key issues identified around local authorities’ approach to successful engagement were:

– senior appreciation of and support for research evidence;
– experience of using research and data to inform decision-making;
– consortia, to spread the cost and reduce risks to reputation;
– support from brokers with the expertise and time to develop proposals;
– the ability and skills to successfully commission research (or access to them); and
– local authority research teams and service managers establishing relationships with local universities.

Following the LGKN work, Solace continued working with the ESRC and LGA through the Research Facilitator, and has established dedicated spokespeople on evidence-based policy.  Recently, it supported the Centre for Public Scrutiny and Nesta in producing the document “Using evidence: A practice guide for local government scrutiny” which launched last month, and which aims to help local government make better use of research evidence.

Recent research has highlighted that local government has particular ways of looking at research that differ from much of academia (including the traditional approaches of public health colleagues).  Local government recognises the importance of ‘place’, and the uniqueness of each area’s situation and background.  As a result, we are particularly interested in evidence, including particular expertise, which relates directly to our place, and this can come across as only being interested in evidence that is ‘home grown’.

When Gemma Phillips and Judith Green recently looked at the transition of public health from the NHS to local government, they found that this different culture reflects local government’s more holistic view of health and wellbeing (rather than healthcare services), and our focus on practicality (rather than the provenance and methodological rigour of research studies).

Austerity has meant less government spending on research and evaluation, particularly at local level, although in 2015/16 national government still invested £5.6 billion in science and research, including £178m in ESRC alone.  So if we want research that government (including local government) will practically use, we probably have to get smarter in terms of more targeted funding, and in particular presenting local government as a key solution to the ‘Impact’ agenda which is vital to universities’ research funding.

This suggests to me that local government needs to take a much more active role in influencing the research agenda locally.  It’s not enough to rely on a kind of ‘Brownian Motion’ in the hope that academics’ research interests will in some way coincide with the policy priorities of local government.  We need to let academics know what our policy priorities are, and to listen to them as they explain what is already known in the relevant fields, and how further research might help us address these priorities.

In the West Midlands, as part of a comprehensive partnership with local universities, the Combined Authority this week set out a clear agenda for research related to its policy priorities for the next three years.  Developed from its Strategic Economic Plan, this includes both economic and social (public service reform) policy priorities, and further development of information sharing and the use of evaluation.  The WMCA ‘Policy Research Plan’ has been developed with input from policy leads and academic experts identified across the local universities and agencies, who will now take forward the agreed activities in a common programme.

So, for example, around ‘connected autonomous vehicles’ we are interested in exploring how emerging technologies can be exploited to improve transport accessibility and reduce subsidy costs whilst supporting enhanced network performance.  Around ‘vulnerable offender pathways’, we need to understand areas where regional working can add most value, together with the offence profile and pathways for specific groups, such as young person and women offenders.

Developing the Plan has compelled policy leads to be much more explicit about the questions they need answering to take forward the policy priorities, and has enabled academic experts to engage in developing these into research-able questions.  As this engagement continues, we expect further synergies to develop giving us much more robust and ‘actionable’ research in future.

References

Local Government Knowledge Navigator reports

http://www.solace.org.uk/knowledge/reports_guides/LGKN_Analysis_to_Action.pdf

http://www.solace.org.uk/knowledge/reports_guides/LGKN_LA_research_collaboration.pdf

Phillips, Gemma, and Judith Green. “Working for the public health: politics, localism and epistemologies of practice”, Sociology of health & illness 37.4 (2015): 491-505.

Using evidence: A practice guide for local government scrutiny

WMCA Policy Research Plan

https://governance.wmca.org.uk/documents/s287/Appendix.pdf

 

lowther-jasonJason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther

Troubled Families: So what can we learn?

Jason Lowther

Over the last five blogs I have looked in some detail at the Troubled Families Programme and in particular its independent evaluation. I’ve argued that the evaluation shows some important impacts from the programme, but has so far missed valuable learning by failing to capture the local angle, covering too short a time horizon, and not designing in a theory-informed experimental approach. This week I want to reflect on four lessons from the experience.

The TFP has delivered real impacts. We know that the TFP has changed how services for these families are delivered. The independent evaluation finds it has mainstreamed “whole-family” approaches, stimulated local multi-agency working, opened up previously impossible data sharing and made employment support more responsive. Families on the programme feel (and told the researchers) that it’s made a big difference to their lives. And the figures local authorities submitted about the changes in families who were classified as “troubled” (out of school, out of work, committing crime, etc) are audited and truthful – they do represent actual changes in people’s circumstances.

The TFP evaluation questions whether these impacts would have occurred in any case, without the TFP itself. But the evaluation was hamstrung by being undertaken too early and for insufficient time, by limited data (for example because academy schools are not required to co-operate on sharing vital information), and by the lack of an experimental and theory-based approach.

So what can we learn from the TFP experience?

First, the TFP isn’t the panacea ministers claimed – trumpeting an incredible 99% success rate whilst delaying publication of the independent evaluation set up the department to face a storm of media criticism. But it has made a big difference: the TFP changed how these services are delivered, the families noticed a significant improvement, and councils have rightly claimed for progress made.

Secondly, the department and evaluators have done a good job at trying to rigorously assess whether the TFP worked better than “business as usual”. Next time, it would be best to build a rigorous experimental approach into the programme design up front – and to develop some testable theories of how the programme is supposed to effect change.

Thirdly, national summaries can only take us so far. The real diamonds of learning are at local level. Departments should fund and support local areas to learn quickly from the natural experiments that happen when different councils adopt and adapt national policy which is based on limited prior knowledge and evidence.

Fourthly, although challenging for politicians with an eye on their ministerial career, pilots need to be given chance to bed-in before being pulled up for evaluation, and evaluation needs to run long enough to know whether we are getting results. Evaluators can learn from past experience and “new” approaches such as theory-based evaluation.

As TFP and other government programmes roll out in future, these four lessons can make sure that we learn and improve outcomes as quickly as possible.

 

lowther-jason

 

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther

Troubled Families: How Experimenting Could Teach Us “What Works?”. Part 2.

Jason Lowther

In my last blog I looked at how designing a more experimental approach into this and future programmes could yield lots of insight into what works where. This week I would like to extend this thinking to look at how “theory-based” approaches could provide further intelligence, and then draw some overall conclusions from this series.

As well as rigorous analysis of quantitative impacts, theory-based approaches to evaluation can help to test ideas of how innovative interventions work in practice – the “how?” question as well as the “what works?” question[1].

For example the Troubled Families practitioners might have developed theories such as:

  • Having consistent engagement with a key worker, and working through a clear action plan, will increase families’ perception of their own agency and progress.
  • Having regular and close engagement with a key worker will enable informal supervision of parenting and reduce risk around child safeguarding concerns.
  • Having support from a key worker and, where needed, specialist health and employment support, will increase entry to employment for people currently on incapacity benefit.

Interestingly each of these appears to be supported by the evaluation evidence, which showed much higher levels of families feeling in control; lower levels of children in need or care; and reduced benefits and employment (compared to controls).

  • Having consistent engagement with a key worker, and working through a clear action plan, will increase families’ perception of their own agency and progress. The evaluation showed almost 70% of TFP families said they felt “in control” and their worst problems were behind them, much higher than in the “control” group of families.
  • Having regular and close engagement with a key worker will enable informal supervision of parenting and reduce risk around child safeguarding concerns. The TFP “final synthesis report”[2] shows the number of children taken into care was a third lower for the TFP families than for the “control” group (p.64).
  • Having support from a key worker and, where needed, specialist health and employment support, will increase entry to employment for people currently on incapacity benefit. Again, the final synthesis report suggest that the weeks on incapacity benefit for TFP families was 8% lower than the controls, and the entry into employment 7% higher (pp.56-57).

 

The TFP evaluation probably rightly writes off these last few examples of apparent positive impacts because there is no consistent pattern of positive results across all those tested. Given that the evaluation didn’t attempt to test particular theoretical hypotheses like this, it is possible that they have occurred through natural random variation. But if a much more targeted search for evidence built on theory delivered these results consistently, that would be worth celebrating.

Next week I will conclude the series by reflecting on the four key lessons we can learn from the TFP evaluation experience.

[1] See Sanderson, I. (2002) ‘Evaluation, policy learning and evidence‐based policy making’, Public administration, 80(1), pp. 1-22. And White, M. (1999) ‘Evaluating the effectiveness of welfare-to-work: learning from cross-national evidence’, Evaluating Welfare to Work. Report, 67.

[2] https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/560499/Troubled_Families_Evaluation_Synthesis_Report.pdf

 

lowther-jason

 

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther

Troubled Families: How experimenting could teach us “what works?”

Jason Lowther

 

In this blog on 3rd Feb, I explored the formal Troubled Families Programme (TFP) evaluation and looked at the lessons we can learn in terms of the timing and data quality issues involved. This week I want to consider how designing a more experimental approach into this and future programmes could yield lots more insight into what works where.

The idea of an “experimental” approach to policy and practice echoes enlightenment period thinkers such as Francis Bacon (1561—1626), who promoted an empirical system built on careful experimentation. Donald Campbell’s ideas[1] on ‘reforms as experiments’ argued that social reforms should be routinely linked to rigorous experimental evaluation. ‘Social engineering’ built on ‘social experiments’ became a popular concept in the USA and social science.

Social experiments in America included work in response to a concern that providing even modest income subsidies to the poor would reduce motivation to find and keep jobs. Rossi and Lyall (1976) showed that work disincentives were in fact less than anticipated. In the field of prison rehabilitation, Langley et al. (1972) tested whether group therapy reduced re-offending rates. The results suggested that this approach to group therapy did not affect re-offending rates.

Unfortunately, meaningful experiments proved more difficult than anticipated to deliver in the field, and even robust experiments were often ignored by policy makers. As a result, until recently this experimental approach fell out of favour in social policy, except in the field of medicine.

The term ‘evidence-based medicine’ appears to have been first used by investigators from a US university in the 1990s where it was defined as ‘a systemic approach to analyze published research as the basis of clinical decision making.’ The evidence-based medicine movement considered experiments – specifically, collections of Randomised Controlled Trials (RCTs) subject to systematic reviews – as the “gold standard” of proof of whether interventions “work” or not.

Randomised controlled trials are sometimes not easy to undertake in social policy environments, but they can be done and they can provide surprising results. Starting in 2007, Birmingham City Council evaluated three evidence-based programmes in regular children’s services systems using RCTs[2]. We found that one programme (Incredible Years) yielded reductions in negative parenting behaviours among parents, reductions in child behaviour problems, and improvements in children’s relationships; whereas another (Triple-P) had no significant effects.

What was interesting for practitioners was that the children in all the trials had experienced improvements in their conduct. Only by use of a formal “control” group were we able to see that these “untreated” children were also improving, and so we were able to separate out the additional impacts of the intervention programmes.

There are a number of lessons from this and other past experience that can help practitioners wanting to deliver robust trials to test whether innovations are working (or not). The most important point is: build the evaluation testing into the design of the programme. The Troubled Families Programme could have built an RCT into the rollout of the programme – for example, selecting first year cases randomly from the list of families who were identified as eligible for the scheme. Or introducing the scheme in some council areas a year earlier than others. Or councils could have done this themselves by gradually rolling out the approach in different area teams.

Sandra Nutley and Peter Homel’s review[3] of the New Labour government’s Crime Reduction Programme stressed the importance of balancing the tensions between fidelity to “evidence based” policy (to maximise the chance of impact) and innovation (to ensure relevance to the local context), short-term wins and long-term learning, and evaluator independence (to ensure rigour) versus engagement (to help delivery).

In my final blog on the TFP next time, I explore the potential for “theory-based” approaches to evaluation helping us to understand “what works and why?” in this and other policy areas.

Campbell, D. T. and Russo, M. J. (1999) Social experimentation. Sage Publications, Inc.

Langley, M., Kassebaum, G., Ward, D. A. and Wilner, D. M. 1972. Prison Treatment and Parole Survival. JSTOR.

Nutley, S. and Homel, P. (2006) ‘Delivering evidence-based policy and practice: Lessons from the implementation of the UK Crime Reduction Programme’, Evidence & Policy: A Journal of Research, Debate and Practice, 2(1), pp. 5-26.

Rossi, P. H. and Lyall, K. (1976) ‘Reforming public welfare’, New York: Russell Sage.

Sanderson, I. (2002) ‘Evaluation, policy learning and evidence‐based policy making’, Public administration, 80(1), pp. 1-22.

White, M. (1999) ‘Evaluating the effectiveness of welfare-to-work: learning from cross-national evidence’, Evaluating Welfare to Work. Report, 67.

[1] Campbell, D. T. and Russo, M. J. (1999) Social experimentation. Sage Publications, Inc.

[2] Little, Michael, et al. “The impact of three evidence-based programmes delivered in public systems in Birmingham, UK.” International Journal of Conflict and Violence (IJCV) 6.2 (2012): 260-272.

[3] Nutley, S. and Homel, P. (2006) ‘Delivering evidence-based policy and practice: Lessons from the implementation of the UK Crime Reduction Programme’, Evidence & Policy: A Journal of Research, Debate and Practice, 2(1), pp. 5-26.

 

 

lowther-jason

 

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther

Troubled Families: Two Secrets to Great Evaluations

Jason Lowther

In this blog last week I explored the (rather flimsy) evidence base available to the developers of the original Troubled Families Programme (TFP) and the potential for “theory of change” approaches to provide useful insights in developing future policy. This week I return to the formal TFP evaluation and look at the lessons we can learn in terms of the timing and data quality issues involved.

The first secret of great evaluation: timing

The experience of the last Labour Government is very instructive here. New Labour appeared as strong advocates of evidence-based policy making, and in particular were committed to extensive use of policy evaluation. Evaluated pilots were completed across a wide range including policies relating to welfare, early years, employment, health and crime. This included summative evaluations of their outcomes and formative evaluations whilst the pilots were underway, attempting to answer the questions “Does this work?” and “How does this work best?”

Ian Sanderson provided a useful overview of Labour’s experience at the end of its first five years in power[i]. He found that one of the critical issues in producing great evaluations (as for great comedy), is timing. Particularly for complex and deep-rooted issues (such as troubled families), it can take a significant time for even the best programmes to have an impact. We now know the (median) time a family remained on the TFP programme was around 15 months.

It can also take significant time for projects to reach the “steady state” conditions, which they would work under when fully implemented. Testing whether there are significant effects can require long-term, in-depth analysis. This doesn’t fit well with the agenda of politicians or managers looking to learn quickly and sometimes to prove a point.

Nutley and Homel’s review[ii] of lessons from New Labour’s Crime Reduction Programme found that “projects generally ran for 12 months and they were just starting to get into their stride when the projects and their evaluations came to an end” (p.19).

In the case of the Troubled Families Programme, the programme started in April 2012, and most of the national data used in the evaluation relates to the 2013-14 financial year. Data on exclusions covered only those starting in the first three months of the programme, whereas data on offending, benefits and employment covered families starting in the first ten months of roll-out.

We know that 70% of the families were still part-way through their engagement with the TFP when their “outcomes” were counted, and around half were still engaged six months later.

It’s now accepted by DCLG that the formal evaluation was run too quickly and for too short a time. There just wasn’t time to demonstrate significant impacts on many outcomes.

The second secret: data quality

Another major element of effective evaluation is the availability of reliable data. Here the independent evaluation had an incredibly difficult job to do. The progress they have made is impressive – for the first time matching a wide range of national data sets, local intelligence and qualitative surveys. But at the end of the day the data quality base of the evaluation is in places poor.

The evaluation couldn’t access data on anti-social behaviour from national data sets, as this is not recorded by the police. This is unfortunate given that the strongest evidence on the effectiveness of TFP-like (Family Intervention) programmes in the past concerns reducing crime and anti-social behaviour[iii].

A chunk of data came from the 152 local authorities. This data was more up to date (October 2015), although only 56 of the councils provided data – which enabled matching to around one quarter of TFP families. The evaluation report acknowledges that this data was “of variable quality”. For example, the spread of academy schools without a duty to co-operate meant there are significant gaps in school attendance data. This will be a serious problem for future evaluations unless academies’ engagement with the wider public service system is assured.

In summary, the TFP evaluation covered too short a period and, despite heroic efforts by DCLG and the evaluators, was based on data of very variable quality and completeness.

Next time we will explore the “impact” evaluation in more detail – looking at how designing a more experimental approach into this and future programmes could yield more robust evaluation conclusions of what works where.

[i] Sanderson, Ian. “Evaluation, policy learning and evidence‐based policy making.” Public administration 80.1 (2002): 1-22.

[ii] Nutley, Sandra, and Peter Homel. “Delivering evidence-based policy and practice: Lessons from the implementation of the UK Crime Reduction Programme.” Evidence & Policy: A Journal of Research, Debate and Practice 2.1 (2006): 5-26.

[iii] DfE, “Monitoring and evaluation of family intervention services and projects between February 2007 and March 2011”, 2011, available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/184031/DFE-RR174.pdf

 

 

lowther-jason

 

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther

 

So: does the Troubled Families Programme work or not? – Part Two

Jason Lowther

In this blog last week I outlined results of the “impact evaluation” element of the Troubled Families Programme (TFP) and the rather limited pre-existing evidence base the TFP had to be built upon. How can government build on existing evidence in designing its initiatives, and what can we do when there isn’t much in the evidence cupboard?

Many government programmes have the luxury of a relatively strong evidence base on which to build. The previous Labour government’s National Strategy for Neighbourhood Renewal and Sure Start programmes, for instance, could draw on decades of research (collated through the 18 Policy Action Teams) on urban initiatives and the impact of early years experiences on achievements in later life. These sometimes honoured the extant evidence more in the theory than in practice[i], but at least they had foundations on which to build.

As evaluations of the Labour government’s Crime Reduction Programme found[ii], it is a difficult task to translate evidence, which is often “fragmented and inconclusive” into practical government programmes. People skilled at this task are in short supply in central government.

But in the case of the TFP, the most robust element of the existing evidence base was a single evaluation using a “control” of 54 families and focussed on addressing anti-social behaviour through Family Intervention Projects. What can government do when the evidence base is thin?

One strong tradition, particularly around medicine and around welfare policies in the USA, has been the idea of “experimental government” using social experiments to determine whether (and if so how) innovative approaches work in practice. For example, in the last three decades of the 20th century, America’s Manpower Demonstration Research Corporation (MDRC) conducted 30 major random assignment experiments involving nearly 300,000 people.

Historically, randomised controlled trials (RCTs) were viewed by many as the “gold standard” of evaluation by allowing statistically robust assessments of “causality” – whether observed changes are due to the intervention being evaluated. More recent thinking emphasises that evaluations need to be designed in the best way to create robust evidence and answer specific questions. Often this will involve a mixture of methods, both quantitative and qualitative. The TFP evaluation used a mixture of methods but without building in a “control” group of “troubled families” not yet receiving the TFP interventions.

Granger[iii] argued (for area based initiatives), that the range and variety of initiatives and the scale of change in government means that a strict statistical “control” is unfeasible. She argued that it is “virtually impossible” to achieve precise and clear-cut causal attribution and that we need clear, strong theories as a basis for counterfactual reasoning and causal inference.

The TFP evaluation did not develop or test a “theory of change” for the programme. This is a pity, because rigorously testing a theory can help illuminate where and how programmes do (or don’t) have real impact.

There are several other lessons we can learn from the existing literature on evaluation in government, for example the importance of timing and data quality. We’ll look at these next time.

[i] Coote, Anna, Jessica Allen, and David Woodhead. “Finding out what works.” Building knowledge about complex, community-based initiatives. London: Kings Fund (2004), esp. pp. 17-18.

[ii] Nutley, Sandra, and Peter Homel. “Delivering evidence-based policy and practice: Lessons from the implementation of the UK Crime Reduction Programme.” Evidence & Policy: A Journal of Research, Debate and Practice 2.1 (2006): 5-26.

[iii] Granger, R. C. (1998) ‘Establishing causality in evaluations of comprehensive community initiatives’, New approaches to evaluating community initiatives, 2, pp. 221-46.
lowther-jason

Jason Lowther is a senior fellow at INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther