What Do We Miss out on When Policy Evaluation Ignores Broader Social Problems?

Daniel Silver and Stephen Crossley

With local government funding being stretched to breaking point over the last decade, it is more important than ever to know whether investment into policy programmes is making a difference.

Evaluation draws on different social research methods to systematically investigate the design, implementation, and effectiveness of an intervention. Evaluation can produce evidence that can be used to improve accountability and learning within policy-making processes to inform future decision making.

But is the full potential of evaluation being realised?

We recently published an article in Critical Social Policy that demonstrated how the Troubled Families programme evaluation remained within narrow boundaries that limited what could be learnt. The evaluation followed conventional procedures by investigating exclusively whether the intervention has achieved what it set out to do. But this ‘establishment oriented’ approach assumes the policy has been designed perfectly. Many of us recognise that the Troubled Families programme was far from perfect (despite what initial assessments and central government announcements claimed).

The Troubled Families programme set out to ‘turn around’ the lives of the 120,000 most ‘troubled families’ (characterised by crime, anti-social behaviour, truancy or school exclusion and ‘worklessness’) through a ‘family intervention’ approach which advocates a ‘persistent, assertive and challenging’ way of working with family members to change their behaviours but, crucially, not their material circumstances.

Austerity, mentioned in just two of the first phase evaluation reports, was not considered as an issue that might have had an impact on families. Discussions of poor and precarious labour market conditions, cuts to local authority services for children, young people and families, and inadequate housing provision are almost completely neglected in the reports. Individualised criteria such as ‘worklessness’, school exclusion and crime or anti-social behaviour were considered but structural factors such as class, gender, and racial inequalities were not; nor were other issues such as labour market conditions, housing quality and supply, household income or welfare reforms.

The first phase outcome of ‘moving off out-of-work benefits and into continuous employment’ did not take into account the type of work that was secured, or the possible impact that low-paid, poor quality or insecure work may have on family life. Similarly, the desire by the government to see school attendance improve did not necessarily seek to improve the school experience for the child, and there is no evidence of concern for any learning that did or did not take place once attendance had been registered. Such issues were outside of the frames in which the policy had been constructed and so were considered to be outside of the boundaries of investigation for the evaluation. The scope for learning was therefore restricted to within the frames that had been set by national government when the programme had been designed.

So what can be done?

While large-scale evaluations of national programmes will still take place, local councils can add to these with independent, small-scale evaluations. These can adopt a more open approach that examined what happened locally and contextualise the programme within the particular social problems that residents experience.

A more contextualised form of evaluation can broaden the scope of learning beyond the original framing of a policy intervention. Collaboration between councils and participants who have experienced an intervention through locally situated programme evaluations can explore people’s everyday problems and the tangible improvements that have been delivered by an intervention (and what caused these outcomes to happen). Such an approach with ‘troubled families’ would recognise the knowledge, expertise and capabilities of many families in dealing with the vicissitudes of everyday life, including those caused by the government claiming to be helping them via the Troubled Families programme. Analysis of the data can be used to identify shared everyday problems and narratives of impact that show improvements to people’s everyday lives. By building up a picture about what approaches have been successful, an incremental approach to improving policy and culture within local institutions can be developed – based on the ethos of learning by doing.

In addition to learning about what works, we can also develop our knowledge of what problems have been left unresolved. Of course, no single policy intervention can possibly solve every dimension of our complex social problems. This does not necessarily mean a failure of the intervention, but rather that there are broader issues that need to be addressed. Knowing about these issues can produce useful evidence to find out about social needs in the local community that are not being met, and which the Council might be able to address or use the new knowledge to inform future strategies.

Evaluation is often seen as a bolt-on to the policy-making process. But re-purposing evaluation to learn more about social problems and the effectiveness of tailored local solutions can create evidence and ideas that can be used to improve future social policy.

 

Daniel Silver is an ESRC Postdoctoral Fellow in the Institute of Local Government Studies (INLOGOV) at the University of Birmingham. He previously taught politics and research methods at the University of Manchester. His research focuses on evaluation, social policy, research methods, and radical politics.

Stephen Crossley is a Senior Lecturer in Social Policy at Northumbria University. He com- pleted his PhD from Durham University examining the UK government’s Troubled Families Programme in August 2017. His most recent publications are Troublemakers: the construction of ‘troubled families’ as a social problem (Policy Press, 2018) and ‘The UK Government’s Troubled Families Programme: Delivering Social Justice?’, which appeared in the journal Social Inclusion.

England’s over-centralisation – Part 2: It IS instinctive

Chris Game

There was much in Jessica Studdert’s recent blog to agree with and applaud, but one sentence particularly struck me – the one opening her fourth paragraph: “The centralised response isn’t just structural, at times it has felt deeply instinctive.”.

So, equally instinctively, I did what even an erstwhile academic does during a lockdown – some heavyweight research, naturally. Like re-watching and content analysing the first 69 Government Covid-19 daily press conferences – one of those crisis features that, like the Thursday evening clapping, lives on because no one knows quite how to stop it.

I exaggerated with the ‘heavyweight’ bit, but I did count – sorry, totalise – the press conferences. So, first question: Which minister, Johnson excepted, was the first to front one?

No, not Foreign Secretary Dominic Raab. As First Secretary of State, he stood in while Johnson was hospitalised, but was actually eighth minister to feature. Surely, then, Health and Social Care Secretary, Matt Hancock. Nope, though he and his permanent pink tie have currently clocked up more appearances than Johnson himself.

Struggling? Chancellor of the Exchequer, Rishi Sunak? Hardly Robert Jenrick, Secretary of State for Housing, Communities and Local Government – for all the considerations touched on in Studdert’s blog. Surely not Home Secretary Priti Patel, despite being apparently the only woman minister capable of reading from a lectern.

They’ve done four, five and three respectively, but the shooting star we are looking for is Environment, FOOD and Rural Affairs Secretary, George Eustice. How short are our memories. His brief includes the so-called food supply chain, and this was late March – panic-buying, pasta-hoarding weekend.

Now the seriously tricky question. How many winning elections to serve as a plain local government councillor – not London Mayor – have all 12 featured Ministers fought between them? Maybe not a huge number? One!

One four-year term of elected local government experience between the lot of them. It was served by then 24-year old Gavin Williamson, now Education Secretary, giving English primary schools his considered judgement on when they should reopen.

It’s easy to mock – really easy – but there are archive pictures of Williamson doing his thing as North Yorkshire County Council’s ‘Champion of Youth Issues’ . Making him, I believe, alone among that TV-trusted Cabinet dozen to have even minimal first-hand insight into how local government operates in the policy field for which he is responsible.

The others can tell you lots, variously, about banking (Hancock), hedge fund management (Sunak), litigation (Raab), corporate finance (Alok Sharma), corporate law (Jenrick), public relations (Eustice, Patel), journalism (Johnson, Gove), marketing (Grant Shapps), Conservative Central Office (Patel, Oliver Dowden).

But actually experiencing what they presumably aspired to do – campaigning, meeting constituents, getting elected, representing people, learning about the provision and funding of public services, the whole government and public administration thing – for some reason never grabbed them or even struck them as career-relevant.

Which today means they know virtually nothing at first-hand about some of the vital stuff local governments do, often to the unawareness of even their own publics: emergency contingency planning, air quality monitoring, water testing, pest control, health and safety at work inspection – oh yes, and communicable disease investigation and outbreak control.

Time for a brief digression on the changing meaning of the word ‘nuisance’. It was one of my mother’s favourite words, applied frequently to my sister and myself, but to almost any usually minor upset to her daily life routine. Mask-wearing and disinfecting supermarket trolley handles would be a ‘nuisance’, not the wretched pandemic itself.

Yet the etymology of ‘nuisance’ is the Latin ‘nocere’ – to harm – and its original 15th Century meaning could quite conceivably be applied to Covid-19 and its capacity to inflict serious and even fatal harm.

The mid-19th Century predecessor of today’s Director of Public Health in Birmingham, Dr Justin Varney, would therefore have boasted the title of Nuisance Inspector – his nuisance agenda including factory air pollution, small-pox and cholera outbreaks, and sanitation, with the first generation of public urinals.

Nuisance Inspectors could not by themselves transform towns and cities, but they played a huge part. As do their modern-day successors – Public or Environmental Health Inspectors. Those successors, however – the ones that have survived the past decade of local government funding and employment cuts – could and should, as Studdert noted, have been doing even more.

The Chartered Institute of Environmental Health reckons there are some 5,000 Environmental Health Officers (EHOs) working in UK local councils. All have job descriptions including responsibilities like “investigating outbreaks of infectious diseases and preventing them spreading further.”

That’s what they do – test, track, trace and treat people with anything from salmonella to sexually transmitted diseases – in areas, moreover, with which they are totally familiar and have networks of contacts. ‘Shoe-leather epidemiology’ is the technical term – seriously.

So presumably, as in other countries – South Korea, Singapore, Germany, Ireland – these EHOs will have been reassigned from other work and spent their time contact tracing?

Rhetorical question – we all know the answers. From early March, contrary to World Health Organisation guidelines, our Government’s big ideas were to ‘delay’ the spread of Covid-19, then develop vital (now less vital) smartphone apps.

This enabled the consequently limited scale of contact-tracing to be undertaken centrally by staff newly recruited by Public Health England – the executive agency of Matt Hancock’s Health and Social Care Department created in the ill-conceived NHS upheaval in 2012.

Insufficient, inexperienced staff doing a job crying out for the skills, knowledge and contacts of council EHOs, who instead were monitoring social distancing rules in pubs, clubs and restaurants.

There are almost always costs in ‘keeping it central’, but, as we have seen, for so many ministers, it must be instinctive. It’s all they and most of their civil servants know at first hand. The alternative would be funding and at least sharing data with pesky local authorities, thereby losing some of their precious control.

Finally, last weekend, all other options exhausted, the Government did allocate a ring-fenced £300 million to English councils to play a leading role, starting immediately, in tracking and tracing people suspected of being at risk of Covid-19.

This time, tragically, the cost of blinkered, prejudiced, self-protective government was paid in lives.

Wider opening of schools during covid-19

Cllr. Ketan Sheth 

Education impacts society and is a measure and driver of our progress as a community.  A good education keeps us physically and mentally strong and plays a key role in the betterment of our socio-economic environment and the communities in which we live. Education is the ultimate pathway of success, providing the support that enables each and everyone of us to keep growing our knowledge and ourselves across the whole of our lives.  That is why education is given the highest status in today’s world. The delivery of our education service, however, has been heavenly impacted at all levels by Covid-19.

We are now starting discussions on easing the Covid-19 lockdown by reopening schools in a bid to restart our economy. The concern is that this might become a breeding ground for a second wave of Covid-19 cases.  Indeed, many parents may decide to keep their children at home, as it is possible that the rate at which the virus spreads may increase when schools open. It is therefore possible that the decline in the number of people infected may be affected. I say ‘possible’ because analysis of international trends suggest there are no definitive indications that opening schools accelerates infection. Schools have not yet been shown to push the reproduction rate (R) above one.

Many of our families and the communities in which we live have actively helped reduce (R) over the last 5 to 6 weeks. As a result the number of hospital admissions of Covid-19, in some communities, has now stabilised. Because of that, the reduction of the reproduction rate has slowed since mid-April, but it is still under 1. This has led to the debate on balancing the needs of the economy and the safety of our communities. In this case, that means our children.

The role of local government is to know and understand its communities and their children. Local government delivers services to local residents every day and is the vital ingredient to finding the best community solution.

As Covid-19 shows, pandemics are not technocratic. They are complex, creating social and behavioural challenges. Parents, teachers, and children are grappling with the threat of contracting the infection, often while dealing with personal loss. Effective management mechanisms between national and local government are therefore critical. We need to strengthen local responses and systems, and respect and build the capacity of local government to manage the policy response from health to the economy, to social protection. Investment in local government will be key to successful recovery and long-term resilience.

Thinking and acting locally will help to ensure that the spread of Covid-19 is curtailed and our communities protected. As far as opening schools goes: this needs to be managed locally and to be responsive to local concerns and needs. A locally crafted step-by-step approach is demanded, setting a code-of-conduct that ensures the highest standards of hygiene, and ensuring all school operations, break times, and classroom divisions meet carefully set social distancing guidelines.

For government to work effectively in the worst of times, it needs to have well-oiled systems, practices and resource flows.  We need to reflect on, and respond to, our population’s needs and changing realities quickly, intelligently, and always with the wellbeing of our communities at the forefront. Anxiety will linger over infection rates, but if we work together at a local level in the communities where we live, we can be agile, and creative, in our services. Together we can do it locally.

 

Picture1.png

Cllr. Ketan Sheth is a Councillor for Tokyngton, Wembley in the London Borough of Brent. Ketan has been a councillor since 2010 and was appointed as Brent Council’s Chair of the Community and Wellbeing Scrutiny Committee in May 2016. Before his current appointment in 2016, he was the Chair of Planning, of Standards, and of the Licensing Committees. Ketan is a lawyer by profession and sits on a number of public bodies, including as the Lead Governor of Central and North West London NHS Foundation Trust.

Covid-19: Is Government Really “Led By The Science”?

Jason Lowther, Director of the Institute of Local Government Studies, University of Birmingham (not representing the views of the university)

In the midst of the EU Referendum campaign, Michael Gove famously commented that “people in this country have had enough of experts”. No longer. Fast forward four years, Gove (and every other minister) is sharing press conferences with professors and claiming to be “led by the science”. But with the UK topping the European tables of Covid-19 deaths, what does that actually mean? And is “science” the only type of knowledge we need to make life-saving policy in the Covid crisis?

Making policy is difficult and complex – particularly in a crisis, and especially one caused by a virus that didn’t exist in humans six months ago but has the potential to kill millions. The information we have is incomplete, inaccurate and difficult to interpret. Politicians (and experts) are under huge pressure, recognising that their inevitable mistakes may well cost lives. My research has shown that even in more modestly stressful and novel contexts, policy makers don’t just use experts to answer questions, but also their public claims to be listening to experts are useful politically. Christina Boswell identified the ‘legitimising’ and ‘substantiating’ functions of experts. Listening (or at least appearing to be listening) to experts can give the public confidence that politicians’ decisions are well founded, and lend authority to their policy positions (such as when to re-open golf courses).

Covid-19 is a global issue requiring local responses, so the spatial aspects of using experts and evidence are particularly important. Governments need to learn quickly from experiences in countries at later stages in the epidemic, including countries where historic relations may be difficult. Central governments also have to learn quickly what is practical and working (or not) on the ground in the specific contexts of local areas, avoiding the vain attempt to manage every aspect from Whitehall. My research shows that the careful use of evidence can help here, developing shared understandings which can overcome historic blocks and enable effective collaboration. But in Covid-19 it seems central government too often is opting out of building these shared understandings. Experience in other countries has sometimes been ignored. Vital knowledge from local areas has not been sought or used. Instead of transparently sharing the evidence as decisions are developed, evidence has been hidden or heavily redacted, breaking a basic principle of good science and sacrificing the opportunity to build shared understandings open to critical challenge.

What counts as “evidence” anyway? Different professional and organisational cultures value different kinds of knowledge as important and reliable. In my work with combined authorities, I found that bringing mental health practitioners into policy discussions had opened up a wide range of new sources of knowledge, such as the voices of people with lived experience. And, carefully managed, this wider range of types of knowledge can lead to better decisions. The Government’s network of scientific advisory committees, once we finally were told who was involved, seems to have missed some important voices. The editor of the Lancet, Richard Horton, argued that expertise around public health and intensive medical care should have been in the room. I would also argue that having practical knowledge from local councils and emergency planners could help avoid recommendations that prove impossible to implement effectively. As Kieron Flanagan has noted recently, we learned in the inquiry into the BSE crisis that esteemed experts can still make recommendations which are impossible to implement in practice.

Making a successful recovery will require government quickly to learn lessons from (their own and others’) mistakes so far. Expert advice and relevant data should be published, quickly and in full – treating the public and partners as adults. Key experts for this phase (including knowledge of local public health, economic development, schools, city centres and transport) should be brought into the discussions as equal partners – not simply the “hired help” to do a list of tasks ministers have dreamt up in a Whitehall basement. Then we can have plans that are well founded, widely supported, and have the best chance of practical success. Our future, in fact our very lives, depend on it.

This post was originally published in The Municipal Journal.

 

lowther-jason

Jason Lowther is the Director of INLOGOV. His research focuses on public service reform and the use of “evidence” by public agencies.  Previously he led Birmingham City Council’s corporate strategy function, worked for the Audit Commission as national value for money lead, for HSBC in credit and risk management, and for the Metropolitan Police as an internal management consultant. He tweets as @jasonlowther

England’s over-centralisation isn’t just a governance issue now – it’s a public health emergency

Jessica Studdert

The concentration of power at Westminster and Whitehall has long frustrated those of us who engage closely with the structures of governance and compare it to decentralised norms across much of Europe. Now, as with so many facets of the Covid-19 crisis, the pandemic has exposed national vulnerabilities and left us grappling with the consequences. The grip on initiative that rests in SW1 is one such weakness, which is impacting how our system is responding to the virus, in turn perpetuating the public health emergency we find ourselves in.

A degree of national direction is clearly needed in the midst of a serious pandemic. People look to the Government for leadership and reassurance. Those in positions of power certainly feel personal responsibility for leading the response. Measures to implement service strategy nationally, such as through the NHS, or to use national heft for international procurement buying power, are certainly necessary. But time after time during the unfolding crisis, the centralised instinct has clouded decision-making, with terrible results.

The structures for the top-down approach to the pandemic were set early on, when the Government chose not to deploy the existing Civil Contingencies Act which set out clear roles, responsibilities and resources for all local and national public bodies. They instead rushed the Coronavirus Act through Parliament, which gave the Executive a greater level of unchecked power and no defined local role. This has had ongoing consequences for the coordination of an effective response. Leaked findings from an internal Whitehall review found that local emergency planning teams believe their abilities have been compromised by a controlling and uncommunicative approach from the central government machine, which persistently withholds data and intelligence.

The centralised response isn’t just structural, at times it has felt deeply instinctive. There has been a repeated preference for big, bold flashy schemes over smaller, sustained but potentially more impactful measures. In the early weeks of the crisis much media attention focussed on the new Nightingale hospitals, yet we are now seeing tragically how that time and resource could have been better invested in the more targeted shielding of hundreds of care homes. When faced with the need to quickly implement testing for Covid-19, the Secretary of State for Health reached for a high-profile 100,000 target and set up new large processing sites. This triumph of tactics over strategy directed the systemic response to focus on numbers over priority need and overlooked existing networks of local lab capacity. Even as attempts are made to set up contact tracing at scale to support the easing of lockdown restrictions, the Government seems to have more confidence in a new mobile app than it does existing local public health teams. This is despite the latter’s expertise in tracing the contacts of people who have highly infectious diseases and clear evidence from countries who have successfully managed their lockdown transition.

The formal power exercised at the centre is in direct contrast to the informal role for local authorities, which is having devastating consequences for their very viability. Because councils’ response has no statutory footing in the context of an emergency, they are left exposed to the whims of a few individuals making decisions in Westminster. At the start of the crisis, the Secretary of State for Local Government told local authorities to spend “whatever it takes” to protect their residents. Councils had immediately set about providing relief to shielded groups, protecting wider vulnerable groups and implementing public safety measures, all while ensuring essential services continued as usual. Rather than support these efforts, Government then rescinded this early clear backing, querying councils’ honesty over their cost assessments and leaving many facing a financial black hole.

The double standards central government imposes on its local counterparts is nowhere more apparent than when it comes to local government finance. An emergency on the scale of a global pandemic has required state-led responses on a scale inconceivable only months ago, and with widespread public approval. Central government spending has snowballed to accommodate unprecedented employee furlough schemes, emergency business support measures, not to mention the enormous costs to the NHS. The Chancellor has the leeway to respond to this through a number of different measures – incurring public debt, raising taxes, freezing public sector wages and reducing public spending, a combination of which he is reportedly considering.

Local government has no such room for fiscal manoeuvre. Councils are legally required to balance annual budgets and have only narrow revenue-raising powers through council tax and business rates which are themselves subject to centrally imposed controls. With a shock to their budgets of this scale they are at the mercy of decisions made by a few in Westminster. These have so far resulted in a couple of ad hoc cash injections of £1.6bn each, and a bit extra cobbled together earmarked for social care and rough sleepers – so far massively short of the estimated £10-13 billon shortfall councils collectively face.

It is no way to run a country. It never was, but in the context of the crisis the contradictions of our top-heavy system of governance are laid bare. The rumblings of discontent from Mayors in the north of England at their regions being side-lined, and from councils over plans to fully reopen schools in the absence of clear local test, track and trace infrastructure, suggest the popular tide is beginning to turn against blanket centrally-imposed measures. As local government is increasingly being seen as better placed to protect their residents, particularly in the context of a Government that is increasingly mis-stepping, there may now be an opportunity for a deeper discussion about how our country should be run in the interests of everyone.

Jessica Studdert is deputy director of the New Local Government Network (NLGN), a Londonbased think-tank. She leads NLGN’s thought leadership and research, and contributes strategic oversight of the organisation. Prior to joining NLGN, Jessica was political adviser to the Labour Group at the LGA. She led policy there, working closely on public service reform and devolution. Previously she worked in policy roles in the voluntary sector for a street homelessness and a childcare charity, and she began her career at the Fabian Society.

The disparities in housing and public health within the BAME community and the pandemic crisis

Cllr. Ketan Sheth

Public Health is important: it prolongs life. A fundamental quality of Public Health is its preventative nature; prevention is far more effective and far less expensive than cure. Public Health is important because we are constantly striving to close the inequality gap between people and encourage equal opportunities for children, all ethnicities and genders. Health is a human right and we should be ensuring no one is disadvantaged, regardless of their background, their ethnicity or where they live. Becoming the voice for people who have no voice is our collective duty. Simply put, our influence on the improvement of someone’s health is a fundamental act of kindness.

Poor housing and living environments cause or contribute to many preventable diseases, such as respiratory, nervous system and cardiovascular diseases and cancer. An unsatisfactory home environment, with air and noise pollution, lack of green spaces, lack of personal space, poor ventilation and mobility options, all pose health risks, and in part have contributed to the spread of Covid-19. The disparities in housing and public health within the BAME community have persisted for decades cannot be doubted, and is underscored by a raft of research over the past six decades as well as highlighted by the recent analysis of the impact of Covid -19. The death rate among British black Africans and British Pakistanis from coronavirus in English hospitals is more than 2.5 times that of the white population.

What are the possible reasons? A third of all working-age Black Africans are employed in key worker roles, much more than the share of the White British population. Additionally Pakistani, Indian and Black African men are respectively 90%, 150% and 310% more likely to work in healthcare than white British men. While cultural practices and genetics have been mooted as possible explanations for the disparities, higher levels of social deprivation, particularly poor housing may be part of the cause, and that some ethnic groups look more likely than others to suffer economically from the lockdown.

Homelessness has grown in BAME communities, from 18% to 36% over the last two decades – double the presence of ethnic minorities in the population. BAME households are also far more likely to live in overcrowded, inadequate or fuel poor housing. What’s more, around a quarter of BAME households live in the oldest pre-1919 built homes. And their homes less often include safety features such as fire alarms, which is striking given the recent Grenfell Tower tragedy. Over-concentration of BAME households in the

neighbourhoods in London, linked to poor housing conditions and lower economic status all ensure negative impacts on health, all of which means lower life expectancy. The roll-out of Universal Credit is having greater effects on the living standards of BAME people since a larger percentage experience poverty, receive benefits and tax credits, and live in large families.
Larger household size also means that ethnic minorities are far more vulnerable to housing displacement because of the Bedroom Tax or subject to financial penalties if they do not move to a smaller home.
These stark facts, sharply bring to our attention the health, social and economic inequalities among our minority ethnic community, all of which are critical to understanding why some ethnic minority groups are bearing the brunt of Covid-19. In this time of reflection, it is not enough to observe; we must think about what more we can do, right now, to reduce the health, housing and economic vulnerabilities that our BAME communities are much more exposed to in these fragile times. Let’s act and prolong life together, as a flourishing community.

Cllr. Ketan Sheth is a Councillor for Tokyngton, Wembley in the London Borough of Brent. Ketan has been a councillor since 2010 and was appointed as Brent Council’s Chair of the Community and Wellbeing Scrutiny Committee in May 2016. Before his current appointment in 2016, he was the Chair of Planning, of Standards, and of the Licensing Committees. Ketan is a lawyer by profession and sits on a number of public bodies, including as the Lead Governor of Central and North West London NHS Foundation Trust.