impact

Impact Measurement Practices of Human Rights Organisations

/ inlogov / Leave a comment

Sahar Khalil

In this post, a recent graduate of INLOGOV’s Master’s of Public Administration degree programme presents findings from their dissertation research. 

The world of impact measurement has expanded greatly over the past 10 years, with many leading NGOs putting in place rigorous impact measurement tools for their organisations. However, these NGOs have mainly been service provision NGOs, many of whom are funded by government grants. Government granting bodies have mandated impact measurement into their grant mechanism, and thus NGOs which receive this funding have to legally report back on the impact that the funding is having. The same cannot be said for human rights organisations, who do not receive any government funding (in order to preserve their independence in investigating all governmental human rights violations). As an operations manager at a leading human rights organisation in the UK, I have seen at first-hand how my organisation and many others in the field have been grappling with the quest of measuring the impact of their work. In this post, I present the findings from my master’s dissertation, conducted at INLOGOV, which looked in more detail at this issue.

Impact Measurement and Human Rights Research

Human rights work stems from the fundamental principle that all humans should have access to basic rights and focuses on protecting and promoting those rights. These principles, which are set out in the 1948 universal declaration of human rights, are backed by numerous international human rights conventions, declaration and resolutions. Human rights projects can create three types of intended positive changes: firstly, policy and legal reforms; secondly, social changes conducive to human rights norms; and lastly, strengthening civil society work. Impact measurement is about assessing the changes introduced by a intervention on policies, communities and the lives of individuals.

Difficulty measuring impact of human rights work

The difficulty with measuring impact of HROs work stems from the complexity of bringing about social or political change. Despite this complexity of change, most monitoring and evaluation toolkits currently used in the NGO sector follow the Linear Theory of Change. This model looks at change as a logical sequence of events, where inputs (funding and resources) lead to specific outputs and outcomes, which ultimately leads to change. This model follows that outcomes can be predicted on the basis of inputs. But change in human rights conditions rarely follow such logical paths. Project implementation teams face limitations in the influence they can exert over the social change process. They can have near total control over project inputs (staffing, funds raised, resources used, and so on) and activities and outputs, but virtually no control over outcomes and impact. No matter how clearly an organisation articulates a pathway to a desired long-term policy goal, it would be virtually impossible to name, predict or explain all the variables that might be at play within that change process. In addition to this, change could be slow and stagnant, while at other times it occurs in sudden leaps and in unpredictable ways.

For these reasons, along with other limitations HROs face, development of impact measurement tools in the sector has been lagging.

Findings from my research

Impact measurement of HROs is under researched, and what little research has been written on this topic is now outdated. I wanted to assess the current practices of human rights organisations in the UK. I spoke to 4 leading experts from major HROs in the UK of varying sizes and funding. The findings are summarised below:

  1. Not all organisations are systematically measuring impact. Some have a clearly developed strategy towards measuring impact; others don’t measure impact in a systematic way but have some limited practices in place. One organisation interviewed still relies on anecdotal evidence of the effectiveness of its work;
  2. Organisations who had systemic impact measurement practices in place stated that having a clear definition of impact that was understood by all employees was a vital and core component of their impact measurement strategy;
  3. Of the organisations that had a systemic impact measurement strategy in place, some are carrying out measurements using in-house staff, whereas others are outsourcing the work to independent private contractors;
  4. Some of these organisations found the Linear Theory of Change useful in helping them confine the scope of impact measurement work, which would otherwise be too broad to assess;
  5. Lastly, while the practices varied, all respondents agreed that there are several benefits to implementing impact measurement practices. Impact measurement has helped them improve the quality and effectiveness of their work, raise more funds, and has improved the transparency and accountability of the organisation in the eyes of the public and stakeholders.

Sahar Khalil is an operations manager at Human Rights Watch and graduate of INLOGOV’s  Masters of Public Administration distance learning degree programme. 

The impact agenda and political agency

/ inlogov / Leave a comment

Matthew Wood

Why should we, as political scientists, ‘bother’ with impact? My answer is that as social actors we cannot avoid ‘impacting’ on society in one way or another, just like any other profession. The question is how we should choose to influence society. As British political scientists our choices are, thankfully, quite broad. Our discipline in this country is eclectic, our research agendas are diverse. Critically, our jobs also allow us significant autonomy to shape our individual identities, practices and relations with the outside world. The crucial thing is how we choose to shape that autonomy. Put in satisfyingly theoretical terms, it’s about how we exercise our political agency. This is why impact appeals to me and why I think it should appeal to others. In this blog I’m going to suggest how we, as researchers, might think about shaping our agency despite some of the problems with the current impact agenda.

The Trouble with Impact

The trouble, as Helen Turton demonstrates, is that there are significant pressures out there that seek to shape our agency as researchers in particular ways, narrow our research agendas and stop us speaking truth to power. Paradoxically, this includes the ‘impact agenda’ itself: everything from the REF’s ‘impact case studies’ to the Lib-Con Coalition’s fetish for behavioural psychology and the suspiciously parochial ‘nudge’. When academics look at this ‘agenda’, they understandably balk about being pushed into a positivistic straight jacket. Though I have yet to come across anyone at conferences who disagrees with the idea of ‘impact’, broadly constituted, many harbour an understandable distaste towards how it is currently being implemented.

So we are faced with a struggle over how to define what we mean by impact – broad or narrow. If we interpret it narrowly, then we risk curtailing our political agency. If we interpret it broadly, then we open up a lot more opportunities. I think we should interpret impact broadly, but the question is how do we do this while still acknowledging there are problems with the existing agenda? There are, I think, two ways we can think about impact, and they involve being reflexive about a few scholarly myths about epistemology and thinking about our communication with the outside world.

Epistemology

Firstly, we should question the myth that ‘positivist’ research, the development of universal laws of politics that can be implemented by policymakers, is somehow the holy grail of ‘impactful’ research. While it is certainly true that policymakers like graphs, stats, and anything that makes a claim to being authoritative, they also prefer research that is easily accessible and straightforwardly communicated, rather than a set of complex regression tables and formulae. The de-funding of political science programmes in the US is evidence enough that a hermetically sealed discipline concerned with establishing causal laws of political life will not wash with a practically focused policy world.

Similarly, we should challenge the myth that post-modern or constructivist research, or theoretically-driven work, is ‘non-impactful’. This clearly isn’t the case. Just take interpretive theory, deliberative theory, postcolonialism, eurocentrism, feminism, etc. These deal with weighty real-world issues of crucial societal importance, and to deny this is to cede ground to those who would cut funding for this kind of work.

The point is we should recognise the value of the full spectrum of political research, and never seek to close down our ontological, epistemological or methodological positions because we think the public will think the findings are less ‘relevant’. The topics we research almost always have connection to relevant societal issues. Often, it’s how we talk about our research to others that matters, and this leads me to the issue of communication.

Lost in Translation?

It’s somewhat of a hackneyed cliché that academics are poor communicators, stuck in an ivory tower talking pretentious gobbledygook. We can easily take umbrage at this, but can also see it as an opportunity for reflecting and improving the language we use. My own doctoral research was on ‘depoliticisation’, a concept with a fairly scholastic lineage, and I find it useful when thinking about communicating my work to do two things.

Firstly, I remind myself that academia has specialist language for a reason. We should cherish our ability to be creative in addressing our research topics, and remember that research starting out in relative theoretical obscurity can become publicly salient. Concepts are also identified with academic career trajectories (if you ‘invent’ a concept then you get a lot of credit for it), and we should never draw an arbitrary barrier at a point where we have ‘enough concepts’ and shut a generation of researchers out from having the potential to be identified with new ideas.

However, as Matt Flinders suggests, the ‘art of translation’ is a good way of thinking about how we can get across the interesting and creative concepts and ideas we come up with to different audiences that speak different languages. So, secondly, I think about how to explain my research in a way that, say, a bus driver or a doctor would understand. This is not me trying to ‘dumb down’ but to reflexively think about how my research would be understood or interpreted by a range of different people. This, I find, is useful not only for getting my research across to the ‘outside world’ on an everyday level, or in blog posts, but also feeding back into my academic work as a fresh perspective. Crucially, if our research is better understood by different audiences, then we have a better chance of getting the insights of our research noticed.

It’s perhaps important to remember that not all the barriers to impact are of our own making. Funding streams, government agendas, etc. are all oriented towards a particular ‘impact’ agenda that is, as Helen and Katie make clear, problematic. The important thing though is that impact is on the agenda, and therefore the potential is there for us as researchers to broaden our capacities for ‘agency’, which we should celebrate and engage with. After all, I began researching Politics because I thought I could change the world for the better by helping our understanding of it. That may be stupidly naïve, but it’s what I keep coming back to.

mattwood2

Matt is a Postdoctoral Research Fellow at the University of Sheffield Department of Politics, and Deputy Director of the Sir Bernard Crick Centre for the Public Understanding of Politics. He is currently researching ‘everyday politics’ and solutions to political disengagement in advanced liberal democracies.

Reclaiming the impact agenda: making impact work for you

/ inlogov / Leave a comment

Helen Louise Turton

When encountering the ‘impact agenda’ the ease of engagement is often dependent upon your discipline and/or the type of research being conducted. Certain forms of research don’t lend themselves to be easily compatible with the requirements of the impact agenda as it is currently defined. Given the pressures placed on academics to bring in external resources and to conduct research that has ‘impact’ academics (especially early career researchers) may then find themselves in a situation where they feel the need to morph their future research in order to meet such expectations. The parameters of the current impact agenda as it currently stands have created a situation where certain forms of research could become marginalised, devalued, and ‘unpopular’ unless we, as academics, exercise our agency and become active in defining impact.

Working in the discipline of International Relations I have experienced first-hand a number of different obstacles and challenges with regards to demonstrating the’ impact’ of my research. For instance;

1) Within IR the issues and topics that we are researching are rarely of local resonance as our research is focused on the ‘international’ level. While I believe that the ‘personal is the international, and the international is the personal’ when it comes to demonstrating the impact of my research on the local level it is difficult, for I cannot claim a causal relation to areas within the UK as demanded by the Research Excellence Framework. For example, if one is researching women’s representation in Rwandan politics it is highly unlikely that such research will “contribute to the economy, society and culture within the UK” (REF, 2012).

2) IR research is often not applicable to certain audiences such as parliamentary committees for example. For instance,  I have researched the sociology of knowledge within the discipline of IR, and the chances of presenting my work to civil servants, or being called to give evidence to a parliamentary select committee, or my being asked to speak to local government forum on this topic are largely improbable.

3) IR theory is a very large subfield within the discipline; however you are part of this prominent research community meaning that your work is theoretically inclined, it is very difficult to demonstrate your impact on a non-academic community. Even though I adopt the view that theory is praxis, theory development and theoretical research is not always evidently applicable to a policy audience and not always accessible to the public.

These obstacles and challenges present us with a danger, that the drive for impact could discourage researchers from focusing on certain issues and theory. The pressures academics are facing to demonstrate their impact could result in scholars trying to make their research fit with the impact agenda by changing the direction of their research. This fear is echoed by Dr Guy Redden, he argues that the narrowly defined criteria for research impact can result in “academics eschew[ing[ worthwhile kinds of work they are good at in order to conform”. The current emphasis on impact, and the way it is becoming locked in to employment, career progression, and grant success could lead to researchers modifying their behaviours and research in order to adapt to the demands of impact. According to Dr Peter Lawrence, the drive for impact has resulted in academics focusing more on their careers and less on understanding and theorising problems.

Do the difficulties and dangers generated mean that we should abandon the impact agenda? Whilst I remain wary and critical of broader changes within academia that the impact agenda is a product of – such as the marketization of knowledge, the corporatization of the university, and the adoption of marker-like behaviours within the academy – I don’t think we should abandon the call for impact per se but we should abandon impact as it is currently defined.

As academics we have a responsibility to share our knowledge and make it readily available and we should be encouraged to publically engage, but the way impact is currently defined is incredibly narrow and tends to orientate primarily around being policy relevant (especially within IR). Rather than making our research fit the impact agenda, we should be making the impact agenda fit with our research. We should be focusing on and thinking about what we want impact to look like. We need to be developing new forums and being more imaginative in the way we approach impact. Crucially, we need to remember that we are not passive in the process, and we should be taking more control over the impact agenda, in order to broaden its definition to prevent excluding certain forms of research. In other words we need to reclaim the impact agenda and define it in our own terms.

turton

Helen Turton is a University Teacher in International Politics and Security Studies at the University of Sheffield. Helen is also the co-convener of the British International Studies Association working group IR as a Social Science. She is currently organising a one-day workshop titled ‘The Impact of IR as a Social Science’ and will be presenting a paper on the relationship between ‘impact’ and IR Theory at the forthcoming International Studies Association’s annual convention in Toronto, March 2014.

Three problems with the impact agenda

/ inlogov / Leave a comment

Katherine Tonkiss

In a recent post for the LSE Impact of Social Sciences Blog, I argued with Catherine Durose that while the idea of delivering policy relevant research is positive, too often our claims to relevance do not deliver genuine impact. A couple of weeks ago I had the opportunity to revisit this issue when I spoke on the PSA Postgraduate Network impact panel, and here I reflect on some of the themes that emerged.

‘Impact’ is a response to the perception that academic work suffers from a relevance gap – that research is not relevant beyond academia and that therefore we should work to address this problem by producing research which has an impact on the wider world. This claim of a relevance gap has been critiqued on the grounds that it may not be true – many disciplines including feminism, and many policy-relevant academic departments (such as INLOGOV) have long had impact beyond their academic niche.

Despite this critique, the imperative to create impactful research is not necessarily something to be avoided. Not all research has to create impact, but it is a good thing that some research does, and maybe more research should do. Impactful research drives a democratisation of knowledge, enabling wider society to challenge and press for change.

There are, however, three key problems with the impact agenda.

1. Privatisation of knowledge

Impact is intended to democratise academic knowledge – to allow it to reach wider audiences. However, as Martin Eve has argued, the need for impact has arisen because of the privatisation of higher education which is privatising knowledge and therefore giving rise to the need for impact. Discussion at the PSA impact panel centred on how, for many, their biggest impact came from teaching but that this form of impact is, in their view, being curtailed. This is not an argument against impact, but rather the narrow way in which having an impact is understood.

2. Quantifying impact

Yesterday, Kate Dommett described in her blog post the various ways in which the impact of academic work is being measured. The problem with this is that, by demanding that impact must be quantifiable and suited to specific measures, we may miss quite a lot of the benefit and value of our research. Again, this is not an argument against impact, but rather what counts as impact. This is particularly evident in how we treat activities such as blogging and engaging with the media. As I argued with Catherine Durose, too often we count hits and retweets as measures of impact, rather than the less quantifiable and more long term dialogue that is opened up by engaging with these media. As such, we are performing claims to relevance rather than genuinely being relevant.

3. Doing research faster

Patrick Dunleavy has recently stated that research should be ‘shorter, better, faster, and free’, and that this should have implications for how research projects are constructed. Dunleavy wants information available to policymakers as quickly as possible and critiques social scientists still ‘trundling on’ doing three year studies and not getting results to policymakers in time for them to be relevant.

Rethinking how we construct large research projects is to be welcomed. However, the claim that this should mean producing research more quickly potentially undermines the quality and rigour of our academic work, as well as the depth that we are able to reach. Rather than thinking about how to shorten projects and deliver results more quickly, a more fruitful avenue may be to think about how to include those affected by the research in that research as it happens. This means co-designing research projects with participants and working with them closely throughout. In an article we wrote for Political Insight, Kate Dommett and I have described how we have taken this approach in the research project that we are currently working on.

Despite the three problems I have described, I do think that there is value in research which creates impact – that is, where it involves participants in the research process and aims to stimulate dialogue beyond the ivory tower. We need to avoid thinking about impact solely as fast research and performances of claims to relevance, but genuinely impactful research can have a positive effect.

tonkiss

Katherine Tonkiss is a Research Fellow in INLOGOV. She has research interests in normative and empirical questions surrounding migration, citizenship and identity, particularly in the UK. Her first book, Migration and Identity in a Post-National World, was published in 2013. Follow Katherine on Twitter @ktonkiss.

Embarking on impact: why do it and what to consider

/ inlogov / Leave a comment

Katharine Dommett

The impact agenda has emerged as a prominent component of academic life. Over the last few years alongside the pressures of writing, teaching and administration scholars have been encouraged (if not expected) to conduct impact and public engagement activities. The origins of this agenda are manifold but derive in the main from the Research Excellence Framework which assesses academic departments on their ‘impact case studies’ and ‘environment statement’ as well as research. In an era of austerity it is being made clear to academics that high quality research is no longer enough, scholars must also demonstrate the public value of their work and the return produced on public money.

This logic has permeated the ethos not simply of evaluation but also of research grant capture. As a statement from the Research Councils UK website says:

‘The impact of a piece of research is a key determinant of its value to a university department given the difference between a 3* impact rating and a 4* impact rating can mean as much as an £80K difference in its annual income. In addition, the impact element has also been fully recognised by each member of the Research Councils UK (RCUK) so the potential impact of research is now a significant factor in assessing whether a funding bid will be successful’

In this context Universities are directing increased resources towards impact activities and training, particularly targeting ‘enthusiastic’ early career academics likely to embrace and pursue publically engaged research. Yet, in spite of these developments few opportunities have been provided for early career academics to engage in debates about the underlying questions of:

  1. Why they should engage with these activities (beyond the need to fulfil REF requirements and gain research grants) and
  2. How they can do so in the most meaningful way possible.

This assertion does not deny the ever increasing online debate about impact (not least on the LSE impact blog) but rather highlights how within departments and specifically PhD training there are few opportunities to explore these issues.

Following this logic a recent roundtable event held by the PSA post-graduate network at the University of Birmingham sought to encourage young PhD scholars to critically consider the impact agenda. The three speakers, Dr Helen Turton, Dr Katherine Tonkiss and Dr Matthew Wood, each have different research backgrounds and perspectives on the impact agenda and provided thoughtful and personal experiences of their early career engagement with impact. Each speaker presented their own account of what they thought impact was, whether they thought it was beneficial and how it could be delivered in practice.

Through critical analysis of this agenda and the sharing of personal experiences each speaker made the audience think about the virtues and vices of the impact agenda, and offered tips to help scholars make conscious choices about whether, why and how to engage with the impact agenda in the future.

In the wake of this conference our three speakers have written up their thoughts and ideas as blog posts aimed at continuing the debate and stimulating ongoing discussion about why to engage and what to consider when dealing with impact. These will be posted here over the course of the next week.

kateDommett

Katharine Dommett is a Research Fellow at the University of Sheffield. She recently convened and chaired the workshop at the University of Birmingham entitled ‘Why Bother with Impact’. She has also written about her experiences of impact with her colleague Dr Katie Tonkiss for PSA’s Political Insight Magazine. Katharine is currently developing training for academics on how to do impact in her role as Deputy Director of the Sir Bernard Crick Centre for the Public Understanding of Politics at the University of Sheffield. For any questions contact [email protected]

The impact factor

/ inlogov / 1 Comment

Katherine Tonkiss

I completed my PhD thesis in early 2012, under the joint supervision of INLOGOV and the Department of Political Science and International Studies (POLSIS). I sat my viva in April of that year and shortly afterwards was interviewed for and gained a three year research fellowship at INLOGOV, working with Professor Chris Skelcher on the Shrinking the State project. A year into the post, this is a good opportunity to reflect on that transition from PhD to post-doc research.

In many ways, working as a Research Fellow on a three year project is similar to PhD study. I am completing a research project in a specified amount of time, with parameters and methods specified in a detailed research proposal. The goal of delivering the thesis in a timely fashion is replaced by the requirement to produce a set of research outcomes with which to wow the funders on completion of the project, and of course the need to publish, publish, publish never really changes.

Yet the transition from PhD to post-doc has also presented some new opportunities and challenges. One aspect is going from sole-authored thesis-writing to team working with colleagues both on the research project and beyond. While in reality I do have quite a lot of autonomy in my current role and the majority of my time is spent working alone on specific areas of the project, this project involves working with a team of five researchers split across three Universities. This kind of collaboration is now very common in academic research – and while it is step change from PhD research, I was quite pleased to enter a more collaborative environment.

Another challenge of the transition is balancing research priorities. I was told on numerous occasions during my PhD studies that researching and writing a thesis is something to really enjoy, because I would never again have the opportunity to immerse myself in my own research to such an extent. In reality, I have been really fortunate to be provided the time to turn my PhD thesis into a book, as well as having some time to pursue my other research interests separate to the project. However, pre-planning was key in achieving this – I had thought about a book contract before my viva was completed, and the three articles I have had published were all accepted before I started working as a Research Fellow. I also don’t think I would be so fortunate if I had the hefty teaching responsibilities which come with many junior academic roles.

The biggest difference between PhD and post-doc research is, though, the emphasis that is placed on ‘impact’ in the latter – that is, producing research that both advances academic knowledge and has a positive economic and societal impact. This impact agenda, while offering rewarding opportunities to engage in debates beyond academia, was challenging in terms of the transition from PhD to post-doc research – and particularly because my PhD was strongly theoretical.

For example, working on this research project has meant developing and maintaining relationships with partners in government and elsewhere, producing research outputs that are useful to them in their work in addition to being relevant to the overall objectives of the project. My colleagues at INLOGOV are very used to this way of working, but it really was a new experience for me after the PhD. Within a couple of months of starting my job, I found myself presenting to senior civil servants in Whitehall. Experiences such as this have allowed me to engage in discussion and debate far beyond the ivory tower, but they are a far cry from the late-night loneliness of thesis writing that I had become accustomed to.

Impact isn’t just about engaging with policy-makers, though. Throughout my PhD, I felt that I was researching and writing about something that I was really passionate about – which, I hope, is a common experience. This isn’t something that’s gone away now that research has become my day job – and indeed exploring intersections between normative ideas and lived experiences has become one of my core research interests. As much as there is a case for producing policy-relevant research in order to work with partners in government, it should also be recognised that research – in whatever discipline – is a tool with which to challenge dominant ideas and assumptions, and the ability to engage in the policy process and to communicate with different audiences in order to enact change is just as – if not more – important.

With the impact agenda now spanning all areas of academic research, it may be argued that it is more important than ever for PhD students to equip themselves with the skills to develop this kind of dual-facing work. I’m not decided on that – I think there can be excellent research without wider impact and these contributions are to be equally valued. However, the opportunity to see your research have an impact beyond academia can be very rewarding, and to that end developing those skills is really worthwhile.

tonkiss

Katherine Tonkiss completed her PhD in 2012, and is now a Research Fellow at INLOGOV. Her first book, Migration and Identity in a Post-national World, will be published later this year with Palgrave Macmillan. Her research interests are focused on the changing nature of migration, citizenship and democracy in a globalising world, the local experience of such transformations, and the intersection between normative and empirical research. Follow her Twitter feed here.