The Significance of Evaluation

Based on the approach that appears to be a best fit for your new initiative, what is an appropriate evaluation plan? What is the purpose of the evaluation plan? Support your choices with literature.

Evaulation( Title of book: A-Z of INteragency working, Author: Jon Glasby and Helen Dickinson, Copyright Palgrave Macmillan 2014, pg 45-47

At its most simple, evaluation is the ‘process of determining the merit, worth or value of something, or the produce of that process’ (Scriven, 1991, p. 139). Evaluation then is something we all do on a daily basis in going about our everyday lives. This is a fairly broad term and so in the context of social sciences has been described as a family of research methods that involves the ‘systematic application of social research procedures in assessing the conceptualization and design, implementation, and utility of social intervention programs. In other words, evaluation research involves the use of social research methodologies to judge and improve the planning, monitoring, effectiveness and efficiency of health, education, welfare and other human service programs’ (Rossi and Freeman, 1985, p. 19).

In the context of inter-agency working, evaluation therefore involves selecting a particular investigative approach that examines some part (or parts) of the processes and/or outcomes of collaborative working and making some judgements as to the worth or value of this. Within the wider literature, there are a number of different evaluations of inter-agency working (see Dickinson, 2008 for a summary). However, these often tend to focus much more on the processes of inter-agency working than on the outcomes. This means that evaluators have a tendency to look at how partners are working together and what the major barriers and facilitators of this are – rather than whether this makes any difference to the kinds of services that are provided or indeed the service user outcomes this might produce.

In thinking about evaluation, it is helpful to distinguish between inputs, outputs and outcomes. The first of these – inputs – are the resources (be that human, material or financial) that are used to carry out activities and produce or accomplish results. Outputs refer to the effects of a process (such as a service) on an administrative structure (Axford and Berry, 2005). They are the direct products or services that stem from the activities of initiatives and are delivered to a specific group or population. Outcomes are the ‘impact, effect or consequences of help received’ (Nicholas et al., 2003, p. 2). That is, outcomes are not just the direct products or services, but are the totality of the consequences of the actions of an organization, policy, programme or initiative. In other words outcomes are the ‘impact on society of a particular public sector activity’ (Smith, 1996, p. 1).

Traditionally health and social care tended to be performance managed on the basis of outputs rather than outcomes. So they would often be more concerned with, for example, how many people have accessed a particular service and not necessarily the difference that this intervention made or the quality of that intervention. However, with the influence of New Public Management (NPM) and pressures from service users calling for better quality of services in recent years we have seen much more of a call to measure services by outcome rather than by output. In the case of inter-agency working, there is little evidence that shows a clear link between collaborative working and improved service user outcomes (Dickinson, 2008). However, it is important to note that there are limited numbers of outcome evaluations of inter-agency working and being able to effectively evaluate collaborative working is very difficult for a range of reasons (e.g. timescales, attributing change to joint working initiatives etc). If you are conducting an outcome evaluation, one of the issues that needs to be clear from the outset is just what outcomes you are trying to deliver. Otherwise it can be very difficult to come to valid conclusions about the degree to which these have been achieved.

In addition to distinguishing between inputs, outputs and outcomes, the evaluation literature has also seen a growing trend towards theory-led approaches to evaluating public services. Theory-led approaches such as realistic evaluation (Pawson and Tilley, 1997) and theories of change (Connell et al., 1995) are often seen in contrast to method-led approaches. Method-led approaches aim at refining the use of particular approaches (methods) with the aim that this should reveal insight into particular initiatives. Such a perspective suggests that if we use the right method to investigate an issue or policy, then we should be able to get to the truth. Theory-led approaches instead aim at mapping out the series of assumptions (or programme theories) underpinning a particular imitative and then to test different aspects of this. So, for example, in the context of inter-agency working, a theory-based approach would map out what is happening in terms of efforts to work jointly and what impacts this should produce on the partner agencies, services and outcomes. Once this has been mapped out, then different aspects of this would be tested using a range of approaches.

Recommendations for policy and practice

• Inter-agency working can be very difficult to evaluate in practice and careful thought needs to be given to this process.

• There are many different possible ways to evaluate inter-agency working and so you need to be clear about what you are trying to evaluate and why to ensure that you select the most appropriate approach.

• Whichever evaluative approach is adopted, trade-offs will need to be made and these need to be acknowledged.

• It is crucial that attempts at inter-agency working are clear what outcomes they are trying to deliver if they are to be evaluated in terms of their outcomes.

• Theory-led approaches offer a lot of potential in terms of evaluating inter-agency working.

KEY TEXTS