Month: February 2014

An Introduction to Evaluation Part I: The Formative/Summative Dualism

Scriven’s (1980) formative and summative dichotomy is arguably the most accepted and renowned evaluation typology. One could suppose that the formative/summative dichotomy is therefore fundamental to understanding evaluation. In this post, the concepts of formative and summative evaluation will be explored.

The label, formative evaluation, refers to an evaluation approach that focuses on evaluation improving a programme, essentially allowing an evaluator an interactive (rather than independent) role (Herman et al, 1987). Reporting occurs throughout an evaluation (rather than at the end or at a certain point) meaning closer and sustained contact with a programme (Clarke, 1999). Summative evaluation focuses on formally reporting findings at a certain point in time.

Prescott et al (2002) emphasise that the two approaches are separate and distinct processes. According to Patton (1996), the formative/summative dichotomy, ‘captures the entire array of evaluation purposes’, as it suggests that, ‘anything that is not formative is summative’.

There is much criticism of the summative/formative dichotomy in modern evaluation: ‘the world of evaluation has grown larger than the boundaries of formative and summative evaluation’ (Patton, 1996). Debate exists over which of these approaches is most vital, Cronbach (1966) suggests that formative evaluation is more important that summative evaluation, whereas Scriven (1967) notes strengths of the summative approach. Patton (1996) also suggests that, ‘formative evaluation rests in the shadow of summative’.

At the simplest level, the dichotomy offers a labelling system to categorise evaluation. Indeed, various labels are applied to evaluation (Scriven’s dichotomy included).  McKie (2003) finds that evaluation labels and typologies are excluding to those who are not familiar with the terms (for example stakeholders within the programme being evaluated). Further, Ussher and Earl (2010) suggest that the terms summative and formative can be confusing, however, they do note some value in applying labels: ‘the identification, definition and consistent use of specific labels are useful for developing understanding and communicating with others’.

Where evaluation is dual-level, that is, an evaluation is required at both national and local levels (Allen and Black, 2006), both summative and formative types could still be utilised. A formative evaluation approach could be applied to local-level evaluation with the increased likelihood of the evaluation being used to generate improvements (according to Herman et al, 1987). A summative evaluation type might occur at a national-level evaluation with the likely audience being policymakers, funders and the public according to Herman et al (1987).

This post has explored formative and summative evaluation (in the context of public programmes). I will save some of the more specific debates of evaluation’s purpose , for instance, of evaluator independence (should an evaluator be assisting a programme to improve through formative evaluation? etc.) for another post.

Part II of the Introduction to Evaluation series (available soon) will consider the history of evaluation.

‘It ain’t what you do, it’s the way that you do it’

Tried to drive change but still feel that things aren’t working? 

 All too often initiatives are considered to have not worked, with little regard for whether the right steps were taken to implement them.

Problem: Believing in a single solution

So you anticipated a problem, you planned for it and still something went wrong? It does happen and will happen. But whatever you do don’t accept one solution as your only solution.

Action: 

1) Take a multi-pronged approach from the start. ‘I will do this, but in case that fails I will also …’. Try also to vary your method – rather than ‘if I email and don’t get a reply, I will email again’, try ‘if I email (written, distant) and don’t get a reply, I will seek face-to-face contact/phone (verbal, personal)’. 

2) Knowing your stakeholders will also help to support this: ‘I know this group are likely to… so I will plan for this by …’ 

3) Learn from your mistakes and ask yourself how you could have approached the problem differently – ‘I could have also ensured that…’. Hindsight is a wonderful thing especially if you learn from it. 

Problem: Assuming others understand

This is particularly the case for internal customers. Often I hear terms like ‘they should know this’, or ‘we’ve told them before’.  Remember though, organisations are messy creatures and stakeholders hear lots of different messages from lots of people, often about the very same thing.

Action: 

1) Consider the issues your stakeholders may have had in the communication of your initiative and (if necessary) engage with them to do this. Plan for this and ensure that details that stakeholders may have heard conflicting messages about are confirmed with them.

2) Consider how small politics, anxiety, internal competition, and culture might have affected the way in which stakeholders perceived your initiative/or the communication of it. Understanding the whole picture might prove highly valuable.

Problem: Impatience

You’ve planned, you’ve done it – you just want it to work. But is impatience stopping your initiative being at its best?

Action: 

1) Treat the initiative as an experiment (be curious) and acknowledge that good things take time and a great deal of adjustment.

Problem: Arrogance

People didn’t get it the first time round? ‘Well they’ll see they have to do it eventually because it is the only option’. Your initiative may well be superior, your department too, but bearing this in mind might not help you to onboard those that you need, and certainly not in a timely way. Don’t get in the mindset that you tried with a great approach and it failed and so it is now a lost cause.

Action: 

If at first you don’t succeed try, and try again (perhaps with some adjustments…). Consider the failings and rather than looking to blame others consider how you could have performed better. Put yourself into the others’ shoes.