Cense

Innovation to Impact

Evaluation for Change

by

Change is everywhere it seems and while it can be said it is the only constant what we are seeing is an increase of change on a massive scale.

However, as the protesters across the United States, Canada and beyond are making clear: there is a big difference between talk of change, the process of change, and the outcome of change efforts. Evaluation can be a powerful tool to help us distinguish these things together as they can be conflated too easily.

Here are three things to consider when seeking to make these distinctions that can be applied anytime, but become more salient when focused on large-scale change efforts where much is happening simultaneously.

Document your baseline

A baseline is a starting point and while it would be great to have data from yesterday, if we are seeking to gather change-related data today that means this is your baseline. Too often baselines are forgotten because any effort to measure or track change needs to answer the question: change in relation to what?

How? Pick the most convenient, proximate moment to gather data. Aim to capture descriptive data of what is happening, time data (see below), and also any numerical aspects of the phenomenon you can. These can be such things as cases of something, number of participants involved, descriptions of the current situation. From this, you can later build a backstory that can help lead to the present moment.

For example, George Floyd was arrested and killed by a police officer on May 25, 2020. It is possible to use that as a baseline for what came next and later build the backstory by showing the many different incidents of a similar nature that may have happened locally, nationally, and beyond to illustrate historical patterns of things like police behaviour, protests, violence, racism or otherwise depending on what changes one seeks to make.

Gather real-time data whenever possible

It’s tempting to gather data after an event (e.g., protest, policy decision, etc.) has taken place (and sometimes that’s unavoidable), however, there is much evidence that we lose perspective and critical information in our post-event reflections that often fail to capture critical details of what actually happens.

How? During the COVID-19 pandemic we have seen many examples of this with live reports from doctors, nurses, and other caregivers working the front-lines of healthcare responses. We’ve seen infectious disease specialists giving interviews on television, exchanging data and opinion via email and Twitter, and through first-hand accounts of citizens dealing with the various policy decisions made. These micro-narratives can make for a strong experiential case for what is happening and what effects the event is having. Reviewing social media posts, proposing online diaries (e.g., selfie video testimonials) or using ‘speakers corner‘ sites or physical booths to allow people to document what they feel, think, say, and do in real-time will provide a more accurate and adaptive means of understanding what is happening as it happens, rather than just retrospectively.

Timestamp your data

Time is a critical contextual factor that can help us understand what happens, why it happens when it does, and to better make sense of the outcomes. The Greeks classified two types of time: Chronos (‘clock time’) and Kairos (‘relative’ time) . Determining what time (as in hour, date etc..) can help you to organize things in chronological order and see relationships between change-making efforts. Relative time — proximity — helps us see the effect of certain activities in relation to others.

How? Modern recording tools often have this built into them, but for the evaluator it is important to record when things happen and document the sequencing of things. Big events like the two we’ve used — the race riots and pandemic — have so many moving parts that it quickly gets difficult to remember retrospectively what happened in what order. This is critical if we want to develop a theory of change or explain what happened as part of the change process.

These three things are all simple and can be done with tools like phone cameras and gathering things in a spreadsheet. More sophisticated ways are available as well and, ideally, there is a method and plan prior to a change initiative taking place. But as we’ve seen, sometimes change just happens. If it does, you’ll be ready to capture it and learn from it before it comes to pass and be able to tell if it doesn’t.

Stay and be safe.

If change is something you need help understanding and documenting, don’t hesitate to reach out and contact us. Evaluating, supporting, and guiding change efforts is what we do.

Photo by mana5280 on Unsplash

Evaluation’s Hidden Value: Clarifying Your Strategy

by

Evaluation is more than just the assessment of merit, worth, and significance of a program, product, or service. Although these are the primary functions of evaluation, it can serve far more than this when designed and implemented in the appropriate manner.

Evaluation focuses attention on the things that your organization builds, has built, and implements and the purpose and role(s) they serve. In doing so, evaluation serves as a primary support tool for strategy. In order for an evaluation to do its job, it must be aligned with the purpose of the thing it is evaluating.

For example, if an evaluation is being asked to assess behaviour change, the focus of the program or stimuli (e.g., app, message, etc..) for that change is a core part of the strategy. By matching the program components with the activities and intended outcomes, evaluators can help determine the degree of alignment and assess (and advise on) the likelihood of success before any data is collected. It is through the data — the evaluation itself — that this alignment and the assumptions behind it are tested.

Theory into practice

The process of aligning the planned activities with the outcomes is part of what a Theory of Change is designed to do. To illustrate, consider a program designed to promote some kind of behaviour change (e.g. engaging in a new activity, doing more or less of something, etc..). For a change to take place, there needs to be a certain logic in how the intervention (i.e., the activity or service) is set up based on some established design principles.

We also know from the wealth of knowledge of behavioural science, that there are many different ways to promote change and that some are more efficacious than others. A skilled evaluator can help determine whether the approach being used is likely to produce the desired change by building out a Theory of Change and then matching that with the appropriate outcomes.

This is what the heart of a strategy is all about: aligning the resources, intentions, and actions together to produce an outcome.

Like the image above, evaluation helps peel back the top coat to reveal what is happening underneath. It helps take the mystery out of why programs work or don’t work and what the reasons are why people do things (or don’t).

It’s one of the reasons why we bring strategy, evaluation, and design work together. By connecting them, you get far more performance out of your program than if you just keep the cover on.

To learn more, contact us and we’ll be happy to show you what’s under your hood.

Image credit: Nathan Van Egmond

Engagement for ideas

by

Engage everyone in idea creation

Among the biggest success factors in any organizational change initiative is engaging staff and stakeholders in participating in the change process; this is true for idea generation through the design cycle to implementation and evaluation. How do we overcome the challenge of disengagement to produce productive creative, innovative ideas in our organizations?

Beyond brainstorming

Ideas are usually the starting point for any change initiative. Ideas produce the raw content of problem identification and the seeds for solution generation. Brainstorming is often the means people first think of as a way to generate ideas and explore concepts. However, there are some substantial problems with this approach and some ways around it.

Brainstorming has been widely criticized for good reasons. Among them: it favours those voices who think (and speak) quickly, speak early, often and loudly. Those early suggestions drive the conversation for what comes next, creating a path dependency that’s hard to escape once initiated. If you’re a quieter, perhaps more contemplative person, you’ll find that you are either late to the conversation or not included. From Jungian Personality Theory, this indicates a bias towards extroversion over introversion, which excludes about 40 per cent of the population according to some estimates.

This approach also favours what Min Basadur would classify as an ‘generator’: someone who’s work style preferences favour idea generation. Those from the other quadrants in the Basadur Profile, particularly ‘conceptualizers’ and ‘optimizers’ (ones who’s work style preference leans toward processing and organizing information) are less likely to respond quickly.

Ideas in private and public

Another problem with many ideation strategies is that many really useful ideas are a bit heretical or outlandish. While design thinking writers often refer to the need to generate ‘wild ideas’ without criticism or judgement, the truth is that’s a lot harder to do in practice, particularly when there are power dynamics in the room, reputations, and the real fear that comes with change and challenges to established practices. The public nature of the ideation process favours transparency, but induces self-censorship and produces disengagement for many and hyper-engagement for a few.

A solution is to have those involved in the process generate ideas independently and anonymously contribute them via a suggestion box (digital or analog). Keep the suggestion box open for a defined period — we recommend no more than one week or as short as three days. This allows time for those who are more reflective to mull through their ideas and contribute them, while those who are more quick to generate ideas will not be affected.

These are then collated independently by someone neutral to the problem and solution set and organized thematically. This has the advantage of potentially embedding some of the wildest ideas within a small set of other ideas that might seem far less threatening. If you create a category of ‘wild ideas’ the risk is that the entire category will be dismissed. A more public, but elaborate, means of doing this can be found in the CoNEKTR Model described elsewhere.

This process can be repeated over the design cycle at different stages – anywhere diverse perspectives and feedback is needed.

Overcoming biases to better ideas

The bias toward ‘rapid ideation’ in design thinking systematically excludes people in favour of a well-meaning intention of trying to avoid participants ‘over-thinking’ a problem. While that might happen, it still prevents many from engaging in the process fully because of personality, work preferences, cognitive style and social pressures.

The strategy listed above is a simple, but highly effective means of getting lots of ideas and engaging your entire team in the process. Try it out. You might be surprised what ideas come from it all.

Unpleasant Design

by

Fall_Bench_Snapseed

Much of what we speak of when we talk about design, products and space is making things more livable, attractive or delightful. But what happens when we want the exact opposite to happen? That is the concept behind Unpleasant Design. The term comes from work being done by Gordan Savičić and Selena Savić who have been curating examples of the way governments and business alike have sought to discourage behaviour rather than encourage it through design choices.

The team at 99% Invisible recently profiled this work drawing on examples of how design has been used to discourage things like homeless people sleeping on benches, kids loitering around a shop, or ‘unintended uses’ of public washrooms.

The hallmark object to define this way of designing for non- or limited use is the Camden Bench, named after the London-based council that commissioned the original. You might have seen these in places you’ve travelled. They are these weirdly angular, almost always uncomfortable, and barely functional slabs that serve as benches. But unlike the one pictured above, they are pretty much impossible to use as a bed if you don’t have one, which is exactly its point.

Unpleasant design represents an illustration of design’s power for multiple purposes and not always good. While these designs address certain problems that are posed by behaviours that some find undesirable, they speak to a larger role that design can play in social life. Another example of unpleasant design is not about what’s designed into something (such as spikes), but also what’s designed out of something. One example is the presence of ‘leaning benches’ to replace places to sit. Another is the absence of water fountains in public spaces. The latter is less about providing hydration to people, but more about getting them to buy water rather than take it for free.

Design is as much about what is invisible as it is visible, which is why the brilliant podcast 99% Invisible is such an appropriate name for a show dedicated to looking the often un-noticed ways we shape the world and how it shapes us.

Photo credit: Fall by Clive Powell under Creative Commons License and adapted for use.