Cense Ltd.

Inspiration, Innovation and Impact

Evaluation for Change

by

Change is everywhere it seems and while it can be said it is the only constant what we are seeing is an increase of change on a massive scale.

However, as the protesters across the United States, Canada and beyond are making clear: there is a big difference between talk of change, the process of change, and the outcome of change efforts. Evaluation can be a powerful tool to help us distinguish these things together as they can be conflated too easily.

Here are three things to consider when seeking to make these distinctions that can be applied anytime, but become more salient when focused on large-scale change efforts where much is happening simultaneously.

Document your baseline

A baseline is a starting point and while it would be great to have data from yesterday, if we are seeking to gather change-related data today that means this is your baseline. Too often baselines are forgotten because any effort to measure or track change needs to answer the question: change in relation to what?

How? Pick the most convenient, proximate moment to gather data. Aim to capture descriptive data of what is happening, time data (see below), and also any numerical aspects of the phenomenon you can. These can be such things as cases of something, number of participants involved, descriptions of the current situation. From this, you can later build a backstory that can help lead to the present moment.

For example, George Floyd was arrested and killed by a police officer on May 25, 2020. It is possible to use that as a baseline for what came next and later build the backstory by showing the many different incidents of a similar nature that may have happened locally, nationally, and beyond to illustrate historical patterns of things like police behaviour, protests, violence, racism or otherwise depending on what changes one seeks to make.

Gather real-time data whenever possible

It’s tempting to gather data after an event (e.g., protest, policy decision, etc.) has taken place (and sometimes that’s unavoidable), however, there is much evidence that we lose perspective and critical information in our post-event reflections that often fail to capture critical details of what actually happens. Aim to get data whenever you can as early as you can.

How? During the COVID-19 pandemic, we have seen many examples of this with live reports from doctors, nurses, and other caregivers working the front-lines of healthcare responses. We’ve seen infectious disease specialists giving interviews on television, exchanging data and opinions via email and Twitter, and through first-hand accounts of citizens dealing with the various policy decisions made. These micro-narratives can make for a strong experiential case for what is happening and what effects the event is having. Reviewing social media posts, proposing online diaries (e.g., selfie video testimonials) or using ‘speakers corner‘ sites or physical booths to allow people to document what they feel, think, say, and do in real-time will provide a more accurate and adaptive means of understanding what is happening as it happens, rather than just retrospectively.

Timestamp your data

Time is a critical contextual factor that can help us understand what happens, why it happens when it does, and to better make sense of the outcomes. The Greeks classified two types of time: Chronos (‘clock time’) and Kairos (‘relative’ time). Determining what time (as in an hour, date etc..) can help you to organize things in chronological order and see relationships between change-making efforts. Relative time — proximity — helps us see the effect of certain activities in relation to others.

How? Modern recording tools often have this built into them, but for the evaluator it is important to record when things happen and document the sequencing of things. Big events like the two we’ve used — the race riots and pandemic — have so many moving parts that it quickly gets difficult to remember retrospectively what happened in what order. This is critical if we want to develop a theory of change or explain what happened as part of the change process.

These three things are all simple and can be done with tools like phone cameras and gathering things in a spreadsheet. More sophisticated ways are available as well and, ideally, there is a method and plan prior to a change initiative taking place. But as we’ve seen, sometimes change just happens. If it does, you’ll be ready to capture it and learn from it before it comes to pass and be able to tell if it doesn’t.

Stay and be safe.

If change is something you need help understanding and documenting, don’t hesitate to reach out and contact us. Evaluating, supporting, and guiding change efforts is what we do.

Photo by mana5280 on Unsplash

Innovation: Why Starting Points Matter

by

If you’re looking to measure some form of progress or impact connected to your innovation (a product, service, or policy) then paying attention to the starting point is critical.

Evaluators call this a baseline and it’s maybe the most important line you can draw. A baseline is really the point of comparison for all you do. When speaking about improvements or change, this is the point you refer to when making those claims.

For something so important, it’s remarkable how few organizations capture baselines well. Let’s look at what it means and how you can do a better job of determining your innovation’s baseline.

Setting a baseline

An ideal baseline is set as far back from the present as possible at the start of your innovation journey. However, as many journeys have starts, stops, and tangents it might be that the start of the innovation journey actually ‘begins’ mid-way through a timeline.

If you are already started your innovation journey, the best time to set a baseline is now. It’s possible in some cases to use retrospective data (looking backward) to assess a baseline, however that can be fraught with certain biases that are unhelpful. If looking retrospectively, consider neutral data points like dates and times, concrete descriptions of product work, and use verifiable sources of data (e.g., work activities, prototypes, expenditures) to support that work.

When setting a baseline, there are some other tips we advise to enable you to capture the most possible useful data you can. If you are innovating in a human system, it’s possible that the innovation may have many effects that go beyond the most obvious so collecting the right data to capture these effects at the beginning is key.

  • List out the resources that have been assembled to develop the innovation such as people, space, and other capital (e.g. funds). These are your starting inputs into the project.
  • Gather a project plan or schedule of activities early to help determine what happens after the project begins. This will help determine where deviations from the plan take place, when, and help you trace back what happens if or when those changes take place to the strategy. Capturing deviations is critical because it helps you go back to see what adaptations you make at the end. Without this data, these activities might appear to be random or haphazard.
  • Capture cultural/environmental factors. Using the STEEP-V (Social, Technological, Economic, Environmental, Political, and Values) model is helpful in knowing what to pay attention to. One of our clients experienced a major, unexpected removal of funding due to rapidly changing political priorities of a government that was supporting their work. By capturing these broader situational variables you can place your innovation work in a context.
  • Document the state of your organization’s readiness and preparedness, which may also include an assessment of innovation readiness. Many innovations fail not out in the market, but within the design studio. Changes to organizational priorities, resources, and personnel can scupper, delay, or change the plans for an innovation. Capturing the state of the organization is an important point as it will allow you to see where things go off track or where they are enabled because of the organization.
  • Develop a project charter and theory of change. While a project may change direction many times, a baseline assessment can help you reflect the desired outcomes and original purpose of the innovation — which are quite likely to change over time. Having this in place can help explain what changes take place and what adaptations take place.

Baselines are the key point for making any claims of change, improvement, or transformation. They are the point where we say “in relation to what?” when speaking about change.

Give yourself some time and use the baseline assessment as a chance to spur reflective and strategic planning about your innovation. You will be grateful you did and amazed at the results later on.

If you’re interested in learning more about baseline development and its role in supporting innovation evaluation, contact us and we’ll gladly help.

Photo by Suad Kamardeen on Unsplash

Expanding Perception Through Evaluation

by

Creating an Innovation Panopticon?

One of Evaluation’s greatest contributions to innovation is its ability to expand the vision of the innovator and attend to matters of perception. What we look at it is not always all we consciously see, nor is what we see the sum of all we perceive. Humans have many conscious and unconscious biases in what they perceive and how they interpret what they perceive. Evaluation can help add clarity to that perception and expand it. 

Evaluation can help distinguish reality from illusion. In this latest in our series on Evaluation: The Innovator’s Secret Advantage we look at the secret of perception and how evaluation can help change what is seen (and reveal what is not) in the journey of innovation. 

Seeing More

The image above is of a jail modeled on the Panopticon, an ancient architectural design to allow someone to see all that is going on at any one time. While its use within prisons is not one that’s particularly comforting or attractive, the model or concept of creating a design that enables a vantage point to see what is happening within an innovation context is a useful metaphor. 

Evaluation can enable innovators to see what is happening from a single vantage point at considerable depth or, as we will discuss, provide an alternative means by seeing the phenomenon from multiple perspectives. Both of these have advantages for innovators looking to assess the performance of a product or service in areas of high complexity and uncertainty. 

It is by reflecting on the vantage point (perspective) that we can determine whether what we are seeing is real or just an illusion and take the appropriate action based on that assessment. 

Seeing Differently

Consider one of the most famous of these visual illusions: the Young Girl / Old Woman Illusion (below). The history of this illusion has been traced back to the late 19th century and has been replicated many times since then to be considered a staple of any introductory psychology text. 

Young Girl/Old Woman Illusion

What’s interesting about this illusion is that you can’t hold both the image of the young girl and the old woman in your mind’s eye at the same time. You can only see one, not both even if you can switch back and forth. The same situation can occur within an innovation where it is difficult to see two ideas at the same time. By asking the right questions, an evaluation can help innovators to see what others see, what they default to, and whether or under what conditions do they see something different (e.g., seeing something as a product or a service). 

Other illusions are more about perceptual shifts that, once made, are difficult or impossible to ‘unlearn’. These can be found in such things as the FedEx logo and its embedded arrow (check the link if you’ve never seen it). Another is the often common hidden ‘8’ within the 8 of diamonds in a deck of cards (see below). 

Eight of diamonds playing card, isolated on white background.

Evaluation for innovation involves asking questions about a program, product, or service that includes taking a perceptual view of many different people and from different situations. It’s a systems-oriented perspective that considers what a particular phenomenon looks like from a particular point of view. 

Seeing What’s Hidden in Plain Sight

Another famous psychological illusion is the one conceived of by Daniel Simons and his colleagues called The Monkey Business Illusion. This selective perception task is designed to show how your attention and focus can blind you to other things going on. The video below provides an updated version of the original video (available via the link above) to illustrate how we can miss things that are right in front of us based on what we attend to. 

The Monkey Business Illusion

Innovators are often focused on the core aspects of their innovation: the product, the process, and the intended outcomes. This focus is often what sets successful innovators apart from others, yet it can also be a liability.  An evaluator can help an innovator (an individual, a team, an entire organization) to see the full picture. 

How? The means to do this is first achieved by asking evaluative questions that look at what is going on, what is new, and what meaning is derived from various activities. 

Taking the Temperature (and Other Innovation Measures)

There is a useful, if not morbid, fable about the frog in the boiling pot. In that fable, the frog fails to notice that the temperature of the water continues to rise because the change is so gradual. The frog ends up boiling to death because of this inability to see that he is being boiled alive because the changes are so hard to detect. A good evaluation designed for innovation provides means to collect data akin to a thermostat calibrated to the particular conditions, situations, and product or service necessary. 

This could include monitoring key inputs and outputs, tracking sales or engagements over time, or looking at resource levels like personnel and how they respond to change. The latter example is a good one, particularly with larger organizations where the duties assigned to a single person could be absorbed by others if that individual left the role. However, as time moves on and the extra work is normalized it is possible that the pattern is repeated over time where those who are left eventually shoulder such a burden that it makes the work impossible, but also the performance degrades. 

These are the kinds of situations where collapse is likely. We’ve seen this with leadership when many people start leaving at the top or when those at the front face of an organize leave in large numbers. The cost in capital, focus, lived experience, and working knowledge can cripple an organization over the long haul. But like the frog in the pot, it might not be until just before the boil that it is noticed. 

Evaluating for Illusion & Reality

Evaluation for innovation recognizes that the attention and focus of those leading the charge is precious and that leaders are likely to miss key things without the right tools to draw their attention to the whole system and the parts. Evaluation can be hired to cover some of these perceptual gaps and illuminate things like the present temperature or noticing gorillas, patterns in data like the 8 of diamonds or discerning the best moment to see a young girl or an old woman. 

In the next piece in this series, we’ll look at how evaluation can uncover layers of impact that go beyond seeing what is in front of us to looking far past it. If evaluation for innovation is something that you need help with, connect with us; we can help you see things differently. 

Title image credit: Cameron Norman