Toolkit

Uncovering Layers of an Innovation’s Impact

2018-12-11 by cense

Innovators — those seeking to take an idea for a product, service, or policy and make it real — usually have a pretty clear sense of what they are trying to achieve with their innovation. This is the primary purpose and may reflect where the innovation achieves the greatest impact. But is this all it does? Could it be doing much more?

Evaluation can play a key role in revealing where an innovation achieves more than just its primary purpose and can serve as a means to uncover layers of impact that can demonstrate various returns on investment (ROI) and open up opportunities for further exploration and exploitation of resources.

Innovation typically involves considerable investment in time, energy, money, and attention and an evaluation can help showcase the return on investment in unexpected ways. Let’s consider something like an event — a learning conference — as an example to illustrate this layering of impact and how evaluation can aid in revealing these layers and supporting innovation.

Change or Impact for Whom?

A look across many conferences finds relatively consistent language in their purpose that includes mission phrases like:

to inspire…, to educate…., to connect…, to showcase…, etc..

Among the first roles an evaluation can play is articulating the theory of change explaining why something is intended to achieve an outcome. A look at the phrases above is likely to prompt questions in an evaluator like: for whom are these to happen? Is the effect expected to be similar for most people? What are the means that people are using to showcase products (e.g., knowledge) and for the audience to engage with content?

These are some of the questions that an evaluator might ask from these initial goals. For an evaluation of a learning event, this might translate into metrics like:

Attendance including details on those in the audience (e.g., professional background, previous participation, basic demographics)
Number of sessions taught and description of those sessions (content)
Overall satisfaction with the conference (including session content, speakers, food and drink, format)
Self-reported learning outcomes from participants
Financial details: Profit, loss, and expenses; sponsorships
Registration information (e.g., online vs. in-person, timing, categories, etc.)

This is a pretty standard set of metrics. We see similar evaluative outcomes across educational programming in different contexts. These might work well for simple purposes, but it only provides a small amount of what it could yield and for innovation, going beyond the usual is one way to separate a new idea from a successful one.

Looking Differently at Outcomes & Impact

An outcome looks different from where we sit in the system that surrounds an innovation. Consider the role of the design of physical space and how that influences outcomes and shifts our understanding of impact.

A common seating format at conferences is round tables in front of a stage (‘Rounds’). This is usually done where there are meals served and for that purpose, the format works well for everyone — except most of those in attendance and the keynote speakers. Rounds are ideal for serving people including the setting up and clearing of dishes. They are generally lousy for talking with people because, with the exception of speaking with one other person, an attendee must either turn their back on another person to speak to someone else or speak over or past someone.

Rounds show high positive impact — efficiency, ability to monitor, reduced errors — for the catering staff. This might be an important outcome for a conference, although probably not. Consider how this format might enhance or degrade the impact of things like the keynote address or the networking expectations of individuals in attendance.

Consider some other potential impacts:

It gets people away from their families and friends (this could be positive or negative).
It pulls them away from work (it gives them a break, provides an incentive, it adds to their workload, disrupts the teams they leave behind, or all or some of these).
It provides supports conference centres and organizers.
It creates connections between people like connecting with ‘old friends’ and colleagues; sustains a connection to a field of practice or discipline; re-affirms a mission; instills a sense of perspective at how much a field changes (or doesn’t) over time; instills a sense of alienation of a field or peers.
It generates or maintains employment in a region
It’s a tourism and prestige generator.

These are all possible outcomes and are a sample of what additional areas of impact that a learning event might influence. This is for illustrative purposes, but should still provide some ways to show how an innovation (service, product, or policy) might have additional outcomes and impacts that could emerge through an evaluation.

Expanding the Field of Vision

Consideration of these additional outcomes might reveal an opportunity and can more fully demonstrate the impact and potential ROI. Asking different questions can also help prioritize what kind of outcomes make the most sense to optimize the design of your innovation.

Conferences like the one pictured above have optimized for creative thinking within a traditional learning structure by including a poet who composed a unique work summarizing each talk, a (literal!) gallery walk showcasing a prominent local artist’s work (Lars Lerin), and hosting a series of interactive conversation sessions over coffee and snacks.

For Service Convention Sweden and others like it the outcomes might include the standard ones and a deeper look at the new professional connections made (and followed up along with the material products generated from them), the application of the lessons learned, and the integration of learning into organizations by those participating sponsors.

Evaluation is not just about the obvious outcomes when deployed in an innovation context. It can demonstrate not only whether an innovation is achieving the expected impact, but the reach of that impact outside of expectations.

And isn’t innovation all about exceeding expectations?

This is the latest in a series on Evaluation: The Innovator’s Secret Advantage. This series looks at how evaluation can be used to support innovation in service design, product development, and policy implementation. For more on how to do this and any help with it, contact us at www.cense.ca/contact.

Three Questions for Evaluative Thinking

2018-08-10 by cense

Evaluative thinking is at the heart of evaluation, yet it’s remarkably challenging to do in practice. To help strengthen those evaluative neural pathways, we offer some questions to aid you in developing your evaluative thinking skills.

To begin, let’s first look at this odd concept of ‘evaluative thinking’.

Tom Grayson’s recent post on the AEA 365 Blog looked at this topic more closely and provided a useful summary of some of the definitions of the term commonly in use. In its simplest term: evaluative thinking is what we do when we think about things from an evaluation perspective, which is to say, a point of view that considers the merit, worth, and significance of something.

Like many simple things, there is much complexity on the other side of this topic. While we have many methods and tools that can aid us in the process of doing an evaluation, engaging in the evaluative thinking supporting it is actually far more challenging. To help foster evaluative thinking we suggest asking three simple questions:

What is going on?

This question is about paying attention and doing so with an understanding of perspective. Asking this question gets you to focus on the many things that might be happening within a program and the context around it. It gets you to pay attention to the activities, actors, and relationships that exist between them by simple observation and listening. By asking this question you also can start to empathize with those engaged in the program.

Ask:

What is going on for [ ] person?

What is going on in [ ] situation?

What is going on when I step back and look at it all together?

Inquiring about what is going on enlists one of the evaluator’s most powerful assets: curiosity.

By starting to pay attention and question what is going on around you in the smallest and most mundane activities through to those common threads across a program, you will start to see things you never noticed before and took for granted. This opens up possibilities to see connections, relationships, and potential opportunities that were previously hidden.

What’s new?

Asking about what is new is a way to build on the answers from the first question. By looking at what is new, we start to see what might be elements of movement and change. It allows us to identify where things are shifting and where the ‘action’ might be within a program. Most of what we seek in social programs is change — improvements in something, reductions in something else — and sometimes these changes aren’t obvious. Sometimes they are so small that we can’t perceive them unless we pause and look and listen.

There are many evaluation methods that can detect change, however, asking the question about what’s new can help you to direct an evaluation toward the methods that are best suited to capturing this change clearly. Asking this question also amplifies your attentive capacity, which is enormously important for evaluation in detecting large and small changes (because often small changes can have big effects in complex systems like those in human services).

What does it mean?

This last question is about sensemaking. It’s about understanding the bigger significance of something in relation to your enterprise. There can be a lot happening and a lot changing within a program, but it might not mean a whole lot to the overall enterprise. Conversely, there can be little to nothing happening, which can be enormously important for an organization by demonstrating poor effects of an intervention or program or, in the case of prevention-based programs, show success.

This question also returns us to empathy and encourages some perspective-taking by getting us to consider what something means for a particular person or audience. A system (like an organization or program) looks different from where you sit in relation to it. Managers will have a different perspective than that of front-line staff, which is different for clients and customers, and different yet from funders or investors. The concept of ‘success’ or ‘failure’ is judged from the perspective of the viewer and a program may be wildly successful from one perspective (e.g., easy to administer for a manager) and a failure from another (e.g., relatively low return on investment from a funder’s point of view).

This question also affords an opportunity to get a little philosophical about the ‘big picture’. It allows program stakeholders to inquire about what the bigger ‘point’ of a program or service is. Many programs, once useful and effective, can lose their relevance over time due to new entrants to a market or environment, shifting conditions, or changes in the needs of the population served. By not asking this question, there is a risk that a program won’t realize it needs to adapt until it is too late.

By asking these three simple questions you can kick-start your evaluation and innovation work and better strengthen your capacity to think evaluatively.

Photo by Tim Foster on Unsplash

Developmental Evaluation: A Short Introduction

2018-04-30 by cense

Developmental Evaluation (DE) was first proposed by Michael Quinn Patton with the support of colleagues who have wrestled with the problem of dealing with complexity in human systems and the need to provide structured, useful, actionable information to make decisions to support innovation.

DE has been described as being akin to taking a classic ‘road trip’ where you have a destination, a planned route, but also a spirit of adventure and willingness to deviate when needed. DE is an approach to evaluation, not a specific method or tool, designed to support decision making for innovation. Innovation, in this case, is about the activities and decisions that allow an organization and its members to create value by design. The design may not turn out as expected or produce surprises, but it is part of an intentional act to create value through new thinking and action.

Ten things about what DE is and is not

Developmental evaluation (“DE” as it’s often referred to as), when used to support innovation, is about weaving design with data and strategy. It’s about taking a systematic, structured approach to paying attention to what you’re doing, what is being produced (and how), and anchoring it to why you’re doing it by using monitoring and evaluation data. DE helps to identify potentially promising practices or products and guide the strategic decision-making process that comes with innovation. When embedded within a design process, DE provides evidence to support the innovation process from ideation through to business model execution and product delivery.

There are a lot of misconceptions about what a DE is and what it is not and we thought it might be worth addressing ten of these to help provide a brief introduction to DE.

DE is an approach to evaluation, not a method. Most standard methods and tools for evaluation can be used as part of a DE. Qualitative, quantitative, administrative, and ‘big’ data can all contribute to an understanding of a program when used appropriately. It is not something that you simply apply to a situation, rather it is an engaged process of refining how you think about the data you have, what data you collect, and how you make sense of it all and apply lessons from it in practice.
DE is about evaluation for strategic decision-making. If the evaluation is not useful in making decisions about a program or service then is it not a DE. What is considered useful in decision-making is context-dependent, meaning that a DE must be tailored toward the specific situational needs of a program or a service.
DE is not about product or service improvement, it’s about product and service development. It involves a shift in mindset from growth and ‘best practices’ to one of mindful, strategic, adaptative strategy and developmental design.
DE is not separate from strategy, but a critical part of it. There must be close ties between those developing and implementing strategy and the evaluation team or evaluator. A bi-directional flow of information is required through regular, ongoing communications so that strategy informs the DE and the DE informs the strategy simultaneously.
DE does not make things easier, but it can make things better. DE helps programs innovate, learn, and adapt more fully, but that isn’t always easy. A strong DE involves deep engagement with data, a commitment to learning, and a willingness to embrace (or at least accept) volatility, uncertainty, complexity, and ambiguity (VUCA). This requires changing the way organizations work and interact with their programs, which requires time, energy, and sustained attention. However, the promise is that with the systematic attention and a methodology that is designed for VUCA, program leaders can put greater confidence in what DE generates than with standard approaches that assume a more linear, stable, set of conditions.
DE can help document the innovation process. Through creating tools, processes, and decision-making structures to support innovation, DE also helps document the decisions and outcomes of those decisions. When people ask: “how did you get here?” DE provides some answers.
DE does not eliminate the risks associated with VUCA. The adaptive strategy that DE is a part of can often be gamed can be a cop-out for those who do not want to make hard decisions. Strategy is not planning, it’s about “an integrated set of choices that determine where the firm should play and how it should win there” (Martin, 2014) and DE provides a means of building the data set and decision tools to support strategy.
DE is not a panacea. Even with the mindset, appropriate decision-making structures, and a good design, DE is not going to solve the problems of innovation. It will give more systematic means to understand the process, outcomes, outputs, and impacts associated with an innovation, but it still means trials, errors, starts and stops, and the usual explorations that innovators need to experience. DE also requires sensemaking — a structured process of ‘making sense’ of the data that emerges from complex conditions. In these conditions, you can’t expect the data will yield obvious interpretations or conclusions, which is why a sensemaking process is necessary.
Not everyone can do DE well. The popularity of DE in recent years has led to a surge in those claiming to be development evaluators. If you are looking for a DE specialist, consider their experience working with VUCA and complexity in particular. You will also want to find someone who understands strategy, group process, design principles, organizational behaviour, organizational sensemaking, and is an experienced evaluator. This last point means adhering to evaluation standards and potentially recruiting someone who has the Credentialed Evaluator (CE) designation to undertake the evaluation. There are many ways to be adaptive and utilization-focused in your evaluation that isn’t considered to be a DE, too.
DE does not have to be complicated. While DE requires greater involvement of more aspects of an organization in its planning and execution, it doesn’t have to be an elaborate, consuming, or complicated endeavor. DE can be done simply on a small scale just as it can be a large, highly involved, participatory process done at a large scale. DE will scale to nearly any program situation. What makes a DE are the things stated above — things like mindset, organizational support, sensemaking

Developmental Evaluation is a powerful way to help innovators learn, demonstrate and showcase the efforts that go into making change happen, and to increase the capacity of your organization to evolve its mindsets, skillsets, and toolsets for innovation.

Are you interested in using DE and learning more about how your innovation — big or small — in your service, products, or systems? Contact us and we can show you what can be done to bring DE and its potential to your organization.

Budgeting for event success

2017-09-21 by cense

Among the most important elements of any project or procurement plan is the budget. A budget not only accounts for costs but also helps articulate the drivers of those expenses, forcing project planners to consider the logistics behind what goes into a project and (hopefully) what comes from it.

One of the mistakes that organizations make in budget development is that it is too often done apart from the planning and operations of the project, particularly with events. How often do we see well thought out project plans scuppered at the last minute because, upon running the numbers, it was determined there weren’t enough funds available to execute the project as designed? Or the intensity, quantity or quality of program components are reduced to save money after the fact?

It’s about more than money

Budget setting is about more than money: it’s about impact. It’s been estimated that 80% of the environmental impact of any product (or project) is determined at the design stage. Whether that number is specifically correct or not, it’s not hard to see evidence of the impact of design on what comes next. Once the budget and other parameters are set it is the work of project staff to figure out how to work within those parameters to achieve a result: that’s the power of the design to shape the outcome.

While money is important, one of the often forgotten budgetary items is time. There used to be a product called the Tim Timer that was a small desk clock that allowed you to input the average cost per hour of time spent per person attending a meeting and then tracked that for the length of the meeting. A meeting of 10 senior managers who might make, on average $80/hour, meeting for two hours would result in a meeting cost of $1600. If half of those managers didn’t have to be there, that means $800 was wasted in just two hours. When we factor in lost productivity for those two hours and the accumulated impact on morale and motivation of attending more meetings that mean little, the costs rise.

When we factor in lost productivity for those two hours and the accumulated impact on morale and motivation of attending more meetings that mean little, the costs rise.

Time and impact

Consider another example: hosting a learning event. If the focus is on introducing people to new content, most planners will focus on the curriculum. However, consider the possible aims: to increase the knowledge, skills, and capacity of attendees to do something. If that is the case, a curriculum is only part of the issue. How the event is organized, how it is facilitated, how the discussion is supported, what learnings are captured, and how attendees are prepared in advance to learn and supported afterwards in integrating that learning is all part of achieving that objective.

This requires consideration of how the event is designed for learning. Each of the elements above can be managed well and skillfully or ignored and poorly executed. How often have we been to learning events that simply didn’t prepare us for learning at all despite having high-quality content available? How much did we retain, forget or ignore as a result?

This is about budgeting not only money but the time, care and attention required to make the most of the investment in learning and people. One example of how these considerations fit in practice is in the CoNEKTR model of learning developed for busy people, diverse groups, and complex problems. CoNEKTR brings together the science of complexity, social networking tools, and design-driven approaches to social learning and innovation in ways that maximize the time and energy of event participants.

Strategies for better budgets

To effectively build budgets that will create the greatest benefit for the time and money spent, consider these tips:

Bring together those with the knowledge of the finances, logistics and strategic intent in co-developing the event plan early in the process
Consider both the cost of time and the opportunity costs associated with the event
Price your event based on value: what are you expecting to get from your investment?
Evaluate the impact of your investment to align your future plans with the outcomes of your current event. This determines how well you managed to accomplish what you set out to do
Invest in time and focus. If you’re organizing the event in-house, is the focus on the event drawing you away from other things related to the operations of your business? Do you have the time to invest in delivering the value you seek from the event? Ensure that you can focus on the event, its purpose, and its outcomes without distraction.

If you need help building better value for your events, facilitating networking, measuring outcome and creating memorable, impactful learning opportunities connect with us and see how we can help you focus and do more.

Better data visualizing, better impact

2017-09-18 by cense

What good is a program evaluation if its findings aren’t used? Not much. Even as an accountability mechanism, evaluations have the potential to demonstrate the impact a program is having in the world and reveal new insights to guide strategy in ways that few other things can. Although there are many ways to convey evaluation findings, one of the most typical is through an evaluation report.

A good report should not only reflect what happened in the program but also inspire its readers to take action based on the report. This involves making sure the key points are made, but also that the findings are communicated in ways that can be easily understood by audiences who may not have access to the evaluator or program staff (which is why reports are written and codified and why, despite their limitations, are unlikely to disappear).

There are courses aimed at preparing stronger evaluation reports, best practice reports, tools for creative visualization and entire literature fields based on knowledge translation. But if you want to create real impact, you need to deliver knowledge to where it is intended for purposes that may not be fully known because great data reports create possibilities, not just communicate results.

Just as the image above illustrates the many ways light and structure can be reflected, so too can the contents of an evaluation report inspire new ways of seeing programs, data, and strategic opportunities. It all comes down to how the data is presented and in what measure. To support this, we present some tools, people and resources that can help you in making better use of the opportunity that an evaluation report offers through visualization and better communication design.

Resources

Stephanie Evergreen’s a specialist in data visualization for evaluation. Her website features a lot of great resources including links to her books, resource cards and includes tips and tricks to take everyday data and transform it into something attractive, engaging and more useful for audiences.

Kylie Hutchinson, an evaluator and a passionate advocate for better communication in program evaluation, has produced a wonderfully appropriate new resource for evaluators called the Evaluation Reporting Guide that is available through her website. It is a useful guide for being more innovative in the way evaluation findings are reported and comes from someone who knows how to do it.

Kumu is a tool that takes networked data and allows anyone with a basic understanding of network theory to create useful, interactive visuals that can be manipulated and presented in different formats for audiences looking to see the bigger picture.

Cole Nussbaumer Knaflic writes a wonderful blog on how to tell better stories through data combining data visualization with tips on narrative writing that can help even the least creative person imagine new possibilities through the data they have available.

Sometimes it’s just about seeing examples. This post from Import provides some classic examples from the history of data visualization and the latest research to illustrate how data has been used to showcase findings from research and evaluation in creative ways to tell better stories.

Lastly, no commentary on ways to see data differently would be complete without a mention of the incredible works of Edward Tufte, one of the pioneers in data visualization and author of some of the most beautiful, provocative works on the subject ever written.

Creating pictures, telling stories.

Each of these resources provide different ways to see, play with, and present evaluation findings and data. There is no ‘right’ way to do it, rather there are many ways to tell stories and some will resonate with your audience. Data visualization and creative report writing is only good if you know what your audience wants, who they are, and what their motivations are to engage with your content.

The best visuals will not help if you aren’t delivering a product that an audience is ready to see or able to act on, but it is a start. A better looking, more coherent report told through visuals utilizes more of our senses and provides greater opportunities for more people to engage with its content by relying on more than just logic, ‘hard numbers’ and evidence to include things like narrative, emotion, and relationships – the things that make us all human whether we are a data whiz or not.

If you’re looking to make more impactful use of the knowledge you have in your organization, connect with us and we can help you to take the best of what you know and transform it into stories about what you do to those that matter.