Cense

Research.Design.Impact.

Developmental Evaluation Traps

by

Developmental evaluation (DE) is a powerful tool for supporting innovation in complex systems. 

Developmental evaluation (DE), when used to support innovation, is about weaving design with data and strategy. It’s about taking a systematic, structured approach to paying attention to what you’re doing, what is being produced (and how), and anchoring it to why you’re doing it by using monitoring and evaluation data. For innovators, this is all connected through design.

DE helps to identify potentially promising practices or products and guide the strategic decision-making process that comes with innovation and provides evidence to support the innovation process from ideation through to business model execution and product delivery.

As introduced over on Censemaking, DE is also filled with traps. 

Over the next couple weeks, we’ll be providing some guidance on how to navigate these traps and what your organization can do to steer clear of them and harness the great potential of this important approach to innovation strategy and evaluation for your organization.

Image credit: 200 pair telephone cable model of corpus callosum by J Brew used under Creative Commons license

Design Skills for Evaluators: Design Loft 2017

by

The Design Loft @ AEA 2017

The Design Loft returns to the American Evaluation Association’s annual conference on Friday, November 10th. The Design Loft will be held on the Mezzanine Level of the conference hotel (note: an earlier program had the Design Loft located on the Exhibition Level). There is a new session every hour.

The Design Loft was launched in 2016 in connection with the Presidential Theme of the American Evaluation Association’s annual conference that focused on evaluation + design. The one-day ‘pop-up conference’ was a huge hit with attendees looking to learn more about design and pick up a skill or two. The Design Loft format is simple: every hour on the hour from 9 am to 5 pm (the last session starts at 4 pm) a different design skill is profiled in a 45-minute, hands-on workshop that allows attendees to meet others, laugh, and gain a practical ‘tool’ for their evaluation practice.

Program

The Design Loft is an experimental space intended to provide conference attendees an opportunity to learn specific tools, techniques, and strategies from the field of design that may have application to evaluation practice.

The Design Loft was first launched in 2016 as an experimental ‘pop-up’ conference within the main annual convention as a means of providing hands-on, practical opportunities for evaluators to learn about design methods and tools that can benefit their work and advance innovation within the field. This year the focus of the Design Loft will be applied to the conference theme of Learning to Action.

Each session runs for about 45 minutes, which will allow attendees to take an ‘active learning break’ between sessions during the conference program. It’s a quick hit of excitement, activity, and learning for action.

AEA Conference Program Description: The Design Loft @ AEA

What is design all about and what can it offer evaluators? The Design Loft is an interactive space where attendees can come and experience design firsthand as part of a series of engaging, short workshops aimed at exposing evaluators to simple tools, techniques, and approaches to using design in practice.

Cameron Norman Ph.D. MDes CE, Principal of Cense Ltd., a Credentialed Evaluator, 20-year AEA member and a professional designer, will guide participants through an interactive, hands-on training experience in a small group format.

The Design Loft is a space for creative thinking, innovation, practical learning, and fun and whether Design is new to you or you wish to broaden your toolkit, time hanging out in the Loft will add much to your evaluation work and budding practice as a designer.

Principles

Design is a structured approach to learning about a topic, identifying needs and developing solutions and responses to problems through the active construction of models and prototypes. Design thinking is the way of approaching this creative process that can be employed by designers and non-designers alike using some of the following principles:

  • Embrace whimsy
  • Show don’t tell (make things visible and tangible)
  • Bias for action
  • Culture of prototyping
  • Time constraints
  • Fail fast to succeed sooner
  • Moving from fail-safe to safe-fail
  • Systems thinking

Schedule of Events

The Design Loft runs Friday, November 10, 2017.

9:00 am:     User Personas: A persona is a tool in design that envisions a typical program user by constructing a profile of their behaviors, perspectives, and lifestyle relevant to the topic drawing on user research. These fictional characters are based on evidence and user-data collected by the design team and can help program designers and evaluators understand and anticipate the issues

10:00 am     Attractor Mapping: Where is the action happening and how we can understand where to focus our energy and evaluations when looking at a complex system? Where might we focus our design and evaluation efforts when so much is going on? This simple, visual approach to system mapping will show us how.

11:00 am     A Day in the Life: What does the typical user of a program go through in their day? How might the reality of a user’s day-to-day experience influence the design of a program and what might it mean for evaluators seeking to understand that experience and its relationship to program outcomes better? This session will show how a simple walk-through of a program using visual tools, acting out, and hypothesis generation might enable program planners, evaluators and collaborators to see new possibilities and insights.

12:00 pm     Journey Mapping: This method helps tell the story of a program user’s experience with a program by tracking the encounters that a person might go through along the program. This allows evaluators and designers to analyze the various touch-point an individual might have with a program and create the right kind of program and data collection opportunity. This allows the evaluator to see where problems and opportunities might lay before implementing a program or looking retrospectively at an existing program.

1:00 pm      5 Whys: This simple set of questions gets us to tap into our inner 5-year old and inquire about not just why something is happening on the surface but toward a more deeper understanding of the cause of a problem. By getting closer to the root of an issue, we are better equipped to design programs that make transformational shifts, not cosmetic ones and evaluations that have the power to transform people and programs alike.

2:00 pm      Role Playing: This physical form of problem exploration and prototyping literally has participants acting out specific actions or scenarios to gain insight into design opportunities, constraints, and challenges. This workshop will provide a perfect mid-afternoon break to get up and move and learn how a simple, imaginative approach to getting out of our head can yield insights and opportunities that will create programs that will resonate with our whole selves.

3:00 pm      Paper Prototyping: This ultra-low tech model of prototyping uses simple tools to construct mock-ups of envisioned products allowing for a quick, low-cost way to see opportunities, challenges and needs without resorting to expensive, time-consuming and potentially harmful full-scale prototypes. Working from an example, participants aid design and assess potential strategic options in a quick, low-cost and effective manner.

4:00 pm      Storyboarding: Movies and plays t and what is needed to make it come alive. We can take the same idea and apply that to evaluations. Visualizing an activity or program through simple drawings — no matter how simply done — can be an engaging way to gain insight into attitudes, beliefs, assumptions, and relationships between concepts, project components, and people. This technique will show you how a simple drawing can yield enormous information to guide a program design and the evaluation questions that follow from it.

Theory of change: An introduction

by

Is the above tree alive and growing or dead and ready to be made into furniture? How does something like a tree connect to providing a swing, becoming a coffee table, or supporting the structure of a home? That is based partly on a theory of change about how a tree does what it does. That might sound strange, but for more sophisticated things like human service programs, linking what something does to what it achieves often requires a tool for explanation and a Theory of Change can serve this need well if used appropriately.

Theory of Change is described as “a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context.” It has taken hold in the non-profit and philanthropic sectors in recent years as a means of providing guidance for program developers, funders, and staff in articulating the value of a program and its varied purposes by linking activities to specific behavioural theory.

Matthew Forti, writing in the Stanford Social Innovation Review (SSIR), suggests a Theory of Change (ToC) contain the following:

To start, a good theory of change should answer six big questions:
1. Who are you seeking to influence or benefit (target population)?
2. What benefits are you seeking to achieve (results)?
3. When will you achieve them (time period)?
4. How will you and others make this happen (activities, strategies, resources, etc.)?
5. Where and under what circumstances will you do your work (context)?
6. Why do you believe your theory will bear out (assumptions)?

Unlike a program logic model, which articulates program components, expected outputs, and outcomes a ToC explains how and why a particular set of actions is to produce a change and the assumptions that underpin it all. ToC can be used with a program logic model or be developed independently.

What a ToC is meant to do is allow you to explain in simple language the connection between a program’s purpose, design, and execution and what it produces in terms of benefit and impact. While it may draw on theories that have been published or tested, it may also be unique to the program itself, but in all cases, it is meant to be understandable to a variety of stakeholders and audiences.

Creating a Theory of Change

A strong ToC requires some understanding of behaviour change theory: what do we know about how change happens? It can’t simply end up with “and then change happens”, it must have some kind of logic that can be simply expressed and, whenever possible, tied to what we know about change at the individual, group, organization, system, or a combination. It’s for this reason that bringing in expertise in behaviour change is an important part of the process.

That is one of the points that Kathleen Kelly Janus, also writing in the SSIR, recently made as part of her recommendations for those looking to better the impact of creating a ToC. She suggests organizations do the following:

  1. Engage outside stakeholders
  2. Include your board and staff
  3. Bring in an outside facilitator
  4. Clearly define the outcomes that will spell success
  5. Track your results rigorously.

Inclusion, consultation, and collaboration are all part of the process of developing a ToC. The engagement with diverse stakeholders — particularly those who sit apart from the program — is critical because they will see your program differently. Outsiders will not get caught up in jargon, internal language, or be beholden to current program structures as explanations for change.

Defining the outcomes are important because change requires an explanation of the current state and what that changed state(s) look like. The more articulate you can be about what these outcomes might be, the more reflective the ToC will be of what you’re trying to do. By defining the outcomes better, a ToC can aid a program in developing the appropriate metrics and methods to best determine how (or whether) programs are manifesting these outcomes through their operations.

Supporting strategy

A ToC is best used as an active reference source for program managers, staff, and stakeholders. It can continually be referred to as a means of avoiding strategy ‘drift’ by connecting the programs that are in place to outcomes and reminding management that if the programs change, so too might the outcomes.

A ToC can be used as a developmental evaluation tool, allowing programs to see what they can do and how different adaptations might fit within the same framework for behaviour change to achieve the same outcomes. Alternatively, it can also be used to call into question whether the outcomes themselves are still appropriate.

By making a ToC accessible, easy to read and to understand the key is to make it visual. Employing someone with graphic design skills to help bring the concepts to life in visual representation can provide a means to clarify key ideas and getting people beyond words. It’s easy to get hung up on theoretical language and specific terms when using words; where possible use visuals, narrative, and representations. Metaphors, colour, and texture can bring a ToC to life.

A ToC, when developed appropriately, can provide enormous dividends for strategy, performance, and evaluation and help all members of an organization (and its supporters and partners) understand what it is all about and how what it does is linked to what it aims to achieve. The ToC can serve your communications, strategy development, and evaluation plans if done well and appropriately facilitated, particularly for complex programs. It doesn’t solve all your problems, but few things will help you understand what problems you’re trying to solve and how you might do it than a good Theory of Change.

If you need help building a Theory of Change, contact us and we can help you develop one and show you how it can support your strategy, innovation, and evaluation needs of your programs and organization as a whole.

Budgeting for event success

by

Among the most important elements of any project or procurement plan is the budget. A budget not only accounts for costs but also helps articulate the drivers of those expenses, forcing project planners to consider the logistics behind what goes into a project and (hopefully) what comes from it.

One of the mistakes that organizations make in budget development is that it is too often done apart from the planning and operations of the project, particularly with events. How often do we see well thought out project plans scuppered at the last minute because, upon running the numbers, it was determined there weren’t enough funds available to execute the project as designed? Or the intensity, quantity or quality of program components are reduced to save money after the fact?

It’s about more than money

Budget setting is about more than money: it’s about impact. It’s been estimated that 80% of the environmental impact of any product (or project) is determined at the design stage. Whether that number is specifically correct or not, it’s not hard to see evidence of the impact of design on what comes next. Once the budget and other parameters are set it is the work of project staff to figure out how to work within those parameters to achieve a result: that’s the power of the design to shape the outcome.

While money is important, one of the often forgotten budgetary items is time. There used to be a product called the Tim Timer that was a small desk clock that allowed you to input the average cost per hour of time spent per person attending a meeting and then tracked that for the length of the meeting. A meeting of 10 senior managers who might make, on average $80/hour, meeting for two hours would result in a meeting cost of $1600. If half of those managers didn’t have to be there, that means $800 was wasted in just two hours. When we factor in lost productivity for those two hours and the accumulated impact on morale and motivation of attending more meetings that mean little, the costs rise.

When we factor in lost productivity for those two hours and the accumulated impact on morale and motivation of attending more meetings that mean little, the costs rise.

Time and impact

Consider another example: hosting a learning event. If the focus is on introducing people to new content, most planners will focus on the curriculum. However, consider the possible aims: to increase the knowledge, skills, and capacity of attendees to do something. If that is the case, a curriculum is only part of the issue. How the event is organized, how it is facilitated, how the discussion is supported, what learnings are captured, and how attendees are prepared in advance to learn and supported afterwards in integrating that learning is all part of achieving that objective.

This requires consideration of how the event is designed for learning. Each of the elements above can be managed well and skillfully or ignored and poorly executed. How often have we been to learning events that simply didn’t prepare us for learning at all despite having high-quality content available? How much did we retain, forget or ignore as a result?

This is about budgeting not only money but the time, care and attention required to make the most of the investment in learning and people. One example of how these considerations fit in practice is in the CoNEKTR model of learning developed for busy people, diverse groups, and complex problems. CoNEKTR brings together the science of complexity, social networking tools, and design-driven approaches to social learning and innovation in ways that maximize the time and energy of event participants.

Strategies for better budgets

To effectively build budgets that will create the greatest benefit for the time and money spent, consider these tips:

  • Bring together those with the knowledge of the finances, logistics and strategic intent in co-developing the event plan early in the process
  • Consider both the cost of time and the opportunity costs associated with the event
  • Price your event based on value: what are you expecting to get from your investment?
  • Evaluate the impact of your investment to align your future plans with the outcomes of your current event. This determines how well you managed to accomplish what you set out to do
  • Invest in time and focus. If you’re organizing the event in-house, is the focus on the event drawing you away from other things related to the operations of your business? Do you have the time to invest in delivering the value you seek from the event?  Ensure that you can focus on the event, its purpose, and its outcomes without distraction.

If you need help building better value for your events, facilitating networking, measuring outcome and creating memorable, impactful learning opportunities connect with us and see how we can help you focus and do more.

Better data visualizing, better impact

by

What good is a program evaluation if its findings aren’t used? Not much. Even as an accountability mechanism, evaluations have the potential to demonstrate the impact a program is having in the world and reveal new insights to guide strategy in ways that few other things can. Although there are many ways to convey evaluation findings, one of the most typical is through an evaluation report.

A good report should not only reflect what happened in the program but also inspire its readers to take action based on the report. This involves making sure the key points are made, but also that the findings are communicated in ways that can be easily understood by audiences who may not have access to the evaluator or program staff (which is why reports are written and codified and why, despite their limitations, are unlikely to disappear).

There are courses aimed at preparing stronger evaluation reports, best practice reports,  tools for creative visualization and entire literature fields based on knowledge translation. But if you want to create real impact, you need to deliver knowledge to where it is intended for purposes that may not be fully known because great data reports create possibilities, not just communicate results.

Just as the image above illustrates the many ways light and structure can be reflected, so too can the contents of an evaluation report inspire new ways of seeing programs, data, and strategic opportunities. It all comes down to how the data is presented and in what measure. To support this, we present some tools, people and resources that can help you in making better use of the opportunity that an evaluation report offers through visualization and better communication design.

Resources

Stephanie Evergreen’s a specialist in data visualization for evaluation. Her website features a lot of great resources including links to her books, resource cards and includes tips and tricks to take everyday data and transform it into something attractive, engaging and more useful for audiences.

Kylie Hutchinson, an evaluator and a passionate advocate for better communication in program evaluation, has produced a wonderfully appropriate new resource for evaluators called the Evaluation Reporting Guide that is available through her website. It is a useful guide for being more innovative in the way evaluation findings are reported and comes from someone who knows how to do it.

Kumu is a tool that takes networked data and allows anyone with a basic understanding of network theory to create useful, interactive visuals that can be manipulated and presented in different formats for audiences looking to see the bigger picture.

Cole Nussbaumer Knaflic writes a wonderful blog on how to tell better stories through data combining data visualization with tips on narrative writing that can help even the least creative person imagine new possibilities through the data they have available.

Sometimes it’s just about seeing examples. This post from Import provides some classic examples from the history of data visualization and the latest research to illustrate how data has been used to showcase findings from research and evaluation in creative ways to tell better stories.

Lastly, no commentary on ways to see data differently would be complete without a mention of the incredible works of Edward Tufte, one of the pioneers in data visualization and author of some of the most beautiful, provocative works on the subject ever written.

Creating pictures, telling stories.

Each of these resources provide different ways to see, play with, and present evaluation findings and data. There is no ‘right’ way to do it, rather there are many ways to tell stories and some will resonate with your audience. Data visualization and creative report writing is only good if you know what your audience wants, who they are, and what their motivations are to engage with your content.

The best visuals will not help if you aren’t delivering a product that an audience is ready to see or able to act on, but it is a start. A better looking, more coherent report told through visuals utilizes more of our senses and provides greater opportunities for more people to engage with its content by relying on more than just logic, ‘hard numbers’ and evidence to include things like narrative, emotion, and relationships – the things that make us all human whether we are a data whiz or not.

If you’re looking to make more impactful use of the knowledge you have in your organization, connect with us and we can help you to take the best of what you know and transform it into stories about what you do to those that matter.