Cense

Research.Design.Impact.

Creating Design Pathways for Learning

by

Capturing learning requires a focus on the journey, not end. Thinking like a designer can shape what we learn and how.

Learning is both a journey and a destination and it’s through recognizing this that we can better facilitate intentional, deliberative learning to support innovation and development. By approaching this journey through the lens of service design — as a design-driven evaluation — we can better design the data and insights that come from it to support learning.

What is learning?

Learning comes from perception, experience, feedback, and reflection. You first encounter something and perceive it with our senses (e.g., read, observe, hear, feel), experience something (e.g., movement, action, tension, emotion), which gives you feedback about the perception and experience that is synthesized through reflection (e.g., memory, comparison with related things, contemplation).

Evaluation is principally a tool for learning because it focuses our perception on things, monitors the experience, provides the feedback and can support reflection through offering a systematic, structured means to makes sense of what’s happened.

Evaluation is simply the means of answering the question “what happened?” in a systematic manner.

For those developing an innovation, looking to change, or seeking to improve the sustainability of our systems, answering ‘what happened?’ is the difference between real impact and nothing.

Mapping the journey with data

A journey map is a tool that is used in service design to help understand how service users (e.g., clients, customers, patients, students) might encounter the service and system to achieve a particular goal. These can be displayed visually with great artistry (see here for a beautiful example of the Indigenous cancer patient journey in BC) or simply with boxes and arrows.

It is one of many types of maps that can be created to illustrate the ways in which a user might navigate or approach a service, decision, or pathway to learning.

For innovators and evaluators, these tools present an opportunity to create touchpoints for data collection and deeper understanding of the service throughout. Too often, evaluation is focused on the endpoint or an overall assessment of the process without considering ways to embed opportunities to learn and support learning throughout a journey.

We feel this is a lost opportunity.

Without the opportunity to perceive, gain feedback, and reflect on what happens we are left with experience only, which isn’t a great teacher on its own and filled with many biases that can shift focus away from some of the causes and consequences associated with what’s happening. This is not to say that there isn’t bias in evaluation, yet what makes it different is that it is systematic and accounts for the biases in the design.

Service design meets evaluation

Design-driven evaluation is about integrating evaluation into the design of a program to create the means for developing systematic, structured feedback to support learning along a service journey. One of the simplest ways to do this is to build a layer of evaluation on the service journey map.

Consider a detailed service journey map like the one illustrating the patient journey map cited above. Along this windy, lengthy journey from pre-diagnosis to the end, there are many points where we can learn from the patient, providers, system administrators, and others associated with the health-seeking person that can inform our understanding of the program or system they are in.

By embedding structured (not rigid) data collection into the system we can better learn what’s happening — in both process and effects. Taking this approach offers us the following:

  • Identify activities and behaviours that take place throughout the journey.
  • Provides a lens on service through the perspective of a user. The same service could be modelled using a different perspective (e.g., caregiver, healthcare professional, health administrator).
  • Identifies the systems, processes, people, and relationships that a person goes through on the way through, by, or in spite of a service
  • Allows us to identify how data can fit into a larger narrative of a program or service and be used to support the delivery of that service.
  • Anchors potential data collection points to service transitions and activities to help identify areas of improvement, development, or unnecessary features.
  • Provides a visual means of mapping the structural, behavioural and social processes that underpin the program to test out the theory of change or logic model (does it hold up?).
  • Offers opportunities to explore alternative futures without changing the program (what happens if we did X instead of Y — how would that change the pathway?).

These are some of the ways in which taking a design-driven approach and using common methods from service design can improve or enhance our understanding of a program. Not a bad list, right? That’s just a start.

Try this out. Service design tools and thinking models coupled with evaluation can provide access to the enormous wealth of learning opportunities that exist within your programs. It helps you to uncover the real impact of your programs and innovation value hidden in plain sight.

To learn more about this approach to evaluation, innovation, and service design contact us. We’d love to help you improve what you do and get more value from all your hard work.

Photo by Lili Popper on Unsplash , Billy Pasco on Unsplash and  Startaê Team on Unsplash . Thank you to these artists for making their work available for use.

Developmental Evaluation: A Short Introduction

by

Developmental Evaluation (DE) was first proposed by Michael Quinn Patton with the support of colleagues who have wrestled with the problem of dealing with complexity in human systems and the need to provide structured, useful, actionable information to make decisions to support innovation.

DE has been described as being akin to taking a classic ‘road trip’ where you have a destination, a planned route, but also a spirit of adventure and willingness to deviate when needed. DE is an approach to evaluation, not a specific method or tool, designed to support decision making for innovation. Innovation, in this case, is about the activities and decisions that allow an organization and its members to create value by design. The design may not turn out as expected or produce surprises, but it is part of an intentional act to create value through new thinking and action.

Ten things about what DE is and is not

Developmental evaluation (“DE” as it’s often referred to as), when used to support innovation, is about weaving design with data and strategy. It’s about taking a systematic, structured approach to paying attention to what you’re doing, what is being produced (and how), and anchoring it to why you’re doing it by using monitoring and evaluation data. DE helps to identify potentially promising practices or products and guide the strategic decision-making process that comes with innovation. When embedded within a design process, DE provides evidence to support the innovation process from ideation through to business model execution and product delivery.

There are a lot of misconceptions about what a DE is and what it is not and we thought it might be worth addressing ten of these to help provide a brief introduction to DE.

  1. DE is an approach to evaluation, not a method. Most standard methods and tools for evaluation can be used as part of a DE. Qualitative, quantitative, administrative, and ‘big’ data can all contribute to an understanding of a program when used appropriately. It is not something that you simply apply to a situation, rather it is an engaged process of refining how you think about the data you have, what data you collect, and how you make sense of it all and apply lessons from it in practice.
  2. DE is about evaluation for strategic decision-making. If the evaluation is not useful in making decisions about a program or service then is it not a DE. What is considered useful in decision-making is context-dependent, meaning that a DE must be tailored toward the specific situational needs of a program or a service.
  3. DE is not about product or service improvement, it’s about product and service development. It involves a shift in mindset from growth and ‘best practices’ to one of mindful, strategic, adaptative strategy and developmental design.
  4. DE is not separate from strategy, but a critical part of it. There must be close ties between those developing and implementing strategy and the evaluation team or evaluator. A bi-directional flow of information is required through regular, ongoing communications so that strategy informs the DE and the DE informs the strategy simultaneously.
  5. DE does not make things easier, but it can make things better. DE helps programs innovate, learn, and adapt more fully, but that isn’t always easy. A strong DE involves deep engagement with data, a commitment to learning, and a willingness to embrace (or at least accept) volatility, uncertainty, complexity, and ambiguity (VUCA). This requires changing the way organizations work and interact with their programs, which requires time, energy, and sustained attention. However, the promise is that with the systematic attention and a methodology that is designed for VUCA, program leaders can put greater confidence in what DE generates than with standard approaches that assume a more linear, stable, set of conditions.
  6. DE can help document the innovation process. Through creating tools, processes, and decision-making structures to support innovation, DE also helps document the decisions and outcomes of those decisions. When people ask: “how did you get here?” DE provides some answers.
  7. DE does not eliminate the risks associated with VUCA. The adaptive strategy that DE is a part of can often be gamed can be a cop-out for those who do not want to make hard decisions. Strategy is not planning, it’s about “an integrated set of choices that determine where the firm should play and how it should win there” (Martin, 2014) and DE provides a means of building the data set and decision tools to support strategy.
  8. DE is not a panacea. Even with the mindset, appropriate decision-making structures, and a good design, DE is not going to solve the problems of innovation. It will give more systematic means to understand the process, outcomes, outputs, and impacts associated with an innovation, but it still means trials, errors, starts and stops, and the usual explorations that innovators need to experience. DE also requires sensemaking — a structured process of ‘making sense’ of the data that emerges from complex conditions. In these conditions, you can’t expect the data will yield obvious interpretations or conclusions, which is why a sensemaking process is necessary.
  9. Not everyone can do DE well. The popularity of DE in recent years has led to a surge in those claiming to be development evaluators. If you are looking for a DE specialist, consider their experience working with VUCA and complexity in particular. You will also want to find someone who understands strategy, group process, design principles, organizational behaviour, organizational sensemaking, and is an experienced evaluator. This last point means adhering to evaluation standards and potentially recruiting someone who has the Credentialed Evaluator (CE) designation to undertake the evaluation. There are many ways to be adaptive and utilization-focused in your evaluation that isn’t considered to be a DE, too.
  10. DE does not have to be complicated. While DE requires greater involvement of more aspects of an organization in its planning and execution, it doesn’t have to be an elaborate, consuming, or complicated endeavor. DE can be done simply on a small scale just as it can be a large, highly involved, participatory process done at a large scale. DE will scale to nearly any program situation. What makes a DE are the things stated above — things like mindset, organizational support, sensemaking

Developmental Evaluation is a powerful way to help innovators learn, demonstrate and showcase the efforts that go into making change happen, and to increase the capacity of your organization to evolve its mindsets, skillsets, and toolsets for innovation.

Are you interested in using DE and learning more about how your innovation — big or small — in your service, products, or systems? Contact us and we can show you what can be done to bring DE and its potential to your organization.

 

Complexity Science for Evaluators

by

Strategy and Complexity

The following is by Cameron Norman,  Principal of CENSE Research + Design  originally shared as an invited post on the American Evaluation Association’s AEA365 blog. AEA365 is a daily blog that shares tips, tricks and resources for professional evaluators worldwide.

My work brings together complexity science and design together with developmental evaluation into something I refer to as developmental design, which is about making decisions in the face of changing conditions.

Lesson Learned: At the heart of developmental evaluation is the concept of complexity and innovation. Complexity is a word that we hear a lot of, but might not fully know what it means or how to think about it in the context of evaluation.

For social programs, complexity exists:

… where there are multiple, overlapping sources of input and outputs

… that interact with systems in dynamic ways

… at multiple time scales and organizational levels

… in ways that are highly context-dependent

Rad Resources: Complexity is at the root of developmental evaluation. So for those who are new to the idea or new to developmental evaluation, here are 7 resources that might help you get your head around this complex (pun intended) concept:

  1. Getting to Maybe is a book co-written by our good friend Michael Quinn Patton and offers a great starting place for those working in community and human services;
  2. Patton’s book Developmental Evaluation (ch 5 in particular) is, of course, excellent;
  3. The Plexus Institute is a non-profit organization that supports ongoing learning about complexity applications for a variety of settings;
  4. Tamarack Institute for Community Engagement has an excellent introduction page including an interview with Getting to Maybe co-author Brenda Zimmerman
  5. Ray Pawson’s new book The Science of Evaluation is a more advanced, but still accessible look at ways to think about complexity, programs and evaluation;
  6. My blog Censemaking has a library section with sources on systems thinking and complexity that include these and many more.
  7. The best short introduction to the concept is a video by Dave Snowden on How to Organize A Children’s Party that is a cheeky way to illustrate complexity that I often use in my training and teaching.

Complexity is part theory, part science and all about a way of seeing and thinking about problems. It doesn’t need to scare you and these resources can really help get you in the right mind-frame to tackle challenging problems and use evaluation effectively as a means to addressing them. It might be complex, but it’s fun.

– See more at: http://aea365.org/blog/cameron-norman-on-complexity-science-for-evaluators/#sthash.Ssewsvsq.dpuf