Cense

Research.Design.Impact.

Living History for Developmental Evaluation

by

Developmental Evaluation (DE) is an approach to supporting the evolution and development of an innovation. This means not only helping guide where an innovation is going but understanding where it is now and where it has been. Just like human development, an innovation is as much a product of its past as it is the present decisions and that past can help inform the strategy for moving forward. But how do we properly account for the past and present context in understanding what comes next? This is where The Living History method comes in.

The Living History method comprises a set of data collection, sensemaking, and design strategies that come together to reflect on how a project comes into being, develops a focus, and evolves up to the present context. It helps provide the ‘backstory’ to the present situation and provides a means of developing a baseline – another key part of a DE.

Why does knowing this matter? Taking a Living History of a program helps surface the path dependencies of a program or innovation. Path dependencies can include those habits of mind that form, patterns of behaviour, and general routines that can develop within a complex environment and consciously or unconsciously shape the context in which decisions are made. By understanding what these patterns are, we can better prepare to address them as the program unfolds.

It’s helpful to the biases and mental models that we hold in guiding our decisions to better account for them both when things work well, but also to help understand how things didn’t work out.

To borrow from the image above: the Living History method helps you use the data collected from preparing the climb, starting up to base camp, and setting the groundwork (the trials, failures, and victories) to initiate the climb as well as the data from the actual ascent to the summit.

Introducing the method

The Living History method is something we developed to ‘backfill’ much of the information that is necessary to understand the present context of a program. Many evaluators do this kind of work with informational interviews, document reviews, and site visits, however, the means in which this data is collated and made sense of isn’t always systematic, nor is it done in a manner that recognizes complexity (which is the space in which DE is meant to be employed).

A Living History involves building that historical narrative that has been constructed by program developers (founders, staff, stakeholders, etc..) and the activities that contribute to this narrative. It draws together these ‘stories’ from the data into an understanding of the present. This present space is where the existing narrative and current construction of the program (‘what is happening now?‘) comes together. It’s also space where strategy is developed and initiated to shape what is to come (‘where are we going and how are we going to get there?‘).

Below is an image that demonstrates this process.

Taking a Living History

A Living History can encompass many different data gathering techniques. Four of the most useful are:

  1. Document review. A thorough review of foundational documents that might include things like the initial application(s) for funding, ‘pitch decks’ used to generate funds or interest, program and policy manuals, staff reviews, board meeting minutes, evaluations, and annual reports (to boards, funders, stakeholders, etc.). Some programs may have an early program logic model or Theory of Change that can be examined. These can give insight into the planning structure, theories, and reasons for establishing the program.
  2. Interviews. Connecting with the program founders, board members (past and present), investors (and funders), as well as leaders involved in the initiative is critical. Having conversations with these people will help surface both explicit and tacit knowledge held by those that helped shape the program. These interviews can be critical in understanding the logic, political context, and social dynamics that underpin the direction of the organization. It is useful to determine not only the direction taken, but the directions considered. The reason is that these ‘unexplored paths’ may linger in the minds of people and serve as a source of tension or inspiration in moving forward.
  3. Site visits. It is helpful to understand that programs – even those that are virtual such as a website or app — originate in the physical world somewhere. Site visits, when possible, help gather information about organizational culture, physical resources, and relational barriers and facilitators. Take the example of the organization that starts up as a maker space in a garage and then evolves to a large corporate high-rise office. Is the culture that was established at the start-up maintained at a different scale or abandoned? We’ve seen this with organizations that began as a start-up, grew enormously, and then abandoned the culture that was created at startup for a more conservative corporate one despite holding on to the belief that things had not changed. Only by pointing out (seeing) the manner in which things physically changed (e.g., open space vs. closed spaces, size and shape of offices) was this able to be pointed out. Further, such visits can help determine developmental milestones such as large investments of capital.
  4. Timeline. A timeline is a simple method of connecting activities — actions + decisions — together in a visual means. This helps determine and organize data about what was done, but also when it was done. This can make an enormous difference in understanding things like time lags and exploring cause-and-consequence connections between actions, reactions, and other effects.

The Role of Sensemaking

Living History is not complete without sensemaking. Sensemaking is required because, like most interesting stories, innovating through complexity is messy and full of contradictions, gaps in logic, and remarkable things that can only be understood by sitting together with all the data in one place. These emergent properties — coherence that forms from diverse sources of information as part of the process of (self) organization — are what make a program a ‘whole’ and not just the ‘sum of the parts’.

Sensemaking is collaborative, discursive, and involves debate, reflection, and a process of drawing conclusions and forming hypotheses based on data and experience. It is the process of taking the strangeness of seeing things in chunks and putting them together. What comes from this process, which is a key part of Developmental Evaluation and design, is a ‘sense’ of what happened, what is going on, and what options might work next.

It helps you understand where you and your program stand now.

Moving forward (and backward)

The Living History method doesn’t end with sensemaking, it also is carried forward.

Developing a project dashboard for innovation evaluation is a helpful means to pull together the data collected from the past, the present, and then populating it as the program evolves. The Living History method views history as being co-constructed as the program evolves and thus thrives on having new data about the present (which is always a second away from being the past) added to the overall corpus of data.

New data, when combined with what has already been collected, can provide new insights into the bigger picture. As data is added and time passes, it is possible that new patterns will emerge, which can be insightful for making decisions. This data, collected now and going forward, will later be recombined as part of an ongoing sensemaking process.

A Living History can also provide a means to document the activities that have taken place up to the present moment by systematically, although retrospectively, capturing (mostly existing) data to show what has happened before, which can be useful for organizations and programs who are starting their evaluations later than they would have liked to.

Consider the Living History approach with your evaluation. By looking at the past you might find yourself better prepared to go forward. We can help.

Photo by Aaron Benson on Unsplash

Three Questions for Evaluative Thinking

by

Evaluative thinking is at the heart of evaluation, yet it’s remarkably challenging to do in practice. To help strengthen those evaluative neural pathways, we offer some questions to aid you in developing your evaluative thinking skills.

To begin, let’s first look at this odd concept of ‘evaluative thinking’.

Tom Grayson’s recent post on the AEA 365 Blog looked at this topic more closely and provided a useful summary of some of the definitions of the term commonly in use. In its simplest term: evaluative thinking is what we do when we think about things from an evaluation perspective, which is to say, a point of view that considers the merit, worth, and significance of something.

Like many simple things, there is much complexity on the other side of this topic. While we have many methods and tools that can aid us in the process of doing an evaluation, engaging in the evaluative thinking supporting it is actually far more challenging. To help foster evaluative thinking we suggest asking three simple questions:

What is going on?

This question is about paying attention and doing so with an understanding of perspective. Asking this question gets you to focus on the many things that might be happening within a program and the context around it. It gets you to pay attention to the activities, actors, and relationships that exist between them by simple observation and listening. By asking this question you also can start to empathize with those engaged in the program.

Ask: 

What is going on for [ ] person?

What is going on in [ ] situation?

What is going on when I step back and look at it all together? 

Inquiring about what is going on enlists one of the evaluator’s most powerful assets: curiosity.

By starting to pay attention and question what is going on around you in the smallest and most mundane activities through to those common threads across a program, you will start to see things you never noticed before and took for granted. This opens up possibilities to see connections, relationships, and potential opportunities that were previously hidden.

What’s new?

Asking about what is new is a way to build on the answers from the first question. By looking at what is new, we start to see what might be elements of movement and change. It allows us to identify where things are shifting and where the ‘action’ might be within a program. Most of what we seek in social programs is change — improvements in something, reductions in something else — and sometimes these changes aren’t obvious. Sometimes they are so small that we can’t perceive them unless we pause and look and listen.

There are many evaluation methods that can detect change, however, asking the question about what’s new can help you to direct an evaluation toward the methods that are best suited to capturing this change clearly. Asking this question also amplifies your attentive capacity, which is enormously important for evaluation in detecting large and small changes (because often small changes can have big effects in complex systems like those in human services).

What does it mean?

This last question is about sensemaking. It’s about understanding the bigger significance of something in relation to your enterprise. There can be a lot happening and a lot changing within a program, but it might not mean a whole lot to the overall enterprise. Conversely, there can be little to nothing happening, which can be enormously important for an organization by demonstrating poor effects of an intervention or program or, in the case of prevention-based programs, show success.

This question also returns us to empathy and encourages some perspective-taking by getting us to consider what something means for a particular person or audience.  A system (like an organization or program) looks different from where you sit in relation to it. Managers will have a different perspective than that of front-line staff, which is different for clients and customers, and different yet from funders or investors. The concept of ‘success’ or ‘failure’ is judged from the perspective of the viewer and a program may be wildly successful from one perspective (e.g., easy to administer for a manager) and a failure from another (e.g., relatively low return on investment from a funder’s point of view).

This question also affords an opportunity to get a little philosophical about the ‘big picture’. It allows program stakeholders to inquire about what the bigger ‘point’ of a program or service is. Many programs, once useful and effective, can lose their relevance over time due to new entrants to a market or environment, shifting conditions, or changes in the needs of the population served. By not asking this question, there is a risk that a program won’t realize it needs to adapt until it is too late.

 

By asking these three simple questions you can kick-start your evaluation and innovation work and better strengthen your capacity to think evaluatively.

Photo by Tim Foster on Unsplash

Developmental Evaluation: A Short Introduction

by

Developmental Evaluation (DE) was first proposed by Michael Quinn Patton with the support of colleagues who have wrestled with the problem of dealing with complexity in human systems and the need to provide structured, useful, actionable information to make decisions to support innovation.

DE has been described as being akin to taking a classic ‘road trip’ where you have a destination, a planned route, but also a spirit of adventure and willingness to deviate when needed. DE is an approach to evaluation, not a specific method or tool, designed to support decision making for innovation. Innovation, in this case, is about the activities and decisions that allow an organization and its members to create value by design. The design may not turn out as expected or produce surprises, but it is part of an intentional act to create value through new thinking and action.

Ten things about what DE is and is not

Developmental evaluation (“DE” as it’s often referred to as), when used to support innovation, is about weaving design with data and strategy. It’s about taking a systematic, structured approach to paying attention to what you’re doing, what is being produced (and how), and anchoring it to why you’re doing it by using monitoring and evaluation data. DE helps to identify potentially promising practices or products and guide the strategic decision-making process that comes with innovation. When embedded within a design process, DE provides evidence to support the innovation process from ideation through to business model execution and product delivery.

There are a lot of misconceptions about what a DE is and what it is not and we thought it might be worth addressing ten of these to help provide a brief introduction to DE.

  1. DE is an approach to evaluation, not a method. Most standard methods and tools for evaluation can be used as part of a DE. Qualitative, quantitative, administrative, and ‘big’ data can all contribute to an understanding of a program when used appropriately. It is not something that you simply apply to a situation, rather it is an engaged process of refining how you think about the data you have, what data you collect, and how you make sense of it all and apply lessons from it in practice.
  2. DE is about evaluation for strategic decision-making. If the evaluation is not useful in making decisions about a program or service then is it not a DE. What is considered useful in decision-making is context-dependent, meaning that a DE must be tailored toward the specific situational needs of a program or a service.
  3. DE is not about product or service improvement, it’s about product and service development. It involves a shift in mindset from growth and ‘best practices’ to one of mindful, strategic, adaptative strategy and developmental design.
  4. DE is not separate from strategy, but a critical part of it. There must be close ties between those developing and implementing strategy and the evaluation team or evaluator. A bi-directional flow of information is required through regular, ongoing communications so that strategy informs the DE and the DE informs the strategy simultaneously.
  5. DE does not make things easier, but it can make things better. DE helps programs innovate, learn, and adapt more fully, but that isn’t always easy. A strong DE involves deep engagement with data, a commitment to learning, and a willingness to embrace (or at least accept) volatility, uncertainty, complexity, and ambiguity (VUCA). This requires changing the way organizations work and interact with their programs, which requires time, energy, and sustained attention. However, the promise is that with the systematic attention and a methodology that is designed for VUCA, program leaders can put greater confidence in what DE generates than with standard approaches that assume a more linear, stable, set of conditions.
  6. DE can help document the innovation process. Through creating tools, processes, and decision-making structures to support innovation, DE also helps document the decisions and outcomes of those decisions. When people ask: “how did you get here?” DE provides some answers.
  7. DE does not eliminate the risks associated with VUCA. The adaptive strategy that DE is a part of can often be gamed can be a cop-out for those who do not want to make hard decisions. Strategy is not planning, it’s about “an integrated set of choices that determine where the firm should play and how it should win there” (Martin, 2014) and DE provides a means of building the data set and decision tools to support strategy.
  8. DE is not a panacea. Even with the mindset, appropriate decision-making structures, and a good design, DE is not going to solve the problems of innovation. It will give more systematic means to understand the process, outcomes, outputs, and impacts associated with an innovation, but it still means trials, errors, starts and stops, and the usual explorations that innovators need to experience. DE also requires sensemaking — a structured process of ‘making sense’ of the data that emerges from complex conditions. In these conditions, you can’t expect the data will yield obvious interpretations or conclusions, which is why a sensemaking process is necessary.
  9. Not everyone can do DE well. The popularity of DE in recent years has led to a surge in those claiming to be development evaluators. If you are looking for a DE specialist, consider their experience working with VUCA and complexity in particular. You will also want to find someone who understands strategy, group process, design principles, organizational behaviour, organizational sensemaking, and is an experienced evaluator. This last point means adhering to evaluation standards and potentially recruiting someone who has the Credentialed Evaluator (CE) designation to undertake the evaluation. There are many ways to be adaptive and utilization-focused in your evaluation that isn’t considered to be a DE, too.
  10. DE does not have to be complicated. While DE requires greater involvement of more aspects of an organization in its planning and execution, it doesn’t have to be an elaborate, consuming, or complicated endeavor. DE can be done simply on a small scale just as it can be a large, highly involved, participatory process done at a large scale. DE will scale to nearly any program situation. What makes a DE are the things stated above — things like mindset, organizational support, sensemaking

Developmental Evaluation is a powerful way to help innovators learn, demonstrate and showcase the efforts that go into making change happen, and to increase the capacity of your organization to evolve its mindsets, skillsets, and toolsets for innovation.

Are you interested in using DE and learning more about how your innovation — big or small — in your service, products, or systems? Contact us and we can show you what can be done to bring DE and its potential to your organization.

 

Building a developmental mindset

by

Build your developmental mindset

Innovating in human systems requires more than ideas, resources, clever strategy or even new thinking; it requires a new mindset.

A mindset is a concept that psychologist Carol Dweck has explored in-depth and described a thinking orientation to the world. It’s not set in stone, but rather a lens through which we see our situations and the possibilities and challenges that they present to us. While Dweck has identified particular mindsets that shape how individuals use their brains, we’d like to explore another mindset here: a developmental one.

Over at Censemaking, Cameron Norman considered the role of the developmental mindset and how organizations engage with developmental evaluation. In that post, Norman argues that developmental evaluation is not possible without having the right mindset in place and we couldn’t agree more.

Witnessing evolution

The critical step toward a developmental mindset is considering viewing programs as a part of living systems. If a program involves a high variety of information and feedback, interacts with other programs and social forces, and exists in a space where there are ongoing disruptions and change, then it is what we would consider complex. If it is complex, then it fits the kind of scenario argued in Censemaking.

To take an evolutionary or developmental mindset requires first viewing programs as dynamic, rather than static. This perspective involves asking different evaluative questions. Static programs allow for questions like:

  • How effective is [the program] at achieving [specific target]?
  • How well does [the program] fit the industry-standard benchmark?
  • What is the impact of [the program] on [specific behaviour]?
  • How did [program component] contribute to [specific outcome]?

These questions allow for answers that have verifiable correlates with expected outcomes, a theory of change and industry benchmarks.

Dynamic programs require asking more sophisticated, nuanced questions that might produce less precise answers, but are more useful in helping a program to develop and evolve. These include:

  • What outcomes are generated through [the program] at this given moment?
  • How have the outputs and possible outcomes of [the program] differed over time?
  • How is [the program] interacting with its social system and what new behaviours and activities are generated through those interactions?
  • What new, beneficial attractors (clusters of organized activity) have established that are attributed in part to the involvement of [the program]?

Seeing opportunity in complexity

A review of the questions above might seem less satisfying in some ways, but they also reveal possibilities. It is why developmental evaluation (DE) is known as evaluation for innovation. DE provides the feedback necessary to understand the role(s) that a program is playing within a system and what comes from the interaction between people, programs, and systems. Thus, DE is a valuable part of the innovation process by allowing organizations to test out ideas and see what kind of influence they have in the world and make adaptations and modifications to it as the conditions, situations, and requirements evolve.

DE is about evaluating programs as part of living systems. This approach recognizes that programs exist in dynamic systems akin to the phrase ‘the river I stand in is not the river I step in’ from Heraclitus. This mindset recognizes that your market competition will change as you change and react to your actions, just as your organization might have to react to their actions.

DE doesn’t solve the problem, but it does allow for new opportunities to be discovered. By asking different questions to see things different, you can see these opportunities, too.

To learn more about developmental evaluation and how the right approach to generating feedback and learning about your programs can help your organization contact us for a consultation.

Image credit: “Mindset” by Steve Davis is licensed under CC BY 2.0

Mindfulness in Developmental Evaluation

by

A place to sit, reflect, observe and evaluate

One of the tools within the toolkit of a developmental evaluator is mindfulness. Mindfulness is a disciplined, regular, and persistent means of paying attention to what is going on. It is a terrific means of collecting data on regular activities, particularly in complex environments where it may be unclear what is worth paying attention to.

In an earlier article on Censemaking (our sister site), Cameron Norman outlines how mindfulness and developmental evaluation work together, highlighting some of the emerging scholarship in the area of organizational mindfulness as an example. As discussed in that piece:

Mindfulness is the disciplined practice of paying attention. Bishop and colleagues (2004 – PDF), working in the clinical context, developed a two-component definition of mindfulness that focuses on 1) self-regulation of attention that is maintained on the immediate experience to enable pattern recognition (enhanced metacognition) and 2) an orientation to experience that is committed to and maintains an attitude of curiosity and openness to the present moment.

It’s one thing to talk mindfulness, but what does this mean for evaluators and organizations in practice? How can we use mindfulness practice and theory to inform developmental evaluation in a practical manner?

We present some strategies we’ve employed in our client work and offer some suggests for those seeking to bring a mindfulness approach to their developmental evaluations.

1. Introduce meditative practice through demonstrations. Let’s get the most obvious link out of the way: meditation. Meditation is usually the first thing that most people think of when hearing the word mindfulness. Meditation is a practice and while it is often linked with spiritual traditions from different cultures it does not need to have a spiritual dimension included if that’s not appropriate or useful. One of the simplest exercises to do is to walk members of the evaluation team through a short mindfulness exercise involving just sitting, closing or dimming the eyes, and paying attention to the breath and the thought patterns going through their head in a non-judgemental manner. A simple one-minute exercise can alert people to the massive amount of stimuli — both inner and outer — that is going on that they either aren’t fully attuned to or wasn’t as clear to them. Developmental evaluation is very much like this: it opens up our awareness of what’s happening in a living system on the go.

2. Build a mindful culture. While meditation is useful, it’s benefits are only accrued when applied collectively as part of the evaluation. Not everyone will take to the idea of meditating (and it needs to be introduced safely), but the idea of paying attention at regular intervals, consistently is effectively at the heart of a developmental evaluation, particularly in a highly complex context. To do this, create regular check-in’s where people answer the simple question: What’s going on? It’s like the equivalent of the “how was your day?” question we might ask our spouse or child. It allows people a momentary space to reflect on what happened within a time period and pick out what was meaningful to them? A follow-up question is: “what did you notice?”

3. Practice non-judgement, at first, and often. Evaluation is about judgement as much as anything else, but in a developmental evaluation context we may not know what benefits, drawbacks, implications or opportunities are present in a situation in the present moment. Encourage participants to simply take note of what is going on, when, and on what was observed without attributing cause and effect, judgement (“good” or “bad”) right away. Attribution is something that can be made later on through a more structured process of sensemaking.

4. Make it social. Mindfulness is generally thought of as a solitary, intra-personal activity. In a developmental evaluation you’re collecting information from across an organization or program and it involves many people and many perspectives: it has to be social to work. This is where sensemaking comes in – a critical component of developmental evaluation. The meaning of any activity won’t be readily apparent without an ability to translate individual observations and reflections into a more collective understanding. Further, because developmental evaluation is used in contexts that are usually complex, the meaning of something will be best determined by having multiple perspectives on the issue — diverse perspectives on the system — brought to bear. Ensure that teams are scheduling regular sensemaking meetings alongside regular reflections.

5. Make it visual. Mindfulness yields a lot of data. In terms of social research methods, mindfulness practice is as close to yielding ‘big data‘ as you’ll get. Lots of small observations over time will produce a lot of data. Sensemaking is a means of identifying potential patterns and meanings in the data, but even then there can be times when it’s hard to make sense of things. Using visual methods such as sketching, map making, and creative visualizations can help, tool. Visualizations allow people to abstract from one concept to the next and literally allow people’s attention to get on the same page. Tools like a gigamap can map and model a problem and potential solutions. Attractor maps can also include different elements that represent certain properties, feelings, experiences, observations and other data and put them together.

The timing of all of these activities is highly dependent upon the complexity and dynamism within the program domain. For example, a 12-week program aimed at educating individuals might have a lot of dynamism in it from week-to-week, enough that weekly check-ins and tri-weekly or monthly sensemaking sessions might be required. However, if the work being done is a collective impact model looking at advocacy for a major policy change, the activities and actions might be more long term. In that case, bi-weekly checkins and maybe quarterly sensemaking sessions are needed.

Mindfulness is a very powerful tool when employed in a developmental evaluation. Creating spaces and practices that allow people to mindfully reflect on their work, capture it and organize it can be a way of collecting data on and detecting subtle, evolving patterns of activity that are the hallmark of complex systems and programs.

Good luck. Be mindful.