Cense

Research.Design.Impact.

Developmental Evaluation Preparedness

by

Pilots have a pre-flight check before they fly a plane. A developmental evaluation journey requires the same kind of preparedness.

Developmental Evaluation (DE) is an approach to strategic learning designed for understanding and supporting innovation. This approach to data-driven learning requires a specific set-up to do well. While we need a mindset, skillset, and toolset to do DE, we also need a receptive system to make it all work.

Here are the things you can do to lay the groundwork for doing DE and making it a success. Think of this as your pre-flight checklist for a journey of learning together.

These are attributes and may be found within individuals, but it’s better if they are distributed widely across the organization.

Mindset

The most important aspect of a Developmental Evaluation is creating the right mindset. This mindset is focused on complexity, adaptation, and development (vs. improvement, linear growth). A list of statements to examine whether you have the right mindset is available on Censemaking and can provide a start.

Some other mindset questions include:

  1. Openness to experience. A DE will likely take a journey that can’t be fully predicted, much like a road trip. An openness to a non-linear pathway toward a preferred destination and the willingness to adapt the path and the destination based on evaluation data and evidence is key.
  2. Tolerance for ambiguity. Few of us enjoy uncertainty, but we all need to deal with it when working on a DE.
  3. Self-awareness. We often get in our own way. Understanding how we think and the biases and perspectives we hold is important to knowing when they serve us and when they are a barrier to innovation. Mindfulness is a part of this quality.
  4. Patience. This is a skill and mindset quality. Knowing that things will unfold at a time and pace that might change is useful — and requires patience in just knowing that.
  5. Evaluative thinking. This is a form of thinking that gets us connecting our activities, outcomes, and effects and asks three key questions about how we know something (and what to do about it).

Skillset

Developmental evaluation is not for the unskilled. While many of the qualities inherent in a DE are natural to people, they are not commonly practiced in organizations. DE is an approach, and while a skilled evaluator might have many methodological skills to do much of what is in a DE, without the following skills — with the evaluator, the evaluation team, the organization, or all of them — you won’t likely succeed.

  • Facilitation. A DE is a collaborative endeavour. You need facilitation skills to engage the diversity of perspectives in the system and coordinate the discussion, feedback, and opportunities for everyone to engage with these perspectives.
  • Sensemaking. Sensemaking is a participatory process that involves taking data that may be difficult to understand, ambiguous in its conclusions, or incomplete and helps people make sense of what it means for the organization within the present context and strategic needs. Sensemaking for DE helps guide decision-making and strategy.
  • Complexity and Systems Thinking. A distinguishing feature of DE is the use of applied thinking from systems science and complexity to guide the entire process. This means creating designs and processes that are sensitive to detecting emergent properties, network effects, organizational behaviour (like self-organization), and non-linear or algorithmic change dynamics.
  • Dynamism. Complexity also requires that an evaluation be designed with consideration of the dynamics of the program, actors, and system. This means adapting or developing from a moving position, not a static position. It also involves designing evaluations that take into account the growth pattern and evolution of a program that suits the system. It’s about a design for changing conditions.
  • Visualization. System complexity is made more so by the inability to effectively ‘see’ the systems in our heads – there’s too much information. The ability to create system maps, visualize the dynamics and relationships within it, and facilitate discussion of those systems is critical to understanding where a program fits within it all. Visualization can be sophisticated or relatively simple in nature.
  • Design. What makes DE distinct is that it is about making modifications to the program as it unfolds using evaluation data to guide the strategy. These modifications require design and the skills — like with evaluation — require designing a program while it is moving. These developmental design skills are often overlooked and bring together service design and innovation with elements of systemic design.

Toolset

The toolset is the least specific of the three ‘sets’ for DE. Almost any method can support a DE, although unlike other approaches, rarely will there be a case where a single method — interview, survey, observation — will work on its own. A multi-method or mixed-method approach to data collection is almost always necessary.

Some additional tools or methods that are not common are ones we’ve advocated for and either developed ourselves or borrowed from other contexts. Three of the most important of these are:

  • The Living History Method. This meta-method brings together artifacts that inform the program’s early development and evolution. It recognizes that the present program is shaped by where it came from and that such a history can influence where things are going. Unless these conscious or unconscious patterns are recognized it is unlikely they will change. This method also helps record decisions and activities that often get missed when creating an innovation.
  • Dashboards. The wealth of potential information generated from a DE and available at our fingertips can overwhelm even the most skilled evaluator. Creating dashboards using simple tools and technologies can help organize the various streams of data. It also can allow someone to gain a sense of where there may be simple patterns laying within a complex layer of data.
  • Spidergrams. The spider diagram is a simple visualization tool that can work well in a DE as it helps pull various data points together in a way that allows comparison between them AND the ability to dive deep within each data stream.

Some additional perspectives on doing DE can be found in the guide (PDF) published by the McConnell Foundation by Elizabeth Dozois and colleagues.

Next time you are considering a DE, look at your present inventory of mindsets, skillsets, and toolsets to see if you have the right resources in place to do the work you need to do.

Need help? Contact us. We can help you set up a Developmental Evaluation, do one, or both. We’d love to hear from you.

Photo by Lucia Otero on Unsplash

Dashboards for innovation evaluation

by

A dashboard system is a way of tracking project activities across a range of different lines of inquiry. This is of particular benefit to those interested in developmental evaluation. Developmental evaluation (DE) is an approach to evaluating programs with an innovation focus and can be used as a means of generating data to support strategy development and deployment. That’s how we use DE.

Developmental evaluations often have multiple components to them that include a variety of methods, data sources, tasks and duties, and sensemaking requirements. All of this information can make a DE difficult for the client to gain a sense of what is happening and even confuse the evaluator at the same time. One way to manage this is to create something we refer to as a dashboard system.

Dashboards: Making strategy and evaluations visible

A dashboard is a means of showcasing multiple information streams in a simple, accessible manner that is easily understood. It might include notes from a meeting, data collection summaries, schedules of activities, and reflective notes. It is not a wholesale repository of all data, rather a higher-level summary that includes enough detail to get a sense of what activities are happening, when, and to what degree.

Among our favourite tools is Trello. Trello is organized into a series of ‘boards’ that allow you to organize content based on a topic or theme and create a simultaneous ‘stream’ of content within each board.

For example, a developmental evaluation might involve something like the following boards:

  1. Interviews with key stakeholder groups
  2. Observations
  3. Document reviews
  4. Sensemaking and strategy notes and outcomes
  5. Meeting minutes
  6. Survey or other data collection results
  7. Reflective notes
  8. Data collection plan

These boards would include any project component that has a distinct, coherent set of activities associated with it. It could include methods, data components, meetings and activities, populations or key document groups.

Tools for thinking and seeing

Trello is just a tool. What we’ve illustrated above could be done as channels within Slack or even as a series of sections within an MS Word or Google Doc. Even an Excel spreadsheet can work. Anything that allows someone to visually track details about discrete areas within a larger project.

(Although tools like MS Word or Excel will work, they are less than ideal as they require strict version controls to articulate changes and this can be difficult to manage, particularly if there are multiple contributors to the document on a regular basis. The reason one might choose MS Word or Excel is less about its functionality and more because it is what people are used to.)

What makes a DE a challenge is that there is a lot of data being generated from many different sources within a complex context. This makes the management of information a challenge and that poses problems for the task of generating helpful insights from the data generated. It also makes it problematic for identifying emergent patterns.

By making data visible, a dashboard allows multiple eyes and multiple perspectives to ‘see’ the data and thus, expands the possibilities associated with seeing possibilities and emergent conditions. The dashboard also increases accountability for all parties. For clients, a dashboard allows them insight into what the evaluator has access to on a regular basis and avoids any surprises. For evaluators, it ensures that clients have the means to engage in the process and don’t sit back waiting for insights to be ‘served up’ to them. DE is best done when it is a collaboration between the client and the evaluator and a dashboard can help this.

Lastly, a dashboard system provides a means of organizing content and keeping track of what is collected, identifying what is needed, and providing a quick-look means to explore what is known and unknown in a project.

Cense employs the dashboard approach on its projects whenever possible. If this approach is of interest to your organization and you want a means of developing strategy and evaluations in a more engaging, transparent manner connect with us and we can show you how this works in practice.