Cense

Research.Design.Impact.

Developmental Evaluation Preparedness

by

Pilots have a pre-flight check before they fly a plane. A developmental evaluation journey requires the same kind of preparedness.

Developmental Evaluation (DE) is an approach to strategic learning designed for understanding and supporting innovation. This approach to data-driven learning requires a specific set-up to do well. While we need a mindset, skillset, and toolset to do DE, we also need a receptive system to make it all work.

Here are the things you can do to lay the groundwork for doing DE and making it a success. Think of this as your pre-flight checklist for a journey of learning together.

These are attributes and may be found within individuals, but it’s better if they are distributed widely across the organization.

Mindset

The most important aspect of a Developmental Evaluation is creating the right mindset. This mindset is focused on complexity, adaptation, and development (vs. improvement, linear growth). A list of statements to examine whether you have the right mindset is available on Censemaking and can provide a start.

Some other mindset questions include:

  1. Openness to experience. A DE will likely take a journey that can’t be fully predicted, much like a road trip. An openness to a non-linear pathway toward a preferred destination and the willingness to adapt the path and the destination based on evaluation data and evidence is key.
  2. Tolerance for ambiguity. Few of us enjoy uncertainty, but we all need to deal with it when working on a DE.
  3. Self-awareness. We often get in our own way. Understanding how we think and the biases and perspectives we hold is important to knowing when they serve us and when they are a barrier to innovation. Mindfulness is a part of this quality.
  4. Patience. This is a skill and mindset quality. Knowing that things will unfold at a time and pace that might change is useful — and requires patience in just knowing that.
  5. Evaluative thinking. This is a form of thinking that gets us connecting our activities, outcomes, and effects and asks three key questions about how we know something (and what to do about it).

Skillset

Developmental evaluation is not for the unskilled. While many of the qualities inherent in a DE are natural to people, they are not commonly practiced in organizations. DE is an approach, and while a skilled evaluator might have many methodological skills to do much of what is in a DE, without the following skills — with the evaluator, the evaluation team, the organization, or all of them — you won’t likely succeed.

  • Facilitation. A DE is a collaborative endeavour. You need facilitation skills to engage the diversity of perspectives in the system and coordinate the discussion, feedback, and opportunities for everyone to engage with these perspectives.
  • Sensemaking. Sensemaking is a participatory process that involves taking data that may be difficult to understand, ambiguous in its conclusions, or incomplete and helps people make sense of what it means for the organization within the present context and strategic needs. Sensemaking for DE helps guide decision-making and strategy.
  • Complexity and Systems Thinking. A distinguishing feature of DE is the use of applied thinking from systems science and complexity to guide the entire process. This means creating designs and processes that are sensitive to detecting emergent properties, network effects, organizational behaviour (like self-organization), and non-linear or algorithmic change dynamics.
  • Dynamism. Complexity also requires that an evaluation be designed with consideration of the dynamics of the program, actors, and system. This means adapting or developing from a moving position, not a static position. It also involves designing evaluations that take into account the growth pattern and evolution of a program that suits the system. It’s about a design for changing conditions.
  • Visualization. System complexity is made more so by the inability to effectively ‘see’ the systems in our heads – there’s too much information. The ability to create system maps, visualize the dynamics and relationships within it, and facilitate discussion of those systems is critical to understanding where a program fits within it all. Visualization can be sophisticated or relatively simple in nature.
  • Design. What makes DE distinct is that it is about making modifications to the program as it unfolds using evaluation data to guide the strategy. These modifications require design and the skills — like with evaluation — require designing a program while it is moving. These developmental design skills are often overlooked and bring together service design and innovation with elements of systemic design.

Toolset

The toolset is the least specific of the three ‘sets’ for DE. Almost any method can support a DE, although unlike other approaches, rarely will there be a case where a single method — interview, survey, observation — will work on its own. A multi-method or mixed-method approach to data collection is almost always necessary.

Some additional tools or methods that are not common are ones we’ve advocated for and either developed ourselves or borrowed from other contexts. Three of the most important of these are:

  • The Living History Method. This meta-method brings together artifacts that inform the program’s early development and evolution. It recognizes that the present program is shaped by where it came from and that such a history can influence where things are going. Unless these conscious or unconscious patterns are recognized it is unlikely they will change. This method also helps record decisions and activities that often get missed when creating an innovation.
  • Dashboards. The wealth of potential information generated from a DE and available at our fingertips can overwhelm even the most skilled evaluator. Creating dashboards using simple tools and technologies can help organize the various streams of data. It also can allow someone to gain a sense of where there may be simple patterns laying within a complex layer of data.
  • Spidergrams. The spider diagram is a simple visualization tool that can work well in a DE as it helps pull various data points together in a way that allows comparison between them AND the ability to dive deep within each data stream.

Some additional perspectives on doing DE can be found in the guide (PDF) published by the McConnell Foundation by Elizabeth Dozois and colleagues.

Next time you are considering a DE, look at your present inventory of mindsets, skillsets, and toolsets to see if you have the right resources in place to do the work you need to do.

Need help? Contact us. We can help you set up a Developmental Evaluation, do one, or both. We’d love to hear from you.

Photo by Lucia Otero on Unsplash

Innovation: Why Starting Points Matter

by

If you’re looking to measure some form of progress or impact connected to your innovation (a product, service, or policy) then paying attention to the starting point is critical.

Evaluators call this a baseline and it’s maybe the most important line you can draw. A baseline is really the point of comparison for all you do. When speaking about improvements or change, this is the point you refer to when making those claims.

For something so important, it’s remarkable how few organizations capture baselines well. Let’s look at what it means and how you can do a better job of determining your innovation’s baseline.

Setting a baseline

An ideal baseline is set as far back from the present as possible at the start of your innovation journey. However, as many journeys have starts, stops, and tangents it might be that the start of the innovation journey actually ‘begins’ mid-way through a timeline.

If you are already started your innovation journey, the best time to set a baseline is now. It’s possible in some cases to use retrospective data (looking backward) to assess a baseline, however that can be fraught with certain biases that are unhelpful. If looking retrospectively, consider neutral data points like dates and times, concrete descriptions of product work, and use verifiable sources of data (e.g., work activities, prototypes, expenditures) to support that work.

When setting a baseline, there are some other tips we advise to enable you to capture the most possible useful data you can. If you are innovating in a human system, it’s possible that the innovation may have many effects that go beyond the most obvious so collecting the right data to capture these effects at the beginning is key.

  • List out the resources that have been assembled to develop the innovation such as people, space, and other capital (e.g. funds). These are your starting inputs into the project.
  • Gather a project plan or schedule of activities early to help determine what happens after the project begins. This will help determine where deviations from the plan take place, when, and help you trace back what happens if or when those changes take place to the strategy. Capturing deviations is critical because it helps you go back to see what adaptations you make at the end. Without this data, these activities might appear to be random or haphazard.
  • Capture cultural/environmental factors. Using the STEEP-V (Social, Technological, Economic, Environmental, Political, and Values) model is helpful in knowing what to pay attention to. One of our clients experienced a major, unexpected removal of funding due to rapidly changing political priorities of a government that was supporting their work. By capturing these broader situational variables you can place your innovation work in a context.
  • Document the state of your organization’s readiness and preparedness, which may also include an assessment of innovation readiness. Many innovations fail not out in the market, but within the design studio. Changes to organizational priorities, resources, and personnel can scupper, delay, or change the plans for an innovation. Capturing the state of the organization is an important point as it will allow you to see where things go off track or where they are enabled because of the organization.
  • Develop a project charter and theory of change. While a project may change direction many times, a baseline assessment can help you reflect the desired outcomes and original purpose of the innovation — which are quite likely to change over time. Having this in place can help explain what changes take place and what adaptations take place.

Baselines are the key point for making any claims of change, improvement, or transformation. They are the point where we say “in relation to what?” when speaking about change.

Give yourself some time and use the baseline assessment as a chance to spur reflective and strategic planning about your innovation. You will be grateful you did and amazed at the results later on.

If you’re interested in learning more about baseline development and its role in supporting innovation evaluation, contact us and we’ll gladly help.

Photo by Suad Kamardeen on Unsplash

Chief Learning Officer

by

C-suite leadership roles focus on an organization’s most important functions. Time to introduce the role of the Chief Learning Officer.

Learning — easier said than done. Yet, learning is vital to the success of an organization that seeks to innovate to gain advantage or merely survive – which is most human service organizations these days.

Learning opportunities abound, yet these require energy and attention in order to take advantage of them. Organizationally, this requires leadership and resources to support people across the organization to learn within their areas of focus and across the institution and networks.

Everyone is responsible for learning and there are some great resources to support that effort, but without someone taking explicit leadership on making sure learning happens within the institution, it’s less likely to happen — at least happen in a way that is designed for innovation.

Introducing: the Chief Learning Officer.

Leading Learning

With the alphabet soup of C-s that we are seeing among organizations’ leadership teams adding another might not seem helpful. What we propose is less about formalizing the title of CLO and more about creating the function of what they can do within an organization.

We envision a CLO role as one that does the following:

  • Establishing a learning plan for the organization and the data structure to support that learning. This means instilling and building a culture of evaluation across the organization, which provides data and feedback on what is happening to allow staff at all levels to learn from what is being done. It involves showing what evaluation can do and co-creating ways to do it across the enterprise to add learning value.
  • Ensures that staff roles and functions include the ability to study and reflect on the work being done and its impact. This means establishing practices and procedures that link evaluation data to program activities. It also involves creating the means to bring in insights from outside sources (e.g., published research and reports, networks, professional communities, customers and clients). Structuring what we do and how and when we do it is part of this function to ensure that roles and learning needs are fit-for-purpose.
  • Organizing the evaluation of program activities and ensuring that staff at all relevant levels of the organization close to each program have access to the information about those programs and can make decisions about those programs without having to go through cumbersome layers of bureaucracy.
  • Create sensemaking channels and opportunities throughout the organization. This allows intra- and cross-departmental/unit collaboration to understand the bigger picture of what’s going on in the organization and industry.
  • Supports the development of self-sustaining communities of practice or learning groups on topics relevant to the organization, yet without specific roles or functions. Topics might include emerging technologies, leadership, creative thinking, or professional development.

Creating your CLO Office

A CLO would link the activities of the organization to the monitoring and evaluation data about those activities with the literature and trend data from outside the organization into a culture of learning within the organization.

This could be a full- or part-time position or something like a fractional CLO role like we can play at Cense.

Whether you create a CLO within your organization or choose to recruit learning support from outside, having a dedicated person shepherding your culture into a learning organization is something that will increase your innovation capacity exponentially.

For more information about how you can build this learning culture within your organization or the fractional CLO role, contact us. We’d love to help you out.

Creating Design Pathways for Learning

by

Capturing learning requires a focus on the journey, not end. Thinking like a designer can shape what we learn and how.

Learning is both a journey and a destination and it’s through recognizing this that we can better facilitate intentional, deliberative learning to support innovation and development. By approaching this journey through the lens of service design — as a design-driven evaluation — we can better design the data and insights that come from it to support learning.

What is learning?

Learning comes from perception, experience, feedback, and reflection. You first encounter something and perceive it with our senses (e.g., read, observe, hear, feel), experience something (e.g., movement, action, tension, emotion), which gives you feedback about the perception and experience that is synthesized through reflection (e.g., memory, comparison with related things, contemplation).

Evaluation is principally a tool for learning because it focuses our perception on things, monitors the experience, provides the feedback and can support reflection through offering a systematic, structured means to makes sense of what’s happened.

Evaluation is simply the means of answering the question “what happened?” in a systematic manner.

For those developing an innovation, looking to change, or seeking to improve the sustainability of our systems, answering ‘what happened?’ is the difference between real impact and nothing.

Mapping the journey with data

A journey map is a tool that is used in service design to help understand how service users (e.g., clients, customers, patients, students) might encounter the service and system to achieve a particular goal. These can be displayed visually with great artistry (see here for a beautiful example of the Indigenous cancer patient journey in BC) or simply with boxes and arrows.

It is one of many types of maps that can be created to illustrate the ways in which a user might navigate or approach a service, decision, or pathway to learning.

For innovators and evaluators, these tools present an opportunity to create touchpoints for data collection and deeper understanding of the service throughout. Too often, evaluation is focused on the endpoint or an overall assessment of the process without considering ways to embed opportunities to learn and support learning throughout a journey.

We feel this is a lost opportunity.

Without the opportunity to perceive, gain feedback, and reflect on what happens we are left with experience only, which isn’t a great teacher on its own and filled with many biases that can shift focus away from some of the causes and consequences associated with what’s happening. This is not to say that there isn’t bias in evaluation, yet what makes it different is that it is systematic and accounts for the biases in the design.

Service design meets evaluation

Design-driven evaluation is about integrating evaluation into the design of a program to create the means for developing systematic, structured feedback to support learning along a service journey. One of the simplest ways to do this is to build a layer of evaluation on the service journey map.

Consider a detailed service journey map like the one illustrating the patient journey map cited above. Along this windy, lengthy journey from pre-diagnosis to the end, there are many points where we can learn from the patient, providers, system administrators, and others associated with the health-seeking person that can inform our understanding of the program or system they are in.

By embedding structured (not rigid) data collection into the system we can better learn what’s happening — in both process and effects. Taking this approach offers us the following:

  • Identify activities and behaviours that take place throughout the journey.
  • Provides a lens on service through the perspective of a user. The same service could be modelled using a different perspective (e.g., caregiver, healthcare professional, health administrator).
  • Identifies the systems, processes, people, and relationships that a person goes through on the way through, by, or in spite of a service
  • Allows us to identify how data can fit into a larger narrative of a program or service and be used to support the delivery of that service.
  • Anchors potential data collection points to service transitions and activities to help identify areas of improvement, development, or unnecessary features.
  • Provides a visual means of mapping the structural, behavioural and social processes that underpin the program to test out the theory of change or logic model (does it hold up?).
  • Offers opportunities to explore alternative futures without changing the program (what happens if we did X instead of Y — how would that change the pathway?).

These are some of the ways in which taking a design-driven approach and using common methods from service design can improve or enhance our understanding of a program. Not a bad list, right? That’s just a start.

Try this out. Service design tools and thinking models coupled with evaluation can provide access to the enormous wealth of learning opportunities that exist within your programs. It helps you to uncover the real impact of your programs and innovation value hidden in plain sight.

To learn more about this approach to evaluation, innovation, and service design contact us. We’d love to help you improve what you do and get more value from all your hard work.

Photo by Lili Popper on Unsplash , Billy Pasco on Unsplash and  Startaê Team on Unsplash . Thank you to these artists for making their work available for use.

The Amazing Spidergram

by

Illustrating action points within a complex system challenges evaluation users. Time to call on your friendly neighbourhood spidergram for help.

Visualizing complex systems is a challenge within strategy, foresight, and evaluation because each component of the system is interconnected with others. Influence on one of these is likely to influence others.

From an action standpoint, it’s easier to focus on one or two small parts of the system than tackle the entire system all at once. How do we reconcile this and provide a means to see the parts of the system without being reductionistic and neglecting the relationship with the whole?

One answer is: look to the spiderweb.

A spiderweb is a good entry-level metaphor for helping people see places they can take action within a system by creating distinctions between the parts (nodes, intersections) and the whole (branches, webs). There are two related, but different spiderweb models worth noting.

Spider Diagrams / Mindmaps

Spider diagrams (or spidergrams or mindmaps) are ways to connect ideas together through the branch-and-thread model akin to a spider’s web (hence the name). These are often called mindmaps and have been shown to facilitate learning about complex topics.

They enable the development of relationships between ideas and possible causal or associative pathways between ideas, concepts, or other data- or evidence-informed concepts. They also enable us to cluster related concepts together to identify sub-systems that may be more amenable to our intervention within the larger whole.

Spidergrams / Radar Diagrams

The other spider-related metaphor that is available to innovators and evaluators is the spidergram, sometimes called a Radar Chart or Spider Chart. These allow for the display of data collected along a scale presented alongside others that use a similar proportioned scale.

This hub-and-spoke model of data allows users to see a variety of performance indicators presented along a similar set of axes related to a common goal.

What this allows for is a view of performance across a variety of metrics simultaneously and can recognize how we make progress on one area often at the expense of another. Strategically, it can enable an organization to balance its actions and foci across a variety of key indicators at the same time.

This can be used with quantitative data such as the financial data above or social data, too.

Spidergrams/charts can also overlay data within the same domain (see example) providing even more depth into recurring or separated data points within the same topic or subsystem.

The web of engagement

What makes these tools powerful is that they display a lot of data at the same time in a manner that can facilitate engagement with a group of people tasked with making decisions. Visualizing data or systems brings the benefit of literally getting people’s focus on the same page.

People around a table can then literally point to the areas they are interested in, concerned with, do not understand, or wish to explore assumptions about.

Complex systems introduce a lot of data and a lot of confusion. Sensemaking through the use of visuals and the discussion that they encourage is one of the ways to reduce confusion and get more from your data to make better decisions.

If you’re stuck in the web of data and complexity, call on your friendly neighbourhood spidergram.

Want to learn more about how this can assist your innovation efforts? Contact us and we’ll gladly swing over and help (without the costume).

Photo by Jean-Philippe Delberghe on Unsplash. Spidergram by