Cense

Research.Design.Impact.

Developmental Evaluation Preparedness

by

Pilots have a pre-flight check before they fly a plane. A developmental evaluation journey requires the same kind of preparedness.

Developmental Evaluation (DE) is an approach to strategic learning designed for understanding and supporting innovation. This approach to data-driven learning requires a specific set-up to do well. While we need a mindset, skillset, and toolset to do DE, we also need a receptive system to make it all work.

Here are the things you can do to lay the groundwork for doing DE and making it a success. Think of this as your pre-flight checklist for a journey of learning together.

These are attributes and may be found within individuals, but it’s better if they are distributed widely across the organization.

Mindset

The most important aspect of a Developmental Evaluation is creating the right mindset. This mindset is focused on complexity, adaptation, and development (vs. improvement, linear growth). A list of statements to examine whether you have the right mindset is available on Censemaking and can provide a start.

Some other mindset questions include:

  1. Openness to experience. A DE will likely take a journey that can’t be fully predicted, much like a road trip. An openness to a non-linear pathway toward a preferred destination and the willingness to adapt the path and the destination based on evaluation data and evidence is key.
  2. Tolerance for ambiguity. Few of us enjoy uncertainty, but we all need to deal with it when working on a DE.
  3. Self-awareness. We often get in our own way. Understanding how we think and the biases and perspectives we hold is important to knowing when they serve us and when they are a barrier to innovation. Mindfulness is a part of this quality.
  4. Patience. This is a skill and mindset quality. Knowing that things will unfold at a time and pace that might change is useful — and requires patience in just knowing that.
  5. Evaluative thinking. This is a form of thinking that gets us connecting our activities, outcomes, and effects and asks three key questions about how we know something (and what to do about it).

Skillset

Developmental evaluation is not for the unskilled. While many of the qualities inherent in a DE are natural to people, they are not commonly practiced in organizations. DE is an approach, and while a skilled evaluator might have many methodological skills to do much of what is in a DE, without the following skills — with the evaluator, the evaluation team, the organization, or all of them — you won’t likely succeed.

  • Facilitation. A DE is a collaborative endeavour. You need facilitation skills to engage the diversity of perspectives in the system and coordinate the discussion, feedback, and opportunities for everyone to engage with these perspectives.
  • Sensemaking. Sensemaking is a participatory process that involves taking data that may be difficult to understand, ambiguous in its conclusions, or incomplete and helps people make sense of what it means for the organization within the present context and strategic needs. Sensemaking for DE helps guide decision-making and strategy.
  • Complexity and Systems Thinking. A distinguishing feature of DE is the use of applied thinking from systems science and complexity to guide the entire process. This means creating designs and processes that are sensitive to detecting emergent properties, network effects, organizational behaviour (like self-organization), and non-linear or algorithmic change dynamics.
  • Dynamism. Complexity also requires that an evaluation be designed with consideration of the dynamics of the program, actors, and system. This means adapting or developing from a moving position, not a static position. It also involves designing evaluations that take into account the growth pattern and evolution of a program that suits the system. It’s about a design for changing conditions.
  • Visualization. System complexity is made more so by the inability to effectively ‘see’ the systems in our heads – there’s too much information. The ability to create system maps, visualize the dynamics and relationships within it, and facilitate discussion of those systems is critical to understanding where a program fits within it all. Visualization can be sophisticated or relatively simple in nature.
  • Design. What makes DE distinct is that it is about making modifications to the program as it unfolds using evaluation data to guide the strategy. These modifications require design and the skills — like with evaluation — require designing a program while it is moving. These developmental design skills are often overlooked and bring together service design and innovation with elements of systemic design.

Toolset

The toolset is the least specific of the three ‘sets’ for DE. Almost any method can support a DE, although unlike other approaches, rarely will there be a case where a single method — interview, survey, observation — will work on its own. A multi-method or mixed-method approach to data collection is almost always necessary.

Some additional tools or methods that are not common are ones we’ve advocated for and either developed ourselves or borrowed from other contexts. Three of the most important of these are:

  • The Living History Method. This meta-method brings together artifacts that inform the program’s early development and evolution. It recognizes that the present program is shaped by where it came from and that such a history can influence where things are going. Unless these conscious or unconscious patterns are recognized it is unlikely they will change. This method also helps record decisions and activities that often get missed when creating an innovation.
  • Dashboards. The wealth of potential information generated from a DE and available at our fingertips can overwhelm even the most skilled evaluator. Creating dashboards using simple tools and technologies can help organize the various streams of data. It also can allow someone to gain a sense of where there may be simple patterns laying within a complex layer of data.
  • Spidergrams. The spider diagram is a simple visualization tool that can work well in a DE as it helps pull various data points together in a way that allows comparison between them AND the ability to dive deep within each data stream.

Some additional perspectives on doing DE can be found in the guide (PDF) published by the McConnell Foundation by Elizabeth Dozois and colleagues.

Next time you are considering a DE, look at your present inventory of mindsets, skillsets, and toolsets to see if you have the right resources in place to do the work you need to do.

Need help? Contact us. We can help you set up a Developmental Evaluation, do one, or both. We’d love to hear from you.

Photo by Lucia Otero on Unsplash

Systems Diagrams Made Simple

by

Sketch diagrams are powerful tools for visualizing complex programs and systems. This simple technique literally gets people on the same page.

Mapping out a service or product ecosystem can be a complicated endeavour. There is the entire field of systemic design that focuses on tools and strategies to engage users for starters. There are approaches like synthesis mapping and service design canvases that can help us walk through the various aspects of a system to find points of leverage, threat, and opportunity.

Nearly all of these methods and tools require user orientation and training — sometimes a little, sometimes a lot. However, in many of our engagements, particularly in healthcare, we find the time (and attention) is so limited it becomes difficult to engage participants using methods that require considerable instruction.

It’s here that we introduce a technique and tool called the system sketch map.

Sketch mapping

A sketch map is a multimedia system map that is created by using any form of representation such as blocks and arrows, stick figures, or elaborate illustration and can be developed at any moment. A sketch map gets participants past the challenge of having to learn a technique or representation convention and can be particularly useful for those participants who feel unskilled at drawing or visualizing their thoughts.

This works well with professionals who may find themselves uneasy about using visual media or do not consider themselves ‘creative’. (Note: Everyone can draw. It’s important to emphasize that this is not an art project).

The exercise works like this:

Begin with the instructions: Draw your system.

That’s it.

Some guidelines: 

  • Any visual formalism can be used. One can even combine visual approaches together.
  • Emphasize the simplest media possible: Pens and paper (or crayons), sticky notes (or stickers) are among the best tools because they are flexible, colourful, and can be combined easily. These are also inexpensive and easy to obtain.
  • Large format paper (e.g., newsprint) or whiteboards are best to use as a canvas to facilitate group participation
  • Group participation is key
  • There are no right or wrong ways to do this. Whatever participants wish to include in that system is all that matters.
  • Give participants a time frame (usually 30 – 50 minutes works best) and try and ensure there are between 4 and 6 people in the group.
  • Emphasize DOing over THINKing. It’s easy for groups to try and do this ‘right’ and analyze everything. The use of simple, inexpensive materials allows people to create ‘do-overs’ easily, erase material, revise and recreate things.
  • Lastly, strive for ‘good enough’ and ‘coherent’ over ‘excellent’ and ‘complete’ (which are highly relative in this context).

What is interesting is that the participants define what their system is and what goes in it. In making these choices it becomes evident what they see as most essential, important, or relevant.:

A completed sketch map then allows everyone (the facilitator and participants) afterward to ‘interrogate’ the map (not the map makers) and ask questions like “does this choice of colour mean anything?“, “is the distance between these two things represent some kind of scale?“, “what might be missing from all of this?”

This interactive discussion process allows everyone to explore what gets placed at different positions, sizes, in different colours, and what gets included and left out of the map. It allows for the use of symbolism (conscious or not), metaphor, and representation without having to shape or bias the participants toward using a particular way of visualizing the system.

What it is, is what it is.

This simple technique can yield enormous insights into the assumptions, structures, relationships, actors, and core components associated with a system and do so within one or two hours and with a small budget.

For more information about sketch mapping and how it can help you with your work or just for more on innovation methods, tools, and strategies, feel free to contact us. We’d love to hear from you and can help.

Photo by Danae Paparis on Unsplash and Kaleidico on Unsplash

The Amazing Spidergram

by

Illustrating action points within a complex system challenges evaluation users. Time to call on your friendly neighbourhood spidergram for help.

Visualizing complex systems is a challenge within strategy, foresight, and evaluation because each component of the system is interconnected with others. Influence on one of these is likely to influence others.

From an action standpoint, it’s easier to focus on one or two small parts of the system than tackle the entire system all at once. How do we reconcile this and provide a means to see the parts of the system without being reductionistic and neglecting the relationship with the whole?

One answer is: look to the spiderweb.

A spiderweb is a good entry-level metaphor for helping people see places they can take action within a system by creating distinctions between the parts (nodes, intersections) and the whole (branches, webs). There are two related, but different spiderweb models worth noting.

Spider Diagrams / Mindmaps

Spider diagrams (or spidergrams or mindmaps) are ways to connect ideas together through the branch-and-thread model akin to a spider’s web (hence the name). These are often called mindmaps and have been shown to facilitate learning about complex topics.

They enable the development of relationships between ideas and possible causal or associative pathways between ideas, concepts, or other data- or evidence-informed concepts. They also enable us to cluster related concepts together to identify sub-systems that may be more amenable to our intervention within the larger whole.

Spidergrams / Radar Diagrams

The other spider-related metaphor that is available to innovators and evaluators is the spidergram, sometimes called a Radar Chart or Spider Chart. These allow for the display of data collected along a scale presented alongside others that use a similar proportioned scale.

This hub-and-spoke model of data allows users to see a variety of performance indicators presented along a similar set of axes related to a common goal.

What this allows for is a view of performance across a variety of metrics simultaneously and can recognize how we make progress on one area often at the expense of another. Strategically, it can enable an organization to balance its actions and foci across a variety of key indicators at the same time.

This can be used with quantitative data such as the financial data above or social data, too.

Spidergrams/charts can also overlay data within the same domain (see example) providing even more depth into recurring or separated data points within the same topic or subsystem.

The web of engagement

What makes these tools powerful is that they display a lot of data at the same time in a manner that can facilitate engagement with a group of people tasked with making decisions. Visualizing data or systems brings the benefit of literally getting people’s focus on the same page.

People around a table can then literally point to the areas they are interested in, concerned with, do not understand, or wish to explore assumptions about.

Complex systems introduce a lot of data and a lot of confusion. Sensemaking through the use of visuals and the discussion that they encourage is one of the ways to reduce confusion and get more from your data to make better decisions.

If you’re stuck in the web of data and complexity, call on your friendly neighbourhood spidergram.

Want to learn more about how this can assist your innovation efforts? Contact us and we’ll gladly swing over and help (without the costume).

Photo by Jean-Philippe Delberghe on Unsplash. Spidergram by

Uncovering Layers of an Innovation’s Impact

by

Revealing Many Layers of Impact

Innovators — those seeking to take an idea for a product, service, or policy and make it real — usually have a pretty clear sense of what they are trying to achieve with their innovation. This is the primary purpose and may reflect where the innovation achieves the greatest impact. But is this all it does? Could it be doing much more? 

Evaluation can play a key role in revealing where an innovation achieves more than just its primary purpose and can serve as a means to uncover layers of impact that can demonstrate various returns on investment (ROI) and open up opportunities for further exploration and exploitation of resources. 

Innovation typically involves considerable investment in time, energy, money, and attention and an evaluation can help showcase the return on investment in unexpected ways. Let’s consider something like an event — a learning conference — as an example to illustrate this layering of impact and how evaluation can aid in revealing these layers and supporting innovation. 

Change or Impact for Whom?

A look across many conferences finds relatively consistent language in their purpose that includes mission phrases like:

to inspire…, to educate…., to connect…, to showcase…, etc..

Among the first roles an evaluation can play is articulating the theory of change explaining why something is intended to achieve an outcome. A look at the phrases above is likely to prompt questions in an evaluator like: for whom are these to happen? Is the effect expected to be similar for most people? What are the means that people are using to showcase products (e.g., knowledge) and for the audience to engage with content?

These are some of the questions that an evaluator might ask from these initial goals. For an evaluation of a learning event, this might translate into metrics like: 

  • Attendance including details on those in the audience (e.g., professional background, previous participation, basic demographics)
  • Number of sessions taught and description of those sessions (content)
  • Overall satisfaction with the conference (including session content, speakers, food and drink, format)
  • Self-reported learning outcomes from participants
  • Financial details: Profit, loss, and expenses; sponsorships
  • Registration information (e.g., online vs. in-person, timing, categories, etc.)

This is a pretty standard set of metrics. We see similar evaluative outcomes across educational programming in different contexts. These might work well for simple purposes, but it only provides a small amount of what it could yield and for innovation, going beyond the usual is one way to separate a new idea from a successful one. 

Looking Differently at Outcomes & Impact

An outcome looks different from where we sit in the system that surrounds an innovation. Consider the role of the design of physical space and how that influences outcomes and shifts our understanding of impact. 

A common seating format at conferences is round tables in front of a stage (‘Rounds’). This is usually done where there are meals served and for that purpose, the format works well for everyone — except most of those in attendance and the keynote speakers. Rounds are ideal for serving people including the setting up and clearing of dishes. They are generally lousy for talking with people because, with the exception of speaking with one other person, an attendee must either turn their back on another person to speak to someone else or speak over or past someone. 

Rounds show high positive impact — efficiency, ability to monitor, reduced errors — for the catering staff. This might be an important outcome for a conference, although probably not. Consider how this format might enhance or degrade the impact of things like the keynote address or the networking expectations of individuals in attendance. 

Consider some other potential impacts:

  • It gets people away from their families and friends (this could be positive or negative).
  • It pulls them away from work (it gives them a break, provides an incentive, it adds to their workload, disrupts the teams they leave behind, or all or some of these).
  • It provides supports conference centres and organizers.
  • It creates connections between people like connecting with ‘old friends’ and colleagues; sustains a connection to a field of practice or discipline; re-affirms a mission; instills a sense of perspective at how much a field changes (or doesn’t) over time; instills a sense of alienation of a field or peers.  
  • It generates or maintains employment in a region
  • It’s a tourism and prestige generator. 

These are all possible outcomes and are a sample of what additional areas of impact that a learning event might influence. This is for illustrative purposes, but should still provide some ways to show how an innovation (service, product, or policy) might have additional outcomes and impacts that could emerge through an evaluation. 

Expanding the Field of Vision

Service Convention Sweden 2018

Consideration of these additional outcomes might reveal an opportunity and can more fully demonstrate the impact and potential ROI. Asking different questions can also help prioritize what kind of outcomes make the most sense to optimize the design of your innovation. 

Conferences like the one pictured above have optimized for creative thinking within a traditional learning structure by including a poet who composed a unique work summarizing each talk, a (literal!) gallery walk showcasing a prominent local artist’s work (Lars Lerin), and hosting a series of interactive conversation sessions over coffee and snacks.  

For Service Convention Sweden and others like it the outcomes might include the standard ones and a deeper look at the new professional connections made (and followed up along with the material products generated from them), the application of the lessons learned, and the integration of learning into organizations by those participating sponsors. 

Evaluation is not just about the obvious outcomes when deployed in an innovation context. It can demonstrate not only whether an innovation is achieving the expected impact, but the reach of that impact outside of expectations.

And isn’t innovation all about exceeding expectations? 

This is the latest in a series on Evaluation: The Innovator’s Secret Advantage. This series looks at how evaluation can be used to support innovation in service design, product development, and policy implementation. For more on how to do this and any help with it, contact us at www.cense.ca/contact. 

Three Questions for Evaluative Thinking

by

Evaluative thinking is at the heart of evaluation, yet it’s remarkably challenging to do in practice. To help strengthen those evaluative neural pathways, we offer some questions to aid you in developing your evaluative thinking skills.

To begin, let’s first look at this odd concept of ‘evaluative thinking’.

Tom Grayson’s recent post on the AEA 365 Blog looked at this topic more closely and provided a useful summary of some of the definitions of the term commonly in use. In its simplest term: evaluative thinking is what we do when we think about things from an evaluation perspective, which is to say, a point of view that considers the merit, worth, and significance of something.

Like many simple things, there is much complexity on the other side of this topic. While we have many methods and tools that can aid us in the process of doing an evaluation, engaging in the evaluative thinking supporting it is actually far more challenging. To help foster evaluative thinking we suggest asking three simple questions:

What is going on?

This question is about paying attention and doing so with an understanding of perspective. Asking this question gets you to focus on the many things that might be happening within a program and the context around it. It gets you to pay attention to the activities, actors, and relationships that exist between them by simple observation and listening. By asking this question you also can start to empathize with those engaged in the program.

Ask: 

What is going on for [ ] person?

What is going on in [ ] situation?

What is going on when I step back and look at it all together? 

Inquiring about what is going on enlists one of the evaluator’s most powerful assets: curiosity.

By starting to pay attention and question what is going on around you in the smallest and most mundane activities through to those common threads across a program, you will start to see things you never noticed before and took for granted. This opens up possibilities to see connections, relationships, and potential opportunities that were previously hidden.

What’s new?

Asking about what is new is a way to build on the answers from the first question. By looking at what is new, we start to see what might be elements of movement and change. It allows us to identify where things are shifting and where the ‘action’ might be within a program. Most of what we seek in social programs is change — improvements in something, reductions in something else — and sometimes these changes aren’t obvious. Sometimes they are so small that we can’t perceive them unless we pause and look and listen.

There are many evaluation methods that can detect change, however, asking the question about what’s new can help you to direct an evaluation toward the methods that are best suited to capturing this change clearly. Asking this question also amplifies your attentive capacity, which is enormously important for evaluation in detecting large and small changes (because often small changes can have big effects in complex systems like those in human services).

What does it mean?

This last question is about sensemaking. It’s about understanding the bigger significance of something in relation to your enterprise. There can be a lot happening and a lot changing within a program, but it might not mean a whole lot to the overall enterprise. Conversely, there can be little to nothing happening, which can be enormously important for an organization by demonstrating poor effects of an intervention or program or, in the case of prevention-based programs, show success.

This question also returns us to empathy and encourages some perspective-taking by getting us to consider what something means for a particular person or audience.  A system (like an organization or program) looks different from where you sit in relation to it. Managers will have a different perspective than that of front-line staff, which is different for clients and customers, and different yet from funders or investors. The concept of ‘success’ or ‘failure’ is judged from the perspective of the viewer and a program may be wildly successful from one perspective (e.g., easy to administer for a manager) and a failure from another (e.g., relatively low return on investment from a funder’s point of view).

This question also affords an opportunity to get a little philosophical about the ‘big picture’. It allows program stakeholders to inquire about what the bigger ‘point’ of a program or service is. Many programs, once useful and effective, can lose their relevance over time due to new entrants to a market or environment, shifting conditions, or changes in the needs of the population served. By not asking this question, there is a risk that a program won’t realize it needs to adapt until it is too late.

 

By asking these three simple questions you can kick-start your evaluation and innovation work and better strengthen your capacity to think evaluatively.

Photo by Tim Foster on Unsplash