tools

Three Questions for Evaluative Thinking

2018-08-10 by cense

Evaluative thinking is at the heart of evaluation, yet it’s remarkably challenging to do in practice. To help strengthen those evaluative neural pathways, we offer some questions to aid you in developing your evaluative thinking skills.

To begin, let’s first look at this odd concept of ‘evaluative thinking’.

Tom Grayson’s recent post on the AEA 365 Blog looked at this topic more closely and provided a useful summary of some of the definitions of the term commonly in use. In its simplest term: evaluative thinking is what we do when we think about things from an evaluation perspective, which is to say, a point of view that considers the merit, worth, and significance of something.

Like many simple things, there is much complexity on the other side of this topic. While we have many methods and tools that can aid us in the process of doing an evaluation, engaging in the evaluative thinking supporting it is actually far more challenging. To help foster evaluative thinking we suggest asking three simple questions:

What is going on?

This question is about paying attention and doing so with an understanding of perspective. Asking this question gets you to focus on the many things that might be happening within a program and the context around it. It gets you to pay attention to the activities, actors, and relationships that exist between them by simple observation and listening. By asking this question you also can start to empathize with those engaged in the program.

Ask:

What is going on for [ ] person?

What is going on in [ ] situation?

What is going on when I step back and look at it all together?

Inquiring about what is going on enlists one of the evaluator’s most powerful assets: curiosity.

By starting to pay attention and question what is going on around you in the smallest and most mundane activities through to those common threads across a program, you will start to see things you never noticed before and took for granted. This opens up possibilities to see connections, relationships, and potential opportunities that were previously hidden.

What’s new?

Asking about what is new is a way to build on the answers from the first question. By looking at what is new, we start to see what might be elements of movement and change. It allows us to identify where things are shifting and where the ‘action’ might be within a program. Most of what we seek in social programs is change — improvements in something, reductions in something else — and sometimes these changes aren’t obvious. Sometimes they are so small that we can’t perceive them unless we pause and look and listen.

There are many evaluation methods that can detect change, however, asking the question about what’s new can help you to direct an evaluation toward the methods that are best suited to capturing this change clearly. Asking this question also amplifies your attentive capacity, which is enormously important for evaluation in detecting large and small changes (because often small changes can have big effects in complex systems like those in human services).

What does it mean?

This last question is about sensemaking. It’s about understanding the bigger significance of something in relation to your enterprise. There can be a lot happening and a lot changing within a program, but it might not mean a whole lot to the overall enterprise. Conversely, there can be little to nothing happening, which can be enormously important for an organization by demonstrating poor effects of an intervention or program or, in the case of prevention-based programs, show success.

This question also returns us to empathy and encourages some perspective-taking by getting us to consider what something means for a particular person or audience. A system (like an organization or program) looks different from where you sit in relation to it. Managers will have a different perspective than that of front-line staff, which is different for clients and customers, and different yet from funders or investors. The concept of ‘success’ or ‘failure’ is judged from the perspective of the viewer and a program may be wildly successful from one perspective (e.g., easy to administer for a manager) and a failure from another (e.g., relatively low return on investment from a funder’s point of view).

This question also affords an opportunity to get a little philosophical about the ‘big picture’. It allows program stakeholders to inquire about what the bigger ‘point’ of a program or service is. Many programs, once useful and effective, can lose their relevance over time due to new entrants to a market or environment, shifting conditions, or changes in the needs of the population served. By not asking this question, there is a risk that a program won’t realize it needs to adapt until it is too late.

By asking these three simple questions you can kick-start your evaluation and innovation work and better strengthen your capacity to think evaluatively.

Photo by Tim Foster on Unsplash

Dashboards for innovation evaluation

2017-09-11 by cense

A dashboard system is a way of tracking project activities across a range of different lines of inquiry. This is of particular benefit to those interested in developmental evaluation. Developmental evaluation (DE) is an approach to evaluating programs with an innovation focus and can be used as a means of generating data to support strategy development and deployment. That’s how we use DE.

Developmental evaluations often have multiple components to them that include a variety of methods, data sources, tasks and duties, and sensemaking requirements. All of this information can make a DE difficult for the client to gain a sense of what is happening and even confuse the evaluator at the same time. One way to manage this is to create something we refer to as a dashboard system.

Dashboards: Making strategy and evaluations visible

A dashboard is a means of showcasing multiple information streams in a simple, accessible manner that is easily understood. It might include notes from a meeting, data collection summaries, schedules of activities, and reflective notes. It is not a wholesale repository of all data, rather a higher-level summary that includes enough detail to get a sense of what activities are happening, when, and to what degree.

Among our favourite tools is Trello. Trello is organized into a series of ‘boards’ that allow you to organize content based on a topic or theme and create a simultaneous ‘stream’ of content within each board.

For example, a developmental evaluation might involve something like the following boards:

Interviews with key stakeholder groups
Observations
Document reviews
Sensemaking and strategy notes and outcomes
Meeting minutes
Survey or other data collection results
Reflective notes
Data collection plan

These boards would include any project component that has a distinct, coherent set of activities associated with it. It could include methods, data components, meetings and activities, populations or key document groups.

Tools for thinking and seeing

Trello is just a tool. What we’ve illustrated above could be done as channels within Slack or even as a series of sections within an MS Word or Google Doc. Even an Excel spreadsheet can work. Anything that allows someone to visually track details about discrete areas within a larger project.

(Although tools like MS Word or Excel will work, they are less than ideal as they require strict version controls to articulate changes and this can be difficult to manage, particularly if there are multiple contributors to the document on a regular basis. The reason one might choose MS Word or Excel is less about its functionality and more because it is what people are used to.)

What makes a DE a challenge is that there is a lot of data being generated from many different sources within a complex context. This makes the management of information a challenge and that poses problems for the task of generating helpful insights from the data generated. It also makes it problematic for identifying emergent patterns.

By making data visible, a dashboard allows multiple eyes and multiple perspectives to ‘see’ the data and thus, expands the possibilities associated with seeing possibilities and emergent conditions. The dashboard also increases accountability for all parties. For clients, a dashboard allows them insight into what the evaluator has access to on a regular basis and avoids any surprises. For evaluators, it ensures that clients have the means to engage in the process and don’t sit back waiting for insights to be ‘served up’ to them. DE is best done when it is a collaboration between the client and the evaluator and a dashboard can help this.

Lastly, a dashboard system provides a means of organizing content and keeping track of what is collected, identifying what is needed, and providing a quick-look means to explore what is known and unknown in a project.

Cense employs the dashboard approach on its projects whenever possible. If this approach is of interest to your organization and you want a means of developing strategy and evaluations in a more engaging, transparent manner connect with us and we can show you how this works in practice.

Engagement for ideas

2017-04-20 by cense

Engage everyone in idea creation

Among the biggest success factors in any organizational change initiative is engaging staff and stakeholders in participating in the change process; this is true for idea generation through the design cycle to implementation and evaluation. How do we overcome the challenge of disengagement to produce productive creative, innovative ideas in our organizations?

Beyond brainstorming

Ideas are usually the starting point for any change initiative. Ideas produce the raw content of problem identification and the seeds for solution generation. Brainstorming is often the means people first think of as a way to generate ideas and explore concepts. However, there are some substantial problems with this approach and some ways around it.

Brainstorming has been widely criticized for good reasons. Among them: it favours those voices who think (and speak) quickly, speak early, often and loudly. Those early suggestions drive the conversation for what comes next, creating a path dependency that’s hard to escape once initiated. If you’re a quieter, perhaps more contemplative person, you’ll find that you are either late to the conversation or not included. From Jungian Personality Theory, this indicates a bias towards extroversion over introversion, which excludes about 40 per cent of the population according to some estimates.

This approach also favours what Min Basadur would classify as an ‘generator’: someone who’s work style preferences favour idea generation. Those from the other quadrants in the Basadur Profile, particularly ‘conceptualizers’ and ‘optimizers’ (ones who’s work style preference leans toward processing and organizing information) are less likely to respond quickly.

Ideas in private and public

Another problem with many ideation strategies is that many really useful ideas are a bit heretical or outlandish. While design thinking writers often refer to the need to generate ‘wild ideas’ without criticism or judgement, the truth is that’s a lot harder to do in practice, particularly when there are power dynamics in the room, reputations, and the real fear that comes with change and challenges to established practices. The public nature of the ideation process favours transparency, but induces self-censorship and produces disengagement for many and hyper-engagement for a few.

A solution is to have those involved in the process generate ideas independently and anonymously contribute them via a suggestion box (digital or analog). Keep the suggestion box open for a defined period — we recommend no more than one week or as short as three days. This allows time for those who are more reflective to mull through their ideas and contribute them, while those who are more quick to generate ideas will not be affected.

These are then collated independently by someone neutral to the problem and solution set and organized thematically. This has the advantage of potentially embedding some of the wildest ideas within a small set of other ideas that might seem far less threatening. If you create a category of ‘wild ideas’ the risk is that the entire category will be dismissed. A more public, but elaborate, means of doing this can be found in the CoNEKTR Model described elsewhere.

This process can be repeated over the design cycle at different stages – anywhere diverse perspectives and feedback is needed.

Overcoming biases to better ideas

The bias toward ‘rapid ideation’ in design thinking systematically excludes people in favour of a well-meaning intention of trying to avoid participants ‘over-thinking’ a problem. While that might happen, it still prevents many from engaging in the process fully because of personality, work preferences, cognitive style and social pressures.

The strategy listed above is a simple, but highly effective means of getting lots of ideas and engaging your entire team in the process. Try it out. You might be surprised what ideas come from it all.

Attractor Mapping

2014-07-12 by cense

Attracting order in the organizational chaos

Attractor mapping is a method that we’ve been developing at CENSE for the past two years. It is a method that brings together a variety of different concepts in the service of assessing where energy is spent within an organization and network. This complexity-oriented method seeks to draw our attention to where emergent phenomenon is likely to be produced and to guide strategy and decision-making around these activities.

Attractor mapping operates much like a heat map, distinguishing areas of activity based on its intensity, which is usually measured by energy. Some examples of heat maps are below and indicate where concentrations of activity are located like cellphone coverage or carbon use. In both of these examples the activity is grafted on to specific areas on a map of physical locations, however it can easily be done with maps made of relationships (like social network maps), emotions or thoughts (like mind maps), or organizational roles (organization charts). Even other systems tools like system dynamic models and causal loop diagrams can be used as the foundation for building an attractor map.

In practice, an attractor map can be made using any map of activity of interest relevant to the issue at hand where complexity plays some role. Complexity simply refers to a set of multiple, overlapping, dynamic influences that are bound by context and time and produce differential effects based on these conditions. It means a simple system that is highly predictive, repetitive and controllable is not the best place for an attractor map. However, even something like a World Cup football game where there are many players operating simultaneously on a pitch can be a focus for an attractor.

Attractor maps are anticipatory in that they help organizations learn where things are happening now so they can pay attention and help anticipate or guide where things are going in the future. This method can be a foresight supporting tool as well as an asset for organizational sensemaking.

In the example of a World Cup match, it can be used to help assess where a play is likely to develop and what a team is paying attention to such as a player, a strategy (defence) or where a particular weakness is in the other team (e.g., less activity on one side or at one position).

If you want to learn more about attractor mapping, details will be presented in future posts. In the meantime, consider how paying attention to where the energy is being spent in your organization is clustered.

Photo credit: Light Chaos by Kevin Dooley used under Creative Commons Licence via Flickr. Check out Kevin’s photostream.

How is Your Cellphone Coverage? by Meg Stewart , Mapping NYC Clean Heat Map (both) by Carbon Visuals all used under Creative Commons License via Flickr.

Thanks to all for making and supporting Creative Commons.