Cense

Innovation to Impact

After-Action Review: Learning Together Through Complexity

by

Complexity science is the study of how systems behave when under conditions of high dynamism (change) and instability due to the number, sequencing, and organization of actors, relationships, and outcomes. Complex systems pose difficulty drawing clear lessons because the relationship between causes and consequences are rarely straightforward. To illustrate, consider how having one child provides only loose guidance on how to parent a second, third or fourth child: there’s no template.

An After-Action Review is one such way to learn from actions you take in a complex system to help shape what you do in the future and provide guidance on what steps should be taken next.

The After-Action Review (AAR) is a method of sensemaking and supporting organizational learning through shared narratives and group reflection once an action has been taken on a specific project aimed at producing a particular outcome — regardless of what happened. The method has been widely used in the US Military and has since been applied to many sectors. Too often our retrospective reviews happen only when things fail, but through examining any outcome we can better learn what works, when, how, and when our efforts produce certain outcomes.

Here’s what it is and how to use it.

Learning Together

An AAR is a social process aimed at illuminating causal connections between actions and outcomes. It is not about developing best practice, rather it is to create a shared narrative of a process from many different perspectives. It recognizes that we may engage in a shared event, but our experience and perceptions of that event might be different and within these differences lie the foundation for learning.

To do this well, you will need to have a facilitator and a note-taker. This also must be done in an environment that allows individuals to speak freely, frankly, and without any fear of negative reprisals — which is a culture that must be cultivated early and ahead of time. The aim isn’t to point blame, but to learn. The facilitator can be within the team or outside the team. The US Military has a process where the teams undertake their own self-facilitated AAR’s.

To begin, gather those individuals directly involved in a project together in close proximity to the ‘end’ (e.g., launch, delivery of the product, etc..). As a group, reflect on the following three question sets:

  1. The Objectives & Outcomes:
    • What was supposed to happen?
    • What actually happened?
    • Practice Notes: Notice whether there were discrepancies between perceptions of the objective in the first place and where there are differences in what people pay attention to, what value they ascribe to that activity (see below), and how events were sequenced.
  2. Positive, Negative, and Neutral Events:
    • What created benefit / what ‘worked’ ?
    • What created problems / what ‘didn’t ‘work’?
    • Practice notes: The reason for putting ‘work’ in quotes is that there may not be a clear line between the activities and outcomes or a pre-determined sense of what is expected and to what degree, particularly with innovations where there isn’t a best practice or benchmark. Note how people may differ on their view of what a success or failure might be.
  3. Future Steps:
    • What might we do next time?
    • Practice notes: This is where a good note-taker is helpful as it allows you to record what happens in the discussion and recommendations. The process should end with a commitment to bringing these lessons together to inform strategic actions going forward next time something similar is undertaken.

Implementing the Method

Building AAR’s into your organization will help foster a culture of learning if done with care, respect and a commitment to non-judgemental hearing and accepting of what is discussed in these gatherings.

The length of an AAR can be anywhere from a half-hour to a full-day (or more) depending on the topic, context, and scale of the project.

An AAR is something that is to be done without a preconceived assessment of what the outcome of the event is. This means suspending the judgement about whether the outcome was a ‘success’ or ‘failure’ until after the AAR is completed. What often happens is successes and ‘wins’ are found in even the most difficult situations while areas of improvement or threats can be uncovered even when everything appeared to work (see the NASA case studies with the Challenger and Columbia space shuttle disasters).

Implementing an AAR every time your team does something significant as part of its operations can help you create a culture of learning and trust in your organization and draw out far more value from your innovation if you implement this regularly.

If your team is looking to improve its learning and create more value from your innovation investments, contact us and we can support you in building AAR’s into your organization and learn more from complexity.

Using Theories for Change

by

The concept of Theory of Change is meant to provide program planners and evaluators with guidance on how to make sense of the mechanisms that guide how something transforms. Theory of Change as a technique is usually visual, participatory and consultative in nature, and is something that is developed alongside the program itself. What is given less attention are the change theories that underpin a Theory of Change.

Confused? You’re not alone.

Clarifying this is critical if your Theory of Change is to have any meaning.

Change Theories & Theory of Change

Change theories are based (largely) on psychological and sociological evidence applied to human behaviour at different levels. These levels include:

  • Individuals
  • Groups (e.g., teams, families)
  • Organizations
  • Communities
  • Societies & Systems

Some change theories will apply at all of these levels, while some are designed more specifically for a specific level. For example, Kotter’s 8-step model for leading change is primarily an organizational change theory.

Change theories are meant to describe what is to change and explain how change is to come about. These serve as the bedrock for what a Theory of Change is meant to convey. A Theory of Change links the structures and resources tied to a specific program, unit, or process with various change theories to explain why it should facilitate transformation.

Design Considerations

While we might have a viable change theory, we might not have a strong design. We often see organizations that seek to make changes that their programs or policies were not designed to accomplish. For example, cognitive rational change theories are built upon the basic assumption that knowledge informs attitudes and beliefs which influence behaviour.

If your program or service doesn’t have a design that facilitates information accessibility that allows your end user (those who are the focus of your service) to understand and use that information, it’s unlikely we will see change. Just-in-time knowledge delivery (e.g., doing a Google search) implies that people have the means (e.g., tools and technology), the literacy, the skills, and the opportunity to access and use that knowledge, otherwise it’s not likely to facilitate change.

Being able to locate a family doctor isn’t useful if you can only do it at a time and place when such a professional isn’t needed.

Theories of Change can help us plan our programs and service offerings and plot the points of impact, but without good change theories and design considerations it’s quite possible we won’t achieve what we set out to do.

Want to learn more about how to develop Theories of Change and what an understanding of social and behavioural science and design can do to help you learn and create impactful programs? Contact us. We’d love to connect.

Expanding Perception Through Evaluation

by

Creating an Innovation Panopticon?

One of Evaluation’s greatest contributions to innovation is its ability to expand the vision of the innovator and attend to matters of perception. What we look at it is not always all we consciously see, nor is what we see the sum of all we perceive. Humans have many conscious and unconscious biases in what they perceive and how they interpret what they perceive. Evaluation can help add clarity to that perception and expand it. 

Evaluation can help distinguish reality from illusion. In this latest in our series on Evaluation: The Innovator’s Secret Advantage we look at the secret of perception and how evaluation can help change what is seen (and reveal what is not) in the journey of innovation. 

Seeing More

The image above is of a jail modeled on the Panopticon, an ancient architectural design to allow someone to see all that is going on at any one time. While its use within prisons is not one that’s particularly comforting or attractive, the model or concept of creating a design that enables a vantage point to see what is happening within an innovation context is a useful metaphor. 

Evaluation can enable innovators to see what is happening from a single vantage point at considerable depth or, as we will discuss, provide an alternative means by seeing the phenomenon from multiple perspectives. Both of these have advantages for innovators looking to assess the performance of a product or service in areas of high complexity and uncertainty. 

It is by reflecting on the vantage point (perspective) that we can determine whether what we are seeing is real or just an illusion and take the appropriate action based on that assessment. 

Seeing Differently

Consider one of the most famous of these visual illusions: the Young Girl / Old Woman Illusion (below). The history of this illusion has been traced back to the late 19th century and has been replicated many times since then to be considered a staple of any introductory psychology text. 

Young Girl/Old Woman Illusion

What’s interesting about this illusion is that you can’t hold both the image of the young girl and the old woman in your mind’s eye at the same time. You can only see one, not both even if you can switch back and forth. The same situation can occur within an innovation where it is difficult to see two ideas at the same time. By asking the right questions, an evaluation can help innovators to see what others see, what they default to, and whether or under what conditions do they see something different (e.g., seeing something as a product or a service). 

Other illusions are more about perceptual shifts that, once made, are difficult or impossible to ‘unlearn’. These can be found in such things as the FedEx logo and its embedded arrow (check the link if you’ve never seen it). Another is the often common hidden ‘8’ within the 8 of diamonds in a deck of cards (see below). 

Eight of diamonds playing card, isolated on white background.

Evaluation for innovation involves asking questions about a program, product, or service that includes taking a perceptual view of many different people and from different situations. It’s a systems-oriented perspective that considers what a particular phenomenon looks like from a particular point of view. 

Seeing What’s Hidden in Plain Sight

Another famous psychological illusion is the one conceived of by Daniel Simons and his colleagues called The Monkey Business Illusion. This selective perception task is designed to show how your attention and focus can blind you to other things going on. The video below provides an updated version of the original video (available via the link above) to illustrate how we can miss things that are right in front of us based on what we attend to. 

The Monkey Business Illusion

Innovators are often focused on the core aspects of their innovation: the product, the process, and the intended outcomes. This focus is often what sets successful innovators apart from others, yet it can also be a liability.  An evaluator can help an innovator (an individual, a team, an entire organization) to see the full picture. 

How? The means to do this is first achieved by asking evaluative questions that look at what is going on, what is new, and what meaning is derived from various activities. 

Taking the Temperature (and Other Innovation Measures)

There is a useful, if not morbid, fable about the frog in the boiling pot. In that fable, the frog fails to notice that the temperature of the water continues to rise because the change is so gradual. The frog ends up boiling to death because of this inability to see that he is being boiled alive because the changes are so hard to detect. A good evaluation designed for innovation provides means to collect data akin to a thermostat calibrated to the particular conditions, situations, and product or service necessary. 

This could include monitoring key inputs and outputs, tracking sales or engagements over time, or looking at resource levels like personnel and how they respond to change. The latter example is a good one, particularly with larger organizations where the duties assigned to a single person could be absorbed by others if that individual left the role. However, as time moves on and the extra work is normalized it is possible that the pattern is repeated over time where those who are left eventually shoulder such a burden that it makes the work impossible, but also the performance degrades. 

These are the kinds of situations where collapse is likely. We’ve seen this with leadership when many people start leaving at the top or when those at the front face of an organize leave in large numbers. The cost in capital, focus, lived experience, and working knowledge can cripple an organization over the long haul. But like the frog in the pot, it might not be until just before the boil that it is noticed. 

Evaluating for Illusion & Reality

Evaluation for innovation recognizes that the attention and focus of those leading the charge is precious and that leaders are likely to miss key things without the right tools to draw their attention to the whole system and the parts. Evaluation can be hired to cover some of these perceptual gaps and illuminate things like the present temperature or noticing gorillas, patterns in data like the 8 of diamonds or discerning the best moment to see a young girl or an old woman. 

In the next piece in this series, we’ll look at how evaluation can uncover layers of impact that go beyond seeing what is in front of us to looking far past it. If evaluation for innovation is something that you need help with, connect with us; we can help you see things differently. 

Title image credit: Cameron Norman

Developmental Evaluation Trap #4: Organizational culture

by

In this latest post in a series on Developmental Evaluation (DE) and its traps, we look at the innovator’s trap of culture. In a quote widely attributed to Peter Drucker

Culture eats strategy for lunch

Culture is basically “the way things are done around here”, which includes the policies, practices, and people that make what happens, happen. Culture can be a trap when we fail to prepare for the change in how we do things that comes with adopting a developmental approach.

Just as we might fear success (discussed in an earlier post), we may also not prepare (or tolerate) it when it comes. Success with one goal means having to set new goals. It changes the goal posts. It also means that one needs to reframe what success means going ahead.

Many successful sports teams face the problem of reframing their mission after winning a championship: what it takes to win the first time is different when the motivation, context, and opportunities shift because of that victory. The same thing is true for organizations.

Building a culture of innovation

We’re not naïve enough to think that we can simply offer you a few bullet points and expect you to change your culture overnight. It is a big, important job. However, there is much research on behaviour change and organizational development that we’ve drawn on to help get you started.

  1. Manage deadlines differently. Deadlines drive much of our organizational behaviour and strategy, yet research has shown that deadline-driven work is less satisfying, that people’s decision-making gets impaired under the stress of urgency, and that priority isn’t always given to the more important tasks. Consider developmental milestones, which are similar to what we use in other areas of life. It’s a mindset that looks at tasks, accomplishments, and expectations in an evolving context with adaptive targets.
  2. Planning your stops as well as your starts. Developmental Evaluation can not only help you build out your innovation, but it can also determine what needs to stop. This means making hard choices about what kind of activities you might continue, what needs to be stopped, and what new things you wish to build out. The Tamarack Stop Start Continue tool provides a practical means to guide an organization through the process of making these strategic decisions. Deciding what you will stop is just as important as creating innovations.
  3. Measure performance compassionately. New research has emerged to show that intensive performance measurement can actually hamper the performance of an organization. Collecting too much data or making it onerous to do, not disclosing the purposes of the data, not using data collected, failing to engage those involved in the program in the design of the data collections, or neglecting to collect context-rich data that can help understand the mediators and moderators of performance are ways to alienate people from evaluation and limit its use. A strong DE approach seeks to engage users, making for more effective evaluation overall.
  4. Give data back. Show those involved in the program and the evaluations associated with it what the data reveals and what decisions are made from it. By making evaluation data transparent, you build trust, but also a culture of innovation by showing the purpose of the data and the ways in which it is used to make decisions. This kind of accounting to your users and stakeholders is part of the culture development for innovation.

By getting the right data, using it respectfully, and making it visible are ways in which the culture of an organization can change through developmental evaluation and lead to the kind of system shifts that allow great ideas to emerge and flourish.

Interested in learning more about what DE is and how implementing evaluation in this manner can support your innovation efforts? Contact us to find out more.

Developmental Evaluation Trap #3: Fearing Success

by

What if you tried to innovate and succeeded? This fourth in a series of posts looking at Developmental Evaluation traps explores that question and the implications that come from being good at innovating.

A truly successful innovation changes things — mindsets, workflows, systems, and outcomes. Some of these changes are foreseeable, some are not and they are rarely ever uniformly positive. Take the strange situation that many non-profits face: most will put themselves out of business if they truly succeed in their mission.

Or consider the corporate manager who is looking to get her or his staff engaged. The benefits of staff engagement are many, including contributing more to the organization, but that also means they will ask more of the organization and require more.

The trap with Developmental Evaluation (DE) is that it provides a window into the organizations’ innovation strategy and disrupts the way in which it learns, adapts, and designs its products. It tests assumptions, meaning that some myths that an organization holds might be challenged. DE emplores organizations to use evidence to support decisions, not just precedent.

DE also provides a means to account for the work that is performed in the quest to innovate (not just what is produced), which sounds like an enormous perk for those who are used to being judged solely on production, unless you really didn’t want to put the work in and are content to simply produce the most marginal of products.

What fear looks like

The most problematic form of fear is self-sabotage.  Self-sabotage refers to the behaviours we engage in — individually or as a group or organization — that keep us from taking the actions we believe will lead to success. The reasons might have to do with social norms. If what you’re doing is normal, then it probably isn’t innovation. Innovating means deviating from what others are doing.

Risk is another fear-factor. Doing something different requires risk-taking and that means holding yourself up to possible criticism, disapproval from others, or financial costs (among others).

Further complicating things is that risk-taking and social behaviour are often linked together.

Fear also manifests in actions not taken. It may manifest in not pursuing opportunities or pursuing only those that are safe. It may be ignoring problems or asking the kind of questions that focus attention away from what is important, to what is easy. Often we see organizations seek to collect lots of data on something unimportant because it’s safer than collecting data on something that matters but is much riskier.

Fear can further manifest in things like over-researching a topic (never having ‘enough’ data’), deferring decisions, blaming the timing (“It’s just not the right time to innovate”), or complexity (“it’s too complex”).

Practical fear-fighting

How do guard against your own fear of success? There are some things we can do to better utilize lessons gained through approaches like DE to assist innovation efforts.

  1. Innovation therapy. Mental health professionals often go through and remain in some form of professional therapy or guided reflection to enable them to attune to their biases, work through their personal issues, and hone their skills with another professional. The same process of ongoing, guided, reflective practice and ‘treatment’ is beneficial to organizations as well because it creates that same social norm to talk about fears, challenges, and ambiguities. Create space for reflection, discussion, and problem-solving in your regular meetings. Make the time for innovation.
  2. Create outcomes from processes. DE is a great tool for showcasing the work that goes into an innovation, not just the innovation itself. Regularly collect the data that goes into the process — ideas generated (quality and quantity), rough sketches, prototypes, and hours worked, to name a few suggestions. These can help show what goes into a product, which is an end in itself. This makes something that seems abstract more tangible.
  3. Make change visible. Make work visible through evaluation and visual thinking – including the ups, downs, sideways and showcase where along the journey you are along a process (e.g., use a timeline)
  4. Create better systems, not just different behaviour. Complex systems have path-dependencies — those ruts that shape our actions, often unconsciously and out of habit. Consider ways you organize yourself, your organization’s jobs and roles, the income streams, the system of rewards and recognition, the feedback and learning you engage with, and composition of your team.  This rethinking and reorganization are what changes DNA, otherwise, it will continue to express itself through your organization in the same way.

These are all part of the developmental evaluator’s toolkit, which is basically a core part of innovation itself. DE can be a means to help you confront your fears, not deny them. Try these out and see how you can avoid the trap of self-sabotage and to become better innovators and learners.

Want to learn more about how to do DE or how to bring it to your innovation efforts? Contact us and we’d be happy to help.