Cense

Research.Design.Impact.

Innovation’s Single Biggest Question

by

There is not a bigger question for innovators — social, product, service, policy — than What are you hiring this [ ] to do for you? 

Let’s break this down a little and then explain why we ask that at the beginning of any engagement and all throughout from the first meeting to the final run of the evaluation data. The question gets us to shape what and how we might design our innovation while the answer is about the ways we tell what kind of value it generates and the impact it produces.

What’s in a question?

The start of the question is about you, the aspiring innovator. This highlights the role of the creator and reminds us that we are generating this ‘thing’ by procurement, by design, or by simply encouraging something to be made. Without us (i.e., you), nothing changes.

The active use of hire is about the reality that we are paying for innovation through our time, our energy, our focus, our social (and often political) capital, and our money. All of these could be spent elsewhere. Design is an investment and it’s purposeful. Asking this big question gets us to pause and think deeply about what we’re putting into innovation and what we’re looking to get out of it.

The [ ] is the thing you’re hiring — the proposed service, experience, product, policy, or ecosystem — and is what you’re purposefully bringing about. This is your idea manifest into something real.

The last part is ‘to do for you‘ is active: it’s about ensuring that you’re clear about what purposes it serves. No matter how beneficial your planned innovation might be for others, you are ultimately asking it to serve a role, fill a need, for you. It is you that wants to solve a problem, build a market, or prevent something from happening, and this requires some clarity to innovate well.

What’s in an answer?

Innovation is not just creating things, it’s about evaluating the things we create. If our novel products, services, experiences, and policies don’t generate value for people, they aren’t really innovations. It’s just stuff.

An evaluation perspective on the question asked above might look at things like:

-The roles people played in the innovation process, including the skills they used, experiences they had, and the insights that they gained along the way. This learning is what feeds into our understanding of how an innovation develops along with the people and organization it is a part of. All of that is part of the innovation dividend or ROI.

-The resources used as part of the ‘hiring‘ process like money, time, human resources, and other capital; all can help look at the value of the initiative to see if the costs and benefits make sense.

-What ‘things’ are produced — the prototypes, their functioning, their benefits, and weakness — as well as provide a means to document the iterations, the steps taken, the new ‘offshoots’ that might emerge, and the resulting products, services, experiences, and policies.

-Lastly, the innovation needs to fulfill some requirements or expectations and evaluation looks at what it does in the world, for whom, under what conditions, and what other impacts might emerge unintentionally. This helps assess risks, benefits, and find new opportunities for further development and innovation.

Better questions, better answers

Innovation is what will drive much of the future value of your organization. It’s what allows you to build, grow, adapt, or sustain what you’re doing because even if you don’t feel a need to change, everything is changing around you and sometimes you need to change just to stay where you are.

By asking this one simple question you might find answers that will lead you to much better innovations to shape and create that future.

We help our clients ask this question. If you want our help, contact us and we’ll gladly help you ask better questions for better answers.

Evaluation As Part of An Innovation Value ROI

by

Follow us for a moment. We’re going to talk about design, innovation, evaluation, and how they all go together.

Design is really the discipline — the theory, science, and practice — of innovation. That means that if you are innovating, you’re designing. Innovating is about adding value through introducing something new to a situation — it might be entirely new, a twist on an existing idea, or an old idea placed in a new context. Innovation and design are about taking ideas and purposefully transforming things into making those ideas real in the form of services, products, or experiences.

Design and innovation are all about creating value.

Thus, understanding the value of design is partly about the understanding of valuation of innovation. At the root of evaluation is the concept of value. One of the most widely used definitions of evaluation (pdf) is that it is about merit, worth, and significance — with worth being a stand-in for value.

Understanding value

Value can only be understood by asking the right questions because it’s a relative question as many people will see the worth of something different from others.

One of the big questions professional designers wrestle with at the start of any engagement with a client is:

“What are you hiring [your product, service, or experience] to do?”

What evaluators ask is: “Did your [product, service, or experience (PSE)] do what you hired it to do?”

AND

“To what extent did your PSE do what you hired it to do?”

“Did your PSE operate as it was expected to?”

“What else did your PSE do that was unexpected?”

“What lessons can we learn from your PSE development that can inform other initiatives and build your capacity for innovation as an organization?”

In short, evaluation is about asking:

“What value does your PSE provide and for whom and under what context?”

Value creation, redefined

Without asking the questions above how do we know value was created at all? Without evaluation, there is no means of being able to claim that value was generated with a PSE, whether expectations were met, and whether what was designed was implemented at all.

By asking the questions about value and how we know more about it, innovators are better positioned to design PSE’s that are value-generating for their users, customers, clients, and communities as well as their organizations, shareholders, funders, and leaders.

This redefinition of value as an active concept gives the opportunity to see what the return on investment — time, money, energy, commitment — can yield an organization in real-time. This means that value is fluid, dynamic and that can be generated on an ongoing basis. It’s not just what you report at the end of the fiscal year or project.

Imagine reporting real-time value at your next stakeholder, staff, or shareholder meeting? Imagine knowing what you’re creating now and having that focus your efforts on what you could create in the near and long-term future?

Evaluation is how you do it. It’s in the name itself.

If you’re looking to hire an evaluation to better your innovation capacity, contact us at Cense. That’s what we do.

 

Living History for Developmental Evaluation

by

Developmental Evaluation (DE) is an approach to supporting the evolution and development of an innovation. This means not only helping guide where an innovation is going but understanding where it is now and where it has been. Just like human development, an innovation is as much a product of its past as it is the present decisions and that past can help inform the strategy for moving forward. But how do we properly account for the past and present context in understanding what comes next? This is where The Living History method comes in.

The Living History method comprises a set of data collection, sensemaking, and design strategies that come together to reflect on how a project comes into being, develops a focus, and evolves up to the present context. It helps provide the ‘backstory’ to the present situation and provides a means of developing a baseline – another key part of a DE.

Why does knowing this matter? Taking a Living History of a program helps surface the path dependencies of a program or innovation. Path dependencies can include those habits of mind that form, patterns of behaviour, and general routines that can develop within a complex environment and consciously or unconsciously shape the context in which decisions are made. By understanding what these patterns are, we can better prepare to address them as the program unfolds.

It’s helpful to the biases and mental models that we hold in guiding our decisions to better account for them both when things work well, but also to help understand how things didn’t work out.

To borrow from the image above: the Living History method helps you use the data collected from preparing the climb, starting up to base camp, and setting the groundwork (the trials, failures, and victories) to initiate the climb as well as the data from the actual ascent to the summit.

Introducing the method

The Living History method is something we developed to ‘backfill’ much of the information that is necessary to understand the present context of a program. Many evaluators do this kind of work with informational interviews, document reviews, and site visits, however, the means in which this data is collated and made sense of isn’t always systematic, nor is it done in a manner that recognizes complexity (which is the space in which DE is meant to be employed).

A Living History involves building that historical narrative that has been constructed by program developers (founders, staff, stakeholders, etc..) and the activities that contribute to this narrative. It draws together these ‘stories’ from the data into an understanding of the present. This present space is where the existing narrative and current construction of the program (‘what is happening now?‘) comes together. It’s also space where strategy is developed and initiated to shape what is to come (‘where are we going and how are we going to get there?‘).

Below is an image that demonstrates this process.

Taking a Living History

A Living History can encompass many different data gathering techniques. Four of the most useful are:

  1. Document review. A thorough review of foundational documents that might include things like the initial application(s) for funding, ‘pitch decks’ used to generate funds or interest, program and policy manuals, staff reviews, board meeting minutes, evaluations, and annual reports (to boards, funders, stakeholders, etc.). Some programs may have an early program logic model or Theory of Change that can be examined. These can give insight into the planning structure, theories, and reasons for establishing the program.
  2. Interviews. Connecting with the program founders, board members (past and present), investors (and funders), as well as leaders involved in the initiative is critical. Having conversations with these people will help surface both explicit and tacit knowledge held by those that helped shape the program. These interviews can be critical in understanding the logic, political context, and social dynamics that underpin the direction of the organization. It is useful to determine not only the direction taken, but the directions considered. The reason is that these ‘unexplored paths’ may linger in the minds of people and serve as a source of tension or inspiration in moving forward.
  3. Site visits. It is helpful to understand that programs – even those that are virtual such as a website or app — originate in the physical world somewhere. Site visits, when possible, help gather information about organizational culture, physical resources, and relational barriers and facilitators. Take the example of the organization that starts up as a maker space in a garage and then evolves to a large corporate high-rise office. Is the culture that was established at the start-up maintained at a different scale or abandoned? We’ve seen this with organizations that began as a start-up, grew enormously, and then abandoned the culture that was created at startup for a more conservative corporate one despite holding on to the belief that things had not changed. Only by pointing out (seeing) the manner in which things physically changed (e.g., open space vs. closed spaces, size and shape of offices) was this able to be pointed out. Further, such visits can help determine developmental milestones such as large investments of capital.
  4. Timeline. A timeline is a simple method of connecting activities — actions + decisions — together in a visual means. This helps determine and organize data about what was done, but also when it was done. This can make an enormous difference in understanding things like time lags and exploring cause-and-consequence connections between actions, reactions, and other effects.

The Role of Sensemaking

Living History is not complete without sensemaking. Sensemaking is required because, like most interesting stories, innovating through complexity is messy and full of contradictions, gaps in logic, and remarkable things that can only be understood by sitting together with all the data in one place. These emergent properties — coherence that forms from diverse sources of information as part of the process of (self) organization — are what make a program a ‘whole’ and not just the ‘sum of the parts’.

Sensemaking is collaborative, discursive, and involves debate, reflection, and a process of drawing conclusions and forming hypotheses based on data and experience. It is the process of taking the strangeness of seeing things in chunks and putting them together. What comes from this process, which is a key part of Developmental Evaluation and design, is a ‘sense’ of what happened, what is going on, and what options might work next.

It helps you understand where you and your program stand now.

Moving forward (and backward)

The Living History method doesn’t end with sensemaking, it also is carried forward.

Developing a project dashboard for innovation evaluation is a helpful means to pull together the data collected from the past, the present, and then populating it as the program evolves. The Living History method views history as being co-constructed as the program evolves and thus thrives on having new data about the present (which is always a second away from being the past) added to the overall corpus of data.

New data, when combined with what has already been collected, can provide new insights into the bigger picture. As data is added and time passes, it is possible that new patterns will emerge, which can be insightful for making decisions. This data, collected now and going forward, will later be recombined as part of an ongoing sensemaking process.

A Living History can also provide a means to document the activities that have taken place up to the present moment by systematically, although retrospectively, capturing (mostly existing) data to show what has happened before, which can be useful for organizations and programs who are starting their evaluations later than they would have liked to.

Consider the Living History approach with your evaluation. By looking at the past you might find yourself better prepared to go forward. We can help.

Photo by Aaron Benson on Unsplash

Three Questions for Evaluative Thinking

by

Evaluative thinking is at the heart of evaluation, yet it’s remarkably challenging to do in practice. To help strengthen those evaluative neural pathways, we offer some questions to aid you in developing your evaluative thinking skills.

To begin, let’s first look at this odd concept of ‘evaluative thinking’.

Tom Grayson’s recent post on the AEA 365 Blog looked at this topic more closely and provided a useful summary of some of the definitions of the term commonly in use. In its simplest term: evaluative thinking is what we do when we think about things from an evaluation perspective, which is to say, a point of view that considers the merit, worth, and significance of something.

Like many simple things, there is much complexity on the other side of this topic. While we have many methods and tools that can aid us in the process of doing an evaluation, engaging in the evaluative thinking supporting it is actually far more challenging. To help foster evaluative thinking we suggest asking three simple questions:

What is going on?

This question is about paying attention and doing so with an understanding of perspective. Asking this question gets you to focus on the many things that might be happening within a program and the context around it. It gets you to pay attention to the activities, actors, and relationships that exist between them by simple observation and listening. By asking this question you also can start to empathize with those engaged in the program.

Ask: 

What is going on for [ ] person?

What is going on in [ ] situation?

What is going on when I step back and look at it all together? 

Inquiring about what is going on enlists one of the evaluator’s most powerful assets: curiosity.

By starting to pay attention and question what is going on around you in the smallest and most mundane activities through to those common threads across a program, you will start to see things you never noticed before and took for granted. This opens up possibilities to see connections, relationships, and potential opportunities that were previously hidden.

What’s new?

Asking about what is new is a way to build on the answers from the first question. By looking at what is new, we start to see what might be elements of movement and change. It allows us to identify where things are shifting and where the ‘action’ might be within a program. Most of what we seek in social programs is change — improvements in something, reductions in something else — and sometimes these changes aren’t obvious. Sometimes they are so small that we can’t perceive them unless we pause and look and listen.

There are many evaluation methods that can detect change, however, asking the question about what’s new can help you to direct an evaluation toward the methods that are best suited to capturing this change clearly. Asking this question also amplifies your attentive capacity, which is enormously important for evaluation in detecting large and small changes (because often small changes can have big effects in complex systems like those in human services).

What does it mean?

This last question is about sensemaking. It’s about understanding the bigger significance of something in relation to your enterprise. There can be a lot happening and a lot changing within a program, but it might not mean a whole lot to the overall enterprise. Conversely, there can be little to nothing happening, which can be enormously important for an organization by demonstrating poor effects of an intervention or program or, in the case of prevention-based programs, show success.

This question also returns us to empathy and encourages some perspective-taking by getting us to consider what something means for a particular person or audience.  A system (like an organization or program) looks different from where you sit in relation to it. Managers will have a different perspective than that of front-line staff, which is different for clients and customers, and different yet from funders or investors. The concept of ‘success’ or ‘failure’ is judged from the perspective of the viewer and a program may be wildly successful from one perspective (e.g., easy to administer for a manager) and a failure from another (e.g., relatively low return on investment from a funder’s point of view).

This question also affords an opportunity to get a little philosophical about the ‘big picture’. It allows program stakeholders to inquire about what the bigger ‘point’ of a program or service is. Many programs, once useful and effective, can lose their relevance over time due to new entrants to a market or environment, shifting conditions, or changes in the needs of the population served. By not asking this question, there is a risk that a program won’t realize it needs to adapt until it is too late.

 

By asking these three simple questions you can kick-start your evaluation and innovation work and better strengthen your capacity to think evaluatively.

Photo by Tim Foster on Unsplash

Developmental Evaluation Trap #4: Organizational culture

by

In this latest post in a series on Developmental Evaluation (DE) and its traps, we look at the innovator’s trap of culture. In a quote widely attributed to Peter Drucker

Culture eats strategy for lunch

Culture is basically “the way things are done around here”, which includes the policies, practices, and people that make what happens, happen. Culture can be a trap when we fail to prepare for the change in how we do things that comes with adopting a developmental approach.

Just as we might fear success (discussed in an earlier post), we may also not prepare (or tolerate) it when it comes. Success with one goal means having to set new goals. It changes the goal posts. It also means that one needs to reframe what success means going ahead.

Many successful sports teams face the problem of reframing their mission after winning a championship: what it takes to win the first time is different when the motivation, context, and opportunities shift because of that victory. The same thing is true for organizations.

Building a culture of innovation

We’re not naïve enough to think that we can simply offer you a few bullet points and expect you to change your culture overnight. It is a big, important job. However, there is much research on behaviour change and organizational development that we’ve drawn on to help get you started.

  1. Manage deadlines differently. Deadlines drive much of our organizational behaviour and strategy, yet research has shown that deadline-driven work is less satisfying, that people’s decision-making gets impaired under the stress of urgency, and that priority isn’t always given to the more important tasks. Consider developmental milestones, which are similar to what we use in other areas of life. It’s a mindset that looks at tasks, accomplishments, and expectations in an evolving context with adaptive targets.
  2. Planning your stops as well as your starts. Developmental Evaluation can not only help you build out your innovation, but it can also determine what needs to stop. This means making hard choices about what kind of activities you might continue, what needs to be stopped, and what new things you wish to build out. The Tamarack Stop Start Continue tool provides a practical means to guide an organization through the process of making these strategic decisions. Deciding what you will stop is just as important as creating innovations.
  3. Measure performance compassionately. New research has emerged to show that intensive performance measurement can actually hamper the performance of an organization. Collecting too much data or making it onerous to do, not disclosing the purposes of the data, not using data collected, failing to engage those involved in the program in the design of the data collections, or neglecting to collect context-rich data that can help understand the mediators and moderators of performance are ways to alienate people from evaluation and limit its use. A strong DE approach seeks to engage users, making for more effective evaluation overall.
  4. Give data back. Show those involved in the program and the evaluations associated with it what the data reveals and what decisions are made from it. By making evaluation data transparent, you build trust, but also a culture of innovation by showing the purpose of the data and the ways in which it is used to make decisions. This kind of accounting to your users and stakeholders is part of the culture development for innovation.

By getting the right data, using it respectfully, and making it visible are ways in which the culture of an organization can change through developmental evaluation and lead to the kind of system shifts that allow great ideas to emerge and flourish.

Interested in learning more about what DE is and how implementing evaluation in this manner can support your innovation efforts? Contact us to find out more.