Cense

Innovation to Impact

Design Skills for Evaluators: Design Loft 2017

by

The Design Loft @ AEA 2017

The Design Loft returns to the American Evaluation Association’s annual conference on Friday, November 10th. The Design Loft will be held on the Mezzanine Level of the conference hotel (note: an earlier program had the Design Loft located on the Exhibition Level). There is a new session every hour.

The Design Loft was launched in 2016 in connection with the Presidential Theme of the American Evaluation Association’s annual conference that focused on evaluation + design. The one-day ‘pop-up conference’ was a huge hit with attendees looking to learn more about design and pick up a skill or two. The Design Loft format is simple: every hour on the hour from 9 am to 5 pm (the last session starts at 4 pm) a different design skill is profiled in a 45-minute, hands-on workshop that allows attendees to meet others, laugh, and gain a practical ‘tool’ for their evaluation practice.

Program

The Design Loft is an experimental space intended to provide conference attendees an opportunity to learn specific tools, techniques, and strategies from the field of design that may have application to evaluation practice.

The Design Loft was first launched in 2016 as an experimental ‘pop-up’ conference within the main annual convention as a means of providing hands-on, practical opportunities for evaluators to learn about design methods and tools that can benefit their work and advance innovation within the field. This year the focus of the Design Loft will be applied to the conference theme of Learning to Action.

Each session runs for about 45 minutes, which will allow attendees to take an ‘active learning break’ between sessions during the conference program. It’s a quick hit of excitement, activity, and learning for action.

AEA Conference Program Description: The Design Loft @ AEA

What is design all about and what can it offer evaluators? The Design Loft is an interactive space where attendees can come and experience design firsthand as part of a series of engaging, short workshops aimed at exposing evaluators to simple tools, techniques, and approaches to using design in practice.

Cameron Norman Ph.D. MDes CE, Principal of Cense Ltd., a Credentialed Evaluator, 20-year AEA member and a professional designer, will guide participants through an interactive, hands-on training experience in a small group format.

The Design Loft is a space for creative thinking, innovation, practical learning, and fun and whether Design is new to you or you wish to broaden your toolkit, time hanging out in the Loft will add much to your evaluation work and budding practice as a designer.

Principles

Design is a structured approach to learning about a topic, identifying needs and developing solutions and responses to problems through the active construction of models and prototypes. Design thinking is the way of approaching this creative process that can be employed by designers and non-designers alike using some of the following principles:

  • Embrace whimsy
  • Show don’t tell (make things visible and tangible)
  • Bias for action
  • Culture of prototyping
  • Time constraints
  • Fail fast to succeed sooner
  • Moving from fail-safe to safe-fail
  • Systems thinking

Schedule of Events

The Design Loft runs Friday, November 10, 2017.

9:00 am:     User Personas: A persona is a tool in design that envisions a typical program user by constructing a profile of their behaviors, perspectives, and lifestyle relevant to the topic drawing on user research. These fictional characters are based on evidence and user-data collected by the design team and can help program designers and evaluators understand and anticipate the issues

10:00 am     Attractor Mapping: Where is the action happening and how we can understand where to focus our energy and evaluations when looking at a complex system? Where might we focus our design and evaluation efforts when so much is going on? This simple, visual approach to system mapping will show us how.

11:00 am     A Day in the Life: What does the typical user of a program go through in their day? How might the reality of a user’s day-to-day experience influence the design of a program and what might it mean for evaluators seeking to understand that experience and its relationship to program outcomes better? This session will show how a simple walk-through of a program using visual tools, acting out, and hypothesis generation might enable program planners, evaluators and collaborators to see new possibilities and insights.

12:00 pm     Journey Mapping: This method helps tell the story of a program user’s experience with a program by tracking the encounters that a person might go through along the program. This allows evaluators and designers to analyze the various touch-point an individual might have with a program and create the right kind of program and data collection opportunity. This allows the evaluator to see where problems and opportunities might lay before implementing a program or looking retrospectively at an existing program.

1:00 pm      5 Whys: This simple set of questions gets us to tap into our inner 5-year old and inquire about not just why something is happening on the surface but toward a more deeper understanding of the cause of a problem. By getting closer to the root of an issue, we are better equipped to design programs that make transformational shifts, not cosmetic ones and evaluations that have the power to transform people and programs alike.

2:00 pm      Role Playing: This physical form of problem exploration and prototyping literally has participants acting out specific actions or scenarios to gain insight into design opportunities, constraints, and challenges. This workshop will provide a perfect mid-afternoon break to get up and move and learn how a simple, imaginative approach to getting out of our head can yield insights and opportunities that will create programs that will resonate with our whole selves.

3:00 pm      Paper Prototyping: This ultra-low tech model of prototyping uses simple tools to construct mock-ups of envisioned products allowing for a quick, low-cost way to see opportunities, challenges and needs without resorting to expensive, time-consuming and potentially harmful full-scale prototypes. Working from an example, participants aid design and assess potential strategic options in a quick, low-cost and effective manner.

4:00 pm      Storyboarding: Movies and plays t and what is needed to make it come alive. We can take the same idea and apply that to evaluations. Visualizing an activity or program through simple drawings — no matter how simply done — can be an engaging way to gain insight into attitudes, beliefs, assumptions, and relationships between concepts, project components, and people. This technique will show you how a simple drawing can yield enormous information to guide a program design and the evaluation questions that follow from it.

Building a developmental mindset

by

Build your developmental mindset

Innovating in human systems requires more than ideas, resources, clever strategy or even new thinking; it requires a new mindset.

A mindset is a concept that psychologist Carol Dweck has explored in-depth and described a thinking orientation to the world. It’s not set in stone, but rather a lens through which we see our situations and the possibilities and challenges that they present to us. While Dweck has identified particular mindsets that shape how individuals use their brains, we’d like to explore another mindset here: a developmental one.

Over at Censemaking, Cameron Norman considered the role of the developmental mindset and how organizations engage with developmental evaluation. In that post, Norman argues that developmental evaluation is not possible without having the right mindset in place and we couldn’t agree more.

Witnessing evolution

The critical step toward a developmental mindset is considering viewing programs as a part of living systems. If a program involves a high variety of information and feedback, interacts with other programs and social forces, and exists in a space where there are ongoing disruptions and change, then it is what we would consider complex. If it is complex, then it fits the kind of scenario argued in Censemaking.

To take an evolutionary or developmental mindset requires first viewing programs as dynamic, rather than static. This perspective involves asking different evaluative questions. Static programs allow for questions like:

  • How effective is [the program] at achieving [specific target]?
  • How well does [the program] fit the industry-standard benchmark?
  • What is the impact of [the program] on [specific behaviour]?
  • How did [program component] contribute to [specific outcome]?

These questions allow for answers that have verifiable correlates with expected outcomes, a theory of change and industry benchmarks.

Dynamic programs require asking more sophisticated, nuanced questions that might produce less precise answers, but are more useful in helping a program to develop and evolve. These include:

  • What outcomes are generated through [the program] at this given moment?
  • How have the outputs and possible outcomes of [the program] differed over time?
  • How is [the program] interacting with its social system and what new behaviours and activities are generated through those interactions?
  • What new, beneficial attractors (clusters of organized activity) have established that are attributed in part to the involvement of [the program]?

Seeing opportunity in complexity

A review of the questions above might seem less satisfying in some ways, but they also reveal possibilities. It is why developmental evaluation (DE) is known as evaluation for innovation. DE provides the feedback necessary to understand the role(s) that a program is playing within a system and what comes from the interaction between people, programs, and systems. Thus, DE is a valuable part of the innovation process by allowing organizations to test out ideas and see what kind of influence they have in the world and make adaptations and modifications to it as the conditions, situations, and requirements evolve.

DE is about evaluating programs as part of living systems. This approach recognizes that programs exist in dynamic systems akin to the phrase ‘the river I stand in is not the river I step in’ from Heraclitus. This mindset recognizes that your market competition will change as you change and react to your actions, just as your organization might have to react to their actions.

DE doesn’t solve the problem, but it does allow for new opportunities to be discovered. By asking different questions to see things different, you can see these opportunities, too.

To learn more about developmental evaluation and how the right approach to generating feedback and learning about your programs can help your organization contact us for a consultation.

Image credit: “Mindset” by Steve Davis is licensed under CC BY 2.0