Innovators — those seeking to take an idea for a product, service, or policy and make it real — usually have a pretty clear sense of what they are trying to achieve with their innovation. This is the primary purpose and may reflect where the innovation achieves the greatest impact. But is this all it does? Could it be doing much more? 

Evaluation can play a key role in revealing where an innovation achieves more than just its primary purpose and can serve as a means to uncover layers of impact that can demonstrate various returns on investment (ROI) and open up opportunities for further exploration and exploitation of resources. 

Innovation typically involves considerable investment in time, energy, money, and attention and an evaluation can help showcase the return on investment in unexpected ways. Let’s consider something like an event — a learning conference — as an example to illustrate this layering of impact and how evaluation can aid in revealing these layers and supporting innovation. 

Change or Impact for Whom?

A look across many conferences finds relatively consistent language in their purpose that includes mission phrases like:

to inspire…, to educate…., to connect…, to showcase…, etc..

Among the first roles an evaluation can play is articulating the theory of change explaining why something is intended to achieve an outcome. A look at the phrases above is likely to prompt questions in an evaluator like: for whom are these to happen? Is the effect expected to be similar for most people? What are the means that people are using to showcase products (e.g., knowledge) and for the audience to engage with content?

These are some of the questions that an evaluator might ask from these initial goals. For an evaluation of a learning event, this might translate into metrics like: 

• Attendance including details on those in the audience (e.g., professional background, previous participation, basic demographics)
• Number of sessions taught and description of those sessions (content)
• Overall satisfaction with the conference (including session content, speakers, food and drink, format)
• Self-reported learning outcomes from participants
• Financial details: Profit, loss, and expenses; sponsorships
• Registration information (e.g., online vs. in-person, timing, categories, etc.)

This is a pretty standard set of metrics. We see similar evaluative outcomes across educational programming in different contexts. These might work well for simple purposes, but it only provides a small amount of what it could yield and for innovation, going beyond the usual is one way to separate a new idea from a successful one. 

Looking Differently at Outcomes & Impact

An outcome looks different from where we sit in the system that surrounds an innovation. Consider the role of the design of physical space and how that influences outcomes and shifts our understanding of impact. 

• 

• 

A common seating format at conferences is round tables in front of a stage (‘Rounds’). This is usually done where there are meals served and for that purpose, the format works well for everyone — except most of those in attendance and the keynote speakers. Rounds are ideal for serving people including the setting up and clearing of dishes. They are generally lousy for talking with people because, with the exception of speaking with one other person, an attendee must either turn their back on another person to speak to someone else or speak over or past someone. 

Rounds show high positive impact — efficiency, ability to monitor, reduced errors — for the catering staff. This might be an important outcome for a conference, although probably not. Consider how this format might enhance or degrade the impact of things like the keynote address or the networking expectations of individuals in attendance. 

Consider some other potential impacts:

• It gets people away from their families and friends (this could be positive or negative).
• It pulls them away from work (it gives them a break, provides an incentive, it adds to their workload, disrupts the teams they leave behind, or all or some of these).
• It provides supports conference centres and organizers.
• It creates connections between people like connecting with ‘old friends’ and colleagues; sustains a connection to a field of practice or discipline; re-affirms a mission; instills a sense of perspective at how much a field changes (or doesn’t) over time; instills a sense of alienation of a field or peers.  
• It generates or maintains employment in a region
• It’s a tourism and prestige generator. 

These are all possible outcomes and are a sample of what additional areas of impact that a learning event might influence. This is for illustrative purposes, but should still provide some ways to show how an innovation (service, product, or policy) might have additional outcomes and impacts that could emerge through an evaluation. 

Expanding the Field of Vision

Consideration of these additional outcomes might reveal an opportunity and can more fully demonstrate the impact and potential ROI. Asking different questions can also help prioritize what kind of outcomes make the most sense to optimize the design of your innovation. 

Conferences like the one pictured above have optimized for creative thinking within a traditional learning structure by including a poet who composed a unique work summarizing each talk, a (literal!) gallery walk showcasing a prominent local artist’s work (Lars Lerin), and hosting a series of interactive conversation sessions over coffee and snacks.  

For Service Convention Sweden and others like it the outcomes might include the standard ones and a deeper look at the new professional connections made (and followed up along with the material products generated from them), the application of the lessons learned, and the integration of learning into organizations by those participating sponsors. 

Evaluation is not just about the obvious outcomes when deployed in an innovation context. It can demonstrate not only whether an innovation is achieving the expected impact, but the reach of that impact outside of expectations.

And isn’t innovation all about exceeding expectations? 

This is the latest in a series on Evaluation: The Innovator’s Secret Advantage. This series looks at how evaluation can be used to support innovation in service design, product development, and policy implementation. For more on how to do this and any help with it, contact us at www.cense.ca/contact.