Innovation @ 50x in Companies and Government Agencies

Posted on August 21, 2015 by steve blank

I’ve spent this year working with corporations and government agencies trying to adapt and adopt Lean Methodologies.  In doing so I’ve learned a ton from lots of people. I’ve summarized my learnings in this blog post, and here and here and here and put it all together in the presentation below.

if you can’t see the presentation click here.

But the biggest surprise for me was getting schooled on how extremely difficult it is to be an innovator inside a company of executors.  More on that in the next post.

Graphic recording - SteveBlank - Innovation at 50x - Trent Wakenight OGSystems 20150814

Filed under: Corporate/Gov't Innovation, Customer Development, Investment Readiness Level, Lean LaunchPad | 5 Comments »

I-Corps at the NIH: Evidence-based Translational Medicine

Posted on December 17, 2014 by steve blank
If you’ve received this post in an email the embedded videos and powerpoint are best viewed on www.steveblank.com

We have learned a remarkable process that allow us to be highly focused, and we have learned a tool of trade we can now repeat. This has been of tremendous value to us.

Andrew Norris, Principal Investigator BCN Biosciences

Over the last three years the National Science Foundation I-Corps has taught over 700 teams of scientists how to commercialize their technology and how to fail less, increasing their odds for commercial success.

To see if this same curriculum would work for therapeutics, diagnostics, medical devices and digital health, we taught 26 teams at UCSF a life science version of the NSF curriculum. 110 researchers and clinicians, and Principal Investigators got out of the lab and hospital, and talked to 2,355 customers. (Details here)

For the last 10 weeks 19 teams in therapeutics, diagnostics and medical devices from the National Institutes of Health (from four of the largest institutes; NCI, NHBLI, NINDS, and NCATS) have gone through the I-Corps at NIH.

87 researchers and clinicians spoke to 2,120 customers, tested 695 hypotheses and pivoted 215 times. Every team spoke to over 100 customers.

Three Big Questions
The NIH teams weren’t just teams with ideas, they were fully formed companies with CEO’s and Principal Investigators who already had received a $150,000 grant from the NIH. With that SBIR-Phase 1 funding the teams were trying to establish the technical merit, feasibility, and commercial potential of their technology. Many will apply for a Phase II grant of up to $1 million to continue their R&D efforts.

Going into the class we had three questions:

  1. Could companies who were already pursuing a business model be convinced to revisit their key commercialization hypotheses – and iterate and pivot if needed?
  2. Was getting the Principal Investigators and CEO out of the building more effective than the traditional NIH model of bringing in outside consultants to do commercialization planning?
  3. Would our style of being relentlessly direct with senior scientists, who hadn’t had their work questioned in this fashion since their PhD orals, work with the NIH teams?

Evidence-based Translational Medicine
We’ve learned that information from 100 customers is just at the edge of having sufficient data to validate/invalidate a company’s business model hypotheses. As for whether you can/should push scientists past their comfort zone, the evidence is clear – there is no other program that gets teams anywhere close to talking to 100 customers. The reason? For entrepreneurs to get out of the building at this speed and scale is an unnatural act. It’s hard, there are lots of other demands on their time, etc. But we push and cajole hard, (our phrase is we’re relentlessly direct,) knowing that while they might find it uncomfortable the first three days of the class, they come out thanking us.

The experience is demanding but time and again we have seen I-Corps teams transform their business assumptions. This direct interaction with potential users and customers is essential to commercialize science (whether to license the technology or launch a startup.) This process can’t be outsourced. These teams saved years and millions of dollars for themselves, the NIH and the U.S. taxpayer. Evidence is now in-hand that with I-Corps@NIH the NIH has the most effective program for commercializing science.

Lessons Learned Day
Every week of this 10 week class, teams present a summary of what they learned from their customers interviews. For the final presentation each team created a two minute video about their 10-week journey and a 8-minute PowerPoint presentation to tell us where they started, what they learned, how they learned it, and where they’re going. This “Lessons Learned” presentation is much different than a traditional demo day. It gives us a sense of the learning, velocity and trajectory of the teams, rather than a demo day showing us how smart they are at a single point in time.

BCN Biosciences
This video from team BCN Biosciences describes what the intensity, urgency, velocity and trajectory of an I-Corps team felt like. Like a startup it’s relentless.

BCN is developing a drug that increases anti-cancer effect of radiation in lung cancer (and/or reduces normal tissue damage by at least 40%). They were certain their customers were Radiation Oncologists, that MOA data was needed, that they needed to have Phase 1 trial data to license their product, and needed >$5 million and 6 years. After 10 weeks and 100 interviews, they learned that these hypotheses were wrong.

If you can’t see the BCN Biosciences video click here

The I-Corps experience helped the BCN Bioscience team develop an entirely new set set of business model hypotheses – this time validated by customers and partners. The “money slides” for BCN Biosciences are slides 22 and 23.

If you can’t see the BCN Biosciences presentation click here

You Can’t Outsource Customer Discovery
What we hear time and again from the Principal Investigators is “I never would have known this” or “I wouldn’t have understood it if I hadn’t heard it myself.” Up until now the NIH model of commercialization treated a Principal Investigator as someone who can’t be bothered to get out of the building (let alone insist that it’s part of their job in commercialization.) In the 21st century using proxies to get out of the building is like using barbers as surgeons.

Clinacuity
While the Clinacuity video sounds like an ad for customer discovery, listen to what they said then look at their slides. This team really learned outside the building.


If you can’t see the Clinacuity video click here

Clinacuity’s technology automatically extracts data in real-time from clinical notes, (the narrative text documents in a Electronic Health Record,) and provides a summary in real time. Their diagrams of the healthcare customer segment in slides 15-18 were outstanding.

If you can’t see the Clinacuity presentation click here

GigaGen
The GigaGen team – making recombinant gamma globulin – holds the record for customer discovery – 163 customer interviews on multiple continents.

If you can’t see the GigaGen video click here

GigaGen’s learning on customer value proposition and who were the real stakeholders was a revelation. Their next-to-last slide on Activities, Resouces and Partners put the pieces together.

If you can’t see the GigaGen presentation click here

Affinity Therapeutics
Affinity came into class with a drug coated Arterial Venous Graft – graft narrowing is a big problem.

One of things we tell all the teams is that we’re not going to critique their clinical or biological hypotheses. Yet we know that by getting out of the building their interaction with customers might do just that. That’s what happened to Affinity.

If you can’t see the Affinity video click here

Affinity was a great example of a team that pivoted their MVP. They realized they might have a completely new product – Vascular wraps that can reduce graft infection.  See slides 17-23.

If you can’t see the Affinity presentation click here

Haro
Haro is making a drug for the treatment of high risk neuroblastoma, the most common extracranial cancer in infancy and childhood. On day 1 of the class I told the team, “Your presentation is different from the others – and not in a good way.”  That’s not how I described them in the final presentation.

If you can’t see the Haro video click here

After 120 interviews the Haro found that there are oncology organizations (NCI-funded clinical development partners) that will take Haro’s compound and develop it at their own expense and take it all the way into the clinic. This will save Haro tens of millions of dollars in development cost.  See slides 12 and 13.

If you can’t see the Haro presentation click here

Cardiax
Caridax is developing a neural stimulator to treat atrial fibrillation. Their video points out some of the common pitfalls in customer discovery. Great summary from Mark Bates, the Principal Investigator: “You don’t know what you don’t know. Scientific discovery is different than innovation. You as a prospective entrepreneur need this type of systematic vetting and analysis to know the difference.”

If you can’t see the Cardiax video click here

After 80 interviews they realized they were jumping to conclusions and imparting their bias into the process. Take a look at slides 8-11 and see their course correction.

If you can’t see the Cardiax presentation click here

The other 15 presentations were equally impressive. Each and every team stood up and delivered. And in ways that surprised themselves.

The Lean Startup approach (hypotheses testing outside the building,) was the first time clinicians and researchers understood that talking to customers didn’t require sales, marketing or an MBA – that they themselves could do a pretty good first pass. I-Corps at NIH just gave us more evidence that’s true.

The team videos and slides are on SlideShare here.

A Team Effort
This blog post may make it sound like there was no one else in the room but me and the teams. But nothing could be farther from the truth. The I-Corps@NIH teaching team was led by Edmund Pendleton. Allan May/Jonathan Fay taught medical devices, John Blaho/Bob Storey taught diagnostics and Karl Handelsman/Keith McGreggor taught therapeutics. Andre Marquis, Frank Rimalovski and Dean Chang provided additional expertise. Brandy Nagel was our tireless teaching assistant. Jerry Engel is the NSF I-Corps faculty director.

Special thanks to Paul Yock of Stanford Biodesign and Alexander Osterwalder for flying across the country/world to be part of the teaching team.

I created the I-Corps/Lean LaunchPad® syllabus/curriculum, and with guidance from Allan May, Karl Handelsman Abhas Gupta and Todd Morrill adapted it for Life Sciences/Health Care/Digital Health. The team from VentureWell provided the logistical support. The I-Corps program is run by the National Science Foundation (Babu Dasgupta, Don Millard and Anita LaSalle.) And of course none of this would be possible without the tremendous and enthusiastic support and encouragement of Michael Weingarten the director of the NIH/NCI SBIR program and his team.

Lessons Learned

  • The I-Corps/Lean LaunchPad curriculum works for therapeutics, diagnostics and device teams
  • Talking to 100 customers not only affected teams’ commercial hypotheses but also their biological and clinical assumptions
  • These teams saved years and millions of dollars for themselves, the NIH and the U.S. taxpayer
  • Evidence is now in-hand that the NIH has the most effective program for commercializing science
  • In the 21st century using proxies to get out of the building is like using barbers as surgeons

Filed under: Customer Development, Investment Readiness Level, Lean LaunchPad, Life Sciences (NIH), NIH (National Institutes of Health), Science and Industrial Policy, Teaching | 2 Comments »

How Investors Make Better Decisions: The Investment Readiness Level

Posted on July 1, 2014 by steve blank

Investors sitting through Incubator or Accelerator demo days have three metrics to judge fledgling startups – 1) great looking product demos, 2) compelling PowerPoint slides, and 3) a world-class team.  Other than “I’ll know it when I see it”, there’s no formal way for an investor to assess project maturity or quantify risks. Other than measuring engineering progress, there’s no standard language to communicate progress.

What’s been missing for everyone is:

  1. a common language for investors to communicate objectives to startups
  2. a language corporate innovation groups can use to communicate to business units and finance
  3. data that investors, accelerators and incubators can use to inform selection

Teams can prove their competence and validate their ideas by showing investors evidence that there’s a repeatable and scalable business model. While it doesn’t eliminate great investor judgment, pattern recognition skills and mentoring, we’ve developed an Investment Readiness Level tool that fills in these missing pieces. Background about the Investment Readiness Level  here and here.

While the posts were theory I was a bit surprised when John Selep, an early-stage investor, approached me and said he was actually using the Investment Readiness Level (IRL) in practice.

Here’s John’s story.

—

As Selections Committee chair for our Sacramento Angels investor group, I review applications from dozens of startup entrepreneurs looking for investment.  I also mentor at our local university, and guest-lecture at a number of Entrepreneurship courses on how to pitch to investors, so the task of helping students and entrepreneurs visualize the process of investor decision-making has often been a challenge.

When I first read about the Investment Readiness Level (IRL) on Steve’s blog, I was excited by Steve’s attempt to bridge the capital-efficient Lean Startup process for founders with the capital-raising process for funders. But the ‘ah-hah!’ moment for me was the realization that I could apply the IRL framework to dramatically improve the guidance and mentorship I was providing to startup company founders .

Prior to having the Investment Readiness Level framework, this “how to get ready for an investor” discussion had been a “soft” conceptual discussion. The Investment Readiness Level makes the stages of development for the business very tangible. Achieving company milestones associated with the next level on the Investment Readiness Level framework is directly relevant to the capital-raising process.

I use the Investment Readiness Level as part of my sessions to help the students understand that being ready for investment means that besides having a pretty PowerPoint, they need to do real work and show Customer Development progress.

Since I began incorporating the Investment Readiness Level framework I’ve made three observations. The Investment Readiness Level (IRL):

  1. Ties the Lean methodology (and capital efficiency) directly to the capital-raising process – closing the loop and tying these two processes together.
  2. Is Prescriptive – offers founders a “what-you-need-to-do-next” framework to reach a higher level of readiness.
  3. Enables better mentoring. The IRL provides a vocabulary and framework for shifting the conversation between investors and entrepreneurs from simply “No”, to the much-more-helpful “Not yet – but here’s what you can do…”.

Selep IRLTying Fundraising to the Lean Startup
The premise of the Lean Startup is that a startup’s initial vision is really just a series of untested hypotheses, and that the Customer Development process is a systematic approach to ‘getting out of the building’ and testing and validating each of those hypotheses to discover a repeatable, scalable business model. The Investment Readiness Level adds to this methodology by tying each phase of this discovery process or ‘hypothesis-validation’ to milestones representing a startup’s increasing readiness for investor support and capital investment.  For investors this is a big idea.

I remind entrepreneurs that investors are implicitly seeking evidence of progress and milestones (but until the Investment Readiness Level never knew how to ask for it).  Entrepreneurs should always communicate their business’ very latest stage of customer development as part of their investor presentation. Given that a startup is continually learning weekly, the entrepreneur’s investor presentation will evolve on a weekly basis as well, reflecting their latest progress.

In our Angel investor group, our Applicant Selections process ranks applicant companies relative to the other applicants.  In the past, the ranking process relied on our Selection Committee members having an intuitive “feel” for whether a startup was worth considering for investment.

As part of our screening process, I’ve embraced the Investment Readiness Level (IRL) framework as a more-precise way to think through where applicant companies would rank. (BTW, this does not mean that the IRL framework has been embraced by rest of our Selections committee – organizational adoption is a lot more complicated than an individual adopting a framework.) I believe the IRL framework offers a more-precise method to discuss and describe ‘maturity’, and will likely become a more explicit part of our selections discussion in the year ahead.

Investment Readiness Level is Prescriptive
At first blush the Investment Readiness Level framework is a diagnostic tool – it can be used to gauge how far a business has progressed in its Customer Development process. A supposition is that startups that have validated hypotheses about key elements of their business have reduced the risks in launching their new business and are more ready for investment.

But the IRL is more than a diagnostic. It enables a much richer investor -> founder dialog about exactly what milestones a startup has actually achieved, and ties that discussion to the stages of the business’ Customer Development and business development progress.  In the same way that Osterwalder’s Business Model Canvas provides a common vocabulary and enables a rich discussion and understanding of exactly what comprises the business’ design and business model, the IRL provides a common set of metrics and enables a rich discussion and understanding of just where the startup is in the maturity of its processes.

This means the IRL is also a Prescriptive tool.  No matter where a startup is in its stage of development, the immediate next stage milestone – where the entrepreneurs should focus their attention next – is immediately clear.  Although every business is unique, and every business model emerges and evolves in its own unique way, the logical sequencing of incremental discovery and validation implicit in the IRL framework is very clear. No ambiguity. Clarity is good.

Investment Readiness Level Enables Better Mentoring
As you might imagine, our Angel group receives applications for funding from a wide, wide variety of businesses, with highly variable quality of the businesses and their applications, and highly variable levels of maturity of those businesses.  Some of our applicants are not scalable, high-growth businesses, and we tell them quickly if they don’t fit our profile. Others have the potential to be scalable, high-growth businesses, but simply aren’t as compelling or as mature as better candidates in our funnel. During every Selections cycle, as we refine our applicant funnel to select the entrepreneurs to present to our membership, I obviously have to say “No” to far more entrepreneurs than those to whom I can say “Yes”.

The Investment Readiness Level adds a new dimension to those conversations, providing a vocabulary and framework for shifting the conversation from simply ‘No’, to the much-more-helpful “Not yet – but here’s what you can do…”.  It has completely changed the nature of the conversations I have with applicants. The prescriptive nature of the IRL means that wherever a business is in its current state of development, the next step on the ladder is nearly always pretty obvious. Of course, there should always be a little latitude for the unique nature of each business, but the IRL framework is a good guidepost. So the “here’s what you can do…” recommendations are clear, logical, and situationally-relevant to the entrepreneur’s business.

I would estimate that perhaps half of the applicants we see have heard of and use some form of Lean Startup or Customer Development methodology. The idea of a “Minimum Viable Product” is something that has entered the general vernacular, but I’m sure that not all of the businesses tossing the term around truly understand the Lean Startup teachings.

So when I’m providing feedback to an entrepreneur applying to our group for funding, I leverage the IRL framework to guide the feedback that I give. I don’t refer to the framework explicitly, but I provide feedback based on where I assess the company to be in their development, and what steps they’d need to pursue to get another rung or two up the ladder.

For example, I might say “The Sacramento Angels have decided that your firm isn’t quite ready for us to consider for potential investment at this point, but if you were able to discuss your prototype with 50-to-100 potential customers and get their feedback, this might help you identify the specific segments that care most-deeply about the advantages you’re offering over the existing alternative. We’d like to stay in touch with you and hear more from you once you’ve identified your initial target segment and how you are going to reach and service them …”

I’ve almost universally found that the entrepreneurs I’m discussing these recommendations with are pleased to have the feedback, even if they’re disappointed that we may not be funding them. For an entrepreneur, receiving guidance of “Not now, but here’s what you can do…” is better than getting a flat, directionless “No”.  For me, the ability to articulate the concept of maturity, and investment readiness as a continuum, is extremely helpful. Being able to articulate that an applicant’s current stage of development, along that continuum, is not aligned with our group’s investment goals but that with further progress on their part, there may be alignment – this is a fundamentally superior message.

The Investment Readiness Level has given me the tools to engage in a consultative, coaching and mentoring conversation that provides much more value to entrepreneurs, resulting in a much more-enjoyable conversation for all involved.

Lessons Learned:

  • Investment Readiness ties capital-raising to the capital-efficient Lean Startup methodology
  • The Investment Readiness Level is Prescriptive
  • The Investment Readiness Level enables better mentoring

Filed under: Customer Development Manifesto, Investment Readiness Level, Venture Capital | 6 Comments »

Is This Startup Ready For Investment?

Posted on February 25, 2014 by steve blank

Since 2005 startup accelerators have provided cohorts of startups with mentoring, pitch practice and product focus. However, accelerator Demo Days are a combination of graduation ceremony and pitch contest, with the uncomfortable feel of a swimsuit competition. Other than “I’ll know it when I see it”, there’s no formal way for an investor attending Demo Day to assess project maturity or quantify risks. Other than measuring engineering progress, there’s no standard language to communicate progress.

Corporations running internal incubators face many of the same selection issues as startup investors, plus they must grapple with the issues of integrating new ideas into existing P&L-driven functions or business units.

What’s been missing for everyone is:

  • a common language for investors to communicate objectives to startups
  • a language corporate innovation groups can use to communicate to business units and finance
  • data that investors, accelerators and incubators can use to inform selection

While it doesn’t eliminate great investor judgment, pattern recognition skills and mentoring, we’ve developed an Investment Readiness Level tool that fills in these missing pieces.

—-

Investment Readiness Level (IRL) for Corporations and Investors
The startups in our Lean LaunchPad classes and the NSF I-Corps incubator use LaunchPad Central to collect a continuous stream of data across all the teams. Over 10 weeks each team gets out of the building talking to 100 customers to test their hypotheses across all 9 boxes in the business model canvas.

We track each team’s progress as they test their business model hypotheses. We collect the complete narrative of what they discovered talking to customers as well as aggregate interviews, hypotheses to test, invalidated hypotheses and mentor and instructor engagements. This data gives innovation managers and investors a feel for the evidence and trajectory of the cohort as a whole and a top-level view of each teams progress. The software rolls all the data into an Investment Readiness Level score.

(Take a quick read of the post on the Investment Readiness Level – it’s short. Or watch the video here.)

The Power of the Investment Readiness Level: Different Metrics for Different Industry Segments
Recently we ran a Lean LaunchPad for Life Sciences class with 26 teams of clinicians and researchers at UCSF.  The teams developed businesses in 4 different areas– therapeutics, diagnostics, medical devices and digital health.  To understand the power of this tool, look at how the VC overseeing each market segment modified the Investment Readiness Level so that it reflected metrics relevant to their particular industry.

Medical Devices
Allan May of Life Science Angels modified the standard Investment Readiness Level to include metrics that were specific for medical device startups. These included; identification of a compelling clinical need, large enough market, intellectual property, regulatory issues, and reimbursement, and whether there was a plausible exit.

In the pictures below, note that all the thermometers are visual proxies for the more detailed evaluation criteria that lie behind them.

Device IRL

Investment Readiness Level for Medical Devices

You can watch the entire presentation here

Therapeutics
Karl Handelsman of CMEA Capital modified the standard Investment Readiness Level (IRL) for teams developing therapeutics to include identifying clinical problems, and agreeing on a timeline to pre-clinical and clinical data, cost and value of data points, what quality data to deliver to a company, and building a Key Opinion Leader (KOL) network. The heart of the therapeutics IRL also required “Proof of relevance” – was there a path to revenues fully articulated, an operational plan defined. Finally, did the team understand the key therapeutic liabilities, have data proving on-target activity and evidence of a therapeutic effect.

Therapeutics IRL

You can see the entire presentation here

Digital Health
For teams developing Digital Health solutions, Abhas Gupta of MDV noted that the Investment Readiness Level was closest to the standard web/mobile/cloud model with the addition of reimbursement and technical validation.

Digital Health

Diagnostics
Todd Morrill wanted teams developing Diagnostics to have a reimbursement strategy fully documented, the necessary IP in place, regulation and technical validation (clinical trial) regime understood and described and the cost structure and financing needs well documented.

Diagnostics IRL

You can see the entire presentation here

For their final presentations, each team explained how they tested and validated their business model (value proposition, customer segment, channel, customer relationships, revenue, costs, activities, resources and partners.) But they also scored themselves using the Investment Readiness Level criteria for their  market. After the teams reported the results of their self-evaluation, the  VC’s then told them how they actually scored.  We were fascinated to see that the team scores and the VC scores were almost the same.

Lessons Learned

  • The Investment Readiness Level provides a “how are we doing” set of metrics
  • It also creates a common language and metrics that investors, corporate innovation groups and entrepreneurs can share
  • It’s flexible enough to be modified for industry-specific business models
  • It’s part of a much larger suite of tools for those who manage corporate innovation, accelerators and incubators

Listen to the blog post here [audio http://traffic.libsyn.com/albedrio/steveblank_hplewis_140225_FULL.mp3]

Download the podcast here

Filed under: Corporate/Gov't Innovation, Investment Readiness Level, Lean LaunchPad, Life Sciences (NIH), NIH (National Institutes of Health), Teaching | 9 Comments »

Moneyball and the Investment Readiness Level-video

Posted on December 21, 2013 by steve blank

Eric Ries was kind enough to invite me to speak at his Lean Startup Conference.

In the talk I reviewed the basic components of the Lean Startup and described how we teach it. I observed that now that we’ve built software to instrument and monitor the progress of new ventures (using LaunchPad Central), that we are entering the world of evidence-based entrepreneurship and the Investment Readiness Level.

This video is a companion to the blog post here. Read it for context.

If you can’t see the video above, click here

You can follow the talk along using the slides below

If you can’t see the slides above, click here

Additional videos here

Startup Tools here

Listen the blog post here [audio http://traffic.libsyn.com/albedrio/steveblank_clearshore_131221.mp3]

Download the podcast here

Filed under: Customer Development, Investment Readiness Level, Lean LaunchPad, Teaching, Venture Capital | 2 Comments »

It’s Time to Play Moneyball: The Investment Readiness Level

Posted on November 25, 2013 by steve blank

Investors sitting through Incubator or Accelerator demo days have three metrics to judge fledgling startups – 1) great looking product demos, 2) compelling PowerPoint slides, and 3) a world-class team.

We think we can do better.

We now have the tools, technology and data to take incubators and accelerators to the next level. Teams can prove their competence and validate their ideas by showing investors evidence that there’s a repeatable and scalable business model. And we can offer investors metrics to play Moneyball – with the Investment Readiness Level.

Here’s how.

————–

We’ve spent the last 3 years building a methodology, classes, an accelerator and software tools and we’ve tested them on ~500 startups teams.

  • A Lean Startup methodology offers entrepreneurs a framework to focus on what’s important: Business Model Discovery. Teams use the Lean Startup toolkit: the Business Model Canvas + Customer Development process + Agile Engineering. These three tools allow startups to focus on the parts of an early stage venture that matter the most: the product, product/market fit, customer acquisition, revenue and cost model, channels and partners.

Lean moneyball

  • An Evidence-based Curriculum (currently taught in the Lean LaunchPad classes and NSF Innovation Corps accelerator). In it we emphasize that a) the data needed exists outside the building, b) teams use the scientific method of hypothesis testing c) teams keep a continual weekly cadence of:
    • Hypothesis – Here’s What We Thought
    • Experiments – Here’s What We Did
    • Data – Here’s What We Learned
    • Insights and Action – Here’s What We Are Going to Do Next

Evidence moneyball

  • LaunchPad Central software is used to track the business model canvas and customer discovery progress of each team. We can see each teams hypotheses, look at the experiments they’re running to test the hypotheses, see their customer interviews, analyze the data and watch as they iterate and pivot.

LPC

We focus on evidence and trajectory across the business model. Flashy demo days are great theater, but it’s not clear there’s a correlation between giving a great PowerPoint presentation and a two minute demo and building a successful business model. Rather than a product demo – we believe in a “Learning Demo”. We’ve found that “Lessons Learned” day showing what the teams learned along with the “metrics that matter” is a better fit than a Demo Day.

“Lessons Learned” day allows us to directly assess the ability of the team to learn, pivot and move forward. Based on the “lessons learned” we generate an Investment Readiness Level metric that we can use as part of our “go” or “no-go” decision for funding.

Some background.

NASA and the Technology Readiness Level (TRL)
In the 1970’s/80’s NASA needed a common way to describe the maturity and state of flight readiness of their technology projects.  They invented a 9-step description of how ready a technology project was.  They then mapped those 9-levels to a thermometer.NASA TRL

What’s important to note is that the TRL is imperfect. It’s subjective. It’s incomplete.  But it’s a major leap over what was being used before.  Before there was no common language to compare projects.

The TRL solved a huge problem – it was a simple and visual way to share a common understanding of technology status.  The U.S. Air Force, then the Army and then the entire U.S. Department of Defense along with the European Space Agency (ESA) all have adopted the TRL to manage their complex projects. As simple as it is, the TRL is used to manage funding and go/no decisions for complex programs worldwide.

We propose we can do the same for new ventures – provide a simple and visual way to share a common understanding of startup readiness status. We call this the Investment Readiness Level . 

The Investment Readiness Level (IRL)
The collective wisdom of venture investors (including angel investors, and venture capitalists) over the past decades has been mostly subjective. Investment decisions made on the basis of “awesome presentation”, “the demo blew us away”, or “great team” is used to measure startups. These are 20th century relics of the lack of data available from each team and the lack of comparative data across a cohort and portfolio.

Those days are over.

Hypotheses testing and data collection
We’ve instrumented our startups in our Lean LaunchPad classes and the NSF I-Corps incubator using LaunchPad Central to collect a continuous stream of data across all the teams.  Over 10 weeks each team gets out and talks to 100 customers. And they are testing hypotheses across all 9 boxes in the business model canvas.

We collect this data into a Leaderboard (shown in the figure below) giving the incubator/accelerator manager a single dashboard to see the collective progress of the cohort. Metrics visible at a glance are number of customer interviews in the current week as well as aggregate interviews, hypotheses to test, invalidated hypotheses, mentor and instructor engagements. This data gives a feel for the evidence and trajectory of the cohort as a whole and a top-level of view of each teams progress.

leaderboard moneyball

Next, we have each team update their Business Model Canvas weekly based on the 10+ customer interviews they’ve completed.

canvas updates moneyball

The canvas updates are driven by the 10+ customer interviews a week each team is doing. Teams document each and every customer interaction in a Discovery Narrative. These interactions provide feedback and validate or invalidate each hypothesis.

disovery 10 moneyball

Underlying the canvas is an Activity Map which shows the hypotheses tested and which have been validated or invalidated.

activty updates moneyball

All this data is rolled into a Scorecard, essentially a Kanban board which allows the teams to visualize the work to do, the work in progress and the work done for all nine business model canvas components.

scorecard update moneyball

Finally the software rolls all the data into an Investment Readiness Level score.

IRL

MoneyBall
At first glance this process seems ludicrous. Startup success is all about the team. Or the founder, or the product, or the market – no metrics can measure those intangibles.

Baseball used to believe that as well. Until 2002 – when the Oakland A’s’ baseball team took advantage of analytical metrics of player performance to field a team that competed successfully against much richer competitors.

Statistical analysis demonstrated that on-base percentage and slugging percentage were better indicators of offensive success, and the A’s became convinced that these qualities were cheaper to obtain on the open market than more historically valued qualities such as speed and contact. These observations often flew in the face of conventional baseball wisdom and the beliefs of many baseball scouts and executives.

By re-evaluating the strategies that produce wins on the field, the 2002 Oakland A’s spent $41 million in salary, and were competitive with the New York Yankees, who spent $125 million.

Our contention is that the Lean Startup + Evidence based Entrepreneurship + LaunchPad Central Software now allows incubators and accelerators to have a robust and consistent data set across teams. While it doesn’t eliminate great investor judgement, pattern recognitions skills and mentoring – it does provide them the option to play Moneyball.

if you can’t see the video above click here

Last September Andy Sack, Jerry Engel and I taught our first stealth class for incubator/accelerator managers who wanted to learn how to play Moneyball.

We’re offering one again this January here.

Lessons Learned

  • It’s not clear there’s a correlation between a great PowerPoint presentation and two minute demo and building a successful business
  • We now have the tools and technology to take incubators and accelerators to the next step
  • We focus on evidence and trajectory across the business model
  • The data gathered can generate an Investment Readiness Level score for each team
  • the Lean Startup + Evidence based Entrepreneurship + LaunchPad Central Software now allows incubators and accelerators to play Moneyball

Listen to the podcast here [audio http://traffic.libsyn.com/albedrio/steveblank_clearshore_131125.mp3]

Download the podcast here

Filed under: Customer Development, Customer Development Manifesto, Investment Readiness Level, Lean LaunchPad, Teaching | 17 Comments »