I was in Boston and was interviewed by The Growth Show about my current thinking about innovation in companies and government agencies.The interviewer was great and managed to get me to summarize several years of learning in one podcast.
It’s worth a listen.
At the end of the interview I got surprised by a great question – “What’s the Problem that Still Haunts You?” I wasn’t really prepared for the question but gave the best answer I could on the fly.
Part of the answer is the title of this blog post.
We just finished our second Hacking for Defense class at Stanford. Eight teams presented their Lessons Learned presentations.
Hacking for Defense is a battle-tested problem-solving methodology that runs at Silicon Valley speed. It combines the same Lean Startup Methodology used by the National Science Foundation to commercialize science, with the rapid problem sourcing and curation methodology developed on the battlefields in Afghanistan and Iraq by Colonel Pete Newell and the US Army’s Rapid Equipping Force.
Goals for the Hacking for Defense Class Our primary goal was to teach students entrepreneurship while they engaged in a national public service. Today if college students want to give back to their country they think of Teach for America, the Peace Corps, or Americorps or perhaps the US Digital Service or the GSA’s 18F. Few consider opportunities to make the world safer with the Department of Defense, Intelligence Community or other government agencies.
Our second goal was to teach our sponsors (the innovators inside the Department of Defense (DOD) and Intelligence Community (IC)) that there is a methodology that can help them understand and better respond to rapidly evolving asymmetric threats. That if we could get teams to rapidly discover the real problems in the field using Lean methods, and only then articulate the requirements to solve them, could defense acquisition programs operate at speed and urgency and deliver timelyand needed solutions.
Finally, we also wanted to show our sponsors in the Department of Defense and Intelligence community that civilian students can make a meaningful contribution to problem understanding and rapid prototyping of solutions to real-world problems.
The Class Here’s a brief description of the Lean Methodology our students used:
Our mantra to the students was that we wanted them to learn about “Deployment not Demos.” Our observation is that the DOD has more technology demos than they need, but often lack deep problem understanding. Our goal was to have the students first deeply understand their sponsors problem – before they started building solutions. As you can imagine with a roomful of technologists this was tough. Further we wanted the students to understand all parts of the mission model canvas, not just the beneficiaries and the value proposition. We wanted them to learn what it takes to get their product/service deployed to the field, not give yet another demo to a general. This meant that the minimal viable products the students built were focused on maximizing their learning of what to build, not just building prototypes.
(Our sponsors did remind us, that at times getting a solution deployed meant that someone did have to see a demo!)
The Hacking for Defense class was designed as “fundamental research” to be shared broadly and the results are not subject to restriction for proprietary or national security reasons. In the 10 weeks the students have, Hacking for Defense hardware and software prototypes don’t advance beyond a Technology Readiness Level 4 and remain outside the scope of US export control regulations and restrictions on foreign national participation.
Results
Eight teams spoke to over 800 beneficiaries, requirements writers, program managers, warfighters, legal, security, customers, etc.
Seven out of the eight teams realized that the problem as given by the sponsor really wasn’t the problem. Their sponsors agreed.
Received from a problem sponsor mid-live stream broadcast “we are working funding for this team now.”
Over half the student teams have decided to continue working on national security projects after this class.
This is the End Each of the eight teams presented a 2-minute video to provide context about their problem and then gave an 8-minute presentation of their Lessons Learned over the 10-weeks. Each of their slide presentation follow their customer discovery journey. All the teams used the Mission Model Canvas, Customer Development and Agile Engineering to build Minimal Viable Products, but all of their journeys were unique.
The teams presented in front of several hundred people in person and online.
The Department of Defense has expanded their use of Hacking for Defense to include a classified version, and corporate partners are expanding their efforts to support the course and to create their own internal Hacking for Defense courses.
Another surprise was how applicable the “Hacking for X…” methodology is for other problems. Working with the State Department we offered a Hacking for Diplomacy class at Stanford.
Both the Defense and Diplomacy classes created lots of interest from organizations that have realized that this “Hacking for X…” problem-solving methodology is equally applicable to solving public safety, energy, policy, community and social issues internationally and within our own communities. This fall a series of new “Hacking for X…” classes will address these deserving communities. These include:
If you’re interested in learning how to apply a “Hacking for X…” class in your workplace or school we’ve partnered with the 1776 incubator in Washington DC to offer a 2-day “Hacking for X…” certification course 26-27 July for those interested in learning how. Sign up here.
It Takes a Village While I authored this blog post, these classes are a team project. The teaching team consisted of:
Joe Felter a retired Army Special Forces Colonel with research and teaching appointments at Stanford’s Center for International Security and Cooperation (CISAC), the Hoover Institution, and the dept. of Management Science and Engineering. Joe is the incoming Deputy Assistant Secretary of Defense for South and Southeast Asia.
Pete Newell is a retired Army Colonel currently a Senior Visiting Research Fellow at the National Defense University’s Center for Technology and National Security Policy and CEO of BMNT Partners.
Steve Weinstein a 30-year veteran of Silicon Valley technology companies and Hollywood media companies. Steve is CEO of MovieLabs the joint R&D lab of all the major motion picture studios.
We were lucky to get a team of 25 mentors (VC’s and entrepreneurs) who selflessly volunteered their time to help coach the teams. Thanks to the team Lean Startup mentors: Paul Dawes, Tom Bedecarre, Kevin Ray, Craig Seidel, Daniel Bardenstein, Roi Chobadi, Donna Slade, and Rafi Holtzman and other advisors; Lisa Wallace, Peter Higgins, Steve Hong, Robert Medve.
We were privileged to have the support of an extraordinary all volunteer team of professional senior military officers representing all branches of service attending fellowship programs at Stanford’s Hoover Institution, and Center for International Security and Cooperation (CISAC) and Asia Pacific Research Center (APARC) at the Freeman Spogli Institute (FSI). These included: Colonel Lincoln Bonner (US Air Force), Colonel Curtis Burns (US Army), Captain Kurt Clark (US Coast Guard), Lieutenant Colonel Kurt Helphinstine (US Air Force), Colonel Seth Krummrich (US Army)), Commander Leo Leos (US Navy), Lieutenant Colonel Eric Reid (US Marine Corps), Colonel Mike Turley (US Army), and Colonel Dave Zinn US Army. Additional volunteers from the active duty military providing support to our teams included Lieutenant Colonel Donny Haseltine (US Marine Corps), Captain Jason Rathje (US Air Force), Major Dave Ahern US Army) and, Major Kevin Mott (US Army).
And finally a special thanks to our course advisor Bill Perry, former Secretary of Defense and Professor Emeritus, and Tom Byers, Professor of Engineering and Faculty Director, STVP.
When Colonel Peter Newell headed up the Army’s Rapid Equipping Force (REF) he used lean methods on the battlefields of Iraq and Afghanistan to provide immediate technology solutions to urgent problems.
Today, his company BMNT does for government and commercial customers what the Rapid Equipping Force did for the U.S. Army.
Pete and I created the Hacking for Defense class (with Joe Felter and Tom Byers.) One of the problems our students run into is that there are always multiple beneficiaries and stakeholders associated with a problem, often with conflicting value propositions and missions. So how do you figure out whose needs to satisfy?
Here’s Pete’s view of how you do it.
Unlike businesses, government organizations don’t sell products, and they don’t earn revenue. Instead, they have missions to accomplish and very hard problems to solve. They use a variant of the Business Model Canvas – the Mission Model Canvas – to map their hypotheses, and they get of the building to do beneficiary discovery. (A beneficiary can be a soldier, program manager, commanding general, government contractor, stakeholder, customer, etc.) And just like in a commercial business they are trying to determine whether the value proposition solves the problem and helps the beneficiary accomplish their mission.
Discovery for both business and government is similar in that the only way to do it is to turn assumptions into facts by generating hypotheses, developing Minimum Viable Products and getting out of the building to test those MVP’s in the trenches where the customers and beneficiaries work. Early in the discovery process, teams are faced with a cacophony of personalities and organizations. Often, they struggle with understanding which person or group represents a beneficiary, supporter, advocate or potential key partner. It’s only through repetitive hypothesis testing that they begin to sort them all out.
It’s in the trenches however, where things become different.
Multiple Beneficiaries, Multiple Conflicts Unlike their commercial counterparts, government problem solvers are often faced with multiple beneficiaries associated with a problem, often with conflicting value propositions. As these differences become apparent, teams must make decisions about the value proposition trade-offs between conflicting beneficiaries – sometimes even pivoting completely in favor of one beneficiary to the detriment of another.
During last year’s Hacking for Defense class at Stanford Team Aqualink experienced the conflicting beneficiaries’ problem. The result was a significant pivot of both beneficiary and value proposition.
Aqualink started with a problem given to them from the chief medical officer of the Navy SEALS – they had no way to understand chronic long-term health issues divers face. Divers work 60 to 200 feet underwater for 2-4 hours, but Navy doctors currently have no way to monitor divers’ core temperature, maximum dive pressure, blood pressure, pulse and the rebreather (air consumption), or the dive computer (dive profile) data.
Having all this new data would give a diver early warning of hypothermia or the bends. More importantly the data would allow the medical director to individually assess the short and long-term health of each diver. And medical researchers would have access to detailed physiological data. The medical director tasked the team with building a wearable sensor system and developing apps that would allow divers to monitor their own physiological conditions while underwater and to download it for later analysis.
In the first week of the class this team got out of the building, suited up in full Navy diving gear and did customer discovery by spending an hour in the life of the beneficiary.
But as the students on the Aqualink team spoke to the SEAL team divers, (another one of their beneficiaries), they experienced an existential crisis. Most of the divers were “ambivalent” (read hostile) about the introduction of a vitals monitoring platform, (“If you gave to us at 0900, it would end up on the bottom of the ocean by 0905.”) Having worked so hard to get into the SEALS, no diver wanted doctors telling them they could no longer dive.
After further questioning, the team discovered the reason the divers were spending so much time underwater – they often did not know where they were. To find out, they had to get a GPS fix. This meant their minisub (called the SEAL Delivery Vehicle) had to rise to within 6 feet of the ocean surface so the GPS antenna could broach the surface. And to do so they had to surface slowly to avoid giving the divers the bends.
The divers told our student team, “Screw the health sensors. Build us a GPS sensor that can be deployed from 100 feet underwater.”
Now the team had a dilemma. They would have to decide which beneficiary to focus on – the SEAL Team medical director, who was the sponsor of their problem, or the operators of the delivery vehicle and divers within SEAL Delivery Vehicle Team One, along with their immediate chains of command in SDVT-1 and Naval Special Warfare Group 3.
When they went back to the medical director with their findings, he was surprised as they were. “Never knew that’s why they spent all that time down there. Heck, yes, fix their problem first.”
Understanding the Problem Context and Problem Ecosystem As Aqualink shows, getting out of the building – interviewing the beneficiaries, drawing their workflows and mapping a day-in -their-life – will give you a more complete picture of the context in which a proposed problem exists. Talking to multiple beneficiaries will lead to better understanding of the entire ecosystem of the problem. Often this will show that the problem you have been given is merely a symptom of a larger problem, or is the result of a different problem.
The solution is to:
Cross check the results of your discovery between different beneficiaries. Often, you’ll find that they seldom have a complete understanding of one another’s workflows and pain points but instead are championing the solution to a mere symptom of a different problem.
Share what you learned in discovery among the different beneficiaries. This will arm you with the tools needed to get them (or their leadership) to agree on the right problem that needs to be solved first. In many cases this will lead to your first pivot!
The goal is to sort out who has a value proposition that must be addressed first.
The power of beneficiaries helping one another While discovery with multiple beneficiaries can be confusing and exhausting, there is immense power when all the beneficiaries work together. Therefore, the goal of customer discovery is not just to understand the pains and gains of individual beneficiaries, but to find a shared purpose between all of them.
Once they understand they share the same goal, they can solve pain points or create gains for each other using the resources they already control. A “shared” sense of purpose is a very powerful step in the pathway towards a deployable solution.
When the Department of Energy asked BMNT to build a training program for getting veterans into advanced manufacturing jobs, we saw the power of a shared purpose between multiple beneficiaries first hand.
The problem we were asked to solve is that of the 10,000 veterans who leave military service every month, many remain unemployed or underemployed, yet at the same time the number of unfilled advanced manufacturing jobs in the U.S. is expected to climb to over a million by 2020. From a business perspective, obtaining technically qualified talent is among the top constraints to growth in the US.
Seems like it would be a match made in heaven, right? Not so fast…
While we initially thought the beneficiaries of the effort were the veterans, we quickly discovered there were other beneficiaries in advanced manufacturing. We found these additional beneficiaries had different pains and gains which in turn required different value propositions to solve their problems.
Our customer discovery taught us that there were three additional beneficiaries:
Universities needed to grow their enrollments. Our discovery showed us universities were willing to create programming for Advanced Manufacturing, but first needed to see a business case for how it would increase their enrollment to make it a worthwhile effort.
Industry needed to attract and hire qualified employees. We learned that technically qualified employees within industry were in such demand that the number one way to get qualified employees was to pilfer them from others.
Government Agencies needed to help their communities build skilled labor pools to attract new industries.
And we learned that our initial beneficiary, veterans leaving service, didn’t need internships or low-paying jobs, but needed jobs that paid enough to support their families.
We found each of these beneficiaries had a shared purpose. And each of them had a value proposition that would create a gain or relieve a pain point for another beneficiary. These were big ideas.
We found that as these overlapping value propositions emerged, we used the results to get the beneficiaries to come together in a workshop designed to jointly create a shared minimum viable product that they could then use to test within their own organizations.
Bringing the groups together in a workshop also served to align value propositions between beneficiaries by demonstrating that there was a way to create a single program that served all their needs. And we created an environment that allowed each beneficiary to discover that the other beneficiaries were partners they could work with in the future.
What was the impact of bringing the beneficiaries together in a workshop and creating this beneficiary ecosystem for advanced manufacturing?
Lawrence Livermore National Lab (LLNL) created a veterans’ jobs program. They teamed with a local college to create internships that allowed veterans to work during the summer. In turn, the local college created additional advanced manufacturing classes to meet LLNL’s technical needs and the regional workforce investment board provided funding.
In Fort Riley, the Army base in Kansas, the military teamed with Kansas State University to create an advanced manufacturing program. Kansas State created a series of advanced manufacturing classes. Soldiers leaving the service can take these courses at a nearby campus beginning up to six months before they leave service.
An unexpected consequence is that today there are soldiers from Fort Riley using advanced manufacturing processes to create parts for vehicles and equipment at the Army base.
Lessons learned
Government problem solvers will often be faced with multiple beneficiaries with different value propositions. Share what you learn from different beneficiaries with each other to sort out which has a value proposition that must be addressed first.
The benefit of having multiple beneficiaries is that their strengths can be used to help one another create gains and relieve pains for one another. Creating a shared sense of empowerment from working together smooths the pathway towards scaling the right solution.
When you’re up to your neck in alligators, don’t forget the goal was to drain the swamp.
I love teaching because I learn something new every class.
This time it was, “Don’t let process distract you from finding the strategy.”
The latest “aha” moment for me when I was at Columbia University teaching an intensive 5-day version of the Lean LaunchPad/I-Corps class. The goal of the class is to expose students to the basics of the Lean Methodology – Business Model Design, Customer Development and Agile Engineering.
In this short version of the class, students (in teams of four) spend half their day out of the classroom testing their hypotheses by talking to customers and building minimum viable products. The teams come back into the class and present what they found, and then they get out and talk to more customers. Repeat for 5 days. All teams talk to at least 50 customers/ partners/ stakeholders, and some manage to reach more than 100.
One of the teams wanted to create a new woman’s clothing brand. The good news is that they were passionate, smart and committed. The not so good news is that other than having been customers, none of them had ever been in the fashion business. But hey, no problem. They had the Lean Startup model to follow. They could figure it out by simply talking to customers and stores that carry unique fashion brands. How hard can this be?!
By the second day the team appeared to be making lots of progress – they had talked to many women about their clothing line, and had marched up and down NY stores talking to buyers in clothing boutiques. They built detailed value proposition canvases for each customer segment (young urban professional woman, students, etc.) – trying to match customer pains, gains and jobs to be done with their value proposition (their new clothing line.) They were busily testing their hypotheses about customer segments and value proposition, seeing if they could find product/market fit.
In listening to them it dawned on me that I had fallen victim to teaching process rather than helping the teams gain insight. I asked them to remind the class what business they were in. “We’re creating a clothing fashion brand,” was the reply. I asked, “And how much fashion brand expertise do you have as a team?” “None, we’re using customer discovery to quickly acquire it.” On the surface, it sounded like a good answer.
But then I asked, “Has anyone on your team asked if any of your 120 classmates are in the apparel/fashion business?” After a moment of reflection they did just that, and eight of their classmates raised their hands. I asked, “Do you think you might want to do customer discovery first on the domain experts in your own class?” A small lightbulb appeared over their heads.
A day later, after interviewing their classmates, the team discovered that when creating a woman’s clothing brand, the clothing itself has less to do with success than the brand does. And the one critical element in creating a brand is getting written about by a small group (less than 10) of “brand influencers” (reviewers, editors, etc.) in fashion magazines and blogs.
Whoah… the big insight was that how you initially “get” these key influencers – not customers or stores – is the critical part of creating a clothing fashion brand. This meant understanding these influencers was more important than anything else on the business model canvas. The team immediately added brand influencers to their business model canvas, created a separate value proposition canvas for them and started setting up customer discovery interviews.
The lessons?
This team was entering an existing market. (The team had already drawn the Petal Diagram mapping the competitive landscape.)
In an existing market there is a track record for how new entrants create a brand, get traction and scale. Many of the key insights about the business model and value proposition canvases are already known.
In an existing market, going through customer discovery (talking to customers, buyers, distribution channel, etc.) without first asking, “Are there any insights that can be gained by understanding the incumbent strategies, can be a trap for the unwary.”
Ironically, when I was entrepreneur I knew and practiced this. When I started a new venture in existing markets I would spend part of my initial customer discovery attending conferences, reading analysts’ reports and talking to domain experts to understand current market entry strategies. (None of this obligated me to follow the path of other companies. At times I took this information and created a different strategy to disrupt the incumbents.) But as an educator I was getting trapped in teaching the process not the strategy.
The fashion brand team’s experience was a great wake-up call.
From now on my first question to startups in an existing market is: “Tell me the critical success factors of the existing incumbents.”
Lessons Learned
In an existing market, draw a Petal Diagram with adjacent companies
Focus part of your initial customer discovery on learning competitive insights.
Describe how those companies entered the market. What was critical?
On the last day Congress was in session in 2016, Democrats and Republicans agreed on a bill that increased innovation and research for the country.
For me, seeing Congress pass this bill, the American Innovation and Competitiveness Act, was personally satisfying. It made the program I helped start, the National Science Foundation Innovation Corps (I-Corps) a permanent part of the nation’s science ecosystem. I-Corps uses Lean Startup methods to teach scientists how to turn their discoveries into entrepreneurial, job-producing businesses. I-Corps bridges the gap between public support of basic science and private capital funding of new commercial ventures. It’s a model for a government program that’s gotten the balance between public/private partnerships just right. Over 1,000 teams of our nation’s best scientists have been through the program.
The bill directs the expansion of I-Corps to additional federal agencies and academic institutions, as well as through state and local governments. The new I-Corps authority also supports prototype or proof-of-concept development activities, which will better enable researchers to commercialize their innovations. The bill also explicitly says that turning federal research into companies is a national goal to promote economic growth and benefit society. For the first time, Congress has recognized the importance of government-funded entrepreneurship and commercialization education, training, and mentoring programs specifically saying that this will improve the nation’s competitiveness. And finally this bill acknowledges that networks of entrepreneurs and mentors are critical in getting technologies translated from the lab to the marketplace.
This bipartisan legislation was crafted by senators Cory Gardner (R–CO) and Gary Peters (D–MI). Senator John Thune (R–SD) chairs the Senate commerce and science committee that crafted S. 3084. After years of contention over reauthorizing the National Science Foundation, House Science Committee Chairman Lamar Smith and Ranking Member Eddie Bernice Johnson worked to negotiate the agreement that enabled both the House and the Senate to pass this bill.
While I was developing the class at Stanford, it was my counterparts at the NSF who had the vision to make the class a national program. Thanks to Errol Arkilic, Don Millard, Babu Dasgupta, Anita LaSalle (as well as current program leaders Lydia McClure, Steven Konsek) and the over 100 instructors at the 53 universities who teach the program across the U.S.
But I haven’t forgotten that before everyone else thought that teaching scientists how to build companies using Lean Methods might be a good for the country, there was one congressman who got it first. IN 2012, Representative Dan Lipinski (D-Il), co-chair of the House STEM Education Caucus, got on an airplane and flew to Stanford to see the class first-hand.
For the first few years Lipinski was a lonely voice in Congress saying that we’ve found a better way to train our scientists to create companies and jobs.
This bill is a reauthorization of the 2010 America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Act, which set out policies that govern the NSF, the National Institute of Standards and Technology (NIST), and federal programs on innovation, manufacturing, and science and math education. Reauthorization bills don’t fund an agency, but they provide policy guidance. It resolved partisan differences over how NSF should conduct peer review and manage research.
I-Corps is the accelerator that helps scientists bridge the commercialization gap between their research in their labs and wide-scale commercial adoption and use.
Why This Matters While a few of the I-Corps teams are in web/mobile/cloud, most are working on advanced technology projects that don’t make TechCrunch. You’re more likely to see their papers (in material science, robotics, diagnostics, medical devices, computer hardware, etc.) in Science or Nature.
I-Corps uses everything we know about building Lean Startups and Evidence-based Entrepreneurship to connect innovation to entrepreneurship. It’s curriculum is built on a framework of business model design, customer development and agile engineering – and its emphasis on evidence, Lessons Learned versus demos, makes it the worlds most advanced accelerator. It’s success is measured not only by the technologies that leave the labs, but how many U.S. scientists and engineers we train as entrepreneurs and how many of them pass on their knowledge to students. I-Corps is our secret weapon to integrate American innovation and entrepreneurship into every U.S. university lab.
“Being in Silicon Valley, a lot of my friends want to work for Google or Apple, but this class showed me that the problems in public service were even more challenging and rewarding. I used to watch the news about Syrian refugees and feel that I was just a bystander to a hopeless situation. But this class helped me realize I can have an impact and be part of the solution.”
Hacking for Diplomacy student
We just held our final week of the Hacking for Diplomacy class, teaching students entrepreneurship and “Lean Startup” principles while they engaged in national public service applying advanced technologies to solve global challenges. Seven student teams delivered their final Lessons Learned presentations documenting their intellectual journey over just 10 short weeks in front of several hundred people in person and online. And what a journey it’s been.
In this class, we partnered with sponsors in the State Department including:
Office of Space and Advanced Technology
Bureau of Political-Military Affairs
Bureau of Conflict and Stabilization Operations
Bureau of Counterterrorism and Countering Violent Extremism
Bureau of Population, Refugees, and Migration
Office of Assistance to Europe, Central Asia, & the Americas
Office of Assistance to the Near East
Office to Monitor and Combat Trafficking in Persons
Our sponsors treated our students like serious problem solvers who could contribute unique technical skills and unfettered customer access. In exchange the sponsors got access to fresh ideas, new technology and a new perspective on serious problems.
By the end of the class our sponsors inside State had experienced a practical example of a new and powerful methodology which could help them better understand and deal with complicated international problems and apply technology where appropriate.
And finally, our students learned that they could serve their country without having to put on a uniform. Today, if college students want to give back to their country, most think of Teach for America, the Peace Corps, or Americorps or perhaps if you wanted to offer your technical skills, the US Digital Service or the GSA’s 18F. Few consider opportunities to make the world safer with the State Department, Department of Defense, Intelligence Community or other government agencies.
(This post is a continuation of a series. See all the posts about Hacking for Diplomacy here.)
Lessons Learned – Not a Demo Day Silicon Valley folks are familiar with Demo Days – presentations where the message is: “Here’s how smart we are right now.” That’s nice, but it doesn’t let the audience know, “Is that how smart you were three months ago, did you get smarter or dumber, what did you learn?”
Hacking for Diplomacy Lessons Learned presentations are different. Each team presents a two-minute video to provide context about their problem and then presents for eight minutes about the Lessons Learned over their ten weeks in the class.
As an example, Team Trace worked with the State Department Office to Monitor and Combat Trafficking in Persons. The team was challenged to help companies push policies of responsible business lower down the supply chain. The key thing to note in this presentation is not only that the team came up with a solution, but also how in talking to 85 people, their understanding of the problem evolved, and as it did, so did their solution. (see Slides 12 and 25).
Team Hacking CT was sponsored by Bureau of Counterterrorism and Countering Violent Extremism with the goal of deterring individuals from joining violent extremist groups. After 100 interviews, the team realized that a bottom-up approach, focusing on support for friends and family of those at risk for radicalization, might be effective.
First, students took the problems they got from their State Department sponsors and transformed those into what we call hypotheses. For instance, one problem was: “We need to improve coordination among all the organizations trying to help Syrian refugees.” That’s a big, unwieldy problem. Students had to break it down into a series of hypotheses. They had to identify who were the beneficiaries and stakeholders, and think about what specific service they were going to provide them, how they were going to get it to them and who was going to pay for it. To help them do that, we have them map their nine critical hypotheses onto a single sheet of paper called the Mission Model Canvas.
Then in step two, the teams got out of the classroom to test these hypotheses through interviews with people in the real world. Every team spoke to close to 100 potential “beneficiaries,” partners and stakeholders including NGOs, tech company executives, supply chain managers, foreign service officers in embassies around the world, and even refugees. While the students were interviewing, they also employed the third piece of the Lean methodology: building the solution incrementally and iteratively. These solutions, called Minimal Viable Products (MVP’s), are what allow the teams to become extremely agile and responsive.
As teams talk to stakeholders they gather evidence to either validate, invalidate or modify their hypotheses. If they find out that their assumptions are wrong (and almost all do,) they Pivot, that is, they make fundamental changes to their hypotheses, instead of blindly proceeding forward simply executing a plan. This ability to gather data, build and test MVPs, and then change course is what gives Lean it’s tremendous speed and agility to deliver rapid solutions that are needed and wanted.
As an example, Team Aggregate DB was working with the State Department Bureau of Conflict and Stabilization Operations (CSO). CSO helps embassies and diplomats to visualize, understand, and stabilize conflict. The team’s challenge was to get helps embassies and diplomats get more information about informal leader networks. Getting out of the building and talking to 87 people gave the team got a firsthand view of the downside when an embassy does not have access to the right local contacts. (Slides 3-9)
As they developed MVPs, our students took these solutions out into the real world for feedback. At first the solutions were nothing more than drawings, wireframes or PowerPoint slides. As they came to understand their problems more deeply, they refined their solutions into the final products we saw.
For example, Team 621 – Fatal Journeys worked with the State Department Bureau of Population, Refugees, and Migration. The team’s challenge: how to get more data on missing or perished refugees. In this presentation, note how the team’s understanding of the problem evolved over the course of talking to 88 people. They realized there was a missing link between key stakeholders that limited identification of perished refugees and prevented emotional and legal closure for their families. The team pivoted three times as they gained deeper and deeper insight into their problem. With each pivot, their solution radically changed. (Their first pass of problem/solution understanding is on Slides 1-29, but then they get additional insight in slides 36-50. Finally, slides 51-64 is their third and final iteration).
Hacking for Diplomacy was profiled this week in the L.A. Times. We’ve also had L.A. Times Beijing bureau chief Julie Makinen, who is on a JSK journalism fellowship at Stanford, helping coach students this quarter on interviewing and research techniques. Julie has been sharing her impressions of the class on this blog. Here’s her last installment:
In the Netflix age, suspense is an increasingly rare commodity. If we’re intrigued by an hour of “House of Cards,” we need not delay gratification – we can just queue up the next episode and push play. But following Stanford’s Hacking for Diplomacy class over the last 10 weeks has been like watching a TV drama the old-fashioned way. There were cliffhangers every time, and you had to wait seven days to find out what would happen next.
The class, which meets just once a week but requires massive outside work, is run not as a traditional lecture where professors drone on in front of passive students — just the opposite. It’s the students standing up in front, discussing what they’ve found out in the past seven days, what progress they’ve made, what obstacles they’ve run smack into. The teachers sit in the back row and lob questions and critiques forth — sometimes very direct critiques. That format keeps students and teachers alike on the edge of their seats.
Conflicts and misunderstandings within student teams — and between students and sponsors — cropped up as the students tried to learn about the State Department, their sponsors problem, and Lean Startup methodology all at once. Students, teachers, and interviewees said surprising, intriguing, even stunning things. Some days, you could see teams going off the rails, but instead of just shouting at your screen, “No, don’t go down that alley!” a professor would actually speak up from the back with something blunt like, “You’re way off track, and we’re firing your idea.”
And just when you thought a team had struck upon a brilliant notion for a product, they’d report back during the next session that everyone they put it in front of hated it. I started looking forward to each Thursday at 4:30 p.m. like my parents looked forward to watching “Dragnet” as kids, because the suspense was killing me.
Thursday’s season finale did not disappoint. Teams that just two or three weeks ago seemed to be foundering pulled off some amazing comebacks.
Take Team Exodus, which had spent a substantial part of the quarter focused on how to match private companies seeking to assist Syrian refugees with NGOs working in the field. Late in the term, the students scrapped that idea after finding competitors who were already deeply engaged in that space. They did a major pivot and decided to concentrate directly on refugees as customers — building on all they had learned during their first eight weeks of interviewing and research.
In week 9, they decided to build an AI chatbot on Facebook’s Messenger platform to allow refugees to ask questions like, “Where can I get clothing?” The bot will tap into a network of NGOs to source answers. A very basic prototype, built primarily by team member Kian Katanforoosh, a master’s student in computer science and management science & engineering, is already up and running.
On the eve of Thursday’s class, team members Katie Joseff and Berk Coker had a call with the United Nations’ refugee agency, UNHCR, and learned that the organization was very interested in working with the students to bring an Arabic chatbot to the field, most likely starting in Jordan.
“At the end, our team kind of came out of the weeds,” said Joseff, an undergrad majoring in human biology. “We finally got to the thing that Steve Blank talks about – where you can see the whites of a customer’s eyes and they just really want the product you’re talking about.”
Team Exodus: Coordinating information to better serve refugees
Watching the students’ process and progress was an eye-opener even for many State Department sponsors and private-sector mentors. Team Space Evaders, who in week 7 seemed to be vying for the title of Team Whose Proposed Products Generated the Most Yawns from Potential Customers, had an “ah-ha” moment and decided instead of focusing on tracking objects already in space, they’d pivot and concentrate on objects that will be launched in the future.
They’re proposing a “debris footprint” that would rate satellites before they’re sent into orbit on how much space junk they could generate. The team hopes that this could lead to international design standards to reduce space debris.
“They had a fundamental insight – don’t track ’em, solve it before they even get into space,” said Jonathan Margolis, deputy assistant secretary of State for science, space and health who came all the way from Washington to meet with the team and sit in on the class in Week 9. “It’s a reconceptualization of a problem we’ve really been struggling with.”
Team members Dave Gabler, a master’s student in business and public policy with an Air Force background, and Matthew Kaseman, an Army vet and freshman in aerospace engineering, said the next step is to produce a white paper that fleshes out the mathematical formulas that could underpin a ratings system, then take that to academic and industry conferences. “That would help start a public discussion and push the debate,” said Gabler.
“The math is probably the easy part,” said Pablo Quintanilla, a former Foreign Service Officer and current head of public policy for Asia for Salesforce, who served as mentor for the Space Evaders team. “There’s so much more to the behavioral side – who in the international space community will adopt this?”
Quintanilla said that working with Space Evaders drove home for him the merits of forming diverse teams to tackle problems. Besides Gabler and Kaseman, the student team included Kate Boudreau, a junior majoring in biomedical computation, and Tyler Dammann, a junior in computer science.
“This cross-functionality and working across disciplines is really effective,” Quintanilla said. “I feel like this is living proof that you should work everywhere like this.”
Professor Jeremy Weinstein, a co-instructor for the class who recently served as deputy U.S. ambassador to the United Nations, acknowledged that 10 weeks is a really short time frame for the students to make any meaningful impact. But unlike an internship, where a lone student “plugs into an existing bureaucratic hierarchy and rules,” the Hacking for Diplomacy students had the advantage of being able to work in teams — and approach the problem more as outsiders.
“The students don’t have to play by the same rules [as insiders]. They can ask the non-PC questions,” said Weinstein. “To be ignorant of the rules is a blessing at times — if you can do it respectfully.”
Getting students to have a healthy appreciation for how government policy is made — sometimes painfully slowly — is part of the educational process. And so perhaps is getting bureaucrats to be more open to fresh ideas. “There’s not going to be a flip of the switch” in State as a result of this class, Weinstein said. “There is some skepticism. But I think more broadly, we’ve won some people over.”
Thursday’s wrap-up session attracted a diverse audience, including representatives from leading Silicon Valley tech companies as well as diplomats from France, Britain and Denmark. Susan Alzner, head of the U.N. Non-Governmental Liaison Service’s New York office, said after watching the student presentations, she wants to take the customer discovery and interview methodology back to her agency.
“The U.N. has lots of small teams of people who often believe they already know the solution to a problem. … And the U.N. does way too much consultation digitally. Interviews are critical. It’s so elaborate to see these students doing 100 interviews to understand a problem, but it’s so important to orient yourself before making a plan to do something.”
Team Hacking 4 Peacekeeping: Better data on, and decision-making about, peacekeeping forces
Weinstein noted that scaling up Hacking for Diplomacy may not be as easy as expanding Hacking for Defense, simply because “there are millions of people who work in the Department of Defense… while the size of the State Department foreign service corps is smaller than the total number of people who play in military bands.” That means there are fewer people who can serve as sponsors.
At the same time, the class could tap a wider array of sponsor organizations. “Scale maybe has to look different — we can look to the [State Department], but also UNHCR, the foreign ministry of the U.K., other international organizations,” Weinstein said. “You have to think of a different array of partners.”
Most of the State Department sponsors for this year’s class, Weinstein noted, were not political appointees but career foreign service officers or career civil servants.
“They are the glue that holds the agency together and they are key to getting anything implemented in government. And so the buy-in is there,” he said. “But they also need permission; they need a blessing to experiment with radically different ideas. And you need political cover in these bureaucracies to do this kind of work.”
“I hope,” he added, “we’ll have that cover in a subsequent administration. More than cover. Endorsement. Enthusiasm. Excitement.”
Our Teaching Team Like the students’ efforts, the teaching of this class was also a team project. I was joined by Jeremy Weinstein, former deputy to the U.S. ambassador to the United Nations and a Stanford professor of political science; Zvika Krieger, the State Department’s representative to Silicon Valley and senior advisor for technology and innovation; retired U.S. Army Col. Joe Felter, who co-created Hacking for Defense and is a senior research scholar at the Center for International Security and Cooperation at Stanford; and Steve Weinstein, chief executive of MovieLabs who teaches entrepreneurship at Stanford and UC Berkeley.
Our teaching assistants were Shazad Mohamed, Sam Gussman and Roland Gillah. We were fortunate to get a team of seven mentors currently or formerly served in the State Department and selflessly volunteered their time to help coach the teams. Each team also got a mentor from the tech industry who helped guide them through creating their final products. Of course, huge thanks to the Stanford students who gave their all through this class.
Going forward While our previous Hacking for Defense class gave us a hint that doing the same for Diplomacy would work, we’re a little stunned about how well this class with the State Department went. A surprising number of students have decided to continue working on foreign policy projects after this class with the State Department or with NGO’s. Other colleges and universities have raised their hands, and said they want to offer Hacking for Diplomacy or potentially a USAID Hacking for Development class at their school.
If you’re interested in offering Hacking for Diplomacy (or Defense) in your school, or if you’re a sponsor in a federal agency interested in solving problems with speed and urgency, join us at our next H4D educators class January 17-19th at Georgetown.
Lessons Learned
Our sponsors inside State saw examples of a new and powerful methodology – Lean which could help them better understand and deal with complicated international problems
Lean offers State speed and agility to deliver rapid solutions that are needed and wanted
Our students learned that they could serve their country without having to put on a uniform
Other universities are willing to have their students work on diplomacy and development problems
We just finished our Lean LaunchPad class at UC Berkeley’s engineering school where many of the teams embedded machine learning technology into their products.
It struck me as I watched the teams try to find how their technology would solve real customer problems, is that machine learning is following a similar pattern of previous technical infrastructure innovations. Early entrants get sold to corporate acquirers at inflated prices for their teams, their technology, and their tools. Later entrants who miss that wave have to build real products that people want to buy.
—–
I’ve lived through several technology infrastructure waves; the Unix business, the first AI and VR waves in the 1980’s, the workstation wave, multimedia wave, the first internet wave. Each of those had a set of common characteristics that the Gartner Group characterizes as the Hype Cycle .
Stage 1: The Technology Trigger: A potential technology breakthrough kicks things off. Early proof-of-concept stories and media interest trigger significant publicity. Often no usable products exist and commercial viability is unproven.
Stage 2: Peak of Inflated Expectations: Early publicity produces a number of success stories—often accompanied by scores of failures. Some companies take action; most don’t.
Stage 3: Trough of Disillusionment: Interest wanes as experiments and implementations fail to deliver. Producers of the technology shake out or fail. Investments continue only if the surviving providers improve their products to the satisfaction of early adopters.
Stage 4: Slope of Enlightenment: More instances of how the technology can benefit the enterprise start to crystallize and become more widely understood. Second- and third-generation products appear from technology providers. More enterprises fund pilots; conservative companies remain cautious.
Stage 5: Plateau of Productivity: Mainstream adoption starts to take off. Criteria for assessing provider viability are more clearly defined. The technology’s broad market applicability and relevance are clearly paying off.
Shiny Object meets First Mover Advantage What’s become apparent in the last few technology hype cycles is that for startups and their investors there is a short multi-year window of opportunity (at the Peak of Inflated Expectations) to sell a startup at an inflated price. This occurs because large technology companies (Google, Facebook, IBM, Microsoft, Twitter, Apple, Salesforce, Intel, et al,) and increasingly other non-tech firms, are in an arms race to stay relevant. For example, according to CBInsights nearly 140 machine intelligence have been acquired since 2011, with over 40 being bought so far in 2016.
Most often the first acquisitions in a hype cycle are for the “shiny objects” – the technology, the team and the tools. The acquired technical teams usually start up or complement the company’s research group in a specific new technology area.
If you’re a startup (or their investors) getting acquired at this point in the hype cycle is exactly where you want to be – short time in business, large acquisition price, value based on a frenzy, perceived scarcity of expertise, and fear of a competitor getting the key talent.
History shows that the acquirers often overpay buying this expertise early. While these acquisitions have teams of great researchers, they rarely contribute actual revenue generating products (because most never reached that stage when they were acquired.) The irony is that the acquisitions made later in the hype cycle – when companies have built real products that customers want, are the ones that generate revenue and profit for the acquirer.
I had all that in mind as we watched our teams present.
Machine Learning Meets Lean – Berkeley Lean LaunchPad Class
Each of our teams in this class followed the canonical Lean model:
Articulate your hypotheses using the business model canvas
Get of the building and test those hypotheses using customer development
Validate learning by building minimal viable products and getting them in front of customers
Each week the teams got out of the classroom and talked to 10-15 customers, testing a new part of the business model canvas. And after week two, they had to build and then update their minimal viable product weekly. And present what they learned each week in an 8-minute presentations.
The presentations below are their final Lessons Learned presentations, along with a 2-minute video summary.
SalesStash Three Berkeley PhD computer science students and an MBA working on machine learning. How can you not hit out of the park on day one?
This team epitomized rapid learning. Once their initial assumptions ran into the wall of actual customer feedback they rapidly built multiple minimum viable products (MVPs) and kept pivoting until they found product/market fit (i.e. a customer segment that was grabbing the product out of their hands.)
Delphi Before this class this team had spent three months in an incubator building the product after talking to only one customer. After week two of the class they realized they had wasted three months building something no one actually wanted. What they next learned was pretty amazing.
Homeslice Homeslice had a great journey. They came together over a personal pain – the inability to afford a house in Silicon Valley. Their initial plan was to provide fractional ownership to solve that problem. But they found that first serving an adjacent market – slices of investment properties – could serve as a launchpad for their initial idea of fractional home ownership.
Exit Strategy Exit Strategy was building the penultimate planning tool. This teams learning that this wasn’t a business was as important as finding one that is. Really impressive process.
This class this was a team effort. Professor Kurt Keutzer and Errol Arkilic (former program director for the National Science Foundation Innovation Corps (NSF I-Corps), now founder of M34 Capital) were the lead instructors. Steve Weinstein (CEO of MovieLabs) and I assisted. Thanks to our TA Kathryn Crimmins and all the team mentors: Lev Mass, Kanu Gulati, Ewald Detjens, James Cham, Kanu Gulati, Patrick Chung, Rick Lazansky, Ashmeet Sidhana, Mike Olson, Michael Borrus, Fabrizo De Pasquale, Amit Kumar, Rob Rodick, Mar Hershenson.
Join a select cross-disciplinary class that takes real problems from the U.S. State Department and asks students to use Lean Methods to test their understanding of the problem and deliver rapid-fire innovative solutions to pressing diplomacy, development and foreign policy challenges.
Syrian Refugees, Human Trafficking, Zika Virus, Illegal Fishing, Weapons of Mass Destruction Detection, ISIS on-line propaganda,Anti-Corruption… What do all these problems have in common? The U.S. Department of State is working on all of them.
The U.S. Department of State manages America’s relationships with 180 foreign governments, international organizations, and the people of other countries with 270 embassies, consulates, and other posts. The management of all these relationships is called diplomacy. 70,000 State Department employees (46,000 Foreign Service Nationals, 14,000 Foreign Service Employees and 11,000 in the Civil Service) carry out the President’s foreign policy and help build a freer, more prosperous, and secure world.
The State Department has four main foreign policy goals:
Protect the United States and Americans
Advance democracy, human rights, and other global interests
Promote international understanding of American values and policies
Support U.S. diplomats, government officials, and all other personnel at home and abroad.
At a time of significant global uncertainty, diplomats are grappling with a set of transnational and cross-cutting challenges that defy easy solution. These include the continued pursuit of weapons of mass destruction by states and non-state groups, the outbreak of internal conflict across the Middle East and in parts of Africa, the most significant flow of refugees since World War II, and a changing climate that is beginning to affect both developed and developing countries. And that’s just on Monday. The rest of the week is equally busy.
Hacking for Diplomacy In a world of complex threats, dynamic opportunities, and diffuse power, effective diplomacy and development require institutions that adapt, embrace technology, and allow for experimentation to ensure continuous learning. This means developing new and innovative ways to think about, organize and build diplomatic strategies and solutions. Stanford’s new Hacking for Diplomacy class is a part of this effort.
Hacking for Diplomacy is designed to provide students the opportunity to learn how to work with the Department of State to address urgent foreign policy challenges. While the traditional tools of statecraft remain relevant, policymakers are looking to harness the power of new technologies to rethink how the U.S. government approaches and responds to the problems outlined above and other long-standing challenges. In this class, student teams will take actual foreign policy challenges and a hands-on approach that will require close engagement with officials in the U.S. State Department and other civilian agencies. They’ll learn how to apply “lean startup” principles, (“mission model canvas,” “customer development,” and “agile engineering”) to discover and validate agency and user needs and to continually build iterative prototypes to test whether they understood the problem and solution.
Each week, teams will use the Mission Model Canvas (a variant of the Business Model Canvas used for government agencies and non-profits) to develop a set of initial hypotheses about a solution to the problem and will get out of the building and talk to relevant stakeholders and users. As they learn, they’ll iterate and pivot on these hypotheses through customer discovery and build minimal viable prototypes (MVPs). Each team will be guided by a sponsor from the State Department bureau that proposed the problem and a second mentor from the local community.
Real Problems In thisclass, student teams select from an existing set of problemsprovided by the Department of State. Hacking for Diplomacy is not a product incubator for a specific technology solution. Instead, as teams follow the Mission Model Canvas, they’ll discover a deeper understanding of the selected problem, the challenges of deploying the solutions, and the host of potential technological solutions that might be arrayed to solve them. Using the Lean LaunchPad Methodology the class focuses teams to:
Deeply understand the problems/needs of State Department beneficiaries and stakeholders
Rapidly iterate technology solutions while searching for product-market fit
Understand all the stakeholders, deployment issues, costs, resources, and ultimate mission value
Produce a repeatable model that can be used to launch other potential technology solutions
National Service Today if college students want to give back to their country they think of Teach for America, the Peace Corps, or AmeriCorps. Few consider opportunities to make the world safer with the State Department and other government agencies. Hacking for Diplomacy will promote engagement between students and the Department of State and provide a hands-on opportunity to solve real diplomacy, foreign policy and national security problems.
Like its sister class, Hacking for Defense, our goal is to open-source this class to other universities. By creating a national network of colleges and universities, Hacking for Diplomacy can scale to provide hundreds of solutions to critical diplomacy, policy, and national security problems every year.
We’re going to create a network of entrepreneurial students who understand the diplomatic, policy, and national security problems facing the country and get them engaged in partnership with islands of innovation in the Department of State. This is a first step to a more agile, responsive and resilient, approach to diplomacy and national security in the 21st century.
Use the Lean LaunchPad methodology to deeply understand the problems/needs of State Department customers
Deliver minimum viable products that match State Department needs in an extremely short time
The class will also teach the islands of innovation in the Department of State:
how the innovation culture and mindset operate at speed
how to identify potential dual-use technologies that exist outside their agencies and contractors (and are in university labs, or are commercial off-the-shelf solutions)
how to use an entrepreneurial mindset and Lean Methodologies to solve foreign policy problems
“Sometimes, you have to look back in order to understand the things that lie ahead.”
We just finished the 6th annual Lean LaunchPad class. This year we made a small but substantive addition to way we teach the class, adding a week for reflection. The results have made the class massively better.
For the last 6 years I’ve taught the Lean LaunchPad class at Stanford and Berkeley. To be honest I built the class out of frustration watching schools teach aspiring entrepreneurs that all they need to know is how to write a business plan or how to sit in an incubator building a product.
If you’ve read any of my previous posts, you know I believe that:
a product is just a part of a startup, but understanding customers, channel, pricing, – the business model – is what turns a product into a business
business plans are fine for large companies where there is an existing market, existing product and existing customers, but they are useless in a startup where most often none of these are known
entrepreneurship is experiential and requires theory and a ton of practice.
Therefore, we developed the 8-week Lean LaunchPad class to teach students how to think about all the parts of building a business, not just the product. We organized the class as:
Team-based
Students apply and learn as teams of 4. Eight teams per class
A “flipped” classroom
Students watch the lectures as homework via our MOOC
Every week we teach a new part of the theory of how to commercialize an insight or invention
using the business model canvas as the framework
Every week we teach the practice of Lean
by having the students get out of the classroom and talk to 10-15 customers a week and build a new Minimum Viable Product weekly
in order to validate/invalidate their business model hypotheses
The teaching team critiques their progress and suggests what they might do next
Every week the teams present their results
“Here’s what we thought, here’s what we did, here’s what we found, here’s what we are going to do next”
The combination of the Business Model Canvas, Customer Development and Agile Engineering is an extremely efficient template for the students to follow. It drives a hyper-accelerated learning process which leads the students to a “information dense, evidence-based” set of conclusions. (Translation: they learn a lot more, in a shorter period of time, than in any other entrepreneurship course we’ve taught or seen.)
Demo Days Versus Lesson Learned Presentations One thing we always kept in mind – we were teaching students a methodology and a set of skills for the rest of their lives – not running an incubator or accelerator. As a consequence, we couldn’t care less about a “Demo Day” at the end of the class. We don’t want our students focused on fund-raising, we want them to maximize their learning. Secondly, even for fund-raising, you couldn’t invent a less useful format to evaluate a startup’s potential then the Demo Days held by accelerators. Demo Days historically have been exactly what they sound like, “Show me how smart your team is at this instant in time.” Everything depends on a demo, presentation and speaking style.
We designed our class to do something different. We wanted the teams to tell the story of their journey, sharing with us their “Lessons Learned from our Customers”. They needed to show what they learned and how they learned it after speaking to 100+customers, using the language of class: interview, iterations, pivots, restarts, experiments, minimal viable products, evidence. The focus of their presentations is on how they gathered evidence and how it impacted the understanding of their business models – while they were building their MVP.
Reflection Week In the past, our teams would call on customers until the last week of the class and then present their Lessons Learned. The good news is that their presentations were dramatically better than those given at demo days – they showed us what they learned over 8 weeks which gave us a clear picture of the velocity and trajectory of the teams. The bad news is since their heads were down working on customer discovery until the very end, they had no time to reflect on the experience.
We realized that we had been so focused in packing content and work into the class, we failed to give the students time to step back and think about what they actually learned.
So this year we made a change. We turned the next to last week of the class into a reflection week. Our goal—to have the students extract the insights and meaning from the work they had done in the previous seven weeks.
We asked each team to prepare a draft Lessons Learned presentation telling us about their journey and showing us their:
Initial hypotheses and Petal diagram
Quotes from customers that illustrated learnings and insights
Diagrams of key parts of the Canvas –customer flow, channel, get/keep/grow (before and after)
Pivot stories
Screen shots of the evolution of Minimum Viable Product (MVP)
Demo of final MVP
The teaching team reviewed the drafts and provided feedback to the teams and to the class as a whole. We discussed what general patterns and principles they extracted from all the customer interaction they had. On the last day of class, each team shared their Lessons Learned presentations, giving everyone in the class the benefit of what every team has learned.
We used this week to help teams reflect that they accomplished more than they first realized. For the teams who found that their ideas weren’t a scalable business, we let them conclude that while it was great to celebrate the wins, they could also embrace and celebrate their failures as low cost learning.
By the time the final week of the final Lessons Learned presentations rolled around, the students were noticeably more relaxed and happier than teams in past classes. It was clear they had a solid understanding of the magnitude of their journey and the size of their accomplishments – eight teams had spoken to nearly 900 customers, built 50 minimum viable products, and tested tons of hypotheses.
Pair Eyeware Be sure to look at how they tested their hypotheses on slides 11 and 12, and the before and after value proposition canvases on slide 13 -17. A great competitive Petal diagram is on slide 22
Share and Tell
Great story and setup in slides 3-7. Understanding their market in week 6, slide 31.
Allocate Notice how they learned about their customer archetypes on slides 12-14. After 80 interviews, a big pivot on slide 16.
Nova Credit Look at the key hypotheses on slide 2 and their journey in the class on slide 5.
Lessons Learned
Dedicating a week for reflections expands what everyone learns
Students extract the insights and meaning from the work they did
Discovery for both business and government is similar in that the only way to do it is to turn assumptions into facts by generating hypotheses, developing Minimum Viable Products and getting out of the building to test those MVP’s in the trenches where the customers and beneficiaries work. Early in the discovery process, teams are faced with a cacophony of personalities and organizations. Often, they struggle with understanding which person or group represents a beneficiary, supporter, advocate or potential key partner. It’s only through repetitive hypothesis testing that they begin to sort them all out.
Our customer discovery taught us that there were three additional beneficiaries:
By the second day the team appeared to be making lots of progress – they had talked to many women about their clothing line, and had marched up and down NY stores talking to buyers in clothing boutiques. They built detailed 
Whoah… the big insight was that how you initially “get” these key influencers – not customers or stores – is the critical part of creating a clothing fashion brand. This meant understanding these influencers was more important than anything else on the business model canvas. The team immediately added brand influencers to their business model canvas, created a separate value proposition canvas for them and started setting up customer discovery interviews.
But I haven’t forgotten that before everyone else thought that teaching scientists how to build companies using Lean Methods might be a good for the country, there was one congressman who got it first. 
The combination of the Business Model Canvas, 