Two of the hot topics in education in the last few years have been Massive Open Online Courses (MOOC’s) and the flipped classroom. I’ve been experimenting with both of them.
What I’ve learned (besides being able to use the word “pedagogy” in a sentence) is
1) assigning students lectures as homework doesn’t guarantee the students will watch them and 2) in a flipped classroom you can become hostage to the pedagogy.
Here’s the story of what we tried and what we learned.
—
MOOC’s – Massive Open Online Courses
A MOOC is a complicated name for a simple idea – an online course accessible to everyone over the web. I created my MOOC by serendipity. Learning how to optimize it in my classes has been a more deliberate and iterative process.
If you can’t see the video above click here
When my Lean LaunchPad class was adopted by the National Science Foundation, we taught our original classes to scientists scattered across the U.S. We adopted WebEx, a web video conferencing tool, to hold our classes remotely. Just like my students at Stanford, these NSF teams got out of the building and spoke to 10-15 customers a week. Back in their weekly class, the scientists would present their results in front of their peers – in this case via Webex, as the teaching team gave them critiques and “guidance”. When their presentations were over, it was my turn. I lectured to these remote students about the next week’s objectives.
Is it Live or Is It a MOOC?
After the first NSF class held via videoconference, it dawned on me that since I wasn’t physically in front of the students, they wouldn’t know if my lecture was live or recorded.
Embracing the “too dumb to know it can’t be done,” I worked with a friend from Stanford, Sebastian Thrun and his startup Udacity, to put my Lean LaunchPad lectures online. Rather than just have me drone on as a talking head, I hired an animator to help make the lectures interesting, and the Udacity team had the insight to suggest I break up my lecture material into small, 2-4 minute segments that matched students’ attention spans.
If you can’t see the video above click here
Over a few months we developed the online lectures, then tried it as a stand-in for me on the NSF videoconferences, and found that because of the animations and graphics the students were more engaged than if I were teaching it in person. Ouch.
Now the NSF teams were learning from these online lectures instead of video conferenced lectures – but the online lectures were still being played during class time.
I wondered if we could be more efficient with our classroom time.
Flipping
Back at Stanford and Berkeley, I realized that I could use my newly created Lean LaunchPad MOOC and “flip” the classroom. It sounded easy, I had read the theory:
1) A flipped classroom moves lectures traditionally taught in class, and assigns them as homework. Therefore my students will all eagerly watch the videos and come to class ready to apply their knowledge, 2) this would eliminate the need for any lecture time in class. And as a wonderful consequence, 3) I could now admit more teams to the class because we’d now have more time for teams to present.
So much for theory. I was wrong on all three counts.
Theory Versus Practice
After each class, we’d survey the students and combine it with a detailed instructor post mortem of lessons learned. (An example from our UCSF Lean LaunchPad for Life Sciences Class is here.)
Here’s what we found when we flipped the classroom:
- More than half the students weren’t watching the lectures at home.
- Without an automated tool to take an attendance, I had no idea who was or wasn’t watching.
- Without lectures, my teaching team and I felt like observers. Although we were commenting and critiquing on students presentations, the flipped classroom meant we were no longer in the front of the room.
- No lectures meant no flexibility to cover advanced topics or real time ideas past the MOOC lecture material.
We decided we needed to fix these issues, one at a time.
- In subsequent classes we reduced class size from ten teams to eight. This freed up time to get lecture and teaching time back in the classroom.
- We manually took attendance of who watched our MOOC (later this year this will be an automated part of the LaunchPad Central software we use to manage the classes.)
- To get the teaching team front and center, I required students to submit questions about material covered in the MOOC lecture they watched the previous evening. I selected the best questions and used them to open the class with a discussion. I cold-called on students to ensure they all had understood the material.
- We developed advanced lectures which combined a summary of the MOOC material with new material such as lectures focused on domain specific perspectives. For example, in our UCSF Life Sciences class the four VC’s who taught the class with me developed advanced business model lectures for therapeutics, diagnostics, medical devices and digital health. (These advanced lectures are now on-line and available to everyone who teaches the class.)
The class, now taught as hybrid flipped classroom, looks like this: 
There’s still more to do.
- While we use LaunchPad Central to have the teams provide feedback to each other, knowledge sharing across the teams still needs to be deeper and more robust.
- While we try to give students tutorials for how to do Customer Discovery we need a better way to integrate these into the short time in quarter/semester.
- While we insist that an MVP is part of the class, we need a more rigorous process for building the MVP in parallel with Customer Discovery
Outcomes
Besides finding the right balance in a flipped classroom, a few other good things have come from these experiments. The Udacity lectures now have over 250,000 students. They are not only used in my classes but are also part of other educators’ classes, as well as being viewed by aspiring entrepreneurs as stand-alone tutorials.
My experiments in how to teach the Lean LaunchPad class have led to a 2 ½ day class for 75 educators a quarter (information here.) And we’ve found a pretty remarkable way to use the Lean LaunchPad to organize corporate innovation/incubator groups. (We opened source our teaching guide we use in the classes here.)
Lessons Learned
- Creating engaging MOOC’s are hard
- Confirming that students watched the MOOC’s is even harder
- The Flipped classroom needs to be balanced with:
- Student accountability
- Instructor time in front of the class
- Advanced lectures
Listen to the blog post here [audio http://traffic.libsyn.com/albedrio/steveblank_hplewis_140211_FULL.mp3]
Download the podcast here
Share this:
Filed under: Lean LaunchPad, Teaching |
Steve, very much appreciate all that you’ve been contributing to the startup community.
At the risk of shameless self-promotion, and nearly a decade running strategy, corp dev, ventures, tech R&D and myriad other functions for one of the world’s largest educational content companies, I left with my partner to solve some of the problems related to building engaging learning experiences online. We think we’ve cracked the code not only here but around collective intelligence.
I realize this is a long shot but perhaps we could share with you what we’ve done and where we’re headed. At minimum it may help you think about online learning a bit differently, at best we may have a new platform you can use to continue to drive engagement (and track what students are doing/ not doing).
I’m the director of the Program in Bioinnovation and Entrepreneurship at the University of Colorado Graduate School. I recently migrated my face to face courses to an online platform and I feel your pain.
Let me know if you are interested in sharing your experiences at 3rd International Bioentrepreneurship Education and Training (BEET) Conference, June 20-21, 2014
Anschutz Medical Center, Denver, CO
http://www.regonline.com/3rdinternationalbioentrepreneurshipeducationconfer
arlen.meyers@ucdenver.edu
303-724-1952
We have also created the Society for Intenational Bioentrepreneurship Education and Training (SIBER) linkedin group
Arlen,
Thanks for the invitation.
steve
As the only non-academic member of the I-Corps-L teaching team, your Lean LaunchPad presentations on Udacity.com have been a helpful supplement to the assigned readings for our cohort over the past few weeks.
Letting participants know they’ll be “on the hook” to submit questions about the Udacity presentation, or letting them know they will be expected to report back on their team’s “Revenue Streams” or “Partnerships”, during the subsequent WebEx is a key insight.
“Pedagogy” is indeed a much bandied about word in the educational realm. I was recently introduced to the analogous “andragogy” which focuses on adult learning.
Thanks for sharing your experiences Steve! K-12 teachers are experimenting with all kinds of approaches these days – 1:1, flipped classrooms, iPads for every student, gamification, etc. What surprised me about your experience is an issue with “student accountability” – that is young adults at a top university we are talking about, not children! Then again, research shows that most people signing up for MOOCs actually drop out and don’t complete the classes either. I guess MOOCs are for truly motivated and committed.
Lilia,
Yep, “student accountability” was and is a big surprise. We also have the issue on customer interviews. We find that they soon devolve to phone calls /emails unless we tell them that they’re failing the class.
Doing this stuff is hard. Holding them accountable for it is just as hard.
steve
There have been quite a few studies which have shown that the average person to complete a course already has a strong first degree. Therefore, they have been through and know the level of work required to do well. I did one of the MITx courses a couple of years ago, and really enjoyed, but it was a reasonably big commitment to get the most out of it. At the moment, their real market is those who already have a good job and do not have to do the course, it’s simply for interest (which is great!). But, the lack of proper accreditation, transcripts, and identity verification means, for the moment, they are a luxury or an introduction to a topic at best. I’ll be really interested if and how this problem can be solved!
ps – I’m doing another one right now, they really do provide a good structure to learning something new!
Beautiful feedback. We have observed the same thing. There is no guarantee the student is watching the content online. Usually they are not.
I like how you adapted the model. Thanks for sharing your experience.
Derrick
There is at least one tool that all of you missed while setting up your flipped learning environment. The students knew that you didn’t have a way to make them watch the video/lectures as homework. This issue is easily resolved and has been taken into consideration as a necessary part of implementing the flipped learning environment. eduCanon is a software that allows the teacher to add questions to the video onto the timeline. When the student first watches the video they have to answer all the questions one by one and are not able to fast forward the timeline. eduCanon generates a report and sends it to you so that you can see who watched the video and what questions they missed. This feedback loop is a necessary part of a successful pedagogy. Without it you don’t know who is watching the videos and you also don’t know if your video lessons are getting the desired results.
Mr Sherman and Dr.Blank. I am no expert but it seems to me that you are trying to “make students study”. I learned a long time ago that knowledge does not move this way from teacher to student. “It is not the teacher who teaches. It is the student who learns.” It is not a good idea to try to increase the students commitment. You have to demand 100% student commitment. If a student does not do his job there will be no learning and thus he will fail. The same thing happens when it comes to start-ups. If the entrepreneur does not do his job he fails. See if you can pivot your way of thinking according to this motto. “It is not the teacher who teaches. It is the student who learns.”
Mr Basile, I agree with some your position. It is the student’s responsibility to learn, however teachers have to account to their administrators who demand that the students do the assignments. So to satisfy the administrators quest for data, these quizzes are “necessary”. Beyond that, they also serve the teacher as feedback about the quality of the lesson. If they see a pattern develop that indicates the lesson is flawed, then they can fix it. When I was in teacher education back in 1975, I learned that good lessons have “tests and measurements” designed into them so that both the student and the teacher could track the student’s progress. Without this feedback loop, the teacher has no way to know how to differentiate the instruction to accommodate for the student’s lack of understanding. The videos and quizzes are not the whole of the lesson. They are made to replace and “flip” the purpose of homework with class time. The videos become the “lecture” that was formally given during class time. In a flipped learning environment, the learning is best accomplished during class time while the students have access to each other and the teacher, now that the time is not being spent on lecturing. So while working on projects and or story problems and applying what they have learned, if they have any questions they can get them answered immediately and move onto more challenging content. I am a strong proponent of project based learning precisely for the reasons you posted about. I am however confused by your statement, “It is not the teacher who teaches. It is the student who learns.” I get the part about putting the emphasis on student learning. However, if the teacher doesn’t teach then what are they doing there in the classroom? This could be a matter of semantics. I for one like to use the term facilitator for the role of teacher because it re-positions the onus’ of learning onto the student and redefines the role of the teacher as one of a facilitator.
Great work and classes to help entrepreneurs all over who may not have the access to this type of education and assistance with creating their startup. You have made the latest information and classes available to anyone willing to study and apply your principles!
Regina Herzlinger and colleagues at HBS will be launching a Health Innovation MOOC. Be interesting to see how or if her platform differs.
Hold on, everything you do is tracked online. As an adviser, or top course on Udacity why can’t they add a “tracking” feature.
You set the schedule and the student list. If they have not watched the automatic reminder email goes to the student. If they don’t watch the site automatically tattles to the teacher.
I find the biggest problem with online learning, including MOOC’s, is there is too much focus on trying to replicate the traditional classroom approach. In online learning, rote techniques don’t work and a social learning environment needs to be created to engage students. My question is why have them write questions about the topic then present the questions in class? They can start the discussion through a social media channel right away allowing for a larger group discussions on the topic, mentor feedback, and lead to a deeper understanding of the material, which will produce better team presentations. The accountability piece can be “you must post and respond to comments X amount of times.”
With regards to problems with MOOC’s besides the lack of social learning and interaction, there is no assessment of students current knowledge, which can lead to students being overwhelmed or underwhelmed and lead to high dropout rate. The rational behind MOOC’s has some fallacies built into its program. The idea of taking the best and brightest professors from prestigious Universities and have them present information to the masses on their expertise does not take into consideration, how these “Rockstars” got where they are now. In higher education, Professors are rewarded on their research and writings, not on how well they present the information to the students. Often the task of presenting the information to students falls on the TA’s. On a whole, “Rockstar Professors” are not really good “Edutainers”, of course Steve your the exception:-) Really enjoy your writing on the intersection of entrepreneurship and education.
Karen,
“start the discussion through a social media channel…”
“you must post and respond to comments X amount of times.”
brilliant.
steve
I tried flipping the classroom for two courses (one of them was a bootlegged version of Lean Launchpad). I knew students will not watch the videos without a carrot/stick. Two things worked:
* 5-minute multiple choice quiz on the videos/readings in the next meeting.
* Assignment = produce output. Videos/readings are resources that would help them produce the output. Output never lies.
I remember research showing how the teenage brain could only handle short-term rewards. This has been my experience. Immediate gratification/failure has been the most effective spur to make them work hard.
A 3rd party (no less than Google) validates the effectiveness of this approach. They have an annual competition participated by 12,000 students worldwide called Google Online Marketing Challenge. In each of the past 3 years, a group of my students have been either regional finalists or regional champions.
I love your work on the flipped classroom, Steve! We MUST find a way to make better use of class time. Afterall, research has shown that half of what we say to students is immediately forgotten. I love facilitating in-person workshops, and there I spend less than 20% of the tie saying anything . . . instead I set up exercises, games, experiences, discussions, and student presentations.
Making people smarter isn’t sufficient. Behavior change is the key to changing the results people get. Muscles get stronger by doing exercises, not reading about exercise.
Great work, Steve!
Onward to more experiments! Woooohoooo! – Kimberly
I work in L&D and I find your comments very interesting. Its always a challenge to get the balance right between ‘input’ and Learning activities by the student outside the classroom.
Thank you Steve for sharing this.
I have also been experimenting with flipping the class with your Udacity “How To Build A Startup” lectures, here in Paris. Students seems to be learning, but it’s too soon to draw any conclusions on my side.
I wanted to forward with you this email one of my student – who is now one of your student.
“I was watching How To Build A Startup the other day and my mom (who is visiting from Canada) was listening in and taking notes on her computer because she does things like that. She sent her notes to my stepfather, who was delighted to see in words what he’s discovered throughout his lifetime as an entrepreneur. He started his career as a developer but has now moved into the hotel business. I just sent him the link for the course because he wants to watch it. I thought you would like to know that your idea to have our class watch the Udacity course has gone all the way to a little city on the west coast of Canada.”
Franck,
Thanks for passing on the note from your student. Always great to hear.
steve
Very interesting article. I am enrolled in both Coursera and edX. It’s amazing that from my little town Trapani I can take courses given by universities in America or anywhere in the world. Who would have imagined this?
Hi, Steve I just finish taking your mooc class from Udacity, it was a great experience. This was my first time taking a mooc and the way that it was presented was very educational. Being an entrepreneur graduate from Baruch I have to say that this class was very refreshing and it gives an extra push to really be innovative.
Thank you.
Hi Steve – I’d also like to add my thanks to all that you do and share.
I’ve completed your How to Build a Startup course, and hope soon to take in the new material you’ve created with the Kauffman Foundation.
A couple of observations:
– Actually watching all the way through your lectures takes a long time; I found that doing my ironing at the same time[!] was actually easier than sitting in front of a screen: otherwise I got fidgety and restless
– You might be interested in Rob Fitzpatrick’s http://momtestbook.com/ which gives tips and insight about conducting customer development interviews.
Hope this helps, and please keep doing what you’re doing. Thank you.
This is not spam. I have no affiliation with eduCanon. eduCanon is a software that is written by a teacher and his code buddy, who recognized the need for a way to make sure that students watched the videos and also that the teachers would have a way to measure the success of the lesson and make changes as necessary. Oh, by the way teachers can use eduCanon for free.
Sure giving quizzes may well be perceived as punishment or an insincere attempt at extrinsic motivation so too are grades and so far as I can tell teachers still use grades. When they no longer need grades then I guess they will no longer need quizzes. That will be the day.
I have been studying the use of video in the classroom because I am a 3D animator who produces educational videos and writes posts about how and why someone might want to use them in their classroom or online.
To quote “Derrick Comfort, on February 11, 2014 at 7:25 am said:
Beautiful feedback. We have observed the same thing. There is no guarantee the student is watching the content online. Usually they are not.” Well eduCanon was developed with the specific intention to resolve that issue and many more that arise out of assigning videos as lessons. So, there it is, if your instructional design uses video assignments and you want to make sure that the students watch the video then use eduCanon and you will get a report that they have done so as well as data about what they learned from the lesson. If you don’t want to collect data through a quiz then don’t bother asking any questions. When using eduCanon for the video lesson the student must watch the video because eduCanon will not let them fast-forward through it on the first viewing. Then at the end or during the video you can ask them questions about the content of the video. The video pauses for the question and will not continue until the question is answered. If you want to learn more about your lesson and how well it worked with the student, then ask some questions and get their answers just like you do with any other lesson.
[…] Steve Blank […]
Reblogged this on Boston Teacher and commented:
I’ve been looking into MOOCs a lot over the past year, especially when and how they work best. How could we make the MOOC model work in your local public elementary school?
Excellent stuff as always. I actually discovered Steve via Udacity and I loved the way you adapted the lessons from RSAnimate. Those guys are on to something.
For those who came in late:
http://www.thersa.org/events/rsaanimate
I got a lot from everyone’s comments too.
Ash,
Thanks for the pointer to RSA Animate, I’ve never seen the site before.
So no, “… they way you adapted the lessons from RSAnimate” isn’t correct.
But they have good examples.
steve