Case study: Udemy’s Content Evolution
How Udemy navigated five stages of evolution to build one of EdTech’s biggest on-demand course marketplaces.
👋 We’ve got a cracker of a case study this week. It’s epic but stick with it, there’s so much to learn.
I’m also running a FREE workshop with “first cheque” EdTech investor Emerge on The Never Ending Road to Product-Market Fit next Tuesday. Sign up here.
We’re into the final weeks of recruiting for our Autumn cohorts. Get in touch if you’re interested in finding out more.
“Product market fit isn't a destination. It's a journey,” says Frank Visciano, Udemy’s former Chief Operating Officer, now a startup advisor and fractional executive.
“Often when people talk about product-market fit, they talk about it as this thing that you need to get, and once you get it, then you build a business. But the way that it actually plays out is that product-market fit is a moving target.”
He expands: “Product-market fit is a recipe for growth that you uncover. And then as soon as you uncover it, it starts to mutate and slip. You have to fix the plane as you're flying it. And when I think about Udemy’s journey and why we were successful, it’s that we built a really effective learning machine to rapidly understand and evolve and improve and fix product-market fit over time. And within the context of a content marketplace business, your product is your content.”
Frank led the on building the supply-side of Udemy’s marketplace. Given that Udemy is one of the most successful EdTech content marketplaces, he is often asked for his reflections of how they did this. In response, he has identified five phases of evolution, which we’re exploring today.
He describes these as:
The Wild West: cracking the chicken and egg dilemma
Brute Force: build the minimum viable content base
Quantity Ramp: shifting from outbound to inbound supply
Scaled Quality: tightening quality to improve student experience and retention
Strategic content: selectively acquiring new content to seed new markets
We’re going to dig into the realities of that journey. What they did during each phase. The signs they recognised to know when to move to the next stage. And how the team evolved alongside each new reality.
“Basically, it’s just us figuring out over and over again, who's our audience, what's their pain point and what are we giving to them to fix that?” smiles Frank.
1. Arriving in ‘The Wild West’
Frank joined Udemy in 2013 when they were only twenty people, split between the US and two of the co-founders’ home country of Turkey. He says they were in what he describes as “the throes of nascent product-market fit.”
He’d taken the job with the goal of hunting down university partnerships. But in the two weeks between accepting the role and arriving, his brief had already changed, “we decided edX and Coursera and Udacity were way better at universities,” he explains.
The company's thesis had rapidly evolved: “You can think about making education accessible as a supply and a demand problem.”
“Coursera and Udemy were both democratising education. But Coursera's approach was to take the best content in the world, which they believed was coming from the universities, and expand and democratise access on demand side. Give students who wouldn't have otherwise been able to take a course from Columbia the ability to do that.”
He contrasts this with Udemy’s new and emerging point of view. “Udemy's approach was almost exactly the opposite. We believed that the best teachers in the world weren't necessarily at universities. And so if you really wanted to get more people learning and democratising the demand side, you need to start by democratising the supply side.”
This perspective reframed the problem they were solving. “It meant that instead of trying to get university professors who already knew how to create a great curriculum and help them bring that online, we were trying to get everyday experts who had never created a curriculum before to become teachers. The fundamental problem Udemy needed to solve is how do you teach thousands of people who've never taught before to become great instructors?”
He describes his initial role as “smiling, dialling and writing 1000 emails a week.” They were trying to recruit instructors to the platform and in doing so figure out some of the fundamentals.
While the day-to-day may have functionally felt like sales, in practice it was very much product discovery and definition for their product of online learning. “What do we mean when we say a course? What do we mean when we say an instructor? What type of content, what categories, what format? Should our courses be free or paid, or both? We did all sorts of different kinds of experiments,” he says.
To start with they needed the minimum amount of content to achieve the headline metric: number of students. Initially they did this by structuring Creative Commons License Content into courses.
Along with realisation that they would focus on democratising supply, there were two other key tipping points in this phase. “We developed a point of view of which topics mattered. Programming was the category that we were going to go after,” he says.
The second was affiliate marketing. “Udemy had a kind of an early days blowout success with a company called Startup Digest. Once we did that, there was a bit of a rinse and repeat.”
He says that in a marketplace business, there's always a bottleneck. You're either supply constrained or demand constrained. At the very beginning, they were demand constrained.
“Once we figured out affiliates was a channel we graduated to the next stage, where all of a sudden we were now supply constrained. It was then a question of how do you get an absolute ton of content really quickly?”
2. Applying ‘Brute Force’
There were still plenty of open questions. “Who is the instructor was still quite top of mind. And there were questions emerging around quality standards. If you believe that anyone can teach, can anyone teach anything? Or do there need to be some guardrails around that?”
Two specific things became crucial to reconcile in order to help answer these bigger questions. One was pricing. The second was the relationship and business model with content creators.
Pricing
“The big conundrum was that an online course was not an established thing,” Frank explains. “We were trying to define it. And we had two parties that we were trying to match together in a marketplace: students and instructors.”
It turned out that these two audiences had very different perspectives on price.
“We talked to instructors, and we asked them, what do you think a course is worth? They would say, ‘all right, well, I give lectures at conferences, and when I do that, I get paid $100 per hour. And when I write a book, I get a $10,000 advance. And when I do consulting work, I make $300 per hour as an advisor… And so therefore, I think probably an online course is worth a couple of thousand bucks. $5000 to $10,000.’”
They then asked the students. “And they would say, all right, well, if I'm not taking an online course, probably I'm reading a book, right? And so, you know, I'll probably pay like, 30 to 50 bucks for that book.”
These positions were wildly different. The challenge was how to reconcile them.
“The ‘ah-ha’ unlock was discounting,” he says. Udemy allowed instructors to set their own list price. But they took control over the sales price by creating opt-in marketing programs for instructors that utilised discounting.
“Instructors could price their course at $1,000,” he explains, “and then we would then use discounts to basically bridge the gap between students’ and instructors' expectations. We would bring the price way down and then convince instructors that this is actually a different equation than they thought.”
Slowly, instructors realised that the model wasn’t a high price multiplied by a low volume. Instead, a lower price times higher volume gave them a much bigger pie.
This approach explains why in the early days, Udemy offered such big discounts. “It wasn’t e-commerce sloppiness,” grins Frank, “it was a reaction to some pretty fundamental customer needs that, in an early stage marketplace, were almost irreconcilable.”
Exclusivity
At this point they were approaching established content producers like Pearson and Wiley who were looking for ways to take their books online, as well as individual experts and YouTubers.
Their investors were asking what their ‘moat’ was. How would they defend their position? “The initial instinct was to steer toward exclusivity,” says Frank. “We got pretty far down that path where we were negotiating with a lot of the big top publishers to lock in their content and make Udemy the only provider for them.”
But in the background, Frank and his team were starting to understand that it wasn’t the Pearsons and Wileys where they were finding most demand. Instead it was courses from ‘homegrown’ individuals.
“Chris Bryant, was teaching IT certification. All of a sudden his business was blowing up. He was making tens of thousands of dollars a month,” remembers Frank. “And so our point of view of who our most important instructors were was changing as we were navigating this decision. Eventually, we started to believe that the answer to who our instructors were was much more about everyday experts.”
Because these kinds of individuals typically had an ecosystem distribution strategy, they realised that it was important not to fight for exclusivity. Instead, they asked for a higher revenue share.
“It was hard to pull off, but ultimately it then gave us more margin to pour into our marketing activities,” he reflects. “This meant that we were able to acquire more students and that made it more lucrative for instructors. It unlocked the flywheel on the marketing side, which then became a really big competitive moat because there was nowhere else you could go to host your course that was going to bring you as many students.”
Revenue share
To begin with, Udemy took 30% of the revenue, dropping to 10% when the instructor brought the student to Udemy. The significant shift was introducing a tiered revenue share structure.
“We changed it so that if Udemy generated the sale via paid or affiliate marketing, we’d take a higher share of revenue. So we take 75% and you as the instructor take 25%,” says Frank. This gave Udemy more margin to expand marketing activities. “You should be okay with that, because your marginal cost to deliver that additional student is zero.”
But they also flipped this around. “We said, if you as an instructor, bring the student in because you do your own promotion, then you keep 100% of that revenue,” he explains. “We were fine with that, because it created a new user acquisition channel for us: instructor promotions. They brought people to the platform.”
By this point they knew that people who came didn’t take just one course, they took multiple courses with different instructors. So Udemy made money from cross fertilisation and instructors were happy because they felt in control. This also further incentivized instructors to do their own marketing as they were rewarded handsomely for it.
“It was a game changer in terms of what it unlocked on the marketing side,” he remembers.
Team and a shared North Star
During the ‘Brute Force’ stage it was all about outbound sales. “It became a bit of a ‘bodies game’,” says Frank, saying this is where they first started to grow the team and rely more on virtual assistants.
They also invested in helping instructors move their courses onto the platform. “Maybe they've written a book, but they've never created an online course,” he says. “We would help them to do that. It was not at all scalable, but that was really helpful to get people to migrate courses over.” They did the same for YouTubers, helping them organise their videos into a coherent curriculum.
They organised around the needs of the instructor. “You had a supply side, general management approach,” says Frank. “Sales, marketing, business development, product and operations, all oriented around the instructor.”
They all rallied around a new North Star: number of courses. “We were religious about doing anything and everything to get as many courses we possibly could. That meant getting courses from new instructors, and also getting existing instructors to create new courses.”
They were also starting to see instructors bringing multiple courses. So this meant as well as sales, it became about account management. They split the team, with some focusing on existing instructors, others going after new instructors. “It gave the team a really, really clear focus about where we were spending our time.”
The ‘flywheel’ had started to spin.
3. Ramping up quantity
The next stage was prompted by the switch from outbound to inbound. Instead of needing to go out and find instructors, they now needed to manage the incoming supply of courses that were now being generated by word of mouth.
This meant “starting to get smart about how to organise the rapidly scaling supply side,” says Frank. By this point they were going from 1000 to 5000 new courses per month, and starting to consider international expansion and category management. They were starting to see new opportunities around categories like design.
“Marketplaces create a lot of data,” he says. “You can pay attention to emerging trends, and then use those to inform your product expansion.”
Once they had spotted a trend and decided that it was important, they would assign a person to it to help accelerate it. These opportunities typically fell into two types: language and category.
Languages and categories
By now, 40% of their customers were outside of the US but only 1.5% of their revenues were coming from local language content.
“If you believe that the promise of a marketplace business is to provide relevant content, we weren't really living up to that promise,” says Frank. “We almost had an imperialist approach: taking English content and selling it to Brazil. So we saw that as an opportunity to expand our product-market fit.”
They picked a handful of languages to begin with and assigned owners with the goal of building a ‘minimum viable supply’. They were back to using brute force.
“We were sort of taking a half step back,” says Frank. “The goal was to acquire the 50 courses that will generate 50% plus of new revenue in that region. This was the gateway to getting people in, in the local language.”
Once they were successful in getting the flywheel to spin in that language, they picked the next market. “The playbook was 80% the same,” says Frank.
But there were nuances. After initially treating the Spanish market as a cohesive whole, “we learnt that Chileans were happy to take courses from Spaniards, but Spaniards wouldn't buy courses from Chileans.”
They also realised that alongside the universal demand for programming and tech, different markets were interested in different categories. “Health and fitness emerged as a category that the US didn't care about, but Brazil really loved,” he remembers. “Germany was weird. Photography blew up, so we followed that track. We developed more great photography content and then it became important in Japan too.”
They got good at spotting trends, assigning owners and exploring if new languages and categories had the potential to scale. The team was now a growing band of entrepreneurial folks that liked the hustle. But they were careful to keep the focus on collective learning and made sure that they passed insights and practices back and forth in team meetings.
“You can't have 180 different playbooks. Your business would just explode!” says Frank. “So we believed that there might be three or four different playbooks of clusters for how you grow a business.” They tested their hypothesis that Brazil, India and Russia might behave in a similar way. That German was typical of Western Europe. And so on.
“We wanted to set up a structure and a process where people were solving for growth, by finding the insights that overlapped, rather than chasing the idiosyncrasies of a given market.”
He notes that at this stage, in terms of team structure they were prioritising accountability for growth opportunities and not ‘organisational cleanliness’. Categories and languages were both given owners and the team had to find ways to collaborate where there were overlaps. This meant hiring for people that were comfortable with this ambiguity and need for collaboration.
4. Scaling up quality
Once they were finding better product-market fit by providing more relevant content, there were two new factors that led them to switch their focus further towards quality.
Firstly, they were starting to explore business-to-business opportunities with enterprises and questions about being able to provide a consistent course quality kept emerging.
Secondly, they were trying to improve retention and the likelihood of students buying a second course. Plus, they were starting to see diminishing returns on offering lots of courses in the same area.
“The difference between getting your first versus your fifth course on Python programming is meaningful in terms of increasing quality,” says Frank. “But the difference between a 50th course and your 100th course is very diluted.”
They asked themselves if they weren’t chasing volume, what would they pay attention to instead? “The answer was, we should pay attention to quality,” he says.
And so the headline metric for the team changed from Number Of New Courses Per Month to Net Promoter Score (NPS), weighted by number of enrolments. “We wanted to both have higher quality courses on average, but also skew our enrollments toward our best courses.”
This changed the game. “It became less about acquisition and more about creating the right standards and processes to level up instructors,” he says.
Defining quality
To figure out how to do this they hired a Data Scientist. “We called them a Learning Scientist,” explains Frank. “His job was to help us figure out how to define what course quality means.”
The team took several hundred courses and coded them for different attributes. “It could be everything from obvious stuff like the topic, length or structure to more superficial elements like the price. Price did impact a customer's perception and actual experience of quality.”
They did the same for the student’s learning objectives. “What was the student who enrolled into this course trying to get out of it?” They began collecting this at the start of the course and then asked students at the end how well the course had met this goal.
The third part was the instructor. They started coding objective criteria but also more subjective things: “For instance, was the instructor charming or funny?”
They ran a regression analysis to understand which of these variables mattered most and came up with 17 data driven standards that they then used to grade courses that entered the marketplace.
Enforcing quality
This then changed their entire approach to working with instructors. “I remember going to an Ask Me Anything event for instructors during this period,” he reminisces. “I said to them, ‘if you have created a low quality course, it's my job as the head of the instructor team to bury it.’ We had to make them understand that the new dimension of competition for them was going to be course quality.”
Once they had enforced quality ‘upstream’ by implementing the 17-point quality check and doing things like providing more education on making good courses, they started making improvements ‘downstream’ in the platform.
They spotlighted quality instructors, not just their top earners as they had done previously. They made course reviews really prominent. Then, more specific reviews highlighting reviews from learners with the same learning objective. The partnerships and product team working together in “a kind of joint venture”.
Revolution for instructors, evolution for the team
Frank says that this was a ‘revolution’ for the instructors. For years they had talked about quantity and revenue. Now they were talking about quality.
“There were some instructors that didn't like the new rules of the game, because they had been winning for years and all of a sudden they were getting dethroned,” he says. “There were a lot of instructors that were able to make the leap to the new world, but there were also incumbents that got toppled by new challengers.”
For the team though, the change was more of an evolution and a welcome, cathartic release. “One of the cool things about EdTech is that you attract people who really have a strong sense of mission and purpose,” he says.
“Even though I had people that loved getting up every day, obsessing about the revenue number for Brazil, they also cared personally and very deeply about the quality and the impact of what they were doing. There was always this tension between, are we driving revenue, or are we driving student experience? And so when we finally got to the place where we were ready to graduate and give up on quantity and focus on quality the team was euphoric.”
5. Strategic content
The final stage is another story of diminishing returns. This time in improvements in quality.
“When the average Net Promoter Score for courses jumped from 30 to 60 it made a big difference to how many students became sticky, lifelong customers,” he says. “But going from an NPS of 60 to 65 was like zero. But the amount of work you need to do to achieve that improvement was really heavy.”
At this point, partly driven by the increasing share of business-to-business, the goal became about efficiency. “How quickly could we bring the right content in to meet new demands?”
On the team front, after the previous phases of fluidly organising to meet new opportunities, the new goal was about reintegration and organisational clarity.
“This was a period of time where the team consolidated,” he says. “It didn’t make sense to have category managers for the consumer marketplace in Brazil, and a totally separate B2B team.” So the team became smaller but they were careful to ensure that the operational side was still effective.
Summary
We look back over the five stages:
The Wild West: identify the right content and build enough supply to create demand via an effective marketing channel
Brute Force: meet the demand by building the minimum viable content base and jump starting the flywheel
Quantity Ramp: shift from outbound to inbound supply and improve matching by introducing new categories and languages
Scaled Quality: introduce tighter quality control to improve student experience and retention
Strategic content: selectively acquire new content to seed new markets and improve efficiency
Over these five stages Frank thinks there are two key things to keep in mind.
First, in education, learning is the product. “Understand the interaction and overlap between your product in terms of your tech experience and your product in terms of your content, and to make sure that those are aligned and pushing in the right direction in a unified way.”
Second, is the evolution of product-market fit. “Things should evolve over time if you're growing. You should hire people that are cool with that evolution, and manage expectations appropriately. Once you figure something out, that's not the end, it's just the beginning of the next chapter. Think about it as a phased growth path, as opposed to a one and done. That is super important.”
Frank is now consulting and advising startups. He’s passionate about green energy, education and healthcare and a marketplace geek. Get in touch if you could use his help!
This case study features in my programme Finding Product-Market Fit in EdTech. Next cohort starts in October.