What happens when the bottoms up, organic growth usually associated with consumer companies starts to go…. enterprise? Part of our continuing podcast series (you can listen to part one on user acquisition and part two on engagement/retention) on growth, this episode explores the increasing trend of enterprise growth shifting to be more “bottoms up” — with a16z general partners Martin Casado and Andrew Chen, and DocSend CEO and co-founder Russ Heddleston, in conversation with Hanne Tidnam.

So what exactly does more bottoms up growth for enterprise look like? And then how does organic growth map into the direct sales model we traditionally see in enterprise? How does it affect company building overall? What changes in how we evaluate growth, what do we look at… and how can those two different models work best together?

Show Notes

• New growth patterns that are emerging [0:36], how DocuSign broke into the market [3:04], and the company profiles that most benefit from a hybrid approach to growth [4:39]
• Important metrics that companies and investors should watch for [7:05]
• When it’s time to hire a sales force [12:38] and other signs of rapid growth [14:55]
• Pricing and packaging issues [17:43] and further discussion around when it’s time to bring in sales [21:20]
• How new growth patterns are leading to organizational changes [25:15]

Transcript

Hanne: Hi, and welcome to the a16z Podcast. I’m Hanne, and today we’re talking about another aspect of growth. This episode is about the growth typically attached to bottoms up consumer companies, but that’s now more and more showing up in enterprise. So what does that more bottoms up growth for enterprise look like? How does it affect company building, how does it change how we evaluate growth, and what do we look at?

Joining us to talk about the tactics and questions we should be thinking about in this kind of hybrid scenario are a16z General Partners Martin Casado and Andrew Chen, and Russ Heddleston, CEO and co-founder of DocSend.

New patterns of growth

Hanne: Let’s start with the super basic question, which is what exactly are you starting to see happen with this shift in enterprise?

Martin: So traditionally in the enterprise, you’d build a product, and that product would be informed by your knowledge of the market. And then once that product was ready, you’d go ahead and sell it by hiring salespeople and the salespeople would go directly engage. You’d probably do some sales-led marketing where maybe the salespeople would go find the customers or you’d have some basic marketing to do it. But the majority of the go-to-market effort in the early days was this kind of direct sale.

And we’re seeing kind of this huge shift, especially in SaaS and in open source where companies establish massive market presence and brand and growth using these kind of more traditional consumer-ish growth motions. And then that very seamlessly leads into sales, and often a very different type of sale. And so I think a lot of people in the industry are on their heels, both investors and people that have started companies in the enterprise before, they’re trying to understand exactly what’s going on.

Hanne: Is it actually seamless? Is it a seamless transition there?

Martin: Well, I mean, that’s often the question, right? So we’ve seen companies moving on either sides of this. Some companies are like, “You know, listen, we’re just going to do organic growth.” And they don’t actually do sales. And in our experience, these tend not to be kind of hyper growth on the revenue side. Right? So they’ll continue to kind of growth customers, but it’s hard for them to get these nice, hyper linear revenue growth.

On the other hand, we see companies that will just do sales. And for them, it’s actually very difficult to grow quickly because they don’t have the type of funnel that you’d get from the growth metrics. And the ones that seemed to have figured it out the best, what they’ll do is they’ll create kind of a brand phenomenon. They’ll get this growth, they’ll get that engine working and then they do kind of tack on some sort of sales on the backend and then those two motions work in tandem.

Russ: So if you’re a small startup, breaking into that big ACV sale is tough. You’ve got to have a really high annual contract value and everything is going to be more crowded. And it happens occasionally but it doesn’t happen as often. And if you’re trying to target a specific buyer, just getting access to them can be very challenging and that’s just a huge hurdle to overcome. Like, how on earth could anybody break into that? Consumer understands a lot of different tips and tricks because you have to be really frugal to acquire a customer that you’re just supporting with advertising to get someone who you make six bucks a year off of. You can’t spend any money to get that person. So there are a lot of tactics there that are really interesting. If you apply those to some of the B2B value propositions, you can actually break in in a way that no one else was really thinking about before.

Hanne: Well, let’s get into those. What are some of those?

Russ: The way we broke into the market is we took a relatively simple workflow which is sending content from one business to another business. And so we said, “Okay, a better way of doing that is to allow the person sending it to create 10 different links to the asset, send them off to 10 different companies and see what happens to them.” How long do they look at each page? Who do they forward it to? You can see what people care about.

And so the first version of DocSend was just free. That actually just gets people using the product, and it’s cheap enough that they can keep everything else in their stack. So we’re not replacing anything, we’re purely additive at that point. And that’s really how we got our toe hold in the market.

Andrew: Russ, how did you get your first 100 users?

Russ: I think the first revenue we got was in the form of a bottle of whiskey that someone gave me as a thank you for giving them a account that they used for their own fundraising process.

Hanne: What kind of whiskey?

Russ: You know, I don’t actually remember it. I think the office consumed it relatively quickly so I don’t think it was around for very long.

Andrew: But from a top of funnel standpoint, where did you get the first…

Russ: It was all word of mouth. Forty-two percent of our signups are still word of mouth. Twenty-eight percent of our signups are from someone viewing a link and then getting interested and coming into the product.

Andrew: when you look back at Dropbox the first thing they did to get traction was to announce on “Hacker News” and also “Dig” at the time was such a big deal, right? These days, maybe the actual platforms have changed, like, maybe you go to “Product Hunt” instead, maybe you go to Twitter. But ultimately, doing a big announcement but then kind of getting the all sort of viral word of mouth means that a lot of your first users end up experiencing it because one of their friends wants to show them the product, or they just decide they want to try it. As opposed to having somebody sort of email you or call you up.

Hanne: Is there a certain kind of company that this works for better than others?

Andrew: I think that there are certain kinds of products that can be all the way pegged to completely self-serve, bottoms up versus maybe what’s kind of in the middle. Is the product a horizontal enough product that literally you can bring almost all of your coworkers — things like Dropbox, Asana, Slack, these are all things that everyone in your company can use, and so naturally is going to spread much faster because at every moment, each node in the network is going to be able to have access to all 15 to 30 people around them where it can spread.

The second thing is products that are actually really front and center in your workflows, all the acquisition that we see, especially virally, happens because of engagement. They’re deeply, deeply linked with each other. Because as you engage and as you’re using the product more, inevitably then you’re sharing links, you’re assigning tasks to people, you’re commenting on people’s files. These are all things that bring people back and bring new people into the product. There’s a whole class of products that aren’t completely horizontal that maybe only apply to a particular job title or function. And so that all of a sudden gets harder because maybe it can spread within the department, within the function, but it’s not going to go really broadly. And eventually you get to the set where it’s like, maybe there’s only a couple buyers in the entire company. And for that, you don’t go bottoms up at all. It’s just literally impossible.

Hanne: So this middle zone is what we’re talking about, where there’s some indication but it’s not completely horizontal and viral. It needs a little bit layered on.

Andrew: The new thing is that the fact that users can then bring these products into their workplace, and you might get a large company of 20,000 people with a patchwork of folks using a whole bunch of different products before IT actually makes a decision. Like, that’s new and very interesting.

Russ: Every company tends to have some form of super power that’s available to it based on just what their business is and what their product does. So we typically add features in one of three buckets. One is to increase the spread of a business to another business. One is to get more lock-in within a company itself, so getting that spread within the company. And then the third is just making our customers more engaged. Because the more they’re using it, the more they’re sending it outside the company. Our top request at one point was, “I need to send a folder of content.” And you’re like, “Okay, that makes sense.” But what they really wanted was this kind of deal room thing. So we ended up building Spaces. And that just really increased engagement of our customers.

Important metrics to consider

Andrew: That is why one of the really interesting things that, Martin, you and I end up talking about with these bottoms up companies is evaluating the engagement on the products using consumer metrics. Because often, it’s the engagement that’s really the leading indicator for growth, but from an acquisition standpoint as well as retention, which then is sort of the leading indicator for, like, are they actually going to renew their subscription over time?

Martin: So to me, this is one of the key questions. We see these companies that fall in between this kind of consumer-ish growth in this enterprise thing. And actually a question I’ve been meaning to ask you that I haven’t yet but this is a good opportunity, so is it the right thing to evaluate these things purely from a consumer lens? Are the growth patterns the same as you would see in consumer XX? Let’s even just put aside the question of sales. Should the growth metrics be the same as a consumer company?

Andrew: When you’re evaluating even purely consumer products, you have to really look at what the expected behavior is. And so I would kind of turn the same question for the kinds of things we’ve been working on, which is obviously if you have users that are trying out some new email security product, let’s say, hopefully they’re not interacting with it that much. But if the whole pitch of the company is, “Hey, this is going to be the system of record for everything that your team’s going to work on for all of their projects, or whatever, and they’re going to use it every day,” then it’s like, “All right, then let’s actually start using, you know, daily active metrics in order to evaluate if that engagement is actually there.

Hanne: What about from your point of view, Martin? Are there metrics that you…

Martin: Well, yeah, I think it starts to get a little complicated. So there are a number of consumer metrics you track. One of them is engagement which gives you a sense of how often it’s used, and maybe that’s something that you can proxy to value. There also is just simply top of funnel growth, right? How many people know about it, what is the brand? The world I come from is nobody knows about the product when you start. There is no organic growth. Marketing is, at best, linear with the dollars you put in, the number of customers that are top of funnel, it’s probably sub-linear. All the value and monetization is driven my direct sales and so you’re…

Russ: It’s account-based sales.

Martin: It’s account-based sales. So your ACV has to be high enough to cover the marketing cap. So that’s one bookend. The other bookend is all of this growth stuff you do acquires tons of customers and then the product will monetize itself, right? So my big question is, is there a slider bar here? If you slide the slider bar all the way to the left, there’s the Atlassian model, and there’s very little sales. And if you slide your slider all the way to the right, then it’s just direct sales and no marketing. And then the question is, what does it look like in the middle? Because you look at it like the slider bar is all the way to the left, and I look at like the slider bar is all the way to the right. But more and more, we’re seeing companies that actually they’re very interesting on both sides, but they’re not classic on either.

Andrew: Totally.

Martin: So let’s assume we take the case of the slider bar as all the way to the organic growth and it’s purely horizontal and it’s growing like crazy. So the question is does it still make sense to build a direct sales force? As in, will it increase the unit economics if you do? I think our experience here with Slack and with Hub and with many companies is…

Andrew: It’s definitely yes, right?

Martin: Yeah, the answer is yes.

Andrew: Because definitely yes.

Martin: Because that’s how you maximize ACV per customer, because there is a procurement process and just finding the budget and maximizing that is something a human can do much better than a product at this point in sales.

Andrew: Right, and in fact, I think actually even the virally spreading products end up going tilting towards enterprise over time for a really simple reason, which is that with larger companies your cohorts will look better because there’s revenue extension. Because no matter what, when you’re working SMBs, I find it very hard to get better than, let’s say, a 5% per month churn rate. All these little companies keep going out of business all the time, they’re fickle, they have small budgets, etc. And so what you quickly find is you have to go to the big guys, all the budget’s there. And so then that inevitably leads you, even when you’re completely bottoms up, to start building stickier products for enterprises and add the sales team, add customer service, and all of that. So I think that is the natural trend.

Hanne: my question is when is that happening? Is that happening in tandem all along? Are they sort of naturally that hybrid from the beginning or do they slide along as things change in the company’s cycle?

Martin: Specifically were you thinking about sales when you started?

Russ: No. Not at all.

Martin: The common refrain.

Russ: When we launched DocSend, we didn’t have any background in B2B. So it kind of caught us by surprise and we got a lot of interest that we weren’t able to convert into dollars because we weren’t even charging people. If we could do DocSend over again, I think we could build it in half the time. Because I think this is a new type of company that there aren’t that many examples for.

Hanne: if you were to put that very broadly as like the type of company you mean what is that type of company?

Russ: If you create a business value, like a B2B value for something, you build some product and you release it for less money than you should or free, you’re going to get some usage of it. If you’re creating a B2B value, you kind of picked your target audience, you get your 100 accounts you want to sell it into, and you have people just pound on their doors to get in there.

Martin: You literally start at the top of the list, you go to the bottom, and then you go back to the top of list.

Andrew: And I think when you compare it to consumer…I mean, for most consumer audience-based plays, you really defer monetization for a really long time. Because you have to aggregate this huge audience and then you start talking about, like, okay, let’s look at ad-based models. And so, and you contrast that to these B2B products where you can actually monetize from early on. And in fact, when you monetize it actually unlocks a bunch of paid acquisition channels, and it’ll unlock sales, and it unlocks a bunch of stuff. I think that’s very confusing for people who, you know, they get started and they’re kind of in this consumer products mindset. And so they often end up kind of like, “Oh, how I do grow? How do I increase acquisition?”

Inflection points to watch for

Hanne: What are the signs that that’s the right time when it begins shifting, the sort of tipping point where you’re like, “Okay, should I need to pay attention to this?”

Russ: We were just selling some small deals on the side. So I was like, “I think we should hire a salesperson.” So we hired our first SMB AE, and in our first month we’re like, “We don’t think she’s going to sell anything.” And she sold twice what the quota was supposed to be. There was just a lot of money laying around where if you actually talked to someone on the phone and explained it to them, they might have bought one seat before but now they’re going to buy 15.

Martin: Didn’t you have a support collecting checks?

Russ: We had a support person selling a lot of DocSend for quite a while.

Martin: That’s a pretty good indication it’s time to do sales.

Russ: Yeah, that’s another really indicator. Also, now that we’re going a little bit more up market, you actually need someone who’s able to run a good sales process even though they’re not doing the outbound part of it once you get them in the door, running a good sales process, having good sales hygiene, really understanding who your buyer is, you need to do all those things too. So you really need to marry both sides of it.

Martin: Another shift I’ve seen, which is important from a company building perspective, so if you think about direct enterprise sales, the actual lead up to the sale can take nine months to 18 months. You’re working the account, you’ve got an SE in the account and you’re educating them, etc. So with these new companies, often the customer is education themselves, they’re already trying, and so much of the actual total value of the account comes after they’re users of the product. So it’s about expanding the account. So now there’s this very interesting relationship between sales and customer success where a lot of the value is actually being driven by customer success. I don’t think the direct enterprise is used to this model.

Russ: Yeah, we always say, “You win the renewal when you do the onboarding.” And getting everyone engaged quickly with an account really helps with expansion and renewal. When we do onboarding, we have a little raffle. So if you’ve got 50 salespeople at your company and if you send a certain amount of DocSend links externally in the first two weeks, then you’re eligible in this raffle and you get one of three different prizes. It’s like a $200 bottle of whiskey or tequila or Amazon gift card. And that’ll actually…

Martin: What kind of whiskey?

Russ: I also don’t know. But that’ll actually get everyone using the product really quickly, and then they look at that and they say, “Oh, we bought the product for our sales team. Man, we should use this for our customer success team or our support team.” And so they build faith in it and then it naturally expands. Sometimes you need a salesperson involved, but more often than not, customer success is just saying, “Yeah, you can use it for that too.” And then they expand.

Hanne: So I want to get into the timing question of when, when this starts happening. When you happen into this moment, when all of a sudden you realize, this would be helpful, how do you begin to actually make that happen? What are the signs and signals that are telling you now is the time?

Andrew: Well, I think one really important one is what kinds of companies and people are signing into your service? Where you’re starting to see both prominent tech companies as well as Fortune 1000s just signing up to try it. Even on a purely bottoms up basis, you create the funnel from signing to using a contact enrichment service and starting to score all of these new users that are coming in. And if you find out that a large proportion of them are actually enterprises, that’s actually pulled demand from the market that you should actually be up leveling faster.

Russ: One of the things we actually did to spread that awareness faster is we decided that marketers will send off tons of things to people, so why don’t we just support the marketing use case? Not because we make more money from that. If we power, for instance, a researcher port for a company, they’re sending that to tens of thousands of people that then get exposure in lots of areas that we weren’t even in before. So it really kind of allows it to hop into other places, and then we generate more of that demand coming in. You need to take a look at who’s signing up for your product and you need to think about what might they be looking for and what problems might we be able to solve for them?

Andrew: Another thing I might add is what kinds of feature requests folks are having. If you’re building something that’s like an email client, something that is really horizontal or it’s a new document editor, everything’s great and all of a sudden, you start getting these future requests for Salesforce integration, and you’re like, oh, okay, this is like a different…

Russ: Another request we’ve always gotten has been DocSend, you can’t actually send anything from DocSend and it’s really nice to be able to send from email and customize it, and there’s a different philosophy around that but we were thinking, like, “Man, just let people send stuff right from DocSend. Because then it’s got a DocSend email that they get.” And so it’s actually a good growth thing, as well. So you can, kind of, reprioritize your product list based on how much it’s going to spread awareness about your product outside of the company, which is a great lens for every company to use when thinking about trying to make these viral loops go faster.

Hanne: That’s interesting. Okay, so say ideally you do have this kind of blended model going on. Are there conflicts ever in the types of information that you’re getting from the different sources?

Martin: At the highest level, I think there actually are a lot of conflicts in these motions and in a number of areas. And the most obvious one and this is something that’s so prevalent in open source is, a good way to get organic growth is to give something away for free. And if you give it away for free, it may be hard to monetize it because a lot of the assumptions here are predicated on organic growth, there’s always an open question of how much do you give away versus how do you monetize it? Enterprise really is all about monetization because there is no conversion between eyeballs and dollars like you do in kind of more advertising-like domains. And so there’s a real tension there.

Pricing and packaging

Hanne: So how do you think about that balance?

Andrew: It’s sort of funny because it sort of implies that you can go one way and not the other. Meaning, if you have a product that’s making a bunch of money and you have a highly functional sales team, and then a product person in the org is like, “Hey, let’s have a free offering,” that is not going to happen. Versus the other way where you have something that’s product led and it generates a lot of users and then you build this whole pipeline off of that and you build the sales org. If you do it in that order, all of a sudden the freemium product actually feels like it’s actually very helpful. Nevertheless, eventually free tends to go away or become pretty crippled as the whole business evolves. But freemium can be so disruptive in these industries because if you’re a large enterprise, B2B software company, you’re not going to be able to do this kind of low end free offering.

Russ: Yeah, a lot of what we’re talking about is just pricing and packaging which is something that’s so hard for everybody. because you’ll look at a company and you’ll look at their pricing and packaging, and you’ll be like, “Congratulations. You’ve done it.” But then when you look at a new company and be like, “What should their pricing be?” Everyone’s like, “I have no idea.” And it’s hard because you can’t AB test it. And so you have examples of what’s worked but it’s really hard to predict what will work for any given business and so you could say on the low end, we got a free thing. On the high end, we got an enterprise thing. And then maybe there’s something in the middle.

We actually just increased the pricing and added a couple new plans. And we thought the conversion would come down but we’d make more money. What happened was that conversion went up and we made way more money.

Hanne: And why do you think that was happening?

Russ: We moved some features around and then we talked about the plans differently and who they’re for. And so people also trusted it a bit more because they’re paying more for it. People then value it more and actually use it more because they’re paying for it.

Andrew: Right. Well, I mean this is the difference between also when Netflix increases their monthly subscription by $2, everyone’s screaming bloody murder. And B2B is obviously less elastic.

Hanne: “Oh, it must be good.”

Andrew: There’s some price signaling as well.

Martin: But it’s also important to compare it to traditional pricing and packaging. the general model used to be when you first come to market, you are as expensive as possible and you know you’re going to go for a limited set, but ACV is high enough to cover it. And the sales cycles are long anyways. And then after you feel like you’re saturating that, you offer lower priced units so that the aggregate market is larger net cannibalization. So you don’t want to cannibalize yourself. And the way you do this is market research of existing customers, you know the target customer base, and you can AB test. You can actually do fairly small rollouts because it’s not marketing led.

That motion is lost in this world because basically, as soon as it’s publicly available for free, everybody knows about it and it’s very difficult then to kind of retract that. So you have to be very thoughtful about pricing and packaging upfront because any experiment basically is reality now. And that’s very, very different from the traditional enterprise motion. I mean we experimented with pricing so much in the early days and the only thing you had to hold sacrosanct was price very expensive early on because you’re only going to get 10 customers anyway and you just can’t do that motion now.

Andrew: Even the way that you do pricing, it can potentially impact engagement. Where do you put your pay wall? Is it a time-based trial, is it a usage-based thing? Those things become really important because, especially when you have a product that is growing virally, it’s building a network inside these companies, you don’t want to cut off the network prematurely, because the network is what makes the whole thing sticky. So for example, it would not make sense for a product like Slack — if they were like, “Well, we’re going to cap the number of people that can join the channel to five,” that doesn’t make sense because the entire network effect is based on having all of your colleagues there. So what you end up wanting to do is you’re gaining these features that the IT admins want, and those are the things that end up being how you differentiate the enterprise customers from purely the consumer ones.

When to focus on sales

Hanne: When you start thinking about forecasting or planning, do you ever get competing signals and information from this blended model where you’re doing two different kinds of growth and sales?

Martin: Well I think this is a really interesting question of…for wherever you are in the lifecycle of the company, let’s say you have $1 to spend on go to market, how much of that $1 goes to brand and marketing, versus how much of that $1 goes to sales? And that is a question I don’t think anybody knows the answer to.

Hanne: But what are some of the ways you start figuring it out?

Martin: The traditional view in the enterprise is you spend it all on sales, basically, until you’ve got a working pipeline or a repeatable sale. Then you have economics you understand and then you start increasing the top of funnel. That’s the traditional model. But now, we’re marketing led. And so, how do you know how to split those dollars up and when to do it?

Russ: A lot of it has to do too with the DNA of the founding team. my two co-founders and I are all engineers and product people. And so we’ve basically used our product as the marketing engine for the company so far. We haven’t done any paid acquisition, we haven’t been doing a lot of marketing stuff that’s been driving a lot of the top of the funnel. The product itself is driving the top of the funnel.

Hanne: But that would be what most of these companies are doing kind of? In this kind of company, that would be common?

Martin: Well, okay, I mean there are a number of companies that will actually just buy their users. I’m totally not used to that. Andrew’s totally used to that. And so this is kind of…

Andrew: …Yeah, and I hate it. Yeah, there’s folks that they’re spending tons and tons of money on Facebook, on Google, etc. That’s very common. The other one as well is a huge focus on content marketing as being one of the primary channels I think that is really different.

Russ: It’s kind of going back to what we said earlier where, should companies invest in sales? And my view on that would be, if you show me a company that’s growing organically, I’ll show you a company that’s performing better if you also add a sales team to it. If you can get it working with the product, you can actually probably get a good baseline of growth, but you should probably spend more on marketing and sales on top of that. And if you can get the unit economics anywhere near reasonable for a paid acquisition, you should probably put everything you can into that channel, knowing it’s just a component of your overall strategy.

Andrew: The thing that makes it hard to normalize a bunch of these efforts is they happen on very different time scales. You can literally increase your paid acquisition budget and see a spike in signups and self-serve conversions within a 24-hour period. If you’re going to go and hire and build out your sales team, it’s going to take you months to build the team, and then months to recruit them. But when the revenue hits from these really large contracts, it’s huge. Hopefully, you have multiple systems that are mutually reinforcing each other as opposed to feeling like they’re in conflict. But that certainly happens if you are trying to figure out, where do I put the next dollar?

Hanne: I mean, what are some ways around dealing with that discrepancy between timeframes and planning and forecasting when you’re trying to match up these two very different chronologies?

Martin: I don’t think there’s any recipe. There’s never a recipe to doing a startup anyway. There’s no recipe to find product market fit. I don’t think there’s any recipe to knowing what’s the right balance between growth and sales and when to do it. But here are things that a founder should think about that has traditional enterprise expertise in the new world. The first one is brand. You normally don’t think about brand, but brand does drive viral growth. Product focus, right? The product itself actually creates virality. The enterprise very rarely thinks about, believe it or not, product. They think more about solving problems.

Hanne: Really? That’s so surprising.

Martin: It’s not about making the product “delightful” or easily consumable. It’s solving a real problem and adding business value and less about consumability, right? Now you have to think a lot more about consumability, like single-player mode, like self-service mode. Right? Very different than traditional enterprise. You need to design your company for bottoms up growth whether you’re open source or you’re doing SaaS or whatever, because this is the new method of consumption. And I do think that the one most important is if you’re doing bottoms up growth, I think you have to expect a lower ACV which is a different way to build a sales team. And so you just have to be more comfortable with your inside, inside/outside models and then you have to be more comfortable with focusing in on expansion rather than upfront ACV.

So these are all very, very different than the traditional enterprise.

Hanne: They’re sort of mind shifts.

Martin: They’re all mind shifts.

New organizational structures

Andrew: There are new organizational structures that end up being built within these companies that sit alongside sales because all of a sudden, you can have multiple revenue centers, right? And that’s a very different approach. Then the people that you hire for this end up being designers and PMs and engineers that are kind of this business-y, metrics focused folks. Going back to Dropbox, I know the most recent incarnation were sort of biz ops people turned PMs that were previously working oftentimes in consulting or banking.

Hanne: So it’s a new hybrid kind of role in organization as well that comes down from this?

Andrew: Right, exactly.

Hanne: That’s interesting.

Andrew: Do you want to hire the nth engineer into this team that can run a whole bunch more of these AB tests? Or do you build out your sales team more? These are the kinds of decisions that these companies have to make these days.

Hanne: Russ, did you see that as well that kind of hybrid role?

Russ: Yeah, there are a lot of things that aren’t just salespeople calling and getting contracts signed. Enterprise sales is like a playbook that makes sense. For the bottoms up company, you’ll see this perfect curve and kind of the outside view of that is they did something brilliant at the beginning and then everyone went on vacation and it just kept growing. But in reality, behind the scenes is a series of every smart things you did to keep that growth going. And what got you from A to B is not going to get you from B to C. So you often have to to redo your organization, you have to add in new roles, and you have to recognize when you’re going to hit points of diminishing return for a type of investment. And you have to get ahead of that and say, “Well, what’s the next type of investment we’re going to be able to do to get us to the next stage of things?”

Hanne: Add on another layer, right?

Russ: Right.

Hanne: As Jeff would say.

Russ: Yeah, it’s different for every company. There’s no one right answer.

Andrew: The really important key thing is the importance of not just a great product but literally great user experience and design, and all the fit and finish that you would expect with a completely modern consumer-facing application.

Hanne: Now that’s coming to this world too.

Andrew: Right, exactly. Like, Envoy, that is an amazing B2B viral story. They’re very rare, But the reason why people use that now is because offices are part of the brand experience. And then after they use the thing, then they’re kind of like, “Oh, yeah, we’re using pen and paper back at the home office. We need to upgrade to this too.” These examples crossover both the consumer sort of design world, all the way to sales, all the way to performance marketing. You really have to leverage a lot of skills in order to execute these strategies.

Russ: The expectation for the usability of software I think is going up in enterprises. Larger companies expect more polish and more usability. And if it’s not there, they start to really worry about it being shelf-ware or not the value proposition. And shelf-ware is a pretty big problem at a lot of big companies.

Andrew: One of the funny anecdotes at Uber was that for a long time, we were officially on Hip Chat but there were so many teams across the company that would have their little secret Slack team chat going because they just didn’t wanna…

Hanne: Illicit Slacking?

Andrew: I feel comfortable saying that now that Hip Chat’s been shut down. Employees will literally rebel and use whatever they want. And so as a result, as companies selling into these, your products have to be really good to compete with everything else that’s out there.

Martin: I didn’t understand how powerful actually just growth tactics were. independent of product. Actually independent of sales. Andrew, you and I were looking at a company which was amazing. Like the growth was amazing. Like all of these numbers were amazing. The engagement, they were monetizing, like everything looked great and the conclusion we came to was, like, it’s because they just had, like, such an amazing growth team that was almost independent of the product that they were selling.

Hanne: Oh my gosh.

Martin: We literally came to the end and we’re like, “Wow, this could be anything. This could be, like, you know, dog food. This could be, like…

Hanne: Doughnuts.

Martin: Yeah, whatever if you figure out how to do it right, it’s a very, very powerful thing. And by the way, that used to be what you said about sales. What you used to say about sales is if you have a very good sales team that understands the buyer, you know, it’s kind of independent of product.

Hanne: Awesome. Thank you guys so much for joining us on the a16z podcast.

Group: Thank you.

• 

  Martin Casado is a general partner at a16z where he invests in enterprise companies. Prior, he was cofounder and CTO of Nicira (acquired by VMware) and is the creator of the software defined networking movement.

  Follow Twitter
• 

  Andrew Chen is a GP at a16z in consumer technology. Prior to that, he led Rider Growth at Uber. He has written for a decade on metrics & growth on his blog/newsletter, and is the author of The Cold Start Problem.

  Follow Twitter
• 

  Russ Heddleston
• 

  Hanne Winarsky

Once you have users, how do you keep them engaged, retain them, and even “resurrect” or re-engage them? That’s the focus of this episode of the a16z Podcast, which continues our series on the basics of growth from user acquisition to engagement and retention — covering, as always, key metrics and how to think about them. Especially as many products and platforms evolve over time, so do the users, some of whom may even use the product in different ways… so what does that mean for engagement, and how can startups analyze their users? “Show me the cohorts!” may be the new “show me the money”…

Featuring a16z general partners Andrew Chen and Jeff Jordan, in conversation with Sonal Chokshi, the discussion also covers everything from how network effects come in to play (is there really a magic number or “aha” moment for a product?) to who are the power users (and the power user curve for measuring, finding, and retaining them). Because at the end of the day, you don’t want a leaky bucket that you’re constantly trying to fill up. That doesn’t work, and definitely won’t scale.

Show Notes

• Discussion of engagement and natural trends, including how to measure both [1:03]
• The importance of analyzing cohorts [5:39] and identifying the “aha” moment [8:44]
• Moving users up the ladder of engagement over time [10:55], further discussion of measuring engagement (DAU/MAU) [15:39], and the reliability of certain measurements [19:56]
• Network effects [22:04], engagement vs. retention [24:50], and how to measure retention [29:11]
• Advice for companies [31:07]

Transcript

Sonal: Hi, everyone. Welcome to the a16z Podcast. I’m Sonal. Today’s episode continues our series on growth — the first part covered the basics of user acquisition — and so this part covers, more specifically, engagement and retention. Including, as always, key metrics and how to think about them.

Joining us to have this conversation, we again have general partners Andrew Chen and Jeff Jordan. And we cover everything from how do network effects come in to is there really a magic number or aha moment for a product? To who are the power users and what is the power user curve for measuring them. But first, we begin with what happens after the initial acquisition phase, as different kinds of users join a product or platform over time — what does that mean for engagement; and how do you analyze them, using cohort analyses?

Strategies for measuring engagement

Andrew: One of the things that you see is that people end up using these products very differently. Because the kinds of users that you’re getting are changing over time. When you look at something like rideshare, you know, all the early cohorts are basically people in urban areas. And in these days all of rideshare is more like suburban or rural folks because you’ve saturated all of the center. And so what you tend to see is as you acquire your folks, your core demographic out that actually ends up showing up in the engagement.

And so, you know, going back to a natural “gravity” to the whole thing [discussed in episode one], this gravity also hits the engagement side of things as well — and then ultimately the LTV because your users were typically getting less valuable. I may take years to see this kind of play out but that’s kind of the natural law of things.

Jeff: There is a progression in these and particularly the ones that are really successful. Early on it’s all about getting users. <Sonal: Right.>  And it’s just like users, users, users. If you’re widely successful at doing that you run out of users (or you start running low on users) and you have to go to engagement. So Pinterest has a very high-quality problem right now. Most women in America, have downloaded the Pinterest app.

Sonal: Oh yah, I’ve had it for years.

Jeff: Some growth can come through, okay, there’s some number of women who never heard of Pinterest somewhere in the country. But much more so they need to engage and re-engage the existing audience. I mean, we love engagement from an investor standpoint because it’s just, you know, that [crosstalk]

Sonal: [crosstalk] It shows stickiness.

Jeff: You can often hack your way into new users. It’s really hard to hack your way into true engagement. <Sonal: Keeping them.> Someone is spending 20 minutes a day on your site. Offerup, Pinterest the major investment thesis was, “Oh, my God!” look at that engagement … And, you know, if they can scale the userbase it’s a beautiful thing.

Sonal: Right. What we mean by engagement is actually interacting with them and seeing their activity. Because to Andrew’s three points of acquisition, engagement, retention, the third piece is keeping them.

Andrew: The way that we’ll often analyze this is looking at cohort analysis.

Sonal: Yes.

Andrew: Where we’ll look at kind of each batch of users that’s joining in each week and really start to dissect like well, how active are they really and to compare all these cohorts over time. You’re basically putting the users that come in from a particular timeframe, let’s say it’s a week, and you’re putting them into a bucket, right? And what you’re doing is you want to compare all of these different buckets against each other.

And so what you typically do is you look at a bucket of a cohort of users and you say, “Okay, well, you know, once they’ve signed up the week after, how active are they?” And what about the week after that and the week after that and you kind of like can build out a curve. And it just turns out that these curves once you’ve looked at enough of them surprisingly, human nature, they all look kind of the same. They kind of all kind of curve down and for the good ones they start to flatten out and plateau and then, for the really good ones they’ll actually swing back up and people will come back to the surface. What you want to do is you want to compare the various cohorts against each other in time to see if you can spot any trends on how the usage patterns are, increasing or decreasing. When you add a new layer to a layer cake, you might unlock a bunch of new behavior. You might unlock a bunch of new frequency that didn’t exist before. Or alternatively, over long thresholds of time, people tend to become less active as you move out of your cohort segment.

Sonal: The cohort graduates.

Andrew: Whether or not a specific cohort of users flattens out is really important, right? Because, you know, if you’re in a world where they kind of slowly degrads and then all of a sudden it’ll actually go to zero, that means that you’re always kind of filling up the bucket — You have a leaky bucket, you’re constantly filling it up.

Sonal: You’re always filling it up. Right.

Andrew: Right, and what happens is that gets progressively harder because, if you want to keep your overall growth rate, because that means you have to double, triple, quadruple your acquisition in order to counteract for that.

One growth accounting equation that’s often thrown around is that you know your incremental — your net — MAUs, right? So your net monthly active users equals all the new people that you’re acquiring, minus all the people that are churned, and then plus all the people that you’re resurrecting…

Sonal: …Re-engaging.

Andrew: Re-engaging, exactly, that are coming back after they’ve churned. And so what happens is for a new startup you are completely focused on new users because you don’t really have that many users to churn, and over time as you get bigger and bigger and bigger what you find is that your churn rate starts to — it’s a percentage of your actives.

And so the evolution of most of these companies as they’re getting bigger tends to start with acquisition, then focus much more on churn and retention, and then ultimately also to layer in resurrection as well.

The value of cohorts

Jeff: And the cohort curves have a couple of other features that I love. Usually in marketplace businesses, the best models are built off of the cohort curves.

Sonal: Interesting!

Jeff: Because you have to understand that degradation and where it goes. Using cohorts really give you a sense of their network effects, and network effect is the business gets more valuable to more users that use it; if it gets more valuable, your newer cohorts should behave better than your early cohorts.

Sonal: Why is that?

Jeff: Because the service is more valuable given how they are.

Sonal: Interesting. So that’s kind of a tip–

Jeff: So in OpenTable if there’s ten times more restaurants you’re going to get a whole lot more reservations per diner because you were serving more of their needs. The OpenTable cores would climb up and get more attractive over time versus, you know, we talk about typically they tend to degrade over time. If you’ve reversed the polarity and they’re growing over time it means you’ve made the business more valuable. And then you start projecting forward. Okay [crosstalk]

Sonal: What a better way to know the business is actually more valuable than thinking it’s valuable and believing your own myth.

Jeff: In a network effects businesses we always ask, show us the cohorts. Everyone is [inaudible] on network effect, I’m a network effect But, you know, when you say, “Show me the data, cohort curves, or [crosstalk].” They don’t like it.

Sonal: It’s like show me the money, it’s now show us the cohorts, I get it.

Jeff: They don’t lie.

Andrew: The other really interesting part is segmenting it.

Sonal: I was about to actually ask you what are “the buckets” of cohorts? Are they all demographic data?

Andrew: For a bunch of hyper-local type businesses, the reason why segmenting it based on market geography, why that’s so valuable is because then you can compare markets against each other. You can say, “Well, you know, this market which is like, has much more density in terms of the numbers of scooters behaves like this.” And you can start to draw conclusions, sort of a natural A/B test in order to do that.

And I think the similar kind of analysis you can do for B2B companies is for products that have different sized teams using it. If you have a really large team that they are all using a product, well, are they all using the product more as a result? And let’s compare that to something that maybe only has a couple. … And so this way you can start to kind of disassemble the structure of these networks and do they actually lead to higher engagement.

Jeff: Slack would be a perfect example of that, you know, just if you have five people in the organization using Slack you get one use curve. If you have the organization it’s the operating system for the organization; you have a very different curve.

Sonal: Though it’s not just an accident, you have to sort of architect it, not just expect, like, serendipity to fall into place.

Andrew: So after you get the new users, the way that you have to think about it is around quality, right? You have to make sure that the new users turn into engaged users. One of the things people often talk about is just sort of this idea of like an “a-ha” moment or a magic moment where the user really understands the true value of the product. But often that involves a bunch of setup. So, for example, you know, for all the different social products (whether that’s Twitter or Facebook or Pinterest, etc.), you have to make sure that when you first bring a new user in, they have to follow all the right people. They have to get, you know…

Sonal: It’s like the onboarding experience.

Andrew: …which, by the way, isn’t just signing up but it’s actually doing all the things to get to this a-ha where you’re like, “Oh.”

Sonal: “I get this product.”

Andrew: I get this product. It’s for me, And once you get that, then they’re kind of, you know, then you have the opportunity to keep them in this engaged state over time.

Sonal: Is that really such a thing that there is, like, an a-ha moment? Or is it sort of like a cumulative… a lot of the later users on Facebook came because everyone else was already there. Is this only tied to new users?

Andrew: In the case of Facebook actually, the fact that everyone was already there makes the a-ha moment that much more powerful, right? Because all your friends and family, they’re already there; your feed’s already full of content. And the first time that you see photos that maybe, you know, someone that you went to high school with, right? That is like whoa.

Sonal: That’s actually what happened to me. I was so excited when I saw an old friend, right?

Andrew: Right. Yeah, exactly. And so what that means is, you get the product and then afterwards, when you actually are getting these push notifications or emails that are like, “Hey, it’s someone’s birthday,” or whatever, you’ve internalized what that product is. And, you know, this moment is different for all sorts of different companies.

Jeff: I’ve always heard this referred to as the magic number. When you show up and it’s a blank slate, it’s like, “What is this about?” But they would drive you maniacally to follow people because when you got to their magic number where they had statistically correlated the number of followers with long-term engagement and retention — they would kill you to get you there, doing what felt like unnatural acts of, like, you log on, follow, and you say no, and they say yes — but when they got you there, it kicked in, and the service then quote/unquote worked for you.

A lot of the entrepreneurs I work with are trying to figure out what is my magic moment that then creates the awareness of the value prop. So take the example of Pinterest. Pinterest when it goes into a new market, first of all, they figured out they need a lot of local content to make it compelling to local users. The U.S. corpus of images doesn’t necessarily…is helpful in international markets but isn’t sufficient. And so they needed to supplement…

Sonal: …You’re right. If I’m Indian, I want, like, saris. I don’t only want, like, skirts, which I may not be able to wear in certain regions.

Jeff: Yeah. Exactly. I haven’t worn a sari in North America in a long time. <team laughs> But then once you have the content set, then you have to get compelling information to that individual in front of them, which, you don’t know the individual when they walk in the door, the faster they do that, the more quickly, the better the business performs; engagement goes up; retention goes up; and it works. So different entrepreneurs had to figure out what is that…what experience do they want to deliver where people get it? And then how do you engineer your way into delivering it?

Increasing engagement

Sonal: Okay. So we’ve come up through acquisition and you’ve gotten new users. They get the product. You even hopefully have a way to measure that and see and track it over time. Do you want then go into trying to get different users? Do you take your existing users? One of the things that we covered very early on is that with SaaS, you always wanna try to take existing users and upsell them because it’s way more expensive to acquire a new customer in that context. (I mean, of course, you wanna grow your customers.) How does this play out in this context? What happens next?

Jeff: In a lot of companies, it’s a progression. So almost all the early activity in a company is, “Okay, how do I get the users?” As you get users, you get more and more leverage from efforts at activation and retention and engagement. So, I mean, use Pinterest as an example: again, a very high percentage of women in America have downloaded Pinterest. Then the leverage quickly goes into, “Okay, how do I keep them engaged? Reactivate the ones who disappear?” And their acquisition efforts in the U.S. get de-emphasized and all the leverage is there except as they’re going international, they’re still in that acquisition part of the curve. And so I think the leverage changes over time based on the situation of the company. Facebook hasn’t had any users in the U.S. in forever because they have them all.

Sonal: This kind of goes back to this portfolio approach to thinking about your users.

Andrew: Once you have an active base of users and customers, what starts to get really interesting is to really analyze what are the things that actually set that group up to be successful really long-term sticky users versus what are the behaviors and profiles where users aren’t successful, right? You actually throw your data science team on it to figure out all the different weights for both behavioral as well as the demographic and sort of profile-based stuff on there. And so one of the first things that you figure out is that each one of these products actually has this ladder of engagement where oftentimes new users show up to do something that’s, valuable but potentially infrequent. And you need to actually level them up to something that happens all the time.

For example, when you first install Dropbox, the easiest thing that you can do is you can use it to just sync your home and your work computers, right? And that’s great but really the way to get those users to become really valuable is for them to share folders at work with their colleagues. Because once they have that and people are dragging files in, and they’re really starting to collaborate on things, that’s like the next level of value that you can actually have on a daily basis versus this thing that kind of is in the background that’s just syncing your storage.

Sonal: So what are some of the things that people can then do to move those users up that “ladder of engagement”?

Andrew: Step one is really segmenting your users into this kind of engagement map, oftentimes you’ll see this visualized as a kind of state machine where you have folks that are new, you have folks that are casual, and you want to track how much they’re moving up or down in each one of these steps.

And then once you have that, then the question is, okay, well, great, how do you actually get them to move from one place to the other? First there’s like content and education; they need to know in context that they can actually do something. So for example, if you can get your users to set their home and work for a transportation product then you can maybe figure out, okay, should I prompt them in the morning to try a ride based on what the ETAs are?

Sonal: Like in the app, there would be some kind of notification.

Andrew: Like lifecycle messaging kind of factors in there. The second is of course if your product has some kind of monetary component, then you can use incentives like $10 bucks off your next subscription if you do this behavior that we know for sure gets you to the next step. And then the third thing is really just like refining the product for that particular use case, maybe there are certain kinds of products that are transacted all the time and so you maybe want to waive fees or you give some credits or you do something in order to get people to get addicted to that as a thing.

Jeff: The really interesting thing is the frequency with which something is consumed. I mean, eBay had enormous levels of engagement early on for an ecommerce app in particular. People would spend hours just browsing because early on it was about collectibles and it was about people’s passion. So if someone’s passionate about Depression-era glass, they will spend hours if you give them that depth of content to say, “Oh, my God. I just found the perfect item.”

OpenTable and Airbnb are both typically much more episodic. Most people don’t dine at fine dining restaurants with high frequency; our median user dined twice a year on OpenTable. And so that has completely different marketing implications and user implications. Measurement is probably even more important in the engagement/retention thing because we got our data scientist to understand the different consumer journeys through our product, and then we tried to develop tactics to accelerate the journeys we wanted and limit the journeys we didn’t want. But in order to develop your strategy, you really need to understand how your users are behaving at a really refined level.

DAU/MAU and other metrics

Sonal: So what are some of the engagement metrics?

Andrew: One really important area is frequency, like, just how often are you using the product regardless of the intensity and the length of the sessions and all that other stuff. Literally just frequency of sessions. We might often ask for a daily active user divided by monthly active user ratio, and that gives you a sense for how many days is a user active?

Jeff: DAU to MAU.

Sonal: You recently put a post out on the DAU/MAU metric.

Andrew: Right.

Sonal: And when it works and when it doesn’t. There’s a lot of nuances around when to apply it and when not to.

Andrew: DAU/MAU was very much popularized by the fact that Facebook used it, including in their public financial statements, and it really makes sense for them because they’re an advertising business and it matters a lot that people use them a lot all the time.

Sonal: It’s like counting impressions and being able to sell that to advertisers.

Andrew: Exactly, their products have historically been 60% plus daily actives over monthly actives. And that’s very high. You’re using it more than half the days in a month. On the flip side, what I was talking about in my essay about this is that DAU/MAU can tell you if something’s really high frequency and if it’s working, but a lot of times products are actually lower DAU/MAU for a very good reason because there’s sort of just a natural cadence, you know, to the product. Like, you’re not gonna get somebody who is using a travel product to use it more than a couple times per year. And yet there are many valuable travel companies, obviously.

Sonal: So you’re saying don’t live and die by that alone.

Andrew: Exactly.

Sonal: Because it really depends on product you have, the type of nature of use it has, etc.

Andrew: You just want to make sure that the metric reflects whatever strategy that you’re putting in place. If you think that your product is a daily use product and you’re gonna monetize using a little bit of money that you’re making over a long period of time but your DAU/MAU is low, is like sub 15%, then it’s probably not gonna work.

And then a metric called L28, which is something else that was pioneered certainly early at Facebook: It’s a histogram and what you want to do is —

Sonal: — A histogram is a frequency diagram.

Andrew: Right. A frequency diagram that basically says, okay, show a bar showing how many users have visited once in that month, then twice in the month, and then three times in the month, and then four times in the month. And you kind of build that all the way out to 28 days.

Sonal: Because there’s 28 days in the month on average.

Andrew: And the 28 days is to remove seasonality and then a related one obviously is like L7, right? So just like last seven days. And so what you want to see…

Sonal: So would this be WAUs (“wows”)? Weekly active users? Is that really a thing, by the way? Or am I just making that up?

Andrew: Right. WAUs, DAUs over WAUs.

Jeff: You just coined it.

Sonal: I know. Great. I coined retainment. Why not?

Andrew: Right. And so the idea with L28 or an L7 is the idea that you should actually start to see when you graph this out that there’s a group of people who just use it 28 days out of 28 days, right? And that there’s a big group of people who use it 27 days out of 28 days, and that there’s a big cluster. And so that’s how you know that you actually have a hardcore segment. And that’s really important because in all these products you typically have a core part of the network that’s driving the rest of it, whether that’s power sellers or power buyers or, in a social network the creators vs. the consumers.

Jeff: I actually have heard this referred to as a smile because the one use is always pretty big. A lot of people show up once, “I don’t understand what this is,” and disappear… And then they typically slide down, more people use it…fewer people use it two days than one, three days than two. Done right, it starts to increase at the end. So you basically get a smile. You just go down. And I mean, that’s really powerful. Facebook had a smile. WhatsApp had a smile. Instagram had a smile. If you take a step back, it’s a precondition for investing in a venture business essentially that there’s growth. If it’s end market you want to see growth, but growth by itself is not sufficient. Investors love engagement. So Pinterest, the key driver of Pinterest, it was growing but the users were using it maniacally.

Sonal: Oh, my God. I think I spent an entire Thanksgiving using Pinterest.

Jeff: It was the engagement that blew my mind much more than the growth. OfferUp has engagement that’s similar to social sites like Instagram and Snap. I mean, a ecommerce site, you know, mobile classifieds, people just sit there and troll looking for bargains, looking for interesting things.

Sonal: It’s a little addictive to see what’s nearby that you can buy. Why not? Yeah.

Jeff: So DAU to MAU, smile, all these metrics are so core to us because a big engaged audience is so rare and, as a result, it’s almost always incredibly valuable.

Andrew: And the engagement ends up being very related to acquisition because when you look at all the different acquisition loops — whether it’s paid marketing or a viral loop or whatever — all of those things are actually powered by engagement ultimately. Like, you need people to get excited about a product in order to share content off of that platform to other platforms in order to get a viral loop going. And so one of the things I was gonna also add on DAU/MAU and L28 is that they’re actually really hard to game, right? Which is fascinating.

Sonal: Yeah, why is that?

Jeff: Whereas growth can be very easy to game.

Andrew: Right, exactly.

Sonal: Why is that? What’s the difference?

Andrew: The typical approach is to say, “Well, you know, I’m gonna add in email notifications. I’m gonna do more push notifications. I’m gonna do more of this and that.” And then automatically, you know, these metrics ought to go up, right? The challenging thing is actually usually sending out more notifications will actually cause more of your casual users to show up because your hardcore users were already kind of showing up already. And what that does is that’ll increase your monthly actives number but actually not increase your daily actives as much. So I’ve actually seen cases where sending out more email decreases your DAU/MAU as opposed to increasing it.

Sonal: That’s really interesting. When you think about this portfolio of metrics, it really tells you a story about people are kind of coming but not really staying–

Andrew: If you get an email or a push notification every day, eventually you turn them off, and then you just stop. So then you get counted as a MAU for that period of time and then you lose them as a DAU. Acquisition is super easy to game because you can just spend money.

Jeff: Or you’ve got a distribution hack that works. Early on in the Facebook platform, companies literally got to a million users and it felt like minutes. Just because there were so many people on Facebook and the ones who were early just got exploding user bases. There were a number of concepts whose mean number of visits was one. They never came back. So you get to see these incredibly seductive growth curves but our job is essentially to be cynical and just say, okay, we need to go be it below that because there are a lot of talented growth hackers who can drive growth. I looked at a number of businesses that had tens of millions of users and no one ever came back.

Sonal: This is why engagement is so, so key.

Network effects

So we’ve talked especially about the fact that growth and network effects are not the exact same thing. Because network effects by definition are that a network becomes more valuable the more users that use it. What happens on the engagement side with network effects? What are the things we should be thinking about in that context?

Jeff: Typically network effects, if they’re real, manifest in data. Things like the cohort curves improve over time. Usually there’s a decay. With network effects, there often is a reversal where they’re growing because it’s more valuable. Another smile, essentially. My diligence at OpenTable was I looked at San Francisco, which was their first market, and sales rep productivity grew over time, restaurant churn decreased over time, the number of diners per restaurant increased over time, the percentage that went that booked through OpenTable versus the restaurant’s own website moved towards OpenTable dramatically. Every metric improved. And so, you know, that’s where it both drives good engagement, but also it just improves the investment thesis.

Sonal: The value overall, right?

Andrew: One of the interesting points about network effects is that we often talk about it as if it’s a binary thing.

Sonal: Right. Or homogenous, like all network effects are equal when they’re not.

Andrew: Exactly right. When you look at the data, what you really figure out is that network effect is actually like a curve, and it’s not like a binary yes/no kind of thing. So for example, [turns to Jeff] I would guess that if you plotted the number, if you took a bunch of cities, every city was a data point, and you graphed on one side the number of restaurants in the city versus the conversion rate for that city, you would quickly find that when cities have more restaurants, the conversion rate is higher, right? I’m just guessing.

Jeff: It’s actually almost perfect but with one refinement. The number of restaurants you have as a percent of that market’s restaurant universe; because having 100 restaurants in Des Moines is different than having 100 restaurants in Manhattan.

Andrew: Makes total sense. So not only that, what you then quickly figure out is that there’s some kind of a diminishing effect to these things often in many cases. So for example, in rideshare, if you are gonna get a car called 15 minutes versus 10 minutes, that’s very meaningful. But if it’s five minutes versus two minutes, your conversion rate doesn’t actually go up.

If you can express your network effect in this kind of a manner, then what you can start to show is, okay, yeah, we have a couple new investment markets that maybe don’t have as many restaurants or don’t have as many cars but if we put money into them and invest in them and build the right products, etc. then you can grow.

You can do this kind of same analysis whether you’re talking about YouTube channels and the number of subscribers you might have, having more videos is better; I’m sure you can show that. If you go into the workplace, and you start thinking about collaboration tools, then what you ought to see is that as more people use your chat platform or your collaborative document editing platform, then the engagement on that is gonna be higher. You should be able to show that in the data by comparing a whole bunch of different teams.

Engagement vs. retention

Sonal: Okay… So we’ve talked about engagement and also how it applies to network effects. Are engagement and retention the same thing? I mean, in all honesty, they sound like they would be the same thing.

Jeff: There’s overlap, but they’re different.

Andrew: Yeah, there’s overlap, right. Just to give a couple examples: So weather is low frequency but high retention because you’re actually gonna need to know what the weather is… <Oh right!>

Sonal: Only once a day, unless you live in San Francisco and you gotta check it, like, 20 times a day with all the microclimates.

Andrew: Right, exactly.

Jeff: Or if you live down here, you have to check it twice a year.

Sonal: That’s true, it’s actually the same year-round.

Andrew: That’s actually what it showed, was actually more that generally people didn’t really check it that often. However, you are highly likely to have it installed and running after 90 days because it’s a reference thing. You might need it.

Sonal: It’s so important, yeah.

Andrew: Like a calculator. Versus if you look at something like games or ebooks or those kinds of products, like, really high engagement because you’re like, “All right. I’m gonna get to…I’m gonna finish this like trashy science-fiction novel that I’ve been reading. I’m just gonna get to it.” But then as soon as you’re done, you’re like, “Okay, there’s no reason why I would go back and read it again.”

Sonal: So the real difference is that engagement obviously varies depending on the product, the type of thing it is, whether it’s weather or ebook, and retention is are you still using it after X amount of time.

Jeff: And different companies have different cadences. If the average user is twice a year, retention is did they book annually. Other businesses are, did they come daily? The model behind retention is completely different and the model behind engagement is completely different.

Andrew: The chart that I’d love to really see is one that was like a bunch of different categories that’s, you know, retention versus frequency versus monetization. I think you got to be, like, really good at least on one of those axes.

Sonal: So we’ve done sort of this taxonomy of metrics. We’ve talked about the acquisition metrics. We’ve talked about some engagement metrics, primarily frequency.

Jeff: On engagement, it’s also time. Not just how frequent someone is, but just how much time did they spend.

Sonal: Right. Time spent on site, on the… piece, writing comments, not just pageviews.

Jeff: Because, I mean, the number of businesses that have great engagement is not high. Because there are only so many minutes in the day. And so, you’re just looking for where, okay, they’re just passing time and enjoying, and they both have obvious monetization associated with that behavior.

Sonal: This is why Netflix is so freaking genius because when they literally invented the format of binge-watching, which you could not do — I love it because it’s a very internet native concept — I mean, they’ve literally fucked up everyone else’s engagement numbers.

Andrew: I think that’s one of the narratives on why building consumer products is much harder these days. Because–

Sonal: –And, do you think it’s true or not?

Andrew: Well, because it used to be. It used to be that you were…what kind of time were you competing for in the first couple years of the smartphone? You were competing against literally I’m gonna stare at the back of this person’s head, or I can like use some cool app that I downloaded, right? Versus these days you actually have to take minutes away from other products.

Sonal: Yes.

Jeff: And it’s typically other [inaudible] because the top apps are almost all done by Facebook, Amazon, Google. And you know, breaking through just — Marc calls it the first page, the people who are on the first screen — are just such the incumbents. And sure, most people have Facebook on the screen and YouTube on the screen and Amazon on the screen.

Sonal: It’s hard to take that down, right?

Jeff: You have that competition. It is a big overhang right now in consumer investing because you have to displace someone’s minutes.

Sonal: Yeah. I would add one more layer to that, at least on the content side, which is I think a lot of people make a lot of category errors because they think they’re competing with like-minded players and, in fact, when it comes to things like content and attention, you’re competing with just about anything that grabs your attention. It’s not just other media outlets. It’s…

Andrew: …Tinder.

Sonal: It’s a dating app. It’s something else.

Jeff: I’m riding in the train for an hour, I could, you know, see what my friends are doing on Facebook, watch videos on YouTube.

Sonal: It actually changes with time blocks. Xerox PARC did a really interesting study on “micro-waiting moments” and they’re literally the little snatches of time, like two seconds here and there, that you might be waiting in line or doing something, so you can do a lot of snack-sized things in that period, which is also another interesting thing to think about for how people engage with various things.

Jeff: So it’s actually funny because there’s some businesses that have good engagement where it’s one session that goes on for a while, YouTube or Netflix or something like that. There are others that are multiple small sections that in aggregate…

Sonal: …Like a podcast which might not finish in one sitting.

Jeff: …Because it’s the micro-opportunities…

Andrew: …And Google is the best example of this, right? In fact if you spend a lot of time on Google.com, you know, refining your searches and clicking around, that means actually the service is doing poorly.

Jeff: They’ve failed. Their goal is to get you to their advertisers as fast as they can.

Andrew: That’s a frequency play and a monetization play ultimately as opposed to an engagement one.

Sonal: Yes, that’s fascinating.

Andrew: And some products are more on the engagement side.

Measuring retention

Sonal: So sometimes you have to optimize it based on how you’re monetizing. What are some of the metrics for retention? I mean, is it just should-I-stay-or-should-I-go? Is that the retention metric?

Andrew: I think the big thing is the concept of churn. Is a tricky one in some cases like subscription Hulu, Netflix, and then also in the SaaS world. Whether or not you’re still continuing to pay or not. And that’s really obvious.

The thing that’s tricky for a lot of these consumer products especially episodic ones — and, it’s actually less whether they’ve quote-unquote churned or not — it’s actually just whether or not they’re active or inactive, and whether or not that’s happening at a rate that you in your business strategy have decided is acceptable or not. If every Halloween, you know how there’s those costume stores that open all over the place. If every Halloween, you go back and you buy a costume, but you’re inactive the rest of the time, have you churned or not? It’s not clear and I would argue you’ve not churned because you’re doing exactly what they want, which is to buy a costume every Halloween.

Sonal: It seems like it smakes assessing the retention of a consumer business very difficult.

Jeff: You adjust the time period that you’re relevant on. If the average diner dines twice a year…

Sonal: …Then that’s your time frame.

Jeff: You can apply that metric. Travel’s a similar thing. Airbnb is for the average user relatively infrequent. You have to tailor your look to what are they trying to do, so if you’re trying to stake up with your friends and you’re doing it twice a year, yeah, that doesn’t work. So Facebook has got a whole different setup.

Andrew: One of the things that companies can often do is to measure upstream signal. So for example, Zillow, you’re probably not gonna buy a house very often. Maybe a couple times in your life. However, what’s really interesting is they can say, “Well, you know, maybe folks aren’t buying houses but at least are we top of mind? Are they checking the houses that are going on sale in their neighborhood? Are they opening up the emails? Are they doing searches?” Right?

Sonal: Interesting. Why do you call that “upstream”?

Andrew: In the funnel. You’re kind of going up in the funnel and you’re tracking those metrics.

Sonal: I get it now!

Andrew: As opposed to, you know, purchases. So even for OpenTable, it’s like, okay, great. Well, maybe if you’re not actually completing the reservations, are you at least checking the app for availability?

Jeff: Or what’s new restaurants where I want to dine? There’s some level of content consumption.

Sonal: So throughout this entire episode, there seems to be this interesting “dance” between architecting and discovering. Like, you might know some things upfront because you’re trying to be intentional and build these things, and then there are things that you discover along the way as your product and your views and your data evolves. How do you advise people to sort of navigate that dance?

Jeff: You iterate. You develop hypotheses. You put it out there and you test the hypothesis. I think my product’s gonna behave this way. And then, did it?

Probably the most important thing is for me, marketing can be art, marketing could be science; in the consumer internet, it’s more science. Some companies can effectively do TV campaigns, large media budgets, things like that. For me, the better companies typically just rip apart their metrics, understand the dynamics of their business, and then figure out ways to improve the business through that knowledge. And that knowledge can feed back into new product executions or new marketing strategies or new something. It’s constant iteration but it’s informed by the data at a level that on the best companies is really, really deep.

Andrew: Ultimately, you have a set of strategies that you’re trying to pursue and you pick the metrics to validate that you’re on the right track, right? And a lot of what we’ve talked about today has really been the idea that actually there’s a lot of “nature versus nurture” kind of parts to this. Your product could just be low cadence but high monetization, and so you shouldn’t look at, you know, DAU/MAU. And so you have to find really the right set of metrics that show that you’re providing value to your customers first and foremost and then really build your team and your product roadmap and everything in order to reinforce that.

Find the loops and the networks that exist within your product because those are the things that are gonna keeps your engagement improving over time even in the face of competition.

Jeff: Growth is good. Growth and engagement is really really, really good.

Sonal: That’s fabulous. Well, thank you, guys, for joining the a16z Podcast.

• 

  Andrew Chen is a GP at a16z in consumer technology. Prior to that, he led Rider Growth at Uber. He has written for a decade on metrics & growth on his blog/newsletter, and is the author of The Cold Start Problem.

  Follow Twitter
• 

  Jeff Jordan is a managing partner at Andreessen Horowitz. He was previously CEO and then executive chairman of OpenTable.

  Follow Twitter
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

Growth is one of the most top of mind questions for entrepreneurs building startups of all kinds (and especially consumer ones) — but how does one go beyond a mindset of “growth hacking” to thinking about growth more systemically and holistically? What are the key metrics to know; why; and how?

This episode of the a16z Podcast — one of two in a series — focuses on the user acquisition aspect of growth, followed by engagement and retention in the next episode. Featuring a16z general partners Andrew Chen and Jeff Jordan, in conversation with Sonal Chokshi, the discussion also covers the nuances of paid vs. organic marketing (and the perils of blended CAC); the role of network effects; where does customer lifetime value (LTV) come in; and much more. Because at the end of the day, businesses don’t grow themselves…

Transcript:

Hi everyone welcome to the a16z Podcast, I’m Sonal. Today’s episode is all about growth, one of the most top of mind questions for entrepreneurs — of all kinds of startups, and especially for consumer ones. So joining to have this conversation, we have a16z general partners Andrew Chen and Jeff Jordan. And we cover everything from the basics of growth and defining key metrics to know, to the nuances of paid vs organic marketing and the role of network effects and more. 

Part one of this conversation focuses specifically on the aspect of user acquisition for growth, and then we cut off and go into the aspects of growth for user engagement and retention, in the next episode. But first, we begin by going beyond the concept of growth hacks — and beginning with the fundamental premise that businesses do not grow themselves…

Sonal: So the topic we wanted to talk about today is growth, which is a big topic. What would you say are the biggest myths and misconceptions that entrepreneurs have about growth?

Andrew: You know, not only is there the misconception that it happens magically, then the next layer I think is that it’s really just like, oh, a series of, you know, tips and tricks and growth hacks that kind of keep things going as opposed to like a really rigorous understanding of, you know, how to think about growth not just, as kind of the top line thing but actually that there’s acquisition, that there’s engagement, that there’s retention, and each one of those pieces is very different than the other and you have to like tackle them systematically.

Jeff: It is a scientific discipline, done right, because it requires you to understand your business and business dynamics at this incredibly micro level.

Sonal: I love that you said that because one of the complaints I’ve heard about “growth hacking” is that it’s just marketing by a different name, and what I’m really hearing you guys say is that there’s a systemic point of view, there’s rigor to it, there’s stages, there’s a program you build out.

Jeff: If you’re fortunate enough to achieve product-market fit and your business starts to take off, typically, you know, when in the wonderful situation do you get this hyper growth where you’ll grow year over year, you know, it’s triple digits. It’s just exploding. And then gradually the law of the large numbers starts to kick in and maybe the 100% growth becomes 50% growth the next year, and then the law of large numbers continue to kick in and there’s 25% and then it’s 12.5% and so growth tends to decay over time even in the best businesses. And so the–

Sonal: — Didn’t you use to call it like “gravity”?

Jeff: I called it gravity, you just would…it comes down to earth. And then the job of the entrepreneur is to be looking years down the road and say, “Okay, at some point growth in business A is going to stop and so I want to keep it going as long as I can and there’s a whole bunch of tactics to do that,” but then the other tactic, the other strategies, okay, I need new layers on the cake of growth.

At eBay the original business was an auction business in the U.S. and so, you know, some of the things we layered on early days we layered on fixed price in the U.S. — it’s not revolutionary but it really did grow then we went international. And then we layered in payment integration and each time we did that the total growth of the company would actually accelerate which is very hard to do at scale.

Sonal: That’s the whole point… like there’s intentionality to it. It’s not an accident that you guys introduce new businesses, new layers on the cake.

Jeff: Businesses don’t grow themselves, the entrepreneur has to grow them. And, you know, occasionally, you stumble into a business that seems to almost grow itself but they’re just aren’t many of those in the world and that growth almost never persists for long periods of time unless the entrepreneur can figure out how to continue its growth.

Sonal: Right. I remember a post you wrote actually a few years ago on “The ‘Oh, Shit Moment!’ When Growth Stops” because people are a little blindsided by it.

Jeff: And that’s the flip side of it. You know, early on you get this great growth, you had to keep it going. When it stops your strategic options had been constrained dramatically.

Andrew: A lot of times when you’re looking at what seemingly is an exponential growth curve. In fact, it’s really something like, oh, you’re opening in a bunch of new markets, right, so there’s sort of a linear line there, but then you’re also introducing products at the same time and you’re also reducing friction and, you know, sign-ups or retention or whatever, and so, the whole combination of those things is really kind of like a whole series of accelerating pieces that looks like it’s, you know, this amazing viral growth curve. But it’s actually like so much work underneath.  <Sonal: Right.> You know, that makes that happen.

Sonal: I’ve also heard you [Andrew] talk about, being able to distinguish what is specifically driving that growth, so you don’t have this like sort of exponential-looking curve without knowing what that lever that you’re pulling to make that happen or knowing what’s happening even if it’s kind of happening naturally or organically. Can we break down some of the key metrics that are often used in these discussions including just what the definitions are and maybe just talk through how to think about them?

Andrew: Right. Yeah, so when you look at a large aggregate number like, you know, total monthly active users, right, or you’re looking at like —

Sonal: — “MAUs”

Andrew: –Yeah, MAUs, right. Or you’re looking at, you know, the GMV like all the…adding up all the transactions in your marketplace–

Sonal: — So, “gross merchandise value”.

Andrew: Yup. And so, you know, when you look at something like that and if it’s going up or down, you don’t have the levers at that level to really understand like what’s really going on. You want to go a couple levels even deeper: How many new customers are you adding? As you’re growing more and more new customers, a bunch of things happen. If you’re using paid advertisement channels, things tend to get more expensive over time because — you know, your initially super, super excited core demographic of customers — like they’re gonna convert the best and as you start reaching into different geographies, different kinds of demos, all of a sudden they’re not gonna convert as well, right?

Sonal: Just to pause in that for a quick moment, you’re basically arguing that growth itself halts growth in that context.

Andrew: Right. Yeah. So the law of large numbers means that you know there’s only a fixed number of humans on the planet, there’s only a fixed number of people that are in your core demographic, right? Once you surpass a certain point, it’s not like it’s it falls off a cliff, it’s just more gradual that you know that the customer behavior really changes.

Sonal: How do you determine what’s what when you don’t have product-market fit? Sometimes aren’t these metrics ways to figure that out or is this all when you have product-market fit… like is there a pre- and a post- difference between these?

Andrew: Very concretely, you want to understand how much of the acquisition is coming from purely organic (people discovering it, people talking to each other), as opposed to, oftentimes you’ll run into the companies that have over 50% of their acquisition coming from paid marketing and that tells you something that you’re, you know, needing to spend that much money to get people in the door.

Sonal: Yeah. So CAC, “customer acquisition cost”, that’s what you’re talking about when you talk about acquisition.

Jeff: CAC is what it cost to acquire a user, “blended CAC” is what it costs to acquire a user on a paid basis plus then also what free users you acquire. So if you’re acquiring half your users through paid marketing you’re paying a $100 to acquire a user but half of your users are coming at zero, paid CAC is 100, blended CAC is 50.

I think blended time is a really dangerous number. Most of the best businesses in the internet age of technology haven’t spent a ton on paid acquisition. And so the truly magical businesses, you know, a lot of them aren’t buying tons of users… Amazon’s key marketing right now is free shipping. And then, yeah, the economics of paid acquisition tend to degrade overtime.

Sonal: As it grows.

Jeff: As it grows and you just try to scale it and, you know, largely you’re cherrypicking the best users and then you’re trying to also scale the number you get to grow. I need twice as many new users this year as last year and you typically pay more so that magical LTV to CAC ratio which early on says, “Oh, we are three to one, you know, in two years it’ll probably be one and a half to one if you’re lucky,” or something like that. So we typically do try to look for these other sources of acquisition be it viral, be it, you know, some other form of non paid <crosstalk>

Sonal: I want to quickly define LTV — it’s “lifetime value” of the customer, but what does that mean?

Jeff: When you’re showing an LTV to CAC ratio you have no idea of what you’re seeing essentially given all the potential variations of the numbers. So we will almost always go for clarity. LTV, lifetime value, should be the profits, the contribution from that user after all direct costs.

Sonal: How do we define the LTV to CAC ratio? What do the two of them in conjunction mean?

Jeff: Well, let’s break them down. LTV is lifetime value. What you’re describing there is the incremental profit contribution for a user over the projected life of that user. So not revenue per CAC is that you know typically there’s cost associated to user. What’s the incremental contribution that the user brought from that <crosstalk> <Sonal: And that you mean the user brought to your company’s value.> To the company, yeah.

Sonal: So it’s a value of your customer to the bottom line?

Jeff: It’s the value of each customer to the bottom line, and then you compare that to the CAC or “cost of acquired customer” to understand the leverage you have between what I need to spend to acquire a customer and how much they’re worth. If your CAC is higher than your LTV you’re sunk. Because it’s costing you more to acquire a user…

Sonal: Than the value you get out of it. Now I get it.

Jeff: …then you’re going to get out of that user.

Sonal: Yeah.

Jeff: If it’s the opposite, at least you’re in the game. You know, I get more profit out of the user than I get the cost to acquire that user. And then there’s this dynamics on how does it scale over time, CAC tends to go up, LTV tends to go down. Because you’re, on the CAC side, you’re acquiring the less interested users over time. So they cost more to acquire and they’re worth less, and so that the LTV to CAC ratio, in our experience, almost always degrades as over time with scale.

And so, you know, when you’re in that conversation, you’re in a very specific conversation of, “Okay, how much room do you have?” “How is it gonna scale?” “You know, what’s gonna impact your CAC like a competitive thing?” So there has to be a lot, it had to be like 10 to kind of get you over that concern that oh, my goodness, those two were so close, that you have no margin for error.

Sonal: Right. This also goes back to the big picture, the layers on the cake, because if you have other layers you don’t have to only worry about one layer CAC to LTV ratio.

Jeff: It really does affect the calculation. If it’s, I’m in a new business, and I have a whole different CAC versus, you know, LTV ratio then that’s a different conversation as well.

Sonal: And the big picture there, is that if you don’t know the difference of what’s doing what when you may get very mistaken signals, mixed signals about your business, and so you guys don’t want blended CAC because you want to know what’s driving the growth.

Andrew: I think what blended CAC gives you is it gives you a sense for at this particular moment in time, you know, what’s happening. The challenge is that when it comes to paid marketing, in particular, it’s easy to just add way more budget and a scale that than it is to scale organic or to scale SEO. So your CAC is giving you a snapshot, but then as you’re trying to scale the business you’re trying to increase everything by 100% over the next, you’re trying to double everything then all of a sudden, you know, your blended CAC starts to approach whatever your dominant channel actually looks like.

And so if you’re spending a bunch of money then it’ll just approach whatever is your paid marketing, you know, CAC. What entrepreneurs should think about is what is the unique organic new thing that’s gonna get it in front of people, without spending a bunch of money, right?

Jeff: A lot of the best businesses have this very interesting, I’ll call it a growth hack. I mean OpenTable, when I was managing it, did not pay any money at all to acquire consumers. Like how can you do that? You know, it had millions of consumers. The restaurants would mark it OpenTable on our behalf.

Sonal: Right.

Jeff: They go to The Slanted Door website like when they were an OpenTable customer and you’d see, you’re looking…you go there to try to get the phone number to make a reservation and they’d say, “Oh, make an online reservation.” And we then got paid to acquire that user in its core form. But that hack was a wonderful thing. It scaled with the business and got us tons of free users.

Sonal: To be fair, and this is another definition we should tease apart really quickly before we move on to more metrics, that also had a quality of network effects which we’ve talked a lot about in terms of these things growing more valuable to more people that use it… is that growth? What’s the difference there?

Jeff: Well, the business grew into the network effect. The key tactic to build the network effect was that free acquisition of consumers that the more restaurants we had, the more attractive it was to consumers the more consumers who came, the more attractive it was to restaurants. So there is a wicked network effect.

Sonal: Like a flywheel effect, right.

Jeff: If you’re not spending anything on paid acquisition of consumers, how do you start it? And the placements that OpenTable got in the restaurant book both physically in the restaurant but particularly in the restaurant’s website was the key engine that got the network effect started. You had to manually sell some restaurants come for the tools, stay for the network, but then once the consumers got enough of a selection and started to use it, it was game over.

Sonal: Right, that was one way of going around the bootstrapping or the chicken-egg problem and seeding a network.

Andrew: Network effects have…there’s a lot of really positive things about them and one of the big pieces is that virality is a form of like something that you get with the network. You know, the larger your network is, the more surface area, the more opportunities you have in order to encounter it, right. And so, you know, in the case of Uber (where I was recently), by seeing all the cars with the Uber logo like those are all opportunities to be like, “Oh, what is this app? I should try it out.” And so it’s mutually reinforcing: then you get more riders and then you get more drivers that are into it and so, I think all of that kind of plays together.

Jeff: I’ll bring two examples up, the pink Lyft mustache when I first got to San Francisco.

Sonal: I remember that.

Jeff: You can see it once in the car and you’d go, “Oh, that’s pretty weird.” You see it twice in the car and you say, “Something is going on here that I don’t know about, and I have to understand what it is.” Lime is the same kind of thing.

Sonal: Right.

Jeff: They’re bright green and they glow essentially. So when someone sees one in the wild, someone bolts by them in a glowing green electric scooter and you’re just like, “Okay…what is that?” And Lime hasn’t spent a penny on consumer acquisition. <Sonal: Right.> Because their model is such that physical cue in the real world leads to it.

Andrew: The other one I’ll throw in as well is within workplace enterprise products there’s a lot of kind of bottoms-up virality that comes out of people, you know, kind of sharing and collaborating.

Sonal: Like with Slack.

Andrew: Yeah, like for example Slack is a great, it’s an example of this. And so, these are all kind of really unique ways that you can, you know, get acquisition for free. And so then your CAC is, you know, “zero” as a result.

Sonal: Yeah. You guys have talked a lot, about organic. It makes it sound to me as a layperson that you don’t want paid marketing! Like what’s your views on this — is it a bad thing, is it a good thing; I don’t mean to moralize it but — help me unpack more where it’s helpful and where it’s not. Are they any rules-of-thumb to use there?

Jeff: I mean a lot of great businesses that have leveraged paid marketing. The OTA sites (online travel agencies – Priceline and Expedia) just spends, you know, they spend a GDP of many large countries in their acquisition; and then it’s often a tactic in some good business. But if it’s your primary engine, a couple of things happen: One is it tends…the acquisition economics tend to degrade over time for the reason we’re saying…  <Sonal: Right this is…> And it leaves you wide open to competition.

Sonal: It gets commoditized basically.

Jeff: If you need to buy users, I mean if you’re selling, you know, the new breed of mattress and you need to buy users and early on, you’re the only person competing for that word, flas-hforward a year or two, they’re like six new age mattress manufacturers with virtually identical products competing for the same consumer. The economics are not going to persist over time. And so, you know, one of the key questions in businesses driven by heavy user acquisition is how does the play end? You know, it usually looks pretty good at the beginning of the play but in the middle it’s starts getting a little complex and there’s tragedies at the ends.

Sonal: There’s literally an arc.

Andrew: And I think, you know, if it is something that you’re using in conjunction with a bunch of other channels and you’re kind of accelerating things, that can be great. For example when Facebook in the past broke into new markets they started with paid marketing to get it going. And so in a case like that really paid marketing is a tactic to kind of get a network affect jumpstarted right? <Sonal: Gotcha.> And then you can kind of like pull off from that if you’d like. <Sonal: Right.>

Andrew: But if you’re super, super dependent on it and you don’t have a plan for a world that you know all the channels atregonna degrade [in] then you’re gonna be in a tough spot in a couple of years.

Sonal: Totally. Do you have sort of a heuristic for when to stop the paid? Is there like a tipping point, you know, THIS is when you move?

Andrew: I think in terms of how much paid should you do as part of your portfolio, I think that’s the right way to think of it is it’s one out of a bunch of different channels, right? And so I would argue the following: First is you really have to measure the CAC and the LTV and be super disciplined about not spending ahead of where you want it to be and not to do it on some, you know, blended number that doesn’t make any sense. <Sonal: Right.> And then I think the other part is you really want it to be kind of a small enough minority of your channels. Such that if you were to get to a point where it turns out to be capped that you’re okay, that you can live with that.

Sonal: Your business will survive and you continue to grow and be healthy.

Andrew: Right, exactly, and you can still get the growth rates you want and you can still, you have such strong product-market fit that you’re able to maintain that.

Jeff: Take a couple of sector examples. You know, ecommerce, a lot of companies struggle with, “Okay, how do I get organic ecommerce traffic?” So most ecommerce companies rely heavily on paid user acquisition, you know, typically one of the interesting things is they degrade over time and they’re all competing for the same user. It’s hard for ecommerce companies in most segments to be profitable and you’d look at the same kinds of dynamic and restaurants delivery. You know, if you can’t differentiate yourself and you’re highly reliant on paid marketing, the movie typically doesn’t end really great, and so, we look for segments where there’s a balance or they come up with that really unique growth hack and they’re not then reliant on paid channels.

And then by the way, paid channels can degrade too. I mean, I’d made a couple of investment mistakes where the paid acquisitions looked really good and then actually what they were doing is they’re arbitraging something like Facebook’s early mobile attempts where the people who participated with Facebook mobile ads early got real deals. They were nowhere near kind of the price they should have been trending at, so you’re like, “Look at these user analytics. They’re awesome!” And then Facebook, you know, kind of got the equilibrium when supply and demand met and the cost went up multiples, and those businesses that looked so good early just got incredibly stressed because they had no alternative to that inflation.

Sonal: That’s the case of platform risk where you’re dependent on the channel of on Facebook mobile or whatever the specific channel was there. But Andrew, you were also earlier talking about just a cap on how much is possible, and you both referenced the fact that things can become very competitive, that your competitors can also buy the same channels and then it gets very crowded or very expensive. So there’s multiple layers of the risk of the paid is what I’m hearing, but you have to be aware of that.

Andrew: Yeah. So I think on the acquisition side today, there’s a couple of really interesting opportunities that might be, you know, temporal, right, and like it may go away, right? <Sonal: Like, anything that crosstalk> For example, I think that if you have a product that is very highly visual, and I think this is, you know, one of the reasons why eSports has gotten so huge is because you have a product that naturally generates a ton of video in an age which all the platforms are trying to rush to video.

Sonal: That’s fascinating.

Andrew: Right? And so, you know, maybe this will be less of an opportunity coming up but like, you know, that’s a thing.

Sonal: Why would you say that’s temporal because it seems like…

Andrew: Because the competition will…

Sonal: …Do the same thing?

Andrew: …Yeah, will do the same thing, right. I think we’re now gonna move to a thing where all of these different kind of software experiences all are incredibly sharable. Like there’s no point these days in building a new game that doesn’t have like built-in recording and publishing the Twitch stream. And built-in tournament systems and all the community features and all that stuff that you need and, you know, I think it used to be that you would think of a game is just the actual IP but in fact, it’s sort of these layers and layers and like social interaction and content around it. And I think that’s about true as well as, all of these different brick-and-mortar experiences that are making themselves highly Instagrammable, they are adding the areas where you actually stand there and pose…

Sonal: Oh, my god, my favorite story about this is the restaurant trend of making square plates and layouts so it really fit beautifully with Instagram. That’s like one of my favorite cases; that’s one of my favorite things in the world is when the physical world adapts to the digital!

Andrew: And then you can go the other way too which is, physical products like scooters that remind you to engage digitally. The other, fun example I always like is everyone’s had the experience now where they’re just like in the room talking and then their Amazon Echo just turns on and it’s trying to go and I’m like, you know, they have no incentive to fix that. <Sonal: Yeah.> Because it reminds you that it’s there and reminds you to talk to it.

I think the big takeaway here is that you have to really be creative and really be on the edge of what everyone’s doing, right? And so if it turns out that everyone’s really into video and they’re really into Instagram right now, you have to think about like how does my product actually fit into that trend? <Sonal: Yeah.> And if you can find it, then you can get an amazing killer way to get jumpstarted and if the trend lasts then great, accelerate it with paid marketing, accelerate it with PR, do all that stuff to kind of keep it going.

I also want to make the distinction that we’re mostly talking about growth and acquisition.

Sonal: Yesss!

Andrew: And that is what startups mostly care about in the early days, because you don’t really have any active users, right? But the other part of this is that you see all the users would show up and how active they are starts to change over time… <overlap/crosstalk>

Sonal: <overlap/crosstalk>…The engagement. Well, thank you guys for joining the a16z Podcast.

[see part two on growth for user engagement and retention]

• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter
• 

  Andrew Chen is a GP at a16z in consumer technology. Prior to that, he led Rider Growth at Uber. He has written for a decade on metrics & growth on his blog/newsletter, and is the author of The Cold Start Problem.

  Follow Twitter
• 

  Jeff Jordan is a managing partner at Andreessen Horowitz. He was previously CEO and then executive chairman of OpenTable.

  Follow Twitter

“The rules of the game are different in tech,” argues — and has long argued, despite his views not being accepted at first — W. Brian Arthur, technologist-turned-economist who first truly described the phenomenon of “positive feedbacks” in the economy or “increasing returns” (vs. diminishing returns) in the new world of business… a.k.a. network effects. A longtime observer of Silicon Valley and the tech industry, he’s seen how a few early entrepreneurs first got it, fewer investors embrace it, entire companies be built around it, and still yet others miss it… even today.

If an inferior product/technology/way of doing things can sometimes “lock in” the market, does that make network effects more about luck, or strategy? It’s not really locked in though, since over and over again the next big thing comes along. So what does that mean for companies and industries that want to make the new technology shift? And where does competitive advantage even come from when everyone has access to the same building blocks (open source, APIs, etc.) of innovation? Because Arthur — former Stanford professor, visiting researcher at PARC, and external professor at Santa Fe Institute who is also known as one of the fathers of complexity theory in economics — has written about the nature of technology and how it evolves, observing that new technology doesn’t come out of nowhere, but instead, is the result of “combinatorial” innovation. Does this then mean there’s no such thing as a dramatic breakthrough?!

In this hour-long episode of the a16z Podcast, we (Sonal Chokshi with Marc Andreessen) explore many of these questions with Arthur. His answers take us from “the halls of production” to the “casino of technology”; from the “prehistory” to the history of tech; from the invisible underground autonomy economy to the “internet of conversations”; from externally available information to externalized intelligence; and finally, from Silicon Valley to Singapore to China to India and back to Silicon Valley again. Who’s going to win; what are the chances of winning? We don’t know, because it’s a very different game… Do you still want to play?

Show Notes

• How the concept of network effects was born [1:05], its initial reception, and eventual adoption [9:53]
• Discussion of diminishing returns [18:09] and the role of timing or luck [22:16]
• How new technologies are created from existing elements [25:12]
• The impact of computer networks [38:44], and the societal impact of machine intelligence [46:42]
• Globalization and economies around the world [53:52]

Transcript

Sonal: Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal. So, today we have a special episode. We talk a lot about network effects as one of the most important dynamics, especially in software-based businesses. You can see much of ours and others’ thinking on the topic at a16z.com/networkeffects. But today, our special guest is W. Brian Arthur. He’s widely credited for first describing network effects, and beyond that, has had a long and very influential career in economics, especially as applied to the tech industry. So I asked Marc Andreessen to co-host and add a little color commentary. But first, more about Brian.

Brian was formerly a professor of economics at Stanford, is a visiting researcher at PARC, formerly Xerox PARC, and is also an external professor at the Santa Fe Institute — because besides his foundational work in network effects, he’s also considered one of the fathers of complexity theory — has written books on the nature of technology and how it evolves, and has also written a number of pieces on AI and the autonomy economy, all of which we’ll touch upon in today’s episode. We also cover a lot of neat history in between, and we end on the topic of innovation clusters around the world, including Silicon Valley. But first, we begin briefly with where Brian’s ideas came from.

The concept of network effects

You’re a really influential economist who’s — and I sometimes make fun of economics.

Prof. Arthur: Feel free.

Sonal: I know. But, you know, your work has really actually driven so much, or described so much, of what actually happens in technology, and there seems to be a gap often between the worlds of economics and technology, and you’re really at the heart of that. So why don’t we start with some of your most seminal work, starting with your famous classic paper around increasing returns and positive feedbacks.

Prof. Arthur: Sure.

Sonal: Like, if you were to just sort of distill and summarize some of the key concepts and how it contributed to the tech industry.

Prof. Arthur: Sure. To go back a little bit, I’ve been interested in technology for a long time. I was trained as an engineer, and then mathematician and operations researcher — basically algorithmic theory. So, my basis is actually technology, and then I added as a layer on top of that — I fell into the wrong company and became an economist. And I arrived in Stanford in 1982. At that time, Silicon Valley was blossoming. We said in ’82, it was all about electronics, then it was about computation, then the web, then the cloud. Now it’s about AI. So, Silicon Valley keeps morphing and changing. I was enormously taken, just by the sheer energy of the place in the early ’80s, and on through 26 years or so since. It keeps recreating itself. It’s like looking into, I don’t know, it’s like looking into some cauldron of everything bubbling and changing all the time. And it became very clear to me that there was a phenomenon going on in technology that you didn’t see so much in the rest of the economy.

Sonal: Right. The phenomenon of network effects, which I should clarify in your papers is also named positive feedbacks, and increasing returns.

Prof. Arthur: Yeah. In standard economics, if you get very large in the market, everybody runs into some sort of diminishing returns, and markets tend to balance. The market’s fairly well-shared. That was the theory when I came along, but it didn’t seem to me that tech worked that way. Go back to about 1982, ’84. At the time, we had VHS and we had Beta. Those were the basic operating systems for video recorders, and one of them happened to be better — Betamax was better — and I started to wonder why VHS dominated the market.

Sonal: I’ve always wondered this, actually.

Prof. Arthur: And then I realized that a host of small events early on had pushed VHS into a slight lead, and if you were going down to your local movie rental store — again, this is…

Sonal: Back when Blockbuster existed.

Prof. Arthur: …Blockbuster, you would tend to see more VHS movies. That meant you’d get a VHS recorder, and that meant that they would stock more VHS.

Sonal: Aren’t those complements, in economic terms?

Prof. Arthur: Oh, yeah. The two were kind of interacting. The more VHS is out there, the more I buy VHS. So I began to realize that I was seeing this in market after market. There weren’t diminishing returns. If VHS got ahead, it would get further advantage. The whole thing was quite unstable, and small events tilted you towards Beta or VHS. My analogy was, this was like bowling a ball perfectly down the middle of an infinitely long bowling alley. It could stay quite long in the middle, but if it started to drift to one side, it would go further, and then it would fall into the gutter at the side, and that side would lock in the market, so to speak.

Sonal: And by the way, you borrowed — I think I remember you telling me that you borrowed the lock-in jargon from the military, like locking in on a target.

Prof. Arthur: Yeah, the — lock-in wasn’t used heavily at the time. I’m sure there are other people who used the phrase, but with fighter jet radar, when you’re going at very high speed and you’re pursuing an enemy or something, or maybe a radar station itself on the ground, you lock into the target. It’s not just that you find the target, but you want to lock on to that target, and then you can release your weapons, and the weapons will stay locked into that particular…

Sonal: Right. I remember this from “Top Gun.” I mean, that was also very popular.

Prof. Arthur: So I borrowed “lock in,” and since, that’s become very popular. We’re locked into this, we’re locked into that. Basically, meaning that small chance events have landed you into something you can’t get out of. So what I realized through quite a few phenomena that have become famous — since this was all very embryonic in my mind — that the sort of firms I was looking at, if one of them got ahead out of half a dozen, it could get further ahead. You couldn’t predict which one would get ahead. It would start to get enough advantage that it could dominate the market and get still further ahead. It would lock in. It would have so much cost advantage — or now we’d say it’s so much user base — that it would be hard to dislodge. Microsoft got ahead with certain contracts very early in the game. They locked in a lot of the personal software in the 1980s.

Similarly, other systems came along since. There were search engines like AltaVista, as well as Google and others. Google gets ahead and began to dominate that market, and now has it pretty well locked in. You could say similar things for social media. So it was a general phenomenon, that anything that got ahead — because you wanted to be with the majority of people — could get further ahead. We now call it network effects. Companies like that set up a network of users, you want to be with the dominant network because your friends are with that, or…

Sonal: It’s more valuable the more you use it — are in it.

Prof. Arthur: Yeah. Or you know more about it, you hear more about it, or you understand it better. Five generations ago, none of our ancestors spoke English, but we’re all speaking English now.

Sonal: English is a network effect? I’ve never thought about that.

Prof. Arthur: We speak English because we wanna be understood by everybody else.

Sonal: Right. You’re right. I never even thought about it.

Prof. Arthur: And if small events had gone otherwise in the 1700s, it might’ve been French. Or, if you were betting in the 1500s, it could have been Latin or whatever.

Sonal: So, how was it received, when you first put out this paper arguing against diminishing returns in tech — more towards positive feedbacks, increasing returns?

Prof. Arthur: Well, I wrote a paper on this in 1983, sent it to four leading economics journals. Not all at the same time, one after another. I finally got it published 6 years later in 1989.

Sonal: So they didn’t really accept it?

Prof. Arthur: They did not like it. I kept getting reviews saying, “We can’t find fault in this, but this isn’t economics.” And in the meantime, the idea was out there, but there was no citation because no journal dared to publish this. There’s a good reason. In those days, what I was saying is that the economy could lock in to technologies — or to products, or even to ways of doing things — that might be inferior, because that came up, maybe, early on by chance and got locked in. And during the Cold War, in the mid-’80s, this was not popular. I gave the talk in Moscow in 1984. I was saying, in a capitalist economy, you can lock in to an inferior product. Hands went up, you know, <in a Russian accent> “Professor, we want to point out that in Soviet Union, such a thing not possible because with socialist planning, we do not make such mistakes.”

Marc: The central planners will dictate the correct outcome.

Prof. Arthur: Yeah. I came back to Stanford, got a Ph.D student. I said, “Figure this out. I don’t believe it.” He did. He wrote a beautiful paper. Robin Cowan is his name. And he showed that even with the best of planning, you can’t foresee what’s gonna happen…

Sonal: That’s fascinating.

Prof. Arthur: …and of course, you can lock in to the wrong thing. Economists hated this. The whole idea was, everybody’s free to choose, and that lands you in the right solution. And I thought, “Is that correct? I’m free to choose. Do we always choose the best spouses? Social statistics might suggest otherwise.” But what it made for was a very different game in Silicon Valley.

Early reception and current views

Sonal: So, speaking of it being a different game. You know, we have a lot of entrepreneurs that listen to our podcast. How does it change the game? Because people always use the phrase “game-changing” very freely.

Prof. Arthur: Well, first of all, entrepreneurs in Silicon Valley are really smart, and they didn’t exactly get all these ideas from me. I’m not being modest, I’m just being realistic. When I brought out this theory, it kind of corroborated their intuition. So, what I’d say is this. If you are thinking in standard terms — go back to brewing beer, or a company like General Foods — if you want to make profits, you’re thinking of getting production up and running properly, getting your costs down, making sure everything’s terribly efficient. The game was different in tech. The whole game was to try to, early on, grab as much advantage as you could. And I remember that I wrote a paper on this — the “Harvard Business Review,” in 1996. And as that paper got circulated very widely in Silicon Valley, I remember hearing one story that Sun Microsystems had developed Java, and naturally, that cost a huge amount of money. So the guys with the green shades…

Sonal: The accountants?

Prof. Arthur: …were saying, naturally enough, we should charge a huge amount of money for anybody who buys this. And the other people, who had read this theory, said, “No, no, no, no, no. Give it away, give it away, give it away for free.” And there was a tremendous hullabaloo over this. And finally, somebody took my article and just slammed it in…

Sonal: Fighting with papers, I love it.

Prof. Arthur: …Scott McNealy’s desk, and it was game over. He got the point immediately that what you do in an increasing returns market is you try to build up your user base. Now, that’s become completely intuitive since. There was a time it wasn’t standard, that the accountants were saying, “We need to amortize all the R&D money, and we need to get that outlay back as fast as we can, so we’ll charge arms and legs. Later, we can drop the cost.”

Marc: Right. It requires deferral of gratification, right? It requires long-term thinking.

Prof. Arthur: That’s right.

Marc: It requires, in other words, not only strategic thinking, but also long-term thinking.

Prof. Arthur: Long-term thinking.

Marc: You have to project forward to what the economics will be when you win.

Prof. Arthur: Yeah. And, again, I think that that makes [a] very different atmosphere in tech. Tech is not about making profits. It’s about positioning yourself in markets, and trying to build up user base, or network advantages — trying to build on those positive feedbacks. Think of amazon.com. For years and years, they kept reinvesting and kept betting on the positive feedbacks, and eventually, they dominated that whole market. Now they can make huge profits and keep expanding. But it gives you a very different way of thinking. I called the standard way of doing things, “The Halls of Production” — you know, these big factories — but it seemed to me that what was happening in tech was not the halls of production. I called that “The Casino of Technology.”

As if you had this huge marquee. There are many tables, with different games going on, you know — oh, yeah, we’re doing a game on face recognition over here, or whatever. And people come up to the table, just as search engines say, “Okay, who’s gonna be here?” “We don’t know. The technology hasn’t really started.” “What’s the technology gonna be like?” “I have no idea, Monsieur.” “How much will I have to put upfront?” “Well, you know, you could join the game, Monsieur for maybe 1 billion.” “What are my chances of winning?” “I have no idea. Perhaps if there are 10 players, your chances might be 1 in 10. Do you still want to play?” So it was a very different game.

And I don’t want to make it sound like too much luck, because the particular entrepreneurs who — kind of, knowing that their technology was right — and they had a sort of instinctive idea of positioning the technology, and building that user base early. Rather than saying, “We want to get profits out of this.” The game keeps changing, but my point is that the basic game in tech is not the same as the basic game in standard production. And every once in a while, you see somebody taken from the standard production side of the economy — some CEO — brought into a tech firm, and they don’t quite get it. The classic case was Apple. The classic case was Sculley.

Sonal: That famous quote, “Do you want to sell sugar water for the rest of your life?” For him to be enticed away from a beverage company to work at Apple?

Prof. Arthur: Yeah. And CEOs are very smart indeed, but it’s not just a matter of intelligence. It’s a different way of thinking. And it’s so familiar to us now — this new way of thinking in the valley, in Silicon Valley — that we take it for granted that we always thought that way. But we didn’t.

Marc: Do you think that — I don’t know if you have a view on this or not. Do you think financial markets understand this, to the degree that they should, even after all this time?

Prof. Arthur: No. I’ll give you two instances. Warren Buffet very, very famously said, “I don’t dabble in high-tech.” He says, “I don’t touch that, simply because I don’t understand it.” A friend of mine, Bill Miller of Legg Mason — I’ve known him for 20 or more years through the Santa Fe Institute. Bill read this stuff, got it, understood it, and did extremely well. So the best answer I can give to that is, it’s not general knowledge among investors fully yet. Certainly wasn’t 20 years ago. But there’s an increasing number of people who get that the rules of the game are different in tech from in standard business.

Marc: One of the smartest hedge fund managers I know — he says there’s still, in financial markets — is what he calls the New York-Palo Alto arbitrage, right? And basically, he said his strategy is [to] spend half his time in New York, and understand what all those assumptions are — which basically are the drivers — New York is the driver of asset prices. That’s where most of the really smart investors are, at least in the U.S. And then he says, basically, come to Palo Alto, figure out all the ways they’re wrong, and then place the contrary bet. And the theory, I think, that you’re laying out underneath that is, basically — you might say that the New York mindset stereotypically might be the “halls of production” mindset, even still.

Prof. Arthur: Yes, that’s right. Yeah.

Marc: Right.

Prof. Arthur: It certainly is that way in Europe. I’m always amazed, and slightly appalled, that people think of technology in Europe as something that’s done by very big companies, and it’s pretty good technology. But they don’t get that this is a game of positioning, of building a user base. And it’s well understood in California. It gets less well-understood on the East Coast, and then not very well understood elsewhere.

Marc: So, question. Another very smart guy, Peter Thiel, takes it a step further. He asserts that in the long run, every kind of industry — every kind of product — either becomes a monopoly or a commodity. In other words, in the long run, the margins either go to infinity, or they’re 100% — or they go to zero. And it’s just a question of time, and if you don’t have increasing returns, you’re on a long-term downward slide to commodity. <Yep.> And he asserts that the things we view as intermediate cases — businesses today that are, like, 20% margins — are fated to decline to zero over time. Is his view, do you think, too extreme, or would you support even a view, kind of, that stark?

Prof. Arthur: I like the idea. I think he’s basically on target, but there are perennially commodity industries — I’m thinking of airlines — where the margins are pretty low. They’re usually lower than 10%, but still these persist, and quite often, governments intervene. Yeah, I have a lot of sympathy for Peter Thiel’s view. I think that in the long, long run, things do tend to get dominated by only one or two players, even in the standard businesses. And the reason that’s not completely and utterly true all the time is that there are new products getting launched all the time in standard product space. And that keeps us in this more standard economic setup.

Diminishing returns and luck

Sonal: When you describe the work on increasing returns, <Yeah.> you also mention the flip side of this, sort of, effect of increasing returns — which is sometimes you might get to the point where the network can go back to a point where it goes to diminishing returns.

Prof. Arthur: Yes.

Sonal: For example, if there’s too many listings in a marketplace, or something. Do you have any thoughts on that, or any new takeaways around that? Because if the network is more valuable as more people use it, why would there be a diminishing return at a certain point, if it gets too big? Like, is there an ideal size?

Prof. Arthur: No, I don’t think so. I think it depends very much on the network itself. Some networks can eventually become commoditized. And so, if it’s a commodity, anybody can, sort of, come in and offer the same thing. But a much more common pattern, and the pattern that I would expect, is that there is a network. Go back to 1984. Microsoft moves in, other companies move in, Microsoft dominates. But eventually what happens in [an] increasing returns market is that the next invention comes along. And some other company [that] is offering web services, or something, comes to dominate. So you can dominate for a while in one large niche in the digital economy, but then the next set of technology comes in, and new players come in with that. Google recognizes this, and Google’s trying to stay ahead of them by being in on the new technologies.

Sonal: Well, it reminds me [of] when they tried to do, like, social networking when Facebook came along, and now they’re, sort of, just decided to become an AI-first company. So it kind of brings that full circle.

Prof. Arthur: Yes, that’s right. But companies don’t always make that transition from one technology to the next very well. Apple’s been very lucky, where they’ve invented some of the technologies, and then they’re able to surf on that new board, so to speak. But the overall thing is that lock-ins tend to last for a certain amount of time, and then they become obsolete, and some new game comes along.

Sonal: Right. Or they become ubiquitous, utility-like — and the new game still comes along. Because I would argue that Google’s always gonna be around for search. <Oh yeah, sure.> Because they’ve, sort of, dominated that market, but they may become, like, utility in that application — if not in something new.

Prof. Arthur: That’s right. And then the advertisers may drift off to something hotter.

Sonal: You mentioned earlier that you don’t think it’s luck, and this discussion makes it almost sound like it’s an accident that there’s a winner-take-all effect. But is there some way of knowing early on — the entrepreneur who maps out the future, who knows the ecosystem — how do we sort of know that these are the ones that will figure out how to tip the market in their favor? What are some of the indicators — it’s not an accident. Like, they’re pulling levers strategically.

Prof. Arthur: Yeah. Let me give you an analogy. Shows how hard this is to predict. I remember sitting — in 1991, I was invited to the Senate building to brief Al Gore, who was a senator then. It was an afternoon, and was quite hot. And they’re all sitting there, everybody was a little bit sleepy. And then Gore says, “Can you give me an example I can latch on to?” And I said, “Yeah, presidential primaries.” <laughter> And they got it immediately. The phenomenon I’m talking about, you know — if something gets ahead, it tends to get further ahead. It’s true in presidential primaries, that if some candidate pulls ahead, they get more financial backing, they can be more visible. The more visible they are, the more likely it looks that they might win the presidency.

So they get further ahead and more backers. You have to be quite a way into the game before it’s pretty clear. That’s the best I can do on that. Meaning, sometimes if there’s a very early tilt, like, within a few months, it’s pretty clear what’s gonna take over. But it can be very much like presidential primaries. It’s all the same mechanism. And predicting exactly who that’s going to be might look easy afterwards, but on the spot, it’s very difficult to do.

Marc: Well, this goes, actually, to the nature, I think, of how history is written, right? Which is, the way history gets written is, the victor is imputed all kinds of positive qualities. Like, genius, visionary, marvelous executer, right? And everybody knew, right? Everybody predicted. And then, of course, the people who don’t win, it’s like, “Oh, idiot, you know, losers, what were they thinking?”

Prof. Arthur: Yeah. Exactly.

Marc: We experience this in venture capital. It’s like, we basically get two kinds of press coverage. One is what a bunch of geniuses we were for backing the successful company, and what a bunch of morons we were for backing the failing company. And I keep pointing out we’re the same people. We don’t whip between genius and moron. We’re somewhere in the middle. But to your point, it’s the nature of the technology casino. The other thing I’ve observed on this point is — I don’t know if it’s cynical sense, or maybe a realistic sense — in a sense, the question of, like, what is the spark that causes one to jump ahead? In a sense, it kind of doesn’t matter.

Sonal: Oh, that’s actually, kind of, very sacrilegious.

Marc: Or even say, a less cynical way to put it might be — there might be 20 different ways somebody gets that initial jump. It might be they start two months earlier, it might be they raise a little more money, it might be they get a key distribution partnership. It kind of doesn’t matter exactly what it is as long as there is — as long as something actually happens. And so, there is a lot of idiosyncratic kind of history to these things.

Prof. Arthur: Yeah. And my shorthand term for all that is luck. Of course, there’s no such thing. It’s just all small events. Who sat beside whom in a airplane, and chatted up somebody or whatever.

Marc: Or whose mother happened to be on the board of United Way — the CEO of IBM, as one example.

Prof. Arthur: Yes. Yeah, yeah. Famous story. Yeah. Yeah.

Sonal: Oh, right. This is that infamous Bill Gates biography story — where, because his mom was on the board of United Way, she met the CEO of IBM, and then that meeting led to Microsoft and IBM striking a software deal that helped Microsoft in the early days.

Marc: Right. Exactly right. Right. The other interesting kind of situation that we run into a lot on this, when we try to figure this out is — it’s fairly often you’ll have a scenario where you’ll have 2 — you might have 20 in the field, but you’ll have 2 companies that you kind of think have the highest probability of winning. And one of them is a little bit further ahead, but has a somewhat less-skilled or experienced founder. And then you’ll have another company that maybe started a little bit later, that will be further behind for the moment, but has a much more experienced and qualified founder CEO. And if you’re going entirely based on current trends, you go with the less-experienced, less-knowledgeable founder. On the other hand, you often have somebody very sharp who’s like, “Oh, yeah, I know exactly what I’m doing. He doesn’t know what he’s doing. I can take him out.” And like, that’s a real — that’s a conundrum that we face every day. <crosstalk> And it really elevates this kind of question of, like, how important actually is skill?

Prof. Arthur: I mean, you’ve pretty much answered your own question, I think. Skill is extremely important, but it’s not tech skill. It’s not even skill in raising money. Those are kind of necessary, but not sufficient. What sort of skill is really, really important is strategic skill. It’s feel for how to build here, how to build up there. Basically, I often thought of this as surfing. You either get a wave or you don’t. If you get the wave, the whole momentum of that wave pulls you forward, and then you’ve got to maneuver and stay in the green water.

How new technology is created

Sonal: Oh, yeah. That was an analogy that Pete Pirolli used to use a lot at PARC for innovation, because he’s such an avid surfer. He would compare the two. And I remember reading an article years ago by JSB as well, that compared executives to surfers. 

But let’s actually now shift gears and talk about, like, you know, once you understand these concepts that we’ve been talking about so far. Once you have these building blocks, like, network effects and positive versus diminishing returns — that you can essentially manipulate to pull levers and get the outcome you want, maybe luck, maybe not. The bigger question is, are there macro forces at play here too? I don’t mean macroeconomic. I mean more around the nature of technology and how it evolves — which coincidentally is the name of the book you published in 2009. I have a copy from you on my shelf. Anyway, it surprised me that you once argued that tech evolution is not like evolution in the obvious sense. So, tell us about that.

Prof. Arthur: Well, yeah. Quite a while ago, about 15, 20 years ago, I got really interested in where technology comes from. And the idea around that we have is that there’s some genius in an attic or something.

Sonal: Usually a garage.

Prof. Arthur: Yeah, a garage. That’s right. Cooking up technology and coming up with inventions. What started to become clear to me as having looked in detail at some inventions — that technologies, in a way, come out of other technologies. If you take any individual technology, say, like, a computer in 1940s, it was made possible by having vacuum tubes. By having relay systems, by having very primitive memory systems, maybe mercury delay tubes. All of those things existed already. And so, it seemed to me that technologies evolved by people not so much discovering something new, or inventing, but by putting together different Lego blocks, so to speak, in a new way. Once something was put together — like, say, a radio circuit for transmitting radio waves — it could be thrown back in the Lego set. And occasionally then, some of the new combinations would get a name and be tossed back in. Things like gene sequencing were put together from existing molecular biology technologies, and then that becomes a component in yet other technologies…

Sonal: Right. I mean, CRISPR is a great example.

Prof. Arthur: CRISPR. Exactly.

Sonal: Now you have CRISPR, which itself is a gene-editing tool, which then creates so many other things.

Prof. Arthur: Yep. And that tool will be a component in future technologies. And I began to realize this wasn’t Darwinian. It wasn’t Darwin’s mechanism. It’s evolution, but it wasn’t that you vary radial circuits, or you vary air piston engines…

Sonal: Right. It’s not like a natural selection effect.

Prof. Arthur: Yeah. You can’t vary radial circuits and then suddenly get a computer out of that, or radar. You can’t vary air piston engines in 1930 and get a jet engine out of that. These things come along as completely new combinations, using new principles, and that keeps adding to your Lego set. And that starts to explain why there’s a controversy or a question, say, in the 1920s. Anthropologists were asking, why don’t you have trams and steam engines in the Trobriand Islands? And they began to say, “Well, it’s not because the islanders are stupid. It’s because they don’t have these building blocks to build it out of.” And that, in turn, has many implications. One of them is, if you get a region like Silicon Valley, with an enormous number of these building blocks — and more important, it has people who understand the craft to put all this together — not just the science, but what parameters — then it can very quickly keep coming up with new combinations.

Sonal: What’s the implication for adoption though, for industry?

Prof. Arthur: The implication is that if you have a new collection of technologies — let me just mention AI, artificial intelligence — those are all building blocks. Industry doesn’t adopt AI. AI is a slew of technologies. It’s a new Lego set. Industry is using its own technologies. And what really happens is that industries — the medical industry, the healthcare industry, the aircraft industry, the financial industry — they encounter this new Lego set of AI, and they pick and choose components to create their own new things.

Sonal: <inaudible> and recombine, right.

Marc: One of the interesting, sort of, aspects of that, I find, is as a consequence of what you’re describing. There, it seems to me, is a long pre-history of almost any “new technology,” right? A couple favorite examples I have of that — the French had optical telegraphy working, I think, 40 years before other people figured out electromechanical telegraphy. So, literally, tubes of glass underground in Paris with light pulses going through, and this is like the 1820s or 1830s.

Prof. Arthur: Really?

Sonal: I had no idea.

Marc: Super early. Another great example…

Prof. Arthur: With telescopes or something?

Marc: Yeah. Some sort of — I mean, they were, like, relay stations, but it was little flashes of light, like lanterns, through glass tubes. And so it was, sort of, fiber optics — 160, 180 years prematurely. My other favorite example is — MIT published a great book called “Tube” years ago. It was about the pre-history of television. And we think of television as being, like, 1930s, 1940s, Philo Farnsworth, all these guys. Well, it turns out the idea for television emerged immediately upon the idea for radio. And there was a Scottish inventor named John Logie Baird. And in the — I think 1910s — he invented mechanical television, because he couldn’t do the electric. He did mechanical.

And so he literally had spinning wooden blocks. So he had pixels. It was almost like a computer display, but made out of, literally, wooden blocks. And the pixels would basically spin — the wooden blocks would spin to form pictures. And one of the funniest scenes in the book is, he takes it to the board of governors of the BBC, in 1912 or something, and they’re like, “You are completely out of your mind.” And he’s just like, “Oh, just let me prove it. Let me prove it. Let me get some sets out there. I’ll prove that people want to do this.” And they finally gave him a programming block. They gave him access to the radio frequency — Thursday night, midnight for 15 minutes — and he was broadcasting for months…

Sonal: That’s amazing.

Marc: …you know, mechanical TV that nobody ever saw, right? And then 30 years later, right, people picked it up and actually made the version that works. So then I go through all this just to kind of say — so then what we can project forward is that all of the breakthrough technologies of the next 30, 40, 50 years — they already, in a sense, exist in some form. Is that…

Prof. Arthur: Yes, that’s right. Pretty much. To get a new technology, you need two things. You need to sort of have a principle — meaning a way of doing things. Early television worked on this idea that you could pick up pixels, or little snippets of light or darkness in the image you’re looking at, and then transmit those by radio — very high frequency — decode it at the other end, and reproduce on some screen or another. So, yeah, you need a principle and you need the components. There’s a famous example Stanford was involved in. In the very early 1900s, the U.S. Navy was very interested in telegraphy, or telegraph.

What they had at the time was spark radio. So, you could sort of, you know, <sound of electrical current> send these Morse code things across the whole spectrum. Anybody could pick it up. So they were looking for a perfect sine wave, as continuous-time radio — a continuous wave, not just a spark wave — at a single frequency. There was a company formed, Pacific Telegraph, sometime around 1906, 1907. They managed to get the guy who invented the triode vacuum tube…

Marc: Oh, de Forest?

Prof. Arthur: Yeah, Lee de Forest. Came out from Yale, kind of on the run from predators. De Forest and Federal Telegraph spent several years trying to get a perfect sine wave so that they could transmit radio waves on a single frequency offshore to naval vessels. They couldn’t really do it. And in 1912, AT&T put out a call for inventions. Their idea was to be able to telephone from New York to Chicago, but you needed to have some sort of repeating circuitry — needed to clean up the wave every 20 miles or so, and then retransmit it. It turned out that within about six months, three inventors — de Forest among them — came up with a triode vacuum tube early amplifier. That amplifier was fed back — that becomes an oscillator, kind of like a microphone shrieking. The oscillator gives you a perfect sine wave. You could modulate that and send that out as a radio message to ships offshore, or to anything. 

And if I recall right — this is quite early in the game. These radio guys with headphones — they were always called sparks, the radio officers in ships — they were listening to Morse code one time, not very far from here, and suddenly, somebody transmitted music. They all, kind of, jumped — what the hell, you know? <laughter> They’re listening to dot-dot, you know? <Beethoven.> And suddenly there’s music coming out of their headphones, and it blew their minds that this was possible. It’s a bit of a long story, but the point is that individual inventions, like the triode vacuum tube, when put together in clever ways with other components, give you an oscillator, which is the basis of radio transmission. They give you radio receivers, etc., and that builds up the broadcasting industry, which, in turn — parts of that are used to give you television. And then in relay form — on or off switches — these things start to give you logic circuits, and in turn, that gives you early computers, etc.

Sonal: Which in turn… the semiconductor industry to — right.

Prof. Arthur: So, technologies don’t come out of nowhere. They come out of a very deep understanding of what’s in the Lego box and how to put those things together.

Marc: Well, so the pessimistic view on that would be — boy, that means by implication, there really aren’t the kind of eureka moments that people think about. And the pessimistic view on that is — then, therefore, there’s really not gonna be anybody sitting around in the next 20 years who’s gonna say, “I wanna build warp drive,” and therefore faster than light travel. And they’re just gonna come up with it. Or immortality, or whatever, you know, these — and so, in a sense, it’s an argument against, kind of, dramatic innovation. Let’s just say determined innovation. On the other hand, it’s an optimistic argument, because it says the number of combinations of the Lego blocks are combinatorially effectively infinite over time.

Prof. Arthur: Well, I did argue that there are breakthroughs, you know, there are eureka moments. They tend to work that — I’m sitting here wondering how I could get some effect. How could I transmit images by radio wave? And I could be sitting there thinking for months, “Well, I could use this combination, that combination, and another combination,” and then suddenly I realize, if I can get this in place, and that in place, and the other thing in place, that’s gonna work. And the interesting thing is — and I’ve read individual accounts by the dozen from inventors, even lab books. You see this again and again, “Can’t do it. Can’t do it. Can’t do it,” and then, “Oh. Oh. Oh.” One of my favorite stories is that the steam engine already existed way before James Watt. And James Watt, in the 1760s — I think it was in Glasgow — was brought in to see if he could improve it.

So Watt thinks it over and he thinks, “Oh, well, you know, you’re heating the steam, you’re expanding it in a cylinder, then you’re suddenly cooling it again, and all of this is pretty slow. What if I allowed the steam to expand the cylinder, and then that steam is ejected into a second cylinder that’s kept at [a] very low temperature?” Suddenly, the steam collapses, there’s a vacuum, etc. So he invented an independent cold cylinder. He thought of it passing the village green on a Sunday, the Sabbath day. He was properly Scottish. It nearly killed him. He says, “And there I was…” And he knows it’s gonna work, but he can’t get into his workshop until Monday. You can just read this stuff and see that it’s half killing him, that he can’t prove the concept until the next day. He’s a machinist, and he got it to work fairly readily.

Sonal: So he’s using the building blocks to basically — people are using existing building blocks to do this sort of combinatorial innovation, combinatorial evolution…

Prof. Arthur: That’s right. Yeah, yeah. The point I’m making is that new technologies don’t build up as just pure inventions. There’s plenty of breakthrough insights, but they build out of what’s already there, the components. And quite often, then, new things come along, some key breakthrough technologies. Deep learning is one. CRISPR is another.

Sonal: Right. These aren’t just isolated components. They themselves are tools, and literally recombine or create other technologies. And, by the way — in that sense, I think it is very much like evolution. I mean, we had Yuval Harari on the podcast too. And basically, in his book “Sapiens,” he argues that tech helps mankind leapfrog natural evolution. And, only in that context — we were talking about it across a much larger timescale. But in this context, I do think of it as a primordial soup for the next phase. 

On that note, you mentioned deep learning, which — we think of it as, basically, machine learning, distributed computing, artificial intelligence. I mean, just for this purpose, we can broadly clump that into one category. And I remember a big piece you did for “McKinsey Quarterly,” right before I left PARC. It was around 2011, and it was on the second economy. Basically an autonomy economy. And actually, you should summarize this, because then I’d like to talk to you about how you might update that today, given all the advances in AI since.

Prof. Arthur: Sure. Yeah. What I was pointing out was that there’s a familiar physical economy, the one we all know about. It has to do with retail stores, and factories, and banks — all the stuff that we see in the physical world. I was checking into a flight in the San Jose airport, sometime around 2011, and when I put my frequent flyer card in, suddenly, it was triggering a lot of processes. Certainly, the flight was being alerted that I was now there, maybe TSA was being alerted. So I began to realize that somehow there’s a huge second economy out there of machines talking to machines. I was thinking of it as a very large underground, unseen, invisible economy — could be in the cloud — of servers talking to servers, of software and algorithms talking to servers, talking to other servers — all being transmitted and in conversation. Always on, and occasionally then, putting out shoots up into the physical world.

And it reminded me as a metaphor of aspen trees. Aspen trees, apparently, are one huge organism — that is, they’re all connected underground with the same root system. And what you see on the surface is the trees themselves, but there is a very, very large underground root system that’s all connected. These roots are all talking to each other, and this would be like the second economy. I now think I should have chosen the term “virtual economy,” or better still, the “autonomous economy,” because all of this is happening without our knowing. It’s autonomous. It’s things talking to things. So I don’t emphasize an internet of things. It’s more like an internet of conversations. Things triggering things, things switching off things, and querying.

Sonal: I mean, just to give it a quick picture. If you have that image of you putting the card in the kiosk at the airport, and you have all these machines talking to each other, if you were to light up all those machines at once, they’d be all around the world. There’d be servers on Amazon’s cloud, there’d be something local. The local printer. There’d be something else, like, a processing payment thing, maybe in Palo Alto. There could be all these different pieces kind of coming together to drive that one transaction.

Prof. Arthur: Yes. And not just a few dozen computers or servers lighting up, because those servers would be lighting up other servers. <Right.> And so, in the end, there could be hundreds of thousands of servers that were lighting up very briefly, maybe only for a few fractions of a second, and then shutting down again, and then passing messages. So I was interested in this autonomous economy. There was general conversation about automation and robots, and 3D printing. And I thought, no, they’re missing the point. I tend to think that the digital revolution — I believe there is such a thing, and I believe it keeps morphing or changing. About every 20 years, the digital revolution gets a new theme. <Right.> And the latest revolution comes almost by accident — that in the 2010s or so, we started to get huge numbers of sensors. Sensing chemicals, sensing visual pixels, sensing images, sensing temperatures — by the hundreds and dozens and hundreds of thousands. And all these sensors out there — and they were maybe feeding back from smartphones or from your car, and huge amounts of data.

About the same time, and this was no coincidence, along comes a new generation of neural networks powered by deep learning. But more than anything, powered by all the data that the sensors are bringing us. And these algorithms started to be able to do one thing very well, and that was pattern recognition. Could recognize your voice much better than before, because of all the data, all the training. It could recognize faces. So, suddenly, we got the ability of algorithms to do things that we thought only humans could do.

As recently as 20 years ago, or 10 years ago, we would have said, “Oh, yeah, computers are great, but they’ll never be good at what humans are good at.” What are humans good at? We’re good at recognizing things, we’re good at fast association. Computers, they can do deduction or logic. We’re not much good at logic, so it seemed that the whole world was nicely divided.

Sonal: But now…

Prof. Arthur: But now, computers have learned to do associative thinking. These patterns mean such and such. And so, suddenly, we’re in an area that we thought only human beings were gonna be good at, and we’re seeing industry after industry change as a result. It’s not just automation, it’s much more than that. It’s redoing or restructuring whole areas of the economy. So, I was looking for an analogy in history that even vaguely resembles what’s happening. The printing revolution, starting around the 1450s — suddenly information went from being very closely guarded by monasteries and abbeys and libraries — these big vellum books chained to desks. And with printing, it became publicly available. So, printing made information externally available, and that changed everything. It very much changes the way people are thinking. Copernicus, for example, had at his disposal data that he could not have got hold of if it just existed in monasteries. It made a huge difference. It brought in modern science, it helped the Renaissance, and this brought us our modern world.

Sonal: I mean, I would agree — but is that the big transformation now? That we have the modern tech equivalent of the printing press?

Prof. Arthur: What’s gone external now is not information. What’s gone external is intelligence. I may be driving in a convoy of 50 driverless cars, and the whole idea of the car adjusting — the car is talking to roadside sensors and servers, it’s talking to other cars, it’s talking to the highway patrol servers, and so on. And it’s basically farming out its intelligence into this other economy, and then getting back intelligent actions in return. So it’s a bit like phone-a-friend, only the friend is incredibly smart, and the friend consists of, again, these hundreds of thousands of servers talking to each other and then adjusting what you do. So suddenly, intelligence doesn’t just exist in human beings. Suddenly, intelligence exists in the cloud, or in this autonomous economy, and we can farm out not just getting information, but getting smart moves back. And this is making all the difference.

Sonal: So it’s not about the form intelligence takes, it’s that intelligence is no longer housed internally in the brains of human workers. Because it’s moved outward into the virtual economy.

Prof. Arthur: Yes, that’s right.

Impact of computer intelligence

Sonal: So, when intelligence is not just information, but, sort of, decision-making, or being able to externalize a lot of this — I mean, one of the things you mentioned earlier is about these building blocks of technology. What happens when all of these things are available to everybody equally? Like, is there not, like, a sort of a Red Queen effect, where everyone’s accessing the same building blocks and tools? So how do companies, how do industries find competitive advantage in that kind of a world?

Prof. Arthur: I think the answer to that question is timing. If I’m a retail bank, whatever that might be, I might be quite a large bank. And I’m saying all these externally intelligent technologies and algorithms are suddenly available. How can I make use of that, and how can I bring those into my operations and combine them with what I’m doing? I’m making mortgage loans, I’m acting as escrow or something, you know — all these various different types of financial operations. I can make a lot of them automatic and autonomous, and get an advantage. The trouble is that that can be rapidly commoditized.

Sonal: So what does that mean for jobs? In this podcast, we talk a lot about how whenever industries are changed in this way — you know, through tech and other shifts — that other new jobs — classic examples include more designers in the age of Adobe design, that new jobs that never existed before, like social media managers — that can only exist today. What’s your take here?

Prof. Arthur: So, what I’m seeing is — about 90 years ago or so, John Maynard Keynes pointed out that he thought by 100 years’ time, 2030, we’d be in an economy where the production problem was largely solved. There’d be enough, in principle, to go around for everyone. There might be plenty, in principle — goods and services around — but getting access to them meant you needed wages, which you needed a job for, and that was not possible. I think that what Keynes said in that regard is becoming true. In other words, the trough is full, but how do the piggies get their share of the trough? So we’re now in a new distributive era. 

What’s counting is not how much is produced, but who gets what. The whole question of growth and getting more economic product out there — physical product and services — that’s a job for entrepreneurs, and it’s a job for engineers. Who gets what is much more a political issue, and that’s not quite a job just for politicians, but it’s a job for society to solve. And we haven’t solved it in Europe or anywhere else, so it’s a new era.

Marc: The problem with that theory is the same problem as that theory in Keynes’s era, right? Which is, sort of, Milton Friedman’s observation in the 1950s, 1960s, when that issue came up again. Which is that human wants and needs are infinite, right? One of the things we are best at as a species is coming up with new things that we want. <crosstalk> And then the things that we want in one generation become the things that we need in the next generation, right? Air conditioning goes from being a luxury to being something that we expect…

Sonal: Cell phones…

Marc: …and we get outraged when we don’t have [it], and cell phones and everything else. And, you know, he speculated as a thought experiment — he said, “Look, you know, we have no way of envisioning the wants and needs of what people will have in the future. We just know they’ll be there.” And he said, “Look, maybe it’ll be that, like, you know, right now, psychiatry is a luxury good. And maybe in the future, it’ll be a basic human right to have access to a psychiatrist, and then we’ll employ half the population being psychiatrists to the other half.” It just is one example, right, of…

Prof. Arthur: I’m looking forward to this new economy.

Sonal: I like that one. The best, actually, among the examples, so far.

Marc: Exactly. And in economic terms, of course, the problem with Keynes’s analysis was, it overlooks the role of productivity growth, right? Which is the scenario that you were describing — is the scenario of, like — rapidly increase your productivity growth. And in a world of rapidly increasing productivity growth, you have gigantic gains in economic welfare. You have gigantic growth in underlying industries, right? You have gigantic amounts of entrepreneurial activity that come out of that. And that, then, generates a fountain of new jobs to satisfy all those new wants and needs. And then, finally, I can’t resist pointing out that you’re making this argument on a day when the unemployment rate in the U.S. dropped below 4%.

There’s certainly no trace — and remember, once there was a day in the American economy — you actually had very low productivity growth, not very high productivity growth. Which counters against the argument that there’s some level of unprecedented technological disruption that’s happening, because you certainly can’t see it in the numbers. And then you have unprecedented levels of job growth and employment. So the facts seem to be on the other side of this argument.

Prof. Arthur: Well, let me both agree and disagree here. I certainly agree that there will be whole new categories of jobs. I very much like the idea that half of us will be therapists.

Sonal: I love that one too.

Prof. Arthur: And the other half — and we can swap couches.

Marc: Oh, yeah, no, no, the therapists will need therapists.

Prof. Arthur: I think there’ll be plenty of new jobs invented. At the same time, though, not just through automation and not just through algorithms, but over the last 20 or 30 years, we’ve had a huge amount of globalization. Jobs have been off-shored, and that’s not just due to the rise of China. It’s due to the rise of telecommunications. Of — I can keep track of all the suppliers in China, all the factories in China, the inventories, and so on, in real time. Couldn’t have done that much in the 1980s because the technology wasn’t there. And that hollowed out an enormous amount of traditional workers in the middle of America, and certainly in Britain and in other countries. So, where I would come out on this question — I like your observation. I agree, yes, we will get new jobs, but quite often there’s a big lag in between the original happening of hollowed-out industries, and then something taking its place.

An analogy that I like is that in Britain in the 1850s. The economy was going gangbusters. New textile companies, the railways were just starting to kick in. There was all kinds of possibilities — steelworks, everything got suddenly very serious. And at the same time — so, there were people getting very rich, but at the same time, there was child labor, there were… 

Sonal: The Dickensian world…

Prof. Arthur: The whole Dickensian world of people almost being worked to death. Both are true. The economy is going gangbusters. Some people are not doing well out of this. It took about 30 to 60 years before the whole thing equalized and workers had safe conditions, they had much better conditions — and eventually, they were able to partake in a decent way in all this wealth creation. So what I would say is that the digital economy, through globalization — and now through algorithms — is pressing us into a scramble to invent new categories of jobs. I’m optimistic. I think eventually, we’ll get on top of this. And I’m hoping we do it in a good way, where we have creative pursuits, not just rote jobs, like we might’ve had 100 years ago. I think things are going quite well.

Marc: Good.

Globalization and international affairs

Sonal: So, it is a global world now, and it depends on what your frame of reference is. For me, my frame of reference is — I have relatives in India, who are now increasing in their middle class. If your frame of reference is global, you see this as a very different kind of shift. It really depends on where you, sort of, put the square, the rectangle of the frame, and where you zoom in. Because there is Africa, you know, another great example, Cambodia. You have all these countries — there’s something interesting happening there. So, speaking of that, I’d love to hear — because you spent a lot of time in Singapore. I’d love to hear your thoughts on, sort of, the evolution of that, because we’ve often made the argument that this kind of form — top-down, government-planned innovation cluster — never works out, and Singapore is a rare exception. How would you distill it, having been on the ground there?

Prof. Arthur: I’m a watcher of countries that look as if they’re in trouble and then make their way out of trouble. Finland’s a good example, because The Cold War shuts down, Finland was a broker — a bit like Hong Kong, in between the West and the East. Then around 1990, suddenly the bridge is there, but the river ceases to exist. And so then they invented their way out of that with Nokia and other companies. Their back was to the wall. And I could say the same thing in Singapore. When the country was set up, about 51 or 52 years ago, they felt very much as if they had been set adrift — so, like a little rowing boat that was being towed behind Malaysia, and then somebody cut the rope. So, I think, again, it was a matter of desperation, very good planning. People like Lee Kuan Yew, who led the government.

And what they did was, they decided that they would go into what was then tech manufacturing. They had inherited shipyards from the British, etc., so they were able to station themselves as a very early manufacturer, a bit like Hong Kong or Taiwan. Produce cheap goods, and take great advantage that the oil tankers had to stop at — and become a commercial and brokerage hub for shipping. Since that, they’ve moved into finance. What I’m finding — and let me broaden into Asian countries, including China — we tend to think of — as recently as 10 years ago, we would have thought of China as being not fully developed. Not at all like Japan, which is developed. Singapore is quite developed.

What we’re now seeing in Asia is that a lot of countries in Asia, including China, their digital revolution is not much more than two to three years behind what’s happening in California or in the West. They’re extremely well-advanced, they’re paying a huge amount of attention to technical education, and it’s not just that they’re following in China. They’re not just following, say, genomics or AI. They’re inventing their own. Singapore, by dint of strong will and going techie, has managed to do that already. What I do notice in Singapore is that they tend to — not so much initiate perfectly new technologies, but they’re very quick to take them up. China, though, is able to initiate things…

Sonal: Initiate them as well, right.

Prof. Arthur: …especially in things like genomics.

Sonal: Do you think the initiation thing matters? Because part of your thesis around there being these building blocks that are widely available, which leads to this combinatorial innovation, combinatorial evolution, as you describe it. I wonder if that even matters so much anymore, because if these building blocks — open source, APIs — all are available. Like, application programming interfaces that people can combine into entirely new companies. It seems like you can actually draw on the best of the best expertise.

Prof. Arthur: I think so. That’s been a long debate, actually, in economics. Why put all the effort into initiating something when you can just position yourself to learn the technology quickly? The other case is, it’s good to be first. I think it’s debatable. What I would say though in China is that when it comes to a country digitizing everything, China isn’t going to be far behind.

Sonal: It’s especially true of AI, actually.

Prof. Arthur: Yes. Especially in artificial intelligence and in genomics, and probably in several other industries.

Sonal: Well, genomics is particularly interesting, because they were the first to do human-scale studies of CRISPR. Because we, regulatorily — rightly so — may not be able to, or maybe not. I don’t know. I don’t have an opinion on that.

Prof. Arthur: What I see is this sort of technology expanding rapidly into the rest of the world — and the other country, of course, to mention is India. For several decades, technological education in India has been excellent. IIT, places like that, Bangalore — and India is not very far behind. China’s in a better position because China is top-down hierarchical. They can quickly reorganize and change their economy.

Sonal: We go back and forth around this all the time, but every past industrial planning, top-down, centralized model of coordination has eventually eaten its own, and fallen on its own, like — hoisted on its own petard, to use that expression. Which is kind of the thing that inevitably seems to happen. That’s what happened in Japan, and it’ll inevitably happen with China.

Prof. Arthur: Well, it may inevitably happen in the United States, too.

Sonal: That’s true. That’s a very good point.

Prof. Arthur: I do think that, occasionally, economies get a bit tired, people get complacent, etc. I was in India — I’ve been there several times. But a long time ago, like 1975, when there were old English cars — Morris Minors driving around, taxis that you wouldn’t have seen since the 1950s in London. And the Indian economy has gone light years beyond that.

Sonal: Well, I would say that one of the other shifts there, which is important to note here for this part of the conversation, is that India, China, Singapore — they’ve moved away. Well, India went through an outsourcing phase, as you know, you described this, to being originators of their own innovation. They’re not just a copycat narrative. And we’ve written about this when it comes to China as well. I mean, just yesterday, Walmart announced it’s buying Flipkart — which that’s, kind of, an inversion of the typical model that would have happened before. So, anyway, I think that’s an important shift that this is playing out against.

Prof. Arthur: Yeah. The rest of the world is very rapidly catching up. I still think that the U.S. economy is going to do extremely well.

Sonal: That’s great. It’s optimistic.

Prof. Arthur: Well, it’s not just optimism. I think it’s pretty well inevitable. Let me restate this. I think what’s gonna happen in the next decade or two — the story in the U.S. economy is simply going to be that huge industries are going to reorganize themselves along the lines of autonomous intelligence.

Sonal: When you described that the economy has these, sort of, 20-year beams that you’ve seen, and you’ve described them as morphings in your writings. Like, sort of fundamental sea changes. And you described integrated circuits already, and fast computation as the first — we talked about the connection of digital processes, and now you mentioned these sensors. The cheap and ubiquitous sensors. My question for you, as someone who’s long studied this, is — how do you know when you’re seeing the beginning of one of these revolutions? That it’s a morphing in the making. Is this, sort of, a hindsight view? Because you are, sort of, seeing it early with everything else. What are the signs that tell you this is a morphing — this is a big theme that’s emerging? That gives you the confidence to say that about, say, deep learning, or CRISPR, even?

Prof. Arthur: I think that a change is usually quite well underway before people pick it up. You wake up one day and you say, “Oh, my god. The game has changed.” In the case of sensors, I remember in 2010 or so, sitting down with the CTO of Intel, and I asked him, “Can you tell me when the average sensor is, for example, at a parking meter — that might sense a car being at the meter — the average sensor is gonna drop below about 10 cents per unit?” And he said, “Yeah, that’ll be around 2013, 2015.” He knew pretty well exactly, and so I thought, that’s gonna be a game-changer, because we will now know what’s happening everywhere. What I didn’t see at the time was that the ubiquity of sensors would bring in big data. Some of us saw that in advance, but the big data — [we] didn’t see [it] would bring in all these smart algorithms. <Right.> And so, it’s the combination. Is there a way to see these new things coming along? Yeah. If you’re waiting for them.

Sonal: This reminds me of a story that Alvy Ray Smith tells. He’s a PARC alum as well. He co-founded Pixar back in the day. And he did a piece for me at “WIRED” about how they knew very early on — they had John Lasseter, they had this creative vision — they knew very early on the kinds of things that they wanted to do. And they later mapped out, like, a trajectory of their movies based on Moore’s law, but it was, like, a tool for them. So they saw it, but yeah, they had to wait.

Prof. Arthur: But usually, it’s hard to see. The best I can hope for, at least in my own case, is that within two or three years, you just go, “Oh, the game has changed.” And when the game changes, you realize you’re in a slightly different era. And when you’re in that era, you realize that it’s not gonna last. That in 10 years’ time, 20 years’ time, or 30 years’ time, there’ll be a different version. I wanna make this comment very quickly. I’ve been physically in Silicon Valley — if you count Berkeley, I’ve been in the…

Sonal: I think we should count Berkeley.

Prof. Arthur: Yeah. Okay. I’ve been in the Bay Area now for very close to 50 years. I was a grad student in Berkeley. And then, in Stanford, I’ve been here since 1982. And in all that time, when the game gets a little tired at times, people say, “Oh, the Valley is over,” but it doesn’t. It discovers new technologies and then reinvents itself. That’s the way capitalism works here in Silicon Valley, but in other countries, where it’s more planned, it may have stopped, and places like that can come to a halt as a result.

Sonal: That’s the perfect note to end on. I’m gonna quote a piece from one of your middle-early papers. You talk about whether there’s any hope in complexity, essentially, and you say, “It shows us an economy perpetually inventing itself, perpetually creating possibilities for exploitation, perpetually open to a response. An economy that is not dead, static, timeless, and perfect, but one that is alive, ever-changing, organic, and full of messy vitality.”

Prof. Arthur: It’s not a coincidence that I wrote that, because that’s the way Silicon Valley operates. Inventing and reinventing itself, and morphing and changing, in a way you can’t quite predict, and in a way that I think is delightfully messy, but ordered at the same time.

Sonal: Fabulous. A messy ordered vitality. Brian Arthur, thank you for joining the “a16z Podcast.”

Marc: Yeah. Thank you, Brian. That was really tremendous.

Prof. Arthur: And thank you very much for having me. I’m delighted. Thank you.

Sonal: Thank you.

• 

  W. Brian Arthur
• 

  Marc Andreessen is a cofounder and general partner at a16z. Marc co-created the highly influential Mosaic internet browser and cofounded Netscape.

  Follow Twitter
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

From the old claim that “IT doesn’t matter” and question of whether tech truly drives organizational performance, we’ve been consumed with figuring out how to measure — and predict — the output and outcomes, the performance and productivity of software. It’s not useful to talk about what happens in one isolated team or successful company; we need to be able to make it happen at any company — of any size, industry vertical, or architecture/tech stack. But can we break the false dichotomy of performance vs. speed; is it possible to have it all?

This episode of the a16z Podcast boldly goes where no man has gone before — trying to answer those elusive questions — by drawing on one of the largest, large-scale studies of software and organizational performance out there, as presented in the new book, Accelerate: The Science of Lean Software and DevOps — Building and Scaling High Performing Technology Organizations by Nicole Forsgren, Jez Humble, and Gene Kim. Forsgren (co-founder and CEO at DevOps Research and Assessment – DORA; PhD in Management Information Systems; formerly at IBM) and Humble (co-founder and CTO at DORA; formerly at 18F; and co-author of The DevOps Handbook, Lean Enterprise, and Continuous Delivery) share the latest findings about what drives performance in companies of all kinds.

But what is DevOps, really? And beyond the definitions and history, where does DevOps fit into the broader history and landscape of other tech movements (such as lean manufacturing, agile development, lean startups, microservices)? Finally, what kinds of companies are truly receptive to change, beyond so-called organizational “maturity” scores? And for pete’s sake, can we figure out how to measure software productivity already?? All this and more in this episode!

Show Notes

• Where DevOps originated historically [1:58] and why it’s important for businesses today [5:45]
• The practicality of analyzing DevOps [7:42] and the results of research to determine DevOps best practices [11:50]
• Discussion around ideal DevOps structures and differences across company types [16:34]
• What the data show about measuring and improving productivity [20:37]
• How companies determine whether they’re ready to implement recommended changes [30:24] and the importance of culture and leadership [36:47]

Transcript

Hi, everyone, welcome to the “a16z Podcast.” I’m Sonal. So, one of our favorite themes to talk about in this podcast is how software changes organizations, and the nature of the firm. And today’s episode takes a different angle on the topic by drawing on the research of one of the largest large-scale studies of software and organizational performance out there. From the authors of the new book “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations,” by Nicole Forsgren, Jez Humble, and Gene Kim.

Joining us to have this conversation, we have Nicole, who did her Ph.D research trying to answer the elusive eternal questions around how to measure this, especially given past debates around “does IT matter?” She’s now the CEO of DORA, DevOps Research and Assessment, which also puts out, with Puppet, the annual State of DevOps Report.

And then we have Jez Humble, who is CTO at DORA, and is also the co-author of the books, “The DevOps Handbook,” “Lean Enterprise,” and “Continuous Delivery.” They share the latest findings about high-performing companies of all kinds, even those who may not think they’re tech companies, and answer whether there’s an ideal organizational type in size, architecture, culture, or people that lends itself to success here. But we began this episode by briefly talking about the history of DevOps, and where it fits in the broader landscape of related software movements.

Dr. Forsgren: So, I started as a software engineer at IBM. I did hardware and software performance. And then I took a bit of a detour into academia, because I wanted to understand how to really measure and look at performance that would be generalizable to several teams in predictable ways, and in predictive ways. And so, I was looking at and investigating how to develop and deliver software in ways that were impactful to individuals, teams, and organizations. And then I pivoted back into industry, because I realized this movement had gained so much momentum and so much traction. And industry was just desperate to really understand — what types of things are really driving performance outcomes and excellence?

Sonal: And what do you mean by this movement?

Historical background

Dr. Forsgren: This movement that now we call DevOps — so the ability to leverage software, to deliver value to customers, to organizations, to stakeholders.

Jez: And I think, from a historical point of view, the best way to think about DevOps, it’s a bunch of people who had to solve this problem of how do we build large distributed systems that were secure, and scalable — and be able to change them really rapidly and evolve them? And no one had that problem before, certainly at the scale of companies like Amazon, and Google. And that really is where the DevOps movement came from, trying to solve that problem. And you can make an analogy to what Agile was about, since the kind of software crisis of the 1960s, and people trying to build these defense systems at large scale — the invention of software engineering, as a field, Margaret Hamilton, her work at MIT on the Apollo program. What happened in the decades after that was, everything became kind of encased in concrete in these very complex processes: “This is how you develop software.” And Agile was kind of a reaction to that, saying we can develop software much more quickly with much smaller teams in a much more lightweight way.

Sonal: So, we didn’t call it DevOps back then but it’s also more agile. Can you guys break down the taxonomy for a moment? Because when I think of DevOps, I think of it in the context of the containerization of code, and virtualization. I think of it in the context of microservices, being able to do modular teams around different things. There’s an organizational element, there’s a software element, there’s an infrastructure component. Like, paint the big picture for me of those building blocks and how they all kind of fit together.

Jez: Well, I can give you a very personal story, which was my first job after college was in 2000, in London, working at a startup where I was one of two technical people in the startup. And I would deploy to production, by FTP and code from my laptop going into production. <yeah> And if I wanted to roll back, I’d say, “Hey, Johnny, can you FTP your copy of this file to production,” and that was our rollback process. And then I went to work in consultancy where we were on these huge teams and deploying to production, there was a whole team with a Gantt chart, which put together the plan to deploy to production. And I’m like, “This is crazy.” Unfortunately, I was working with a bunch of other people who also thought it was crazy.

And we came up with these ideas around deployment automation, and scripting, and stuff like that. And suddenly, we saw the same ideas have popped up everywhere, basically. I mean, it’s realizing that if you’re working in a large, complex organization, Agile is gonna hit a brick wall. Because, unlike the things we were building in the ’60s, product development means that things are changing and evolving all the time. So it’s not good enough to get to production the first time, you’ve got to be able to keep getting there, on and on. And that really is where DevOps comes in. So it’s like, with Agile we’ve got a way to build novel products, but how do we keep deploying to production, and running the systems in production in a stable, reliable way, particularly in a distributed context?

Dr. Forsgren: So, if I phrase it another way, sometimes there’s a joke that says, “Day one is short and day two is long.”

Sonal: What does that mean?

Dr. Forsgren: Right. So day one is when we, like, create all this…

Sonal: By the way, it’s sad that you have to explain the joke to me. <crosstalk> But you do actually have to explain that joke to me. <laughter>

Dr. Forsgren: No, which is great, though, because — so day one is when we create all of these systems, and day two is when we deploy to production. We have to deploy and maintain forever and ever and ever and ever.

Sonal: So day two is an infinite day.

Dr. Forsgren: Right. Exactly. And…

Jez: For successful products.

Dr. Forsgren: Hopefully. We hope that day two is really, really long. And we’re fond of saying Agile doesn’t scale. And sometimes I’ll say this and people shoot laser beams out of their eyes. But when we think about it, Agile was meant for development, just like Jez said, it speeds up development. But then you have to hand it over, and especially infrastructure, and IT operations — what happens when we get there? So DevOps was sort of born out of this movement, and it was originally called Agile System Administration. And so then DevOps sort of came out of development and operations. And it’s not just dev and ops, but if we think about it, that’s sort of the bookends of this entire process.

Sonal: Well, it’s actually like day one and day two combined into one phrase.

Dr. Forsgren: Day one and day two.

Why DevOps is essential

Sonal: The way I think about this is I remember the stories of, like, Microsoft and the early days, and the waterfall cascading model of development. Leslie Lamport once wrote a piece for me about why software should be developed like houses, because you need a blueprint. And I’m not a software developer, but it felt like a very kind of old way of looking at the world of code.

Jez: I hate that metaphor.

Sonal: Tell me why.

Jez: If the thing you’re building has well-understood characteristics, it makes sense. So if you’re building a truss bridge, for example, there’s well-known understood models of building truss bridges. You plug the parameters into the model, and then you get a truss bridge and it stays up. Have you been to Sagrada Familia in Barcelona?

Sonal: Oh, I love Gaudi. Yeah.

Jez: Okay, so if you go into the crypt of the Sagrada Familia, you’ll see his workshop. And there’s a picture where, in fact, a model that he built of the Sagrada Familia, but upside down with the weight simulating the stresses. And so he would build all these prototypes and small prototypes because he was fundamentally designing a new way of building. <Oh, right.> All Gaudi’s designs were hyperbolic curves and parabolic curves, and no one had used that before.

Sonal: Things that had never been pressure tested, literally, in that case.

Jez: Exactly. He didn’t want them to fall down. So he built all these prototypes and did all this stuff.

Sonal: He built his blueprint as he went by building and trying it out, which is a very rapid prototyping kind of model.

Jez: Absolutely. So, in the situation where the thing you’re building has known characteristics, and it’s been done before, yeah, sure, we can take a very phased approach to it. And you know, for designing these kinds of protocols that have to work in a distributed context, and you can actually do formal proofs of them, again, that makes sense. But when we’re building products and services where, particularly, we don’t know what customers actually want, and what users actually want, it doesn’t make sense to do that. Because you’ll build something that no one wants, you can’t predict.

Dr. Forsgren: And we’re particularly bad at that, by the way. Even companies like Microsoft, where they are very good at understanding what their customer base looks like. They have a very mature product line. Ronny Kohavi has done studies there and only about 1/3 of the well-designed features deliver value.

The practicalities of DevOps analysis

Sonal: That’s actually a really important point. The mere question of — does this work? — is something that people really clearly don’t pause to ask. But I do have a question for you guys, a push back, which is — is this a little bit of the cult, “Oh, my God, it’s like so developer-centric, let’s be agile, let’s do it fast, our way — you know, two pizzas, that’s the ideal size of a software team.” And, you know, I’m not trying to mock it. I’m just saying that, isn’t there an element of actual practical realities, like technical debt and accruing a mess, underneath all your code? And a system that, you may be there for two or three years and you can go off to the next startup, but okay, someone else has to clean up your mess. Tell me about how this fits into that big picture.

Dr. Forsgren: This is what enables all of that.

Sonal: Oh, right. Interesting. So it’s not actually just creating a problem because that’s how I’m kind of hearing it.

Dr. Forsgren: No, absolutely. So you still need development, you still need tests, you still need QA, you still need operations, you still need to deal with technical debt, you still need to deal with re-architecting really difficult large monolithic code bases. What this enables you to do is to find the problems, address them quickly, move forward…

Jez: I think that the problem that a lot of people have is that we’re so used to couching these things as trade-offs and as dichotomies, the idea that if you’re gonna move fast, you’re gonna break things. The one thing, which I always say is if, you know, if you take one thing away from DevOps is this, high-performing companies don’t make those trade-offs. They’re not going fast and breaking things, they’re going fast and making more stable, more high-quality systems. And this is one of the key results in the book, in our research, is this fact that high performers do better at everything, because the capabilities that enable high performance in one field, if done right, enable it in other fields.

So, if you’re using version control for software, you should also be using version control for your production infrastructure. If there’s a problem in production, we can reproduce the state of the production environment in a disaster recovery scenario — again, in a predictable way — that’s repeatable. I think it’s important to point out that this is something that happened in manufacturing as well.

Sonal: Give it to me. I love when people talk about software as drawn from hardware analogies. That’s my favorite type of metaphor.

Jez: Okay. So I’ve got — so Toyota didn’t win by making shitty cars faster. They won by making higher-quality cars faster and having a shorter time to market.

Sonal: The Lean manufacturing method, which by the way, also spawned Lean startup thinking and everything else connected to it.

Dr. Forsgren: And DevOps pulls very strongly from Lean methodologies.

Research methods and findings

Sonal: So, you guys are probably the only people to have actually done a large-scale study of organizations adopting DevOps? What is your research and what did you find?

Dr. Forsgren: Sure, so the research really is the largest investigation of DevOps practices around the world. We have over 23,000 data points, all industries…

Sonal: Give me, like, a sampling — like, what are the range of industries?

Dr. Forsgren: So I’ve got entertainment, I’ve got finance, I have healthcare and pharma, I have technology…

Jez: Government.

Dr. Forsgren: Government, education.

Sonal: You guys have every vertical. And then when you take it around the world…

Dr. Forsgren: So we’re primarily in North America, we’re in AMEA, we have India, we have a small sample in Africa…

Sonal: Right. Just to quickly break down like the survey methodology, questions that people have. In the ethnographic world, the way we would approach it is that you can never trust what people say they do, you have to watch what they do. However, it is absolutely true, and especially in a more scalable sense, that there are really smart surveys that give you a shit ton of useful data.

Dr. Forsgren: Yes. And part two of the book covers this in almost excruciating detail.

Sonal: We like knowing methodologies, that’s nice to share that.

Dr. Forsgren: Yes. Well, and it’s interesting, because Jez has talked about in his overview of Agile and how it changes so quickly, and we don’t have a really good definition. What that does is it makes it difficult to measure, right? And so, what we do is we’ve defined core constructs, core capabilities, so that we can then measure them. We go back to core ideas around things like automation, process, measurement, Lean principles. And then I’ll get that pilot set of data and I’ll run preliminary statistics to test for discriminant validity, convergent validity, composite reliability — make sure that it’s not testing what it’s not supposed to test. It is testing what it is supposed to test. Everyone is reading it consistently the same way that I think it’s testing, like, even run checks to make sure that I’m not inadvertently inserting bias or collecting bias, just because I’m getting all of my data from surveys.

Sonal: Sounds pretty damn robust. So tell me, then, what were the big findings? That’s a huge question but give me the hit list.

Dr. Forsgren: Well, okay, so let’s start with one thing that Jez already talked about: speed and stability go together.

Sonal: This is where he was talking about there not being, necessarily, a false dichotomy. And that’s one of your findings, that you can actually accomplish both.

Jez: Yeah. And it’s worth talking about how we measure those things as well. So we measure speed or tempo, as we call it in the book, or sometimes people call it throughput as well…

Sonal: Which is a nice full-circle manufacturing idea, like the semiconductor, like, circuit throughput.

Jez: Yeah, absolutely.

Sonal: I love hardware analogies for software, I told you.

Jez: A lot of it comes from Lean. So lead time, obviously one of the classic Lean manufacturing measures we use, how long does it take — we look at the lead time from checking into version control to release into production. <Right.> So that part of the value stream because that’s more focused on the DevOps end of things.

Dr. Forsgren: And it’s highly predictable.

Jez: The other one is release frequency. So how often do you do it? And then we’ve got two stability metrics, and one of them is time to restore. So in the event that you have some kind of outage, or some degradation in performance in production, how long does it take you to restore service. For a long time, we focused on not letting things break. And I think one of the changes, paradigm shifts, we’ve seen in the industry, particularly in DevOps is moving away from that. We accept that failure is inevitable because we’re building complex systems. So, not how do we prevent failure, but when failure inevitably occurs, how quickly can we detect and fix it?

Dr. Forsgren: MTBF, right? Mean time between failures. If you only go down once a year, but you’re down for three days, and it’s on Black Friday — but if you’re down, very small, low blast, very, very small blast radius, and you can come back almost immediately and your customers almost don’t notice? That’s fine.

Jez: The other piece around stability is, change fail rate. When you push a change into production, what percentage of the time do you have to fix it because something went wrong.

Sonal: By the way, what does that tell you, if you have a change fail?

Jez: So, in the Lean kind of discipline, this is called percent complete and accurate. And it’s a measure of a quality of your process. So in a high-quality process, when I do something for Nicole, Nicole can use it, rather than sending it back to me and say, “Hey, there’s a problem with this.” And in this particular case, what percentage of the time, when I deploy something to production, is there a problem because I didn’t test adequately, my testing environment wasn’t my production-like enough.

Sonal: Those are the measures for finding this. But the big finding is that you can have speed and stability together through DevOps. Is that what I’m hearing?

Dr. Forsgren: Yes, high performers get it all, low performers kind of suck at all of it, medium performers hang out in the middle. I’m not seeing trade-offs four years in a row — so anyone who’s thinking, “Oh, I can be more stable if I slow down,” I don’t see it.

Sonal: It actually breaks a very commonly held kind of an urban legend around how people believe these things operate. So tell me, are there any other sort of findings like that because that’s very counterintuitive.

Dr. Forsgren: Okay. So this one’s kind of fun. One is that this ability to develop and deliver software with speed and stability drives organizational performance. Now, here’s the thing…

Sonal: I was about to say, that’s a very obvious thing to say.

Dr. Forsgren: So, it seems obvious, right? Developing and delivering software with speed and stability drives things like profitability, productivity, market share. Okay, except, if we go back to Harvard Business Review, 2003, there’s a paper titled, “IT Doesn’t Matter.” We have decades of research — I wanna say at least 30 or 40 years of research — showing the technology does not drive organizational performance. It doesn’t drive ROI. And we are now starting to find other studies and other research that backs this up. Erik Brynjolfsson out of MIT, James Bessen out of Boston University, 2017.

Sonal: Did you say, James Bessen?

Dr. Forsgren: Yeah.

Sonal: Oh, I used to edit him too.

Dr. Forsgren: Yeah, it’s fantastic. Here’s why it’s different. Because before, right, in like the 80s, and the 90s, we did this thing where, like, you’d buy the tech and you’d plug it in, and you’d walk away.

Sonal: It was an on-prem sales model where you, like, deliver and leave, as opposed to software as a service in other ways that things… 

Dr. Forsgren: Yep.

Jez: And people complain if you try to upgrade too often.

Sonal: Oh, right. <laughter>

Dr. Forsgren: The key is that everyone else can also buy the thing and plug it in and walk away. How is that driving value or differentiation for a company? If I just buy a laptop to help me do something faster, everyone else can buy a laptop to do the same thing faster. That doesn’t help me deliver value to my customers or to the market. It’s a point of parity not a point of distinction.

Sonal: Right. And you’re saying that point of distinction comes from how you tie together that technology process and culture through DevOps.

Jez: Right. And it can provide a competitive advantage to your business. If you’re buying something that everyone else also has access to, then it’s no longer a differentiator. But if you have an in-house capability, and those people are finding ways to drive your business — I mean, this is the classic Amazon model, they’re running hundreds of experiments in production at any one time to improve their product. And that’s not something that anyone else can copy. That’s why Amazon keeps winning. So what people are doing is copying the capability instead. And that’s what we’re talking about. How do you build that capability?

Organizational types

Sonal: The most fascinating thing to me about all this is honestly not the technology, per se, but the organizational change part of it, and the organizations themselves. So of all the people you studied, is there an ideal organizational makeup that is ideal for DevOps? Or is it one of these magical formulas that has this ability to turn a big company into a startup and a small company into — because that’s actually the real question.

Dr. Forsgren: From what I’ve seen, there might be two ideals. The nice happy answer is the ideal organization is the one that wants to change.

Sonal: I mean, given this huge n=23,000 data set, is it not tied to a particular profile of a size of company? They’re both shaking their heads, just for the listeners.

Dr. Forsgren: I see high performers among large companies. I see high performers in small companies. I see low performers in small companies. I see low performers in highly regulated companies. I see low performers in non-regulated companies…

Sonal: So, tell me the answer you’re not supposed to say.

Dr. Forsgren: So, that answer is, it tends to be companies that are like, “Oh, shit.” And there are two profiles — either, one, they’re like way behind, and “Oh, shit,” and they have some kind of funds. Or, they are, like, this lovely, wonderful bastion of like — they’re these really innovative high performing companies. But they still realize they’re a handful of like two or three companies ahead of them. And they don’t wanna be number two, they are gonna be number one.

Sonal: So, those are the ideal — I mean, just to, like, anthropomorphize it a little bit. It’s like the 35- to 40-year-old who suddenly discovers you might be pre-diabetic, so you better do something about it now before it’s too late. But it’s not too late, because you’re not so old, where you’re about to reach sort of the end of a possibility to change that runway. And then there’s this person who’s, sort of, kind of already like in the game running in the race, and there might be two or three but they want to be like, number one.

Jez: And I think to extend your metaphor, the companies that do well are the companies that never got diabetic in the first place because they always just ate healthily like…

Sonal: They were already glucose monitoring. They had continuous glucose monitors on, which is like DevOps, actually.

Dr. Forsgren: They were always athletes.

Jez: Right. You know, diets are terrible because at some point, you have to stop the diet.

Sonal: And it has a sudden start and stop, as opposed to a way of life, is what you’re saying.

Jez: Right, exactly. So, if you just always eat healthily and never eat too much, or very rarely eat too much and do a bit of exercise every day, you never get to the stage where, “Oh, my God, now I can only eat tofu.”

Dr. Forsgren: Yeah, totally. So, like, my “loving professorness nurture Nicole” also has one more profile that, like, I love, and I worry about them, like mother hen. And it’s the companies that I talk to, and they come to me and they’re struggling. And I haven’t decided if they wanna change, but they’re like, “So we need to do this transformation. And we’re gonna do the transformation.” And it’s either because they want to, or they’ve been told that they need to. And then they will insert this thing where they say, “But I’m not a technology company.” I’m like, but we just had this 20-minute conversation about how you’re leveraging technology to drive value to customers, or to drive this massive process that you do. <Yeah.> And then they say, “But I’m not a technology company.” I could almost see why they had that in their head because they were a natural resources company. But there was another one where they were a finance company.

Sonal: I mean, an extension of “software eats the world” is really every company is a technology company. It’s fascinating to me that that third type exists, but it is a sign of this legacy world moving into software.

Dr. Forsgren: Right. And I worry about them. Also, at least for me, personally, you know. I’ve lived through this, like, mass extinction of several firms, and I don’t want it to happen again. <Yeah.> And I worry about so many companies that keep insisting they’re not technology companies, and I’m like, “Oh, honey child…”

Sonal: You’re a tech company.

Jez: You know, one of the gaps in our data is actually China. And I think China is a really interesting example because they didn’t go through the whole, you know, “IT doesn’t matter” phase, they’re jumping straight from no technology to Alibaba and Tencent, right? I think U.S. companies should be scared, because at the moment, Tencent and Alibaba are already moving into other developing markets, and they are gonna be incredibly competitive because it’s just built into their DNA.

Productivity and performance

Sonal: So, the other fascinating thing to me is that you, essentially, were able to measure performance of software, and clearly productivity. [Are] there any more insights on the productivity side?

Jez: Yes, yes, I wanna go.

Dr. Forsgren: This is his favorite rant.

Sonal: For everybody else, he’s like jumping around and, like, waving his hands. So tell us!

Jez: The reason the manufacturing metaphor breaks down is because in manufacturing, you have inventory.

Sonal: Yes.

Jez: We do not have inventory in the same way in software. In a factory, like the first thing your Lean consultant is gonna do, walking into the factory is point to the piles of thing everywhere. But I think if you walk into an office where there’s developers, where’s the inventory?

Dr. Forsgren: By the way, that’s what makes talking about this to executives so difficult — they can’t see the process.

Jez: Well, it’s a hard question to answer, because is the inventory the code that’s being written? And people actually have done that and said, “Well, listen, lines of code are an accounting measure and we’re going to capture that as, you know, capital, which we’re gonna…

Sonal: That’s insane, it seems like an invitation to write crappy, unnecessarily long code.

Jez: That’s exactly what happens and then…

Sonal: It’s like the olden days of getting paid for a book by how long it is. And it’s like, actually really boring when you can actually write it in like one third of the length.

Dr. Forsgren: Let’s write it in German.

Jez: Right, you know…

Sonal: <laughter> I was thinking of Charles Dickens.

Jez: In general, you know, you prefer people to write short programs, because they’re easier to maintain, and so forth. But lines of code have all these drawbacks, we can’t use them as a measure of productivity.

Sonal: So if you can’t measure lines of code, what can you measure? Because I really want an answer. Like how do you measure productivity?

Jez: So velocity is the other classic example. Agile, there’s this concept of velocity, which is the number of story points a team manages to complete in an iteration. So before the start of an iteration, in many Agile — particularly Scrum-based processes, you’ve got all this work to do, like, we need to build these five features, how long will this feature take? And the developers fight over it, and they’re like, “Oh, it’s five points.” And then this one’s gonna take three points, this one’s gonna take two points. And so you have a list of all these features, you don’t get through all of them. But at the end of the iteration, the customer signs off, “Well, I’m accepting this one, this one’s fine, this one’s fine, this one’s a hot mess, go back and do it again,” whatever. The number of points you complete in the iteration is the velocity.

Sonal: So it’s like the speed at which you’re able to deliver those features.

Jez: So, a lot of people treat it like that. But actually that’s not really what it’s about. It’s a relative measure of effort. And it’s for capacity planning purposes. So you basically, for the next iteration, will only commit to completing the same velocity that we finished last time, so it’s relative, and it’s team dependent. And so what a lot of people do is they start comparing velocities across teams. Then what happens is, a lot of work you need to collaborate between teams. But hey, if I’m gonna help you with your story, that means I’m not gonna get my story points and you’re gonna get your story points.

Sonal: So, it’s, like, bad incentive structure, right?

Jez: Right, people can game it as well. You should never use story points as a productivity measurement.

Sonal: So, lines of code doesn’t work, velocity doesn’t work — what works?

Jez: So, this is why we like two things, in particular — one thing that it’s a global measure, and secondly, that it’s not just one thing, it mixes two things together, which might normally be in tension. And so this is why we went for our measure of performance. So measuring lead time, release frequency, and then time for a store and change fail rate. Lead time is really interesting because lead time is on the way to production, right? So all the teams have to collaborate — it’s not something where, you know, I can go really fast in my velocity, but nothing ever gets delivered to the customer. That doesn’t count in lead time, so it’s a global measure.

Sonal: So, it takes care of that problem of the incentive alignment around the competitive dynamic.

Jez: Exactly.

Dr. Forsgren: Also, it’s an outcome, it’s not an output.

Jez: There’s a guy called Jeff Patton. He’s a really smart thinker in the kind of Lean-Agile space. He says, “Minimize output, maximize outcomes,” which I think is simple but brilliant.

Sonal: It’s so simple because it just shifts the words to impact.

Jez: And even we don’t get all the way there because we’re not yet measuring, did the features deliver the expected value to the organization to the customers?

Dr. Forsgren: Well, we do get there because we focus on speed and stability, <Mmhmm> which then deliver the outcome to the organization, profitability, productivity, market share.

Sonal: But the second half of this, which I am also hearing is, did it meet your expectations? Did it perform to the level that you wanted it to? Did it match what you asked for? Or even if it wasn’t something you specified that you desired or needed? That seems like a slightly open question?

Jez: So, we did actually measure that. We looked at nonprofit organizations, and these were exactly the questions we measured. We asked people, did the software meet — I can’t remember what the exact questions were.

Dr. Forsgren: Effectiveness, efficiency, customer satisfaction, delivering mission goals…

Sonal: How fascinating that you do it nonprofits, because that is a larger movement along nonprofit measurement space, to try to measure impact?

Dr. Forsgren: But we captured it everywhere <Yeah.> because even profit-seeking firms still have these goals.

Jez: Yeah, in fact, as we know, from research, companies that don’t have a mission, other than making money, do less well than the ones that do. But I think, again, what the data shows is that companies that do well on the performance measures we talked about outperform their low-performing peers by a factor of two. Our hypothesis is, what we’re doing when we create these high-performing organizations, in terms of speed and stability, is we’re creating feedback loops. What it allows us to do is build a thin slice, a prototype of a feature, get feedback through some UX mechanism, whether that’s showing people with a prototype and getting their feedback, whether it’s running A/B tests or multivariate tests in production. It’s what creates these feedback loops that allow you to shift direction very fast.

Sonal: I mean, that is at the heart of Lean startup, it’s the heart of anything you’re putting out into the world — is you have to kind of bring it full circle. It is a secret of success to Amazon, as you cited earlier. I would distill it to just that. I think I heard Jeff Bezos say the best line. It was at the Internet Association dinner in DC last year, where someone asked about an innovation. He’s like, to him an innovation is something that people actually use. <Right.> And that’s what I love about the feedback loop thing is it actually reinforces that mindset of that’s what innovation is.

Jez: Right. So to sum up, the way you can frame this is, DevOps is that technological capability that underpins your ability to practice Lean startup and all these very rapid iterative processes.

Sonal: So I have a couple of questions then. So one is, you know, going back to this original taxonomy question, and you guys describe that there isn’t necessarily an ideal organizational type…

Dr. Forsgren: Which, by the way, should be encouraging.

Sonal: I agree! I think it’s super encouraging and more importantly, democratizing, that anybody can become a hit player.

Jez: We were doing this in the federal government.

Sonal: I love that. But one of my questions is, when — we had Adrian Cockcroft, on this podcast a couple of years ago, talking about microservices. And the thing that I thought was so liberating about what he was describing the Netflix story, was that it was a way for teams to essentially become little mini product management units, and essentially self-organize. Because the infrastructure, by being broken down into these micro pieces, versus, say, a monolithic kind of uniform architecture — I would think that being a, you know, organization that’s containerized its code in that way, that has this microservices architecture, would be more suited to DevOps — or is that a wrong belief? I’m just trying to understand, again, that taxonomy thing of how these pieces all fit together.

Jez: So we actually studied this. There’s a whole section on architecture in the book <Oh, great!> where we looked at exactly this question. Architecture has been studied for a long time, and people talk about architectural characteristics. There’s the ATAM, the architectural trade-off model that Carnegie Mellon developed. There’s some additional things we have to care about — testability and deployability. Can my team test its stuff without having to rely on this very complex integrated environment? Can my team deploy its code to production without these very complex orchestrated deployments? Basically, can we do things without dependencies? That is one of the biggest predictors, in our cohort of IT performance, is the ability of teams to get stuff done on their own without dependencies on other teams, whether that’s testing or whether it’s deploying or is planning…

Dr. Forsgren: Or, even just communicating. <Yeah.> Can you get things done without having to do, like, mass communication and checking and permissions.

Sonal: Question I love love love asking on this podcast is, we always revisit the 1937 Coase paper about the theory of the firm <Yes!> and this idea that transaction costs are more efficient. And this is, like, the ultimate model for reducing friction <Yeah.> and those transaction costs, communication coordination costs, all of it.

Jez: That’s what — like all the technical and process stuff is about that. I mean, Don Reinertsen, once came to one of my talks on continuous delivery, at the end, he said, “So continuous delivery, that’s just about reducing transaction costs, right?” And I’m like…

Sonal: Huh, an economist’s view of DevOps. I love it.

Jez: You’re right. You reduced my entire body of words to one sentence.

Dr. Forsgren: It’s so much Conway’s Law, right?

Sonal: Remind me what Conway’s Law is.

Dr. Forsgren: So organizations which design systems are constrained to produce designs, which are copies of the communication structures of these organizations.

Sonal: Oh, right, it’s that idea, basically, that your software code looks like the shape of the organization itself.

Jez: Right.

Dr. Forsgren: And how we communicate, right? So which, you know, Jez just summarized, if you have to be communicating and coordinating with all of these other different groups…

Sonal: Command and control looks like waterfall, a more decentralized model looks like independent teams.

Jez: Right. So the data shows that. One thing that I would say is, a lot of people jump on the microservices containerization bandwagon, there’s one thing that is very important to bear in mind. Implementing those technologies does not give you those outcomes we talked about. We actually looked at people doing mainframe stuff — you can achieve these results with mainframes. Equally, you can use, you know, Kubernetes and, you know, Docker and microservices and not achieve these outcomes.

Dr. Forsgren: We see no statistical correlation with performance, whether you’re on a mainframe, or a Greenfield or Brownfield system, if you’re building something brand new, or if you’re working on an existing build. And one thing I wanted to bring up that we didn’t before, is I said, you know, day one is short, day two is long. And I talked about things that live on the internet and live on the web. <Yeah.> This is still a really, really smart approach for packaged software. And I know people who are working in and running packaged software companies that use this methodology because it allows them to still work in small, fast approaches. And all they do is they push to a small package, pre-production database, and then when it’s time to push that code onto some media, they do that.

Determining readiness to change

Sonal: Okay. So what I love hearing about this is that it’s actually not necessarily tied, again, to the architecture or the type of company you are, that there’s opportunity for everybody. But there is this mindset of, like, an organization that is ready, it’s like a readiness level for a company.

Dr. Forsgren: Oh, I hear that all the time. I don’t know if I’d say there’s any such thing as readiness, right? Like, there’s always an opportunity to get better, there’s always an opportunity to transform. The other thing that really, like, drives me crazy and makes my head explode is this whole maturity model thing. Okay, are you ready to start transforming? Well, like, you can just not transform and then maybe fail, right? Maturity models, they’re really popular in industry right now, but I really can’t stress enough that they’re not really an appropriate way to think about technology transformation.

Sonal: I was thinking of readiness in a kind of, like, NASA Technology Readiness Levels, or TRLs, which is something we used to think about a lot for very early-stage things. But you’re describing maturity of an organization, and it sounds like there’s some kind of a framework for assessing the maturity of an organization. And you’re saying that doesn’t work. But first of all, what is that framework? And why doesn’t it work?

Dr. Forsgren: Well, so many people think that they want a snapshot of their like DevOps or their technology transformation, and spit back a number, right? And then you will have one number to compare yourself against everything. The challenge, though, is that a maturity model usually is leveraged to help you think about arriving somewhere. And then here’s the problem — once you’ve arrived, what happens?

Jez: Oh, we’re done.

Dr. Forsgren: You’re done! And then the resources are gone. And by resources, I don’t just mean money, I mean time, I mean attention. We see, year over year over year, the best, most innovative companies continue to push. So what happens when you’ve “arrived,” I’m using my finger quotes.

Sonal: You stop pushing.

Dr. Forsgren: You stop pushing. What happens when executives or leaders or whomever decide that you no longer need resources of any type?

Sonal: I have to push back again, though. Doesn’t this help — because it is helpful to give executives in particular, particularly those that are not tech native, coming from the seeds of the engineering organization, some kind of metric to put your head around. “Where are we, where are we at?”

Dr. Forsgren: So you can use a capability model. <Mmm.> You can think about the capabilities that are necessary to drive your ability to develop and deliver software with speed and stability. Another limitation is that they’re often kind of a lockstep or a linear formula, right?

Sonal: No, right. It’s like a stepwise A, B, C, D, E, 1, 2, 3, 4. And in fact, the very nature of anything iterative is its very nonlinear and circular. Feedback loops are circles.

Dr. Forsgren: Right. Maturity models just don’t allow that. Now another thing that’s really, really nice is that capability models allow us to think about capabilities in terms of these outcomes. Capabilities drive impact. Maturity models are just this thing where you have this level one, level two, level three, level four.

Sonal: That’s a bit performative.

Dr. Forsgren: And then, finally, maturity models just sort of take this snapshot of the world and describe it. How fast is technology and business changing? If we create a maturity model now, let’s wait — let’s say, four years — that maturity model is old and dead and dusty and gone.

Sonal: Do new technologies change the way you think about this? Because I’ve been thinking a lot about how product management for certain types of technologies changes with the technology itself, and that machine learning and deep learning might be a different beast. And I’m just wondering if you guys have any thoughts on that.

Jez: Yeah. I mean, me and Dave Farley wrote the “Continuous Delivery” book back in 2010. And since then, you know, there’s Docker and Kubernetes and large-scale launch from the cloud, and all these things that you had no idea would happen. People sometimes ask me, you know, isn’t it time you wrote a new edition of the book? I mean, yeah, we could probably rewrite it. Does it change any of the fundamental principles? <Mmhmm.> No. Do these new tools allow you to achieve those principles in new ways? Yes.

So I think, you know, this is how I always come back to any problem is, go back to first principles. The first principles, I mean, they will change over the course of centuries. I mean, we’ve got modern management versus kind of scientific management, but they don’t change over the course of, like, a couple of years. The principles are still the same, technologies give you new ways to do them, and that’s what’s interesting about them.

Equally, things can go backwards. A great example of this is — one of the capabilities we talk about in the book is working off a shared trunk, or master, in version control, not going on these long-lived feature branches. And the reason for that is actually because of feedback loops. You know, developers love going off into a corner, putting headphones on their head, <Yeah.> and coding something for, like, days. And then they try to integrate it into trunk, you know, and that’s a total nightmare. And not just for them, more critically, for everyone else, who then has to merge their code into whatever they’re working on. So that’s hugely painful. Git is one of these examples of a tool that makes it very easy for people, like, “Oh, I can use feature branches.” So I think, again, it’s nonlinear in the way that you describe, gives you new ways to do things. Are they good and bad? It depends.

Sonal: But the thing that strikes me about what you guys have been talking about, as a theme in this podcast, that seems to lend itself well to the world of machine learning and deep learning, where that technology might be different, is it sort of lends itself to a probabilistic way of thinking, and that things are not necessarily always complete. <Yes, absolutely.> And that there is not a beginning and an end, and that you can actually live very comfortably in an environment where things are, by nature, complex, and that complexity is not necessarily something to avoid. So in that sense, I do think there might be something kind of neat about ML and deep learning and AI, for that matter, because it is very much lending itself to that sort of mindset.

Jez: Yeah. And in our research, we talk about working in small batches. <Mmhmm.> There’s a great video by Bret Victor called “Inventing on Principle,” where he talks about how important it is to the creative process to be able to see what you’re doing. And he has this great demo of this game he’s building where he can change the code, and the game changes its behavior instantly. When you’re doing things like that…

Sonal: You don’t get to see that.

Jez: No. And the whole thing with machine learning is, how can we get the shortest possible feedback from changing the input parameters, to seeing the effects so that the machine can learn? And that the moment you have very long feedback loops, the ML becomes much much harder because you don’t know which of the input changes caused the change in output that the machine is supposed to be learning from. So the same thing is true of organizational change and process, and product development as well, by the way, which is working in small batches, so that you can actually reason about cause and effects. You know, I changed this thing, it had this effect. Again, that requires short feedback loops, that requires small batches. That’s one of the key capabilities we talk about in the book. And that’s what DevOps enables.

Getting started and company culture

Sonal: So we’ve been in this hallway-style conversation around all these themes of DevOps, measuring it, why it matters, and what it means for organizations. But practically speaking, if a company — and you guys are basically arguing it, any company — not necessarily a “company that thinks it’s a tech company,” and necessarily a company that has, like, this amazing, modern infrastructure stack — it could be a company that’s still working off mainframes. What should people actually do to get started, and how do they know where they are?

Dr. Forsgren: So what you need to do is, take a look at your capabilities, understand what’s holding you back, right? Try to figure out what your constraints are. <Mmhmm.> But the thing that I love about much of this is, you can start somewhere, and culture is such a core important piece. We’ve seen across so many industries, culture is truly transformative.

Jez: And in fact, we measure it in our work and we can show that culture has a predictive effect on organizational outcomes and on technology capabilities. We use a model from a guy called Ron Westrum, who was a social scientist studying safety outcomes, in fact, in safety-critical industries like healthcare and aviation. He created a typology where he organizes organizations based on whether they’re pathological, bureaucratic, or generative.

Sonal: That’s actually a great typology. I want to apply that to people I date. <laughter>

Dr. Forsgren: I know, right? Too real.

Sonal: I want to apply that to people. <laughter>

Jez: There’s a book in there, definitely.

Sonal: I like how I’m trying to anthropomorphize all these organizational things into people. But anyway, go on Jez.

Jez: Yeah, so instead of the five love languages, we can have the three relationship types. So pathological organizations are characterized by low cooperation between different departments and up and down the organizational hierarchy. How do we deal with people who bring us bad news? Do we ignore them, or do we shoot people who bring us bad news? How do we deal with responsibilities? Are they defined tightly so that when something goes wrong, we know whose fault it is so we punish them? Or do we share risks? Because we know we’re all in this together and it’s the team…

Sonal: You all have skin in the game, you’re all accountable, right.

Jez: Exactly. How do we do with bridging between different departments? And crucially, how do we deal with failure? As we discussed earlier, in any complex system, including organizational systems, failure is inevitable. So failure should be treated as a learning opportunity. Not whose fault was it, but why did that person not have the information they needed, the tools they needed? How can we make sure that when someone does something, it doesn’t lead to catastrophic outcomes, but instead, it leads to contained small blast radiuses?

Sonal: Right, not an outage on Black Friday.

Jez: Right, exactly. And then also, how do we deal with novelties? Is novelty crushed or is it implemented or does it lead to problems? One of the pieces of research that kind of confirms what we were talking about was some research that was done by Google. They were trying to find what makes the greatest Google team — you know, is it four Stanford graduates, and no developer, and fire all the managers…

Sonal: Right.

Dr. Forsgren: Is it a data scientist, an OGS programmer, and a manager…

Sonal: Right, one product manager paired with one system engineer, with one…

Dr. Forsgren: Yep.

Jez: And what they found was the number one ingredient was psychological safety. <Ah, interesting.> Does the team feel safe to take risks? And this ties together failure and novelty. If people don’t feel that when things go wrong, they’re gonna be supported, they’re not gonna take risks, and then you’re not gonna get any novelty because novelty, by definition, involves taking risks. So we see that one of the biggest things you can do is create teams where it’s safe to go wrong and make mistakes, and then where people will treat that as a learning experience. This is a principle that applies, again, not just in product development, you know, the Lean startup, “fail early, fail often,” but also in the way we deal with problems at an operational level as well.

Dr. Forsgren: And how we interact with our team when these things happen.

Sonal: To just to kind of summarize that, you have pathological, this is a power-oriented thing where, you know, the people are scared, the messenger is gonna be shot. Then you have this bureaucratic kind of rule-oriented world where the messengers aren’t heard. And then you have this sort of generative — and again, I really wish I could apply this to people, but we’re talking about organizations here — for culture, which is more performance-oriented.

Jez: And I just wanna add one thing about this, you know, working in the federal government, you would imagine that to be a very bureaucratic organization.

Sonal: I would, actually.

Jez: And actually, what was surprising to me was that, yes, there’s lots of rules — the rules aren’t necessarily bad, that’s how we can operate at scale, is by having rules. But what I found was, there [were] a lot of people who are mission-oriented. And I think that’s a nice alternative way to think about generative organizations is to think about mission orientation. The rules are there, but if it’s important to the mission, we’ll break the rules.

Dr. Forsgren: And we measure this at the team level, right? Because you can be in the government, and there were pockets that were very generative. <Right.> You can be in a startup <Yep.> and you can see startups that act very bureaucratic.

Jez: Or pathological, even.

Dr. Forsgren: Or very pathological.

Sonal: Right, and that’s like the people where…

Jez: The cult of the CEO…

Sonal: …where it’s not charismatic, inspirational vision, but to the expense of actually being heard and the messenger is shot, etc.

Dr. Forsgren: And we have several companies around the world now that are measuring their culture on a quarterly cadence basis, because we show in the book how to measure it. Westrum’s typology was the table itself, and so we turned that into a scientific, psychometric way to measure it.

Sonal: Now, this makes sense why I’m putting these anthropomorphic analogies because, in this sense, organizations are like people.

Dr. Forsgren: They’re made of people.

Sonal: Teams are organic entities. And I love that you said that the unit of analysis is a team because it means you can actually do something. You can start there and then you can like, see if it actually spreads or doesn’t spread, bridges, doesn’t bridge, etc. And what I also love about this framework is, it also moves away from this cult of failure mindset that I think people tend to have, where it’s like failing for the sake of failing. And you actually wanna avoid failure. <Right.> And the whole point of failing is to actually learn something and then be better and take risks so you can implement this new phase.

Dr. Forsgren: And very smart risks.

Final thoughts and recommendations

Sonal: So what’s your final — I mean, there’s a lot of really great things here, but, like, what’s your final sort of parting takeaway for listeners or people who might wanna get started or think about how they are doing?

Jez: So I think, you know, we’re in a world where technology matters. Anyone can do this stuff, but you have to get the technology part of it right. That means investing in your engineering capabilities, in your process, in your culture, in your architecture. We’ve dealt with a lot of things here that people think are intangible, and we’re here to tell you, they’re not intangible. You can measure them, they will impact the performance of your organization. So take a scientific approach to improving your organization and you will reap the dividends.

Sonal: When you guys talk about, you know, anyone can do this, the teams can do this — but what role in the organization is usually most empowered to be the owner of where to get started? Is it like the VP of engineering, is it the CTO, the CIO?

Dr. Forsgren: I was gonna say, don’t minimize the role of and the importance of leadership. DevOps sort of started as a grassroots movement. But right now we’re seeing roles like VP and CTO being really impactful, in part because they can set the vision for an organization but also in part because they have resources that they can dedicate to this.

Sonal: We see a lot of CEOs and CTOs and CIOs in our business. We have like a whole briefing center, we hear what’s top of mind for them all the time. Everyone thinks they’re transformational. So like, what actually makes a visionary type of leader who has that, not just the purse strings and the decision-making power, but the actual characteristics that are right for this?

Dr. Forsgren: Right. And that’s such a great question. And so we actually dug into that in our research, and we find that there are five characteristics that end up being predictive of driving change and really amplifying all of the other capabilities that we found. And these five characteristics are vision, intellectual stimulation, inspirational communication, supportive leadership, and personal recognition. And so, what we end up recommending to organizations is, absolutely invest in the technology. Also, invest in leadership in your people because that can really help drive your transformation home.

Sonal: Well, Nicole, Jez, thank you for joining the “a16z Podcast.” The book, just out, is “Accelerate: Building and Scaling High Performing Technology Organizations.” Thank you so much, you guys.

Jez: Thanks for having us.

Dr. Forsgren: Thank you.

• 

  Nicole Forsgren is the VP, Research & Strategy at GitHub. Author of the award-winning book “Accelerate: The Science of Lean Software and DevOps”, she was also lead investigator on the largest DevOps studies to date.

  Follow Twitter Website
• 

  Jez Humble
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

Alongside the excitement and hype about our growing reliance on artificial intelligence, there’s intense fear about the way the technology works. A 2017 MIT Technology Review article titled “The Dark Secret at the Heart of AI” warned, “No one really knows how the most advanced algorithms do what they do. That could be a problem.” Thanks to this uncertainty and lack of accountability, a report by the AI Now Institute at NYU recommended that public agencies responsible for criminal justice, health care, welfare, and education shouldn’t use such technology.

Given these types of concerns, the unseeable space between where data goes in and answers come out is often referred to as a “black box” — seemingly a reference to the hardy (and in fact orange, not black) data recorders mandated on aircraft and often examined after accidents. In the context of AI, the term more broadly suggests an image of being in the “dark” about how the technology works: We put in and provide the data and models and architectures, and then computers provide us answers while continuing to learn on their own, in a way that’s seemingly impossible — and certainly too complicated — for us to understand.

There’s particular concern about this in health care, where AI is used to classify which skin lesions are cancerous, to identify very early stage cancer from blood, to predict heart disease, to determine what compounds in people and animals could extend healthy life spans, and more. But these fears about the implications of black box are misplaced. AI is no less transparent than the way in which doctors have always worked — and in many cases it represents an improvement, augmenting what hospitals can then do for patients and the entire health care system. After all, the black box in AI isn’t a new problem due to new tech: Human intelligence itself is — and always has been — a black box.

Let’s take the example of a human doctor making a diagnosis. Afterward, a patient might ask that doctor how she made that diagnosis, and she would probably share some of the data she used to draw her conclusion. But could she really explain how and why she made that decision, what specific data from what studies she drew on, what observations from her training or mentors influenced her, what tacit knowledge she gleaned from her own and her colleagues’ shared experiences, and how all of this combined into that precise insight? Sure, she’d probably give us a few indicators about what pointed her in a certain direction — but there would also be an element of guessing, of following hunches. And even if there weren’t, we still wouldn’t know that there weren’t other factors involved, of which she wasn’t even consciously aware.

If the same diagnosis had been made with AI, we could draw from all available information on that particular patient — as well as data anonymously aggregated across time and from countless other relevant patients everywhere, in order to make the strongest evidence-based decision possible. It would be a diagnosis with a direct connection to the data, rather than human intuition based on limited data and derivative summaries of anecdotal experiences with a relatively small number of local patients.

But we make decisions in areas that we don’t fully understand every day — often very successfully — from the predicted economic impacts of policies and weather forecasts to how we conduct much of science in the first place. We either oversimplify things or accept that they’re too complex for us to break down linearly, let alone explain fully. It’s just like the black box of AI: human intelligence can reason and make arguments for a given conclusion, but it can’t explain the complex, underlying basis for how we arrived at a particular conclusion. Think of what happens when a couple gets divorced because of one stated cause — “infidelity” — when in reality there’s an entire unseen universe of intertwined causes, forces, and events that contributed to that outcome. Why did they choose to split up when another couple in a similar situation didn’t? Even those inside of it can’t fully explain it. It’s a black box.

The irony is that compared to human intelligence, AI is actually the more transparent of intelligences! Unlike the human mind, AI can — and should — be interrogated and interpreted. From the ability to audit and refine models and expose knowledge gaps in deep neural nets to the debugging tools that will inevitably be built and the potential ability to augment human intelligence via brain-computer interfaces, there are many technologies that could help interpret artificial intelligence in a way we can’t do in interpreting the human brain. In the process, we may even learn more about how human intelligence itself works.

Perhaps the real source of critics’ concerns isn’t that we can’t “see” AI’s reasoning — it’s that as AI gets more powerful, the human mind becomes the limiting factor. It’s that, in the future, we’ll basically need AI to understand AI. In health care as well as in other fields, this means we will soon see the creation of a new category of human professionals who don’t have to make the moment-to-moment decisions themselves, but instead manage a team of AI workers — just like commercial airplane pilots who engage autopilots to land in poor weather conditions. Doctors will no longer “drive” the primary diagnosis; instead, they’ll ensure that the diagnosis is relevant and meaningful for a patient, and oversee when and how to offer more clarification and more narrative explanations. The doctor’s office of the future will very likely include computer assistants, on both the doctor’s side and the patient’s side, as well as data inputs that come from far beyond the office walls.

When this happens, it will become clear that the so-called “black box” of AI is more of a feature, not a bug — because it’s more possible to capture and explain what’s going on there than it is in the human mind. None of this dismisses or ignores the need for AI oversight. It’s just that instead of worrying about the black box, we should focus on the opportunity — and therefore better address a future — where AI not only augments human intelligence and intuition, but perhaps even sheds light on and redefines what it means to be human in the first place.

This op-ed originally appeared in The New York Times. 

• 

  Vijay Pande is a general partner at a16z where he invests in biopharma and healthcare. Prior, he was a distinguished professor at Stanford. He is also the founder of Folding@Home Distributed Computing Project.

  Follow Twitter

There’s been a lot of talk about technology — and AI, deep learning, and machine learning specifically — finally reaching the healthcare sector. But AI in medicine isn’t actually new; it’s actually been there since the 1960s. And yet we didn’t see it effect a true change, or even become a real part our doctor’s offices — let alone routine healthcare services. So: what’s different now? And what does AI in medicine look like, practically speaking, whether it’s ensuring the best data, versioning software for healthcare, or other aspects?

In this episode of the a16z Podcast, Brandon Ballinger, CEO of Cardiogram; Mintu Turakhia, cardiologist at Stanford and Director of the Center for Digital Health; and general partner and head of a16z bio fund Vijay Pande in conversation with Hanne Winarksy discuss where will we start to see AI in healthcare first — diagnosis, treatment, or system management — to what it will take for it to succeed. Will we perhaps see a “levels” of AI framework for doctors as we have for autonomous cars?

Show Notes

• Discussion of how AI has been used so far in medicine [0:45] and its potential for the future [3:55]
• Questions about data sets [7:46] and how AI might be scaled [12:45]
• Unknowns about integrating AI into medicine [15:01], and a discussion of incentives for making the transition [21:04]

Transcript

Hanne: Hi, and welcome to the “a16z Podcast.” I’m Hanne, and today we’re talking about AI in medicine. But we want to talk about it in a really practical way. What it means to use it in practice and in a medical practice, what it means to build medical tools with it — but also what creates the conditions for AI to really succeed in medicine, and how we design for those conditions, both from the medical side and from the software side. 

The guests joining for this conversation, in the order in which you will hear their voices, are Mintu Turakhia, a cardiologist at Stanford and director of the Center For Digital Health; Brandon Ballinger, CEO and founder of Cardiogram, a company that uses heart rate data to predict and prevent heart disease; and Vijay Pande, a general partner here at a16z and head of our bio fund. 

So, let’s maybe just do a quick breakdown of what we’re actually talking about when we talk about introducing AI to medicine. What does that actually mean? How will we actually start to see AI intervene in medicine and in hospitals and in waiting rooms?

AI’s history and its future potential

Dr. Turakhia: AI is not new to medicine. Automated systems in healthcare have been described since the 1960s. And they went through various iterations of expert systems and neural networks and [were] called many different things.

Hanne: In what way would those show up in the ’60s and ’70s?

Dr. Turakhia: So, at that time, there were no high resolutions. There weren’t too many sensors. And it was about a synthetic brain that could take what a patient describes as the inputs, and what a doctor finds on the exam as the inputs.

Hanne: Using verbal descriptions?

Dr. Turakhia: Yeah. Basically words. People created what are called ontologies and classification structures. You put in the ten things you felt, and a computer would spit out the top ten diagnoses in order of probability. And even back then, they were outperforming average physicians. So, this is not a new concept.

Hanne: So, basically doing what hypochondriacs do with Google today, but verbally.

Dr. Turakhia: Right. So, Google is, in some ways, an AI expression of that, where it’s actually used ongoing inputs and classification to do that over time. [It’s a] much more robust neural network, so to speak.

Brandon: So, an interesting case study is the MYCIN system, which is from 1978, I believe. And so, this was an expert system trained at Stanford. It would take inputs that were just typed in manually, and then it would essentially try to predict what a pathologist would show. And it was put to the test against five pathologists, and it beat all five of them.

Hanne: And it was already outperforming.

Brandon: It was already outperforming doctors, but when you go to the hospital, they don’t use MYCIN or anything similar. And I think this illustrates that sometimes, the challenge isn’t just the technical aspects or the accuracy, it’s the deployment path. Some of the issues around there are — okay, is there a convenient way to deploy this to actual physicians? Who takes the risk? What’s the financial model for reimbursement? And so, if you look at the way the financial incentives work, there are some things that are backwards. For example, if you think about a hospital from the CFO’s perspective, a misdiagnosis actually earns them more money…

Hanne: What?

Brandon: …because when you misdiagnose, you do follow-up tests, right? And those — and our billing system is fee-for-service. So, every little test that’s done is billed for.

Hanne: But nobody wants to be giving out wrong diagnoses, so where’s the incentive? The incentive is just in the system — the money that results from it.

Brandon: No one wants to give an incorrect diagnosis. On the other hand, there’s no budget to invest in better diagnoses.

Hanne: In making sure it doesn’t happen.

Brandon: I think that’s been part of the problem. And so, things like fee-for-value are interesting because now, you’re paying people for an accurate diagnosis or for a reduction in hospitalizations, depending on the exact system. And so, I think that’s the case where accuracy is rewarded with a greater payment, which sets up the incentives so that AI can actually win in this circumstance.

Dr. Turakhia: Where I think AI has come back at us with a force is — it came to healthcare as a hammer looking for a nail. What we’re trying to figure out is where you can implement it easily and safely, with not too much friction and with not a lot of physicians going crazy, and where it’s going to be very, very hard. And that, I think, is the challenge in terms of building, developing these technologies, commercializing them, and seeing how they scale. And so, the use cases really vary across that spectrum.

Brandon: Yeah, I think about there as being a couple different cases where AI can intervene. One is to substitute what doctors do already, and so people use the example of radiology as an example. The other area that I think is maybe more interesting is that AI can complement what doctors can’t do already.

So, it would be possible for a doctor to, say, read an ECG and tell you whether you’re in an abnormal heart rhythm. No doctor right now can read your Fitbit data and tell you whether you have a condition like sleep apnea. I mean, if you look at your own data, you can kind of see restful sleep as real structured REM cycles, so you can see some patterns there. That said, the gold standard that a sleep doctor would use is a sleep study, where they wire you up with six different sensors and tell you to sleep naturally. There’s a big difference here between the very noisy consumer sensors that may be less interpretable, and what a doctor is used to seeing.

Vijay: Or it could be that the data is on the device, but the analysis can’t be done yet. Maybe the analysis needs a gold standard data set to compare to. There are a lot of missing parts beyond just gathering the data from the patient in the first place.

Dr. Turakhia: I think there’s some inherent challenges in the nature of the beast. Healthcare is unpredictable. It’s stochastic. You can predict a cumulative probability — like a probability of getting condition X, or diagnosis X, over a time horizon of 5 or 10 years — but we are nowhere near saying, “You’re going to have a heart attack in the next 3 days.” Prediction is very, very, very difficult, and so where prediction might have a place is where you’re getting high fidelity data, whether it’s from a wearable or a sensor. 

It’s so dense that a human can’t possibly do it. Like, a doctor’s not going to look at it. And two, it’s relatively noisy. Inaccurate, poor classifiers, missing — periods where you don’t have this continuous data that you really want for prediction. In fact, the biggest predictor of someone getting ill with a lot of wearable studies is missing data, because they were too sick to wear the sensor.

Hanne: Oh, so the very absence of the data is a big indicator.

Dr. Turakhia: Yes. Exactly.

Hanne: That’s so interesting. I don’t feel well enough to put on my what-have-you, and that means something’s not right.

Dr. Turakhia: Possibly, or you’re on vacation. And so that’s the problem. That’s other challenge of AI — is context. And so, what are some of the more simple problems where you have clean data structures, you have less noise, you have very clear training for these algorithms. And I think that’s where we’ve seen AI really pick up, in imaging-like studies. It’s a closed-loop diagnosis. You know, there is a nodule on an x-ray that is cancer-based on a biopsy, proven later in the training dataset, or there isn’t. In the case of an EKG, we already have expert systems that can give us a provisional diagnosis on an EKG. They’re not really learning. And so, that’s a great problem, because most arrhythmias don’t need context. You can look at it and make the diagnosis.

Hanne: We don’t need them to learn, so that’s why it’s good to use right away, to apply this technology immediately.

Dr. Turakhia: You don’t need everything. You don’t need to mine the EMR to get all this other stuff. You can look at the image and say, “Is it probably — does it have a diagnosis or does it not?” And so, imaging of the retina, imaging of skin lesions, x-rays, MRIs, echocardiograms, EKGs — that’s where we’re really seeing AI pick up.

The importance of big data

Brandon: I sort of divide the problems into inputs and outputs. We talked a little bit about some of the inputs that have become newly available, like EMR and imaging data. I think it’s also interesting to think about what the outputs of an AI algorithm would be. And these examples are self-contained, well-defined outputs that fit into the existing medical system. But I think it’s also interesting to imagine what could happen if you were to reinvent the entire medical system with this assumption that we have a lot of data — intelligence is artificial, and therefore cheap — so we can do continuous monitoring. So, one of the things I think about is, “What are the gaps of people who do not have access to EKGs?” I’ve actually never had an EKG done aside from the ones I do myself. So, and most people in the U.S. — you get your first EKG when you turn 65 as part of your Medicare checkup, and they won’t reimburse for anything after that.

Hanne: Oh, wow, I didn’t realize it’s so late.

Brandon: My dad’s an excavator, so he digs foundations for houses, and he hasn’t seen a doctor in 20 years. And if he leaves a job site, the entire job site would shut down. So, it’s hard for some people, I think, to go into the doctor’s office between the hours of 9:00 a.m. to 5:00 p.m. If you look at that in aggregate, about half of people in the U.S. have a primary care physician at all, which seems astonishingly low, but that’s the fact. There’s a gap — about a third of people with diabetes don’t realize they have it, about a fifth of people with hypertension, for AFib, it’s 30% or 40%. For sleep apnea, it’s like 80%.

Vijay: I think it’s one thing just finding out but not being able to do anything about it, but the actionable aspect, I think, really is a huge game-changer. It means that you can have both better outcomes for patients, and, in principle, lower costs for payers.

Hanne: Right. And these are areas where there are clear ways of addressing these specific conditions.

Dr. Turakhia: I will take a little bit of a different view here, which is that I don’t know if AI — artificial intelligence — is needed for earlier and better detection and treatment. To me, that may be a data collection issue.

Hanne: How is that different from what we’re saying about finding it early? How can that not be good?

Dr. Turakhia: Because that may have to do with getting sensors out of hospitals and getting them to patients. And that’s not inherently an AI problem. It could be a last mile AI problem — so that if you want to scale the ability to get this stuff. So, let’s say we get to a point where our bathroom tiles have built-in EKG sensors and scales, and the data is just collected while we brush our teeth. It’s the sensing technology that may detect things discreetly, like an arrhythmia. You may not necessarily need intelligence, but who’s going to look at the data? And so that’s a scaling issue.

Vijay: The AI could look at the data. And the other thing is, if you’re using this as screening, you want to make the accuracy as high as possible to avoid false positives. And AI would have a very natural role there too.

Hanne: But it’s interesting that you’re saying it’s not necessarily about the analysis, it’s about where the data comes from and when.

Dr. Turakhia: I think there are two different problems. There may be a point that it truly outperforms the cognitive abilities of physicians. And we have seen that with imaging so far, and some of the most promising aspects of the imaging studies and the EKG studies are that the confusion matrices — the way humans misclassify things — is recapitulated by the convolutional neural networks.

Hanne: Can you actually break that down for a second? So, what are those confusion matrices?

Dr. Turakhia: So, a confusion matrix is a way to graph the errors and which directions they go. For rhythms on an EKG, a rhythm that’s truly atrial fibrillation could get classified as normal sinus rhythm or atrial tachycardia, or supraventricular tachycardia. The names are not important. What’s important is that the algorithms are making the same type of mistakes that humans are doing. It’s not that it’s making a mistake that’s necessarily more lethal and just nonsensical, so to speak. It recapitulates humans. 

To me, that’s the core thesis of AI in medicine. Because if you can show that you’re recapitulating human error, you’re not going to make it perfect. But that tells you that, in check and with control, you can allow this to scale safely since it’s liable to do what humans do. And so, now you’re automating tasks that, you know — I’m a cardiologist, I’m an electrophysiologist, but I don’t enjoy reading 400 ECGs when it’s my week to read them.

Hanne: So, you’re saying it doesn’t have to be better — it just has to be making the same kinds of mistakes to feel that you can trust the decision-making.

Dr. Turakhia: Right. You dip your toe in the water by having it be assistive, and then at some point, we as a society will decide if it can go fully auto. Fully autonomous without a doctor in the loop. That’s a societal issue. That’s not a technical hurdle at this point.

Hanne: Right.

Vijay: Well, you can imagine, just as — let’s say, self-driving cars, you have different levels of autonomy. It’s not nothing versus everything.

Dr. Turakhia: It’s not.

Vijay: You can imagine level one, level two, level four, level five in self-driving cars. I think that would be the most natural way because we wouldn’t want to go from nothing to everything.

Dr. Turakhia: Exactly. And just like a self-driving car, we as a society have to define who’s taking that risk on. You can’t really sue a convolutional neural network, but you might be able to make a claim against the physician, the group, the hospital that implements it. You know, how does that shake out?

Hanne: To figure out, literally, how to insure against these kinds of errors.

Brandon: I think the way you think about some of these error cases depends on whether the AI is substituting for part of what a doctor does today, or if the AI is doing something that’s truly novel. I think, in the novel cases, you might not actually care whether it makes mistakes that would be similar to humans.

Hanne: That’s an interesting point, because it’s doing something that we couldn’t achieve. What kinds of novel cases like that can you imagine?

Brandon: Wearables are an interesting case, because they’ll generate about 2 trillion data points this year. So, there’s no cardiologist or team of cardiologists who could even possibly look at those. That’s a case where you can actually invert maybe the way the medical system works today. Rather than being reactive to symptoms, you can actually be proactive, and the AI can be essentially something that tells you to go to the doctor rather than something that the doctor uses when you’re already there.

Vijay: Let’s take radiology as an example, where you could have one level where it’s as good as a common doctor, another level where it’s as good as the consensus of doctors.

Hanne: Right.

Vijay: Another level is that it’s not just using the labels [that a] radiologist would say on the image. It’s using a higher-level gold standard. It’s predicting what the biopsy would say. And so, now you’re doing something that…

Hanne: Which would be into your kind of novel, plus…

Vijay: Yeah, something that no human being could do. It can do that because in principle, it could fuse the data from the image, and maybe blood work, and other things that are easier to get and much less risk inducing than removing tissue in a biopsy.

Hanne: So, pulling those multiple streams of information into one and sort of synthesizing them is another area that…

Vijay: Yeah. It’s very difficult for a human being to do, and very natural for a computer.

The unknowns of AI in healthcare

Dr. Turakhia: It is very natural, but I think we need a couple of things to get there. We need really dense, high-quality data to train. And the more data you put in a model — I mean, so, machine learning, by definition, is statistical overfitting, and sometimes…

Vijay: Well, actually, I think that’s wrong. Machine learning done poorly — I mean, it’s like saying driving is driving a car off a cliff. Poor driving is poor driving, but machine learning tries to avoid statistical overfitting.

Dr. Turakhia: It does. My point is that one of the unknowns with any model, it doesn’t matter if it’s machine learning or regression or a risk score, is calibration. And as you start including fuzzy and noisy data elements in there — first of all, often the validation data sets don’t perform as well as the training dataset, no matter what math you use.

Hanne: Okay. And why is that?

Vijay: Well, that’s a sign of overfitting, and usually it’s because there wasn’t sufficient regularization during the training process.

Dr. Turakhia: So overfitting is a concept in statistics to effectively indicate that your model has been so highly tuned and specified to the data you see in front of it, that it may not apply to other cases.

Hanne: It can’t generalize.

Dr. Turakhia: If you had to use a model to identify a bunch of kids in a classroom and pick the kid who’s the fastest, an overfitted model might say it’s the red-headed kid wearing Nikes. Because in that class, that was the case.

Hanne: That was the one child.

Dr. Turakhia: But that has no plausible biological or other plausibility.

Hanne: You can’t extrapolate, it doesn’t generalize, yeah.

Dr. Turakhia: You can’t use that. If you take that to a place where the prevalence of Nike shoes or redheads is low, you might miss the fastest person, right?

Hanne: Not helpful, yeah.

Dr. Turakhia: These are some of the issues. The underlying shifts in population, the natural language processing that’s embedded in AI, the lexicon that people use. How doctors and clinicians write what it is that they’re seeing with their patient is different, from not even specialty to specialty, but hospital to hospital, sort of mini subcultures.

Brandon: It’s going to be different at Stanford than it was at UCSF, which is going to be different at Cleveland Clinic. I think that’s actually a nice thing about wearable data, is that Fitbits are the same all over the world. This label problem though is interesting because, in our context, each label represents a human life at risk. It’s a person who came into the hospital with an arrhythmia, and so you’re not going to get a million labels the way you might for a computer vision application. It would be unconscionable to ask for a million labels in this case. So, I think one of the interesting challenges is training deep learning-based models, which tend to be very data-hungry, with limited label data.

Dr. Turakhia: The kitchen-sink approach of taking every single data element — even if you’re looking at an image, can lead to these problems of overfitting. And what Brandon and Vijay are both alluding to is, you limit the labels to really high-quality labeling, and see if you can go there. And so don’t complicate your models unnecessarily.

Vijay: And don’t build models that are overly complicated for the amount of data that you have. Because if you have the case where you’re doing so much better on the training set than the test set, that’s proof that you’re overfitting, and you’re doing the ML wrong.

Brandon: Modern ML practitioners have a whole set of techniques to deal with overfitting. So, I think that problem is very solvable with well-trained practitioners. One thing you’ve alluded to, which is the interpretability aspect. So, let’s say you train on a population that’s very high in diabetes, but then you’re testing on a different population, which has a higher or lower prevalence. That is kind of interesting — so, identifying shifts in the underlying data and how you get…

Hanne: What would that mean?

Brandon: So, let’s say we train on people in San Francisco, and everyone runs to work and eats quinoa all day. But then we go to a different part of the country where maybe obesity is higher, or you could be somewhere in the stroke belt where the rate of stroke is higher. It may be that the statistics you trained on don’t match the statistics that you’re now testing on. It’s fundamentally a data quality problem. If you collect data from all over the world, you can address this. But it’s something you have to be careful with.

Hanne: But it will take a while for that to happen as we start gathering the data in different ways. How does that actually even happen? How are these streams of data funneled in and examined and fed into a useful system?

Brandon: So, used to be, the way you’d run a clinical trial, you would have one hospital. You’d recruit patients from that hospital, that’d be it. If you got a couple of hundred patients, that might actually be quite difficult to attain. I think with ResearchKit, HealthKit, Google Fit, all of these things, you can now get 40,000 or 50,000 people into a study from all over the world, which is great, except the challenge that the first five ResearchKit apps had is that they got 40,000 people, and then they lost 90% of them in the first 90 days.

Hanne: So, everybody just drops out?

Brandon: Everyone just drops out, because the initial versions of the apps weren’t engaging. So, this adds an interesting new dimension. As a medical researcher, you might not think about building an engaging, well-designed app, but actually, you have to bring mobile design in as now a discipline that you’re good at.

Hanne: So, there has to be some incentive to continue to engage.

Brandon: Yeah, exactly. You need to measure cohorts the same way Instagram, or Facebook, or Snapchat does. So, I think the teams that are starting to succeed here tend to be very interdisciplinary. They bring in the clinical aspect, because you need that to choose the right problem to solve, but also design the study well so that you have convincing evidence. You need the AI aspect, but you also often need mobile design, if it’s a mobile study. You may need domain expertise in other areas if your data is coming from somewhere else.

Hanne: And then it all has to be gamified and fun to do.

Brandon: Yeah. Well, I mean, gamification is sort of extrinsic motivation, but you can also give people intrinsic motivation — giving them insights into their own health, for instance. It’s a pretty good way to hook people.

Urging the current system to adopt AI

Hanne: What’s the system’s incentives? I mean, of course, doctors want it if it makes them more accurate or to scale better. Patients want it if you can predict whether or not you’re going to have a problem. How do we incentivize the system as a whole?

Dr. Turakhia: I believe fundamentally it is going to come down to cost and scale, and what willingness does a healthcare entity, whoever that may be — whether it’s employer-based programs, insurer-based programs, accountable care organizations. Are they going to be willing to take on risk to see the rewards of cost and scale? And so, the early adopters will be ones who’ve taken on a little more risk.

Vijay: Yeah. I think, you know, it is — the challenge is where the hope is, and in terms of value, and in terms of better outcomes. But one has to prove it out and hospitals will want to see.

Dr. Turakhia: The regulatory risk thing is being largely addressed by this new office of digital health and the FDA, and they really seem much more forward-thinking about it. But there are going to be challenges that we have to solve, and I’ll give you one just to get the group’s input here. Should you be versioning AI, or do you just let it learn on the fly? And so, normally, when we have firmware, hardware, software updates in regulated and FDA-approved products, they’re static. They don’t learn on the fly. If you keep them static, you’re losing the benefit of learning as you go. On the other hand, bad data could heavily bias the system and cause harm, right? So, if you start learning from bad inputs that come into the system for whatever reason, you could intentionally or unintentionally cause harm. And so, how do we deal with versioning in deep learning?

Vijay: I mean, to just freeze the parameterization — so versioning, from a computer science point of view, is trivial. There’s the deeper statistical question, which you could version every day, every week, every month.

Hanne: Right. It’s when and how often.

Vijay: And just freeze the parameters. What you want to do is, to the point we were talking about earlier — you want to bring in new validation sets. Things that it has never seen before, because you don’t want to just test each version on the same validation set, because now you’re intrinsically overfitting into it. What you always want to be doing [is] holding out bits of data [so that] you can test each version separately, because I want to make sure that they have very strict confidence that this is doing no harm, and this is helpful.

Hanne: Right. It’s like, we’re introducing this whole new data set of a different kind of thing, and that’s when you make new considerations.

Vijay: Yeah. Data’s coming in all the time, and so you just version on what came in today, and that’s it. It’s pretty straightforward. And as you’re training it…

Brandon: This is the way speech recognition works on Android phones. Obviously, data is coming in continuously, and every time someone says, “Okay, Google,” or, “Hey, Siri,” it’s coming into either Google or Apple. But you train a model in batch and then you test it very carefully and then you deploy it. The versions are indeed tagged by the date of the training data.

Hanne: It’s already embedded in the system. Who are the decision-makers that are green lighting when, like, “Okay, we’re going to try this new algorithm. We’re going to start applying this to these radiology images.” What are the decision points?

Dr. Turakhia: So, with EKGs, the early companies used expert systems to just ease the pain points of me having to write out and code out every single diagnosis.

Hanne: The super low-hanging fruit.

Dr. Turakhia: Yeah. Can you improve the accuracy of physicians with this? Can you increase their volume and bandwidth? Can you actually use it to see which physicians are maybe going off course? What if you start having a couple of physicians whose error rates are going up? 

Right now with quality, the QI process isn’t really based on random sampling. There’s actually no standardized metrics for QI in any of this. When people read EKGs and sign them off, they just sign them. There’s nothing telling anyone that this guy has a high error rate. And so, that is a great use case of this, where you’re not making diagnoses, but you’re helping anchor and show that, well, if you believe this algorithm is good and broadly generalizable across data, you’re restating the calibration problem now.

It’s not that the algorithm has gotten necessarily worse, because, in fact, in seven of the eight doctors, it’s right on par with them. But in this other doctor, it could be if that doctor — if that doctor is not agreeing with the algorithm which is agreeing with the other seven, that doctor is actually not agreeing with the other seven. So now you have an opportunity to train and relearn. Those are the use cases to go off of.

Vijay: You can train and relearn the person?

Dr. Turakhia: The person. Address their reading errors, coding errors, see what’s going on. And that qualitative look, I think, is very, very valuable.

Hanne: So, what are the ways we’re actually going to start seeing it in the clinical setting? You know, the tools that we might see our doctor actually use with us or not.

Dr. Turakhia: I think it’s going to be these adjacencies around treatment with management. There are a lot of things that happened in the hospital that seem really primitive and arcane, and no one wants to do them. I’ll give you a simple one, which is OR scheduling.

Hanne: So, is it actually the way it looks like it is in “Grey’s Anatomy?” Is it just a whiteboard and an eraser?

Dr. Turakhia: It is a whiteboard and somebody on the phone. The OR front desk.

Hanne: That’s unbelievable.

Dr. Turakhia: There’s a backend of scheduling that happens for the outpatients, but you have add-ons, you have emergencies, you have finite…

Hanne: I mean, it seems like even an Excel sheet would be better than a whiteboard.

Dr. Turakhia: The way OR scheduling works now is primitive, and it also involves gaming. It involves convincing staff X, Y, and Z to stay to get the case done, or do it tomorrow.

Hanne: So, there’s so much behind the scenes, like, human negotiation?

Dr. Turakhia: When I do catheter ablations, we have many different moving parts. Equipment, the support personnel of the equipment manufacturer, anesthesia, fellows, nurses, whatever. Everyone has little pieces of that scheduling. It all comes together, but it comes together in the art of human negotiation and very simple things like, “This is your block time, and if you want to go outside your block time, you need to write a Hallmark card to person X.” So, very simple problem where there’s huge returns inefficiency if you could have AI do that. With the AI inputs over time, you could be like, well — you can really know which physicians are quick and speedy, which ones tend to go over their allotted times, which patient cases might be high risk, which ones may need more backup, which should be done during daytime hours.

Brandon: You could add their Fitbit data and then you could tell who’s drowsy at any given moment, for a little elaboration there.

Hanne: Oh, that’s fascinating. Yeah. Whether or not they want to do it.

Brandon: How stressed are they feeling.

Dr. Turakhia: And so, people stay at times that they’re really needed. That kind of elasticity can come with automation where we fail right now. This is a great place where you are not making diagnoses. There’s nothing you’re being committed to from a, kind of, basic regulatory framework. You’re just optimizing scheduling.

Hanne: So, who actually — so, say that that technology is available. How do you actually get it in — where’s the confluence of the regulation and the actual rollout, and how does it actually make its way into a hospital and to a waiting room?

Brandon: There’s an alternative model I’ve seen, which is startups acting as full-stack healthcare providers. So, Omada Health or Virta Health would be examples of this, where if you have pre-diabetes or diabetes, respectively, the physician can actually refer the patient to one of these services. They have on-staff physicians. They’re registered as providers with national provider IDs. They bill insurance just like a doctor would, and they’re essentially acting as a provider who addresses the whole condition end to end. 

I think that case actually simplifies decision-making, because you don’t necessarily have to convince both Stanford and United Healthcare to adopt this thing. You can actually convince just a self-insured employer that they want to include one of these startups as part of their health plan. And so, I think that simplifies the decision-making process and ensures that the physicians and the AI folks are under the same roof. I think that’s a model that we’re going to see probably get the quickest adoption, at least in the <inaudible> world.

Vijay: There are many models, and which is the best model will depend on how you’re helping in the indication, and on the accuracy, and what you’re competing against, and so on.

Brandon: This is a case where, probably, we’ll see the healthcare industry maybe reconstitute itself by vertical, with AI-based diagnostics or therapeutics. Because, if you think — right now, providers are geographically structured. But with AI, every data point makes the system more accurate. Presumably, in an AI-based world, providers will be more oriented around a particular vertical. So, you might have the best data network in radiology, the best data network in pathology, the best data network in…

Hanne: That’s interesting. Yeah. Thank you so much for joining us on the “a16z Podcast.”

Vijay: Great. Thank you.

Dr. Turakhia: Thank you.

Brandon: Thanks for having us.

• 

  Brandon Ballinger
• 

  Vijay Pande is a general partner at a16z where he invests in biopharma and healthcare. Prior, he was a distinguished professor at Stanford. He is also the founder of Folding@Home Distributed Computing Project.

  Follow Twitter
• 

  Mintu Turakhia
• 

  Hanne Winarsky

There are many reasons why we’re in an “AI spring” after multiple “AI winters” — but how then do we tease apart what’s real vs. what’s hype when it comes to the (legitimate!) excitement about artificial intelligence and machine learning? Especially when it comes to the latest results of computers beating games, which not only captures our imaginations but has always played a critical role in advancing machine intelligence (whether it’s AI winning Texas Hold’em poker or beating the world human champ in the ancient Chinese game of Go).

But on learning that Google DeepMind’s AlphaGo can master the game of Go without human knowledge — or more precisely: “based solely on reinforcement learning, without human data, guidance, or domain knowledge beyond game rules” — some people leap too far towards claims of artificial generalized intelligence. So where can we then generalize the findings of such work — unsupervised learning, self-play, etc. — to other specific domains? What does it mean for entrepreneurs building companies (and what investors look for)? And what does it mean for how we, as humans, learn… or rather, how computers can also learn from how we learn?

Deal and research operating team head Frank Chen and a16z board partner Steven Sinofsky ponder all this and more, in conversation with Sonal Chokshi, in this episode of the a16z Podcast. We ended last time with the triumph of data over algorithms and begin this time with the triumph of algorithms over data … is this the end of big data?

Show Notes

• Discussion of how AlphaGo’s algorithm learned Go [0:45], and the potential limits of machine learning [9:41]
• Questions of humans interacting with AI, and our expectation of accuracy [18:46]
• Potential bias in algorithms [29:46] and final thoughts about the future of AI/ML [34:21]

Transcript

Sonal: Hi, everyone, welcome to the “a16z Podcast.” I’m Sonal. And I’m here today, bringing back the band together again — the AlphaGo band, I guess — I don’t know how else to describe us. But we have the head of our Deal and Research Team Operation, Frank Chen. We have Steven Sinofsky, a16z board partner.

Just to give people some quick context, you don’t have to have heard our previous podcast on AlphaGo. But when AlphaGo, the algorithm produced by DeepMind, beat the world champ in Korea in playing Go, which is an ancient Chinese game, we had a podcast where we discussed a lot of the themes and some of the broad things around that. And what we’d like to talk about today is their latest paper — but not just only specifically, but more broadly — what this means for where we actually are. What’s hype, what’s real in AI or artificial intelligence. So, welcome, guys.

Background on the AlphaGo algorithm

Steven: Welcome. So, like one thing that, like — you read this paper, and the paper is published in “Nature.” It’s pretty dense. It has 17 authors on it. And so, it’s quite the force. But the thing that sort of jumps out is the paper, the blogs, everybody — is, the first thing you read is, “one step closer to creating general purpose AI.” And immediately, like, my AI winter, fears of hype antennae pop up, because, like, everything is one step closer. But like, you could take very, very, very tiny little steps, or you could overhype them.

Frank: And we’ve heard these promises throughout history, and especially around board games. So, we solved checkers, and people were like, “Oh, one step away from general intelligence.” Then we solved chess, same thing, we’re one step away from general intelligence.” Then we did the, “Oh, I can find bacteria that causes infection; therefore, we must be one step away.”

And I think the fallacy is that because we’re doing these things that are considered, sort of, high cognition — like the smart people play chess, smart people figure out…

Sonal: Yeah, strategy.

Frank: …then solving one thing that a smart person does must lead us to the next thing that a smart person does, which will lead us to the next. And that’s always been the fallacy, which is, it actually hasn’t quite generalized.

Steven: Well, in fact, they don’t even really compound all that much. They’re all fairly discrete. The one thing that’s different now is that they are all building on these new artificial intelligence or machine learning techniques, and taking a whole bunch of data, training a model on it, and developing solutions that beat all their algorithmic ones. And that was a big thing about the first AlphaGo was, like, “This is data over algorithms.” And all of a sudden, here we are…

Sonal: Algorithms over data.

Steven: Or, a model over data. But you know, Frank, you have an interesting way of looking at this, because, you know, you play Go, and you understand that…

Sonal: Oh, you play Go? I didn’t know that.

Frank: Yeah. Well, enough, yeah.

Steven: More than me.

Sonal: I only play chess. I don’t even know…

Frank: Yes, Chinese chess and Go. All right, I play a few games.

Steven: But, like, it’s not like a generalized problem. You know, it comes with, like, a whole bunch of constraints and things that make it solvable by algorithm.

Frank: Yeah. So, what are some traits about Go that are not like the real world? All the rules are completely well known. The state of play is completely well known, right? In the real world, mostly, we live in a “fog of war” situation, where we know some things, and we don’t know other things. It would cost us something to go figure it out. In Go, and in particular the representations they chose, the entire board state is known to both players, right? So, there’s a lot of things that aren’t like the real world at all when we do these board games. Now, having said that, why are people getting breathless, yet again, around one step closer that AI…

Sonal: I think, because the authors don’t claim generalized intelligence. They’re talking about being able to do — apply some of these techniques in other domains.

Frank: Yeah, so to give credit where credit’s due. Let’s, sort of, review what they actually achieved. And they are, like, very impressive achievements. So, what they achieved was, they now have a Go player that beat all of their other Go players.

Sonal: AlphaGo Zero.

Frank: Well, they named them by the people they beat. So, they had AlphaGo Fan, AlphaGo Lee. This one’s AlphaGo Zero. So AlphaGo Zero, their latest one, has beat every iteration, which has beat all of the best human players, right? So, they have the best Go-playing algorithms in the world. And the difference in how they got to this one is, there was no training data from human games. So, all the other ones had been bootstrapped by, “Let me go watch what the human players do and see if I can mimic that.” And that, very broadly speaking, is the approach to machine learning called supervised learning.

Steven: And not a trivial amount. I mean, they had 100,000 games that they started from to train it on, like, 48 of Google’s TPUs for days on end.

Frank: That’s exactly right. So, a fleet of machines, a ton of data. This one started from nothing, which is, it didn’t even really know the rules. It had a loss function, which is reinforcement learning’s way of sort of improving the algorithms over time. So, let me just give a quick intuitive explanation of reinforcement learning. So, it’s exactly like the game that you used to play when you were a kid and called hotter or colder. Somebody would go hide something, and then you would try to get to discover it.

Sonal: Are you getting hotter? I’m getting hotter, hotter, cold, cold again.

Frank: Exactly. And then, the hotter or colder is the loss function. That is the thing that tells you if you’re getting closer or not.

Sonal: It’s basically trial and error, simply put.

Frank: That’s exactly right. And so, that’s the fundamental approach in AlphaGo Zero, which is, they have this loss function that describes, are you more likely to win the game having made this move or not, right? Ao, that’s all they had. And then they didn’t have human input. They didn’t have human games. They basically said, “I have this loss function that tells you whether you’re more likely to win this game or not.” And then it played itself.

Steven: One other important thing, I think, it has — it also knows — it has the codified rules of the game.

Sonal: That is the one human input it actually got.

Steven: Well, right. And it makes it a very, very constrained problem. Because there’s a whole bunch of the decision trees — like, hot or cold, you could climb up the sofa, you could leave the house, you could go all over the place for days on end. Whereas, this just tells, “No, it’s gonna only be on the floor.”

Sonal: These are the rules of the game.

Steven: Like, this is not gonna be under the sofa.

Frank: It’s a 17 by 17…

Sonal: Very little of real life is actually, like, you get the real Shogi that way.

Frank: Yeah, it’s a 17 by 17 board, it’s black and white pieces, there are rules.

Steven: Oh, yeah, you switch turns, like all of that stuff matters a lot. 

Sonal: It does. And one more thing to add, by the way, too, because you mentioned the 48 TPUs — it’s really significant that they got it down to 4 in this case. That’s a huge, like — on the power side energy, like — you know, it’s a simpler architecture.

Frank: It’s a simple architecture, and thinking from the point of view of a software developer, now you’re back to one machine. You’re not like, “Oh, my God, I need to go rent this massive cloud with massive storage and massive interconnects. And, like, I need to figure out how to provision the cluster and manage the cluster.” You’re back down to one machine, right?

Steven: So, this stuff was pretty impressive. It’s, you know, they did four TPUs. It was three days of playing.

Sonal: Three days total.

Steven: Like, it’s very achievable, you know, like, on your Amazon credits.

Frank: That’s exactly right. And if you think about sort of the approach that most startups take to artificial intelligence today, they basically take the supervised learning approach. And step one, raise money so that you can go get a data set that’s annotated, train your neural network, make recommendations, right? And you could be $1, or $5 or $10, or $50 million in getting that data set, depending on how complicated the data set is. All right, so the reason people are so excited about this is, look, this had no data, aside from rules of the game. It basically played itself. And by day three, it was better than everything that had been trained before it.

Sonal: Nearly an order of magnitude.

Frank: Yeah, it was, orders of magnitude on this…

Sonal: And the results would be a hundred to zero.

Frank: …48 TPUs versus 4. It was 3 days versus 40. It was 30 million train games versus 4.9. So, order of magnitude improvement on all of those dimensions, so, like, let’s give credit where credit is due — this is a very impressive technical achievement. And then, the question that we sort of entered the session with, “Okay, does this make us more likely to be able to create an artificial general intelligence, where the learning algorithm is generalized across domains?” In other words, can I take the breakthroughs here, and make a better pick and pack robot for Amazon? Or make a better healthcare predictor to discover whether you have cancer?

Steven: Or Salesforce forecasting, code generation, or a whole bunch of stuff. And the interesting thing is that — also the techniques here — this is just proof of something that people have been talking about for a long time.

Sonal: Oh, yeah, reinforcement learning has been around for ages.

Steven: And so, part of what’s interesting to me is that reinforcement learning has been around for a long time. Obviously, the modeling has been around for a long time…

Sonal: Self-play.

Steven: …self-play, all of these things. And, like, so many times these steps are, like, somebody new to the domain looking and pulling together a bunch of unrelated things, and just coming up with a very elegant, incredibly elegant solution. And it’s super impressive, but it’s not clear it generalizes. And I think that was one of the things that jumped out for me in reading about it, you know. When you hear, “Oh, and then the next step, on this train of generalized intelligence is drug discovery, and protein folding, and quantum chemistry, and material…” And it’s like, all of a sudden, I’m trying to figure out — protein folding. Like, what are the constraints on protein folding? Well, we know they’re amino acids, and we know that they have to be in three dimensions. But actually, nobody else knows…

Sonal: There’s no codified rules. 

Steven: Like, nobody has the rules of protein folding.

Frank: So, there’s no rules, there’s no perfect understanding of the search space. There’s like — what’s the loss function? Like, how would you even write a loss function?

Sonal: I do want to push back a little bit, though, because far be it from any of us in here to hype this up. But there’s something unique happening here, which at least — I perceive this in the paper. Which is, that I was struck by the analogy to evolution. Like, this is how human beings have evolved. This is evolution — that we learn by trial and error on a massive million scale. So, I don’t want to completely dismiss the idea that we can get to some kind of generalized intelligence. I mean, of course, I understand and agree, but what are the limits? And what are the possibilities that can actually take us there? And where are we constrained? Just to break that down a bit more.

The limitations of machine learning

Frank: Yeah. So, I love the evolution analogy, right? Because in the paper, they talked about how, you know, when it started out, it was making, sort of, naïve moves. It was, sort of, greedy and trying to capture all the tokens. And then, it got to very sophisticated patterns that humans have discovered over thousands of years playing this game, teaching this game, codifying it in books. And it figured that out. And not only did it figure it out, it figured out things that humans haven’t quite codified, right?

Sonal: Right. New ways of playing.

Frank: If you play more games, sort of, it developed an intelligence that humans haven’t yet, because it’s going through its own reinforcement thing. And thousands of generations of games playing each other, sort of, arrive at places that maybe humans would have gotten to if we played another thousand years. But like, you know, it figured out in three days. So, I think there is something incredibly profound going on here, where you’re, basically — you’re accelerating natural selection cycles in computers.

Now, where I think the analogy breaks down is, “Oh, and therefore we can apply this approach to every other problem.” And it’s just going to be a straightforward application of the set of ideas to those problem sets. And then, we’re going to have evolution at that scale.” In other words, eating different problems, right, in exactly the same way. And I think that’s where Steve and I have a little skepticism.

Steven: Yeah, well, and I think also like the key with all of these, if you just take an abstract view of it, you know, you have this awesomely elegant solution to an intractable problem, which is just, on any measure, super cool. But that doesn’t make it generalizable to any other space. There are many, many super cool things that surface. And you have to be careful as, like, an engineer, or a founder, or a person applying this to [go], “Okay, well, what are the elements of this solution that one would need to have as a precursor to applying it.”

And we saw the same thing with all of the work on supervised learning. Like, first you need a data set that is clean, and then it has to be, like, labeled really well. And then, you have to have a neural network model. And you have to do all the weights and all this modeling. And so, there are all these things where you couldn’t just say, “Hey, I’d like to make the most money in our Q4 sales. Let’s machine learn our way there.” And you, like, you can’t just show up on Thursday and do that. Every year, salespeople — they develop a model for how they want to sell, how the customers — what the prospects are. They know the rules — like, there are all these rules. Like, there are this many salespeople, they can only call so many people per day. They know which customers to call. They know what the quotas are going to look like. They know what the product — and you start to think, well, maybe there is something here, because it’s a space where, like, actually, the history might not be as well as applicable to supervised learning as you would like. And so, if there was a way to look at this through the lens of, like, what would be the optimal system to navigate? Or what would be the sales matter? And I think that there are more things like that than fewer. I don’t know how many, you know, properties of nature are amenable to this. Because, for the most part, we don’t understand them.

Sonal: Right. It’s limited by what the human knows, right? And we can’t codify those rules. What are the domains that you see some transferability of these types of things?

Frank: Well, I love the idea of, sort of, sales forecasting, right? Because, essentially, what you’re doing — the intuition is that if I could play act my salespeople doing a million different things, in a million different situations, against a million different set of prospects, you know, I could sort of simulate that, then maybe best practices would just come out of that, right? Because, essentially, that’s what I’m doing in real life. I raise a ton of money, I hire a sales leader, that sales leader hires a bunch of people, gives them a playbook, and has them call a bunch of prospects. I can only run so many experiments, right? Because every call is an experiment. So, the idea would be, if I could simulate what happens inside these calls, and simulate 10, 100, 1000, a million times more, then I’d get much better best practices emerging from that. So, super intriguing idea.

Steven: Off the top of my head, you start wanting to think about, like, “Okay, what about, you know, cybersecurity.” And you think, you’re looking at your code, and you know at any given moment, like, the type of code that can be in a specific place. And you know, the rules and the syntax, and it’s well understood, and you know patterns that are also bad. And so, today, what you do — and the people that have tried to apply machine learning to code — they just have a lot of examples of, like, missing equal signs, or missing semicolons, or operators are wrong. But if you think about it, like, the syntax of the language is completely fixed. And so, you’re back at very much a…

Sonal: A rule based…

Steven: A rule based kind of — you know, rule base is a constraint of, like…

Sonal: Constraint, right.

Steven: These are the only ways you can put the…

Sonal: Definable constraint.

Steven: …symbols together. What you don’t have is right or wrong, win or lose with code.

Frank: Yeah, you don’t have a loss function that’s that objective.

Steven: They’re interesting things to me that, like, there’s just complexity of line that is a very common measure. Like, “Boy, something with three sets of parentheses in it, it’s likely to have a bug in it just because it has three sets of parentheses in it.” You know, something that’s missing, you know, some enclosure kind of rules, like using brackets, not using brackets. These kinds of things, you can actually flag, like — you can think of lint. Just flags them in C and C++ as just, sort of, high-risk behaviors.

So, there are things that you can put in to, like, sort of — and so then you start to think, “Wow, it’d be really interesting to have a tool that is able to look at code, and, sort of — just very different than previously, which was just literally looking at the syntax — but doing millions of examples of generating code and finding bad examples.” Would it be able to do a better job at finding bad examples in the next piece of code that falls into it?

Sonal: Right. I mean, one of the points you guys made last time, in our last podcast is that, at the end of the day, these things aren’t working in isolation. It’s not like there’s one magic approach. You know, there’s always a combination of techniques that come together to actually build real products. How would this, sort of, fit into that? Because one of the thoughts that I had is that — clearly, this kind of approach, even if you don’t have clearly defined rules, will always be more beneficial in places where we don’t have any data. Like, any big data. Just like humans, like kids, learning from N equals 2, like their parents. Or N equals 1, if they’re a single parent.

Frank: I think if we think about what humans do, they have many different types of intelligences. They have many different types of strategies for solving problems. So, my guess is that the artificial intelligences that we create will be similar. Lots of different strategies. And one of the interesting research items is, which strategy should I employ to solve this problem? Because this is something the brain does effortlessly. The strategies it employs to understand conversation are different than planning a trip, than, you know, making sure you don’t fall down, versus, like, long-range planning. Like, how do I choose the best career, right? So, all of these are very different problems. And somehow your brain kind of picks a good strategy for each one. Pedro Domingos talks about this as, sort of, you know, in his book, “The Master Algorithm,” which is — we know that there’s all of these different techniques, but kind of what we’re missing is sort of the synthesizer. The thing that we’ll know…

Sonal: Yes, putting it all together.

Frank: …what strategy should I pick to solve this problem? And it’s something that the brain seems to just do.

Steven: What’s interesting about that observation is, that’s where we are with machine learning today to begin with.

Sonal: Which is?

Steven: Which is just, like, there are all of these different networks that you can model, with so many different layers and how many parameters you want to use. And there’s, like, this art to it right now. And I find that particularly interesting, because anytime there’s an art to something, there’s an opportunity to start a company around that art and to build out a product that surpasses, you know, the best practices in whatever field you’re going after, whether it’s analyzing traffic or figuring out how to drive a car. And so, that is the opportunity, because I don’t think, you know — there’s not, like, some path where in the next two years, there’s the meta-algorithm that knows what way to pick, like, that’s — so, the best thing to do now is to become well versed in all of them.

And my gut just tells me that anytime you’re at a point like this, the most interesting solutions are actually going to end up being a hybrid of, like, the thing that used to work, that everybody said doesn’t work well enough, plus the new thing that everybody says is gonna replace the old thing. And that’s basically been the entire history of AI.

Sonal: Computing in general.

Steven: Well, in computing, for sure, but AI specifically, which is, like, everybody always says, “The new thing — that’s it, we finally got it. Huge step, the old stuff is all done. And the best example for me of that is how everybody said machine learning was going to replace all of natural language processing. But if you dig into any of the work that’s been going on, even the most state-of-the-art translation, which, you know, goes any language pair to any language pair — well, the input and the output all rely on the old school, like, from the 1970s, natural language stuff, just to do some very basic bookkeeping, very basic stuff.

Sonal: Right. You talk about, like, building a spell checker, whatever it is.

Steven: I mean, all the…

Frank: Parts of speech finder.

Steven: Right, and all the image stuff. Like, wow, you know, if you want to figure out features, it turns out, when you start doing features of image recognition, you’re using a whole bunch of old school edge detection and contrast and finding objects and all the stuff. And so, you can’t just, like, show up and say, “Now, we’re going to understand — we’re going to be the unsupervised learning company.” Because the question I was gonna ask is, well, how are you going to make whatever you’re doing practical?

Human interactions with AI/ML

Sonal: What I do remember about hearing the stories of the early days of NLP, and observing parts of this firsthand as well, is how the entire field and community — a lot of them had very strong opinions about, you know, there’s this whole phase of like, expert domain knowledge building. And really, that’s the only way to actually make NLP work at scale. There were all these things they had to do because it was before the days of big data. They couldn’t even conceive of the Google scale big data. And then, they went to this world where, “Oh, my God, we don’t even have to have these kinds of constraints and ways of doing things, because we have all this data.” And now, it’s sexy to me that you can flip that model again and almost say, “You don’t even need big data with the results like this kind of paper because you don’t have to have anything.”

Frank: This is the revenge of the algorithm’s movement, which is, it’s always been about the data. If we had more data, we’d have more accurate models. And what this experiment showed is, “Look, all I needed was the rules and a really good loss function, some very clever programming, and I get better performance than I had when I had the data set.” Right? So, that’s the tantalizing, “Oh, look, you could do it without data.” And like, look, as an investor, that’d be awesome. If I could fund AI experiments without this step of — collect data, annotate data, train model — we could run a ton more experiments

Sonal: Exactly. Because it’s age of abundance. You brought up Pedro Domingos, and it was funny because one of the comments he made on the heels of this announcement, which I thought was quite interesting, is, like, “Well, you know, AlphaGo Zero learned after, like, 5 million games, humans took only so many thousands of years.”

Steven: I love that.

Sonal: And I’m sort of like, “That’s the whole point of computers, is to learn it in three days.” You know, time is money. I don’t care if it took, like, you know, 5 million games, it took three days. But time is more important than amount.

Steven: You know, again, it’s an amazing accomplishment. It’s just, I always try to look at these in the context of how these innovations tend to happen. You know, if you just look, the same thing happened with search, like search itself. When people were inventing all the techniques of search, they were working on these tiny machines with all of these physics constraints about how much they could compute. And like, it was a whole thesis, just to be able to go get, like, the archives of “The New York Times” to search through it.

And the same thing happened with spell checkers. Like, wow, we’d love to have 50,000 words in the spell checker, but we don’t have that much memory. And then, there was a whole — well, now we have a lot of memory, and so now we’re spending all this money to try to compile all the words. And then, someone said, “Why don’t we just use all of the internet as the spelling dictionary. Then there’s no spelling dictionary, or the IME if it’s an Asian language. And so, this curve just keeps repeating itself. And I think that it — neither is going to win out, because data is always going to be valuable, because it’s an input that you can’t just say…

Sonal: Yeah, it’s a great way of thinking about it.

Steven: …whereas, the flipside is, not everything has data. So, if there’s this opportunity — like, you know, a great example is, like, how many of your Lyft drivers also have Waze turned on, just because they know the maps are entirely accurate, but they don’t know about accidents, or emergency closures, or weather, or whatever. And so having that data, combined with the data plus algorithms, that hybrid…

Sonal: Is always gonna win, right.

Steven: …is gonna continue to win, because only if you really want to be practical and actually solve the problem.

Sonal: I have to say one thing about this, which — it’s so fascinating that you use the example of search, and you describe this curve that we’re on. And part of startups is, sort of, you know, getting at the right place and point in time along that curve and where you are in the moment. Because sometimes, I think a lot of academics, or even people who are really in love with their own ideas, sometimes lose a big picture. Like they’ve had this build up, this expertise in one area. And they don’t realize, like, practically speaking, the world has changed around you. Because the point that I think is fascinating, as well, in the innovation story, using the search example, is that Google was, like, the 15th search company to come around before it hit success. And that is kind of relevant to think about.

Steven: Right. And it’s super important too, from the company building perspective, which is — they have this algorithm, which we all talked about back then. PageRank, you know, appropriately named but not…

Frank: After Larry.

Sonal: Or web page ranking as well.

Steven: Right, right. But the interesting thing was very rarely is, like, an algorithm, like, this secret sauce for a company. Because you could look at what goes on from the outside and pretty much reverse engineer an algorithm. So, again, back to — but if you have, like, a data source that you can actually…

Sonal: And a monetization model.

Steven: And turns out, one of the things that Google built out — was, like — they were crawling the web faster than AltaVista. They made this bet because they were machine learning people before machine learning was cool. They made the bet that having the data was going to be — and, it turns out, that was the barrier to entering the search market. Even for Microsoft and Bing [it] was, like, sucking in the entire internet fast enough.

Frank: Step one, crawl the inner web.

Steven: Step one, crawl the internet while it’s growing at exponential rates. And so, you know, I actually want to bring up one more thing that I just think is, kind of, really interesting from a practical point of view.

Sonal: I love it, you’re like the practical person on this podcast.

Steven: I know. I feel, like, particularly practical today because, well, I can’t play Go, so I gotta…

Sonal: You got to add something. I’m just kidding.

Steven: Which is, the thing that I find the most fascinating about all of these solutions in the space, is the engineering of a product — that you can make a commitment to customers that works. And then, when it doesn’t work, you can figure out why. And so, one of the things that’s so interesting about all of this is debugging.

Sonal: Hmm, interesting.

Steven: And how does that really fit in? And sure, you know, with Go, you lost the game. And of course, while they’re building all of this, they’re figuring out, “Whoa, what did we do wrong to make this move repeatedly?” They’re doing all of that debugging over the past N months. But, you know, if you just all of a sudden apply this to the enterprise space, or to adjusting news feeds, or a zillion other things that you can think of — like, figuring out where it goes wrong, like, that’s actually really critical to a business. Like, you can’t put a product out there if — pick our sales forecasting example, and then it’s wrong, like, you can’t just go, “Whew, the machines make mistakes just like people.” Your VP of sales is kinda messed up too. Because nobody pays money to, like, a computer for it to be wrong. And so how do we think about that?

Frank: Yeah, this whole idea is, sort of, transparency behind these models. In other words, do we know why they’re behaving the way they’re behaving?

Sonal: Demystifying the black box.

Frank: You know, a super active area of research right now, right? Which is, how do I make the deep learning models more transparent so that I can debug them, I can verify them, I can make sure there’s no systematic bias in them, right? Because until that, you couldn’t do important things like, “Hey, can this person have a loan or not?” Because the government will say you cannot make that decision unless we understand why it is that you’re making that decision.

Sonal: But you’ve made the argument, Frank, that, you know, when it comes to, say, self-driving cars, we kind of — it’s no better or different than what the human mind does. We don’t know we can’t interrogate the black box of the human mind that’s driving that car. So, the counterpoint of that is that, well, you’re right, Steven, that, you know, you’re paying for this computer to be smarter. But the reality also is, this stuff is not that smarter than humans anyway, so who cares?

Steven: Right. But the problem is first, that’s everybody else, not me. Yeah, like, I’m the best driver on the road. It’s all the other people. I mean, you know, I like to always go back to this — the wonderful, wonderful research on computers and society that Stanford professors Nass and Reeves did, because one of the things that they really realized, really back in the early ’80s…

Sonal: Oh, Clifford Nass. Oh, all right, rest in peace.

Frank: And Byron Reeves.

Steven: And Byron Reeves, which was that there’s something about a mechanical device that produces answers that makes the human brain ascribe way more authority to it than there necessarily should be.

Sonal: In fact, they even apply this to the world of voice recognition systems and the interaction.

Steven: They applied it to voice recognition, they applied it to chatbots. Before they were chatbots. I mean, so the interesting thing is that I don’t know how — it’s going to take a major change in society for people collectively to just go, “Ah, people make mistakes.” And that’s okay. But machines can’t. Like, we have, especially in the United States, a very, very low tolerance for devices making mistakes.

Sonal: Except when machines are doing a kind of alien intelligence that humans cannot do. Because, again, the most fascinating thing about our last talk about AlphaGo, this current talk — is that, at the end of the day, even though it opened — the AlphaGo Zero opened and closed with similar moves to what humans would do, it converged very quickly — there was a whole set of things in the middle that it did that were just things that humans would never have done. It’s actually, then, augmenting us in a very different way, because it’s adding a completely foreign intelligence. So, it’s not even comparable to our own to just judge…

Frank: Yeah, it’s a different type of intelligence, in the same way that animals have a different type of intelligence. And then, you make all kinds of category errors when you say giraffes aren’t intelligent, or bats aren’t intelligent, they’re just intelligent in a different way.

Sonal: Exactly. And we’re more for…

Frank: We can’t use the human yardstick to compare them.

Sonal: It’s the reason we are more forgiving when a dog pees all over your couch and your five-year old kid does the same freaking thing.

Steven: That’s definitely the case. The challenge that we have in technology is, just, the perception. Like, I mean, you can just see it in the discussion of news feeds, and algorithms, and how people are, like, they should just be right. I mean, like, and people should just have debugged this before it all happened. And actually, it’s not even all that crazy sophisticated, what’s going on. And, what’s weird is, of course, at the same time people are critical of what’s on the front page of newspapers, or what’s in the first five minutes of TV newscast — which is literally the same decision made by a human being, who’s just deciding what should we show first on CNN versus Fox News. Some human just made up their — with some black box in their brain, which is augmented by their title of, “in charge of production.” And yet infinitely forgiving of those choices. And so, I actually think that there’s a lot here. And I don’t have an answer, but I think it’s no different than any other software. Which is, if you’re going to make something and offer it to people in a commerce situation…

Sonal: It better damn work. And it better not be wrong.

Steven: It better be clear, like, how it does it to you. I mean, like — people, like — Excel was really great at doing math very quickly. And then, one day, I find myself at the Naval War College having to explain myself to a bunch of generals. Like, how do we know it’s right. I literally just had to sit there going, “I mean, it’s just right.”

Sonal: It just works.

Steven: And those are the moments that contribute to me feeling, like, a lot of empathy for what’s going on in the marketplace now about this, and why I’m so alarmed — not alarmed, that’s totally the wrong word — so focused on this, you know, know the outcome. Because, like, until you’ve just sat there with a bunch of people who control nuclear weapons, telling you we’re using a spreadsheet to calculate it, the weight of being right doesn’t really hit you. Because before that, we were just like, “Oh, my God, it doesn’t seem to make mistakes.” Then we were, like, super ecstatic. And it made, “Look, the charts are cool. Yay, ship it.” And then one day, they’re like, “Is it gonna work or not work?” You’re like, we couldn’t prove it. Like, that was the essence of it. Like, I couldn’t go to people at Boeing, or people at the Navy or wherever, and Wall Street, and prove that Excel works.

And then, of course, 30 years later, like, every time there’s a mistake in Excel, like, it’s a mistake in the human that typed it. Like, Ken Rogoff did a bunch of economic predictions that forecast the recession in 2008. And then, all of a sudden you find out his model was wrong, but the recession happened anyway. Did he make the right prediction or not the right prediction? And, like, how does that work? And I think that’s what’s going on here too, with these things.

Potential bias in algorithms

Sonal: I have a philosophical question, then, for you guys.

Steven: That wasn’t philosophical enough.

Sonal: Well, one more philosophical. We’re being very philosophical here. You know, you said something about how people make judgment calls for what news to show on television, etc. And we have these expectations about algorithms. And one of the topics we’ve discussed on this podcast — Frank, you and I discussed with Fei Fei — is this idea of bias in algorithms, and how algorithms, by definition, can be biased — is one possibility of this type of work. Because they kept using the phrase tabula rasa, which of course I find so fascinating, because in human development, there’s an analogous world of this, where there was this theory that the human brain was also a tabula rasa or blank slate. And then they quickly learned, like, “No, we have millions of years of evolution and DNA, that’s actually — [we’re] actually coming in inheriting things.” Is there now a possibility that algorithms can write themselves, [in] a true tabula rasa-like way, given this type of work? Is that just way out there?

Steven: Oh, that one’s above my paygrade.

Sonal: These guys are throwing up their hands for our listeners, by the way.

Frank: I will give an example of, sort of, the things that are hardwired in your DNA. So, they’ve done a bunch of experiments that if you watch somebody’s hand getting pricked, your muscles in your own hand will involuntarily contract. Now, the big caveat is, if that person has a different skin color than you, your hand doesn’t contract.

Sonal: Really? I didn’t know that.

Frank: In other words, there’s something going on in your lizard brain that says, “That’s an ‘other’s’ hand.”

Sonal: Primal

Frank: Therefore, it’s not relevant to me. Therefore, whatever reflexes caused your hands to contract in sympathy, when somebody jammed a needle into that hand, that’s an example of this, sort of, prewiring. You know, the question is, can algorithms be prewired that way? I mean, you could…

Sonal: Or un-prewired even.

Frank: Or un-prewired.

Sonal: Because that to me is where the opportunity lies.

Frank: I mean, you could definitely write a set of rules that said, “You know, treat other people in a different way,” right? That would be top down. The peril that most people talk about today is not so much that the algorithms are biased, but you’ve fed it not enough data so that your prediction is biased. So, the classic example of this is, in the early days of vision recognition. Some of the image classification algorithms were categorizing people with dark skins as gorillas, that happened because they didn’t feed it enough data of dark-skinned people. So, when people talk about bias in algorithms, they are mostly talking about this phenomenon…

Sonal: Limited by data.

Frank: The human researcher or the human programmer selected an incomplete data set, and therefore you got biased results, as opposed to somehow the architecture of the neural network is biased inherently.

Steven: And I think that that’s a very important point that is being well studied, it’s well articulated. But particularly in the supervised learning case, the data that’s input, at least today — almost every data set that you have is going to be and have some inherent bias, because you weren’t aware of these factors when it was being collected in the first place. Because I think it’s fair to say, the awareness of all of these issues is at an all-time high relative to that. But then again, you look at all the medical studies, and you’re like, “Well, there haven’t been very many women in most of these studies, or there have been only women but studying a drug developed by…”

Sonal: That is true in biology and genetics research too. There’s a lot of limitations. Right.

Steven: But I do think that anybody today embarking on using supervised learning, whether for all or part of the solution — that data set is implicitly challenged, and in particular, it’s the labels. Because even though which labels you pick or which labels you omit, it’s going to create some bias in the model that you’re unaware of.

Frank: That’s exactly right. Your cultural background — your history, the way you grew up — will lead you to label an image a certain way. That may be different than somebody who — right? So what is the ground truth?

Steven: Well, the best example of this or just looking at images and, like, sentiment analysis is such a big thing. But images — like, facial expressions — have just been studied all around the world for decades, multiple decades, about, like, what is happy, what is sad, what is questionable? The same with speech and intonation, like, sometimes, you know, ending on a high note sounds like you’re asking a question, unless you’re in another culture where that’s making an exclamation.

Sonal: Or it can be vocal fry, and people are complaining that women shouldn’t speak that way.

Steven: Well, the vocal fry thing is a perfect example of that going on right now. And actually, the high note was the — sort of the ’80s version of that same thing.

Sonal: It goes in Vogue, too, right?

Steven: And be super careful about the labels. And that’s — to Frank’s point, that’s a great place for transparency. Because if you can go to a future or a potential customer and talk about, “This is what our model is based on when we’re forecasting your sales,” or “telling you to optimize your assembly line.” That could really matter to that process.

Advice for entrepreneurs

Sonal: So, high level takeaways. Hybrid works. Always — this is a thing — this is, like, a refrain. I feel like I should get you a T-shirt that says, “Hybrid works, God damn it.”

Steven: Well, be careful, because I’m really against a lot of hybrids.

Sonal: Oh, yeah. Like, hybrid cloud computing.

Steven: Yeah, hybrid in terms of solutions between old and new for coding. The old stuff that works definitely does work.

Sonal: Old and new, academic and industrial, like, exactly — all the things that make it practical versus pure, so to speak, purist.

Frank: I think another big takeaway of this is this “revenge of the algorithms” moment. Where there’s so much momentum right now that says, basically, we’re one labelled data set away from glory. Right? And this result basically shows you, “Wow, there’s a lot of mileage that you can get out of reinforcement learning, where there’s no data, no labels.”

Sonal: One take away from me is just the element of surprise, because I’ve been thinking a lot, even just, you know, how — I mean, I used to work in the world of how humans learn. But what’s fascinating to me is that the system played itself at a level calibrated to itself at every level. And in human learning, that doesn’t happen. You’re taught by your parents, you learn from, you know, adults, you learn from people who have more experience. There’s all these different things that happen.

And the thing that’s so fascinating to me, is that to me, it is very evolution-like. It’s like a big bang moment. And while I wouldn’t hype and say that means we’re going to end up here, I do think that’s very amazing, especially when you think about the surprises. Like, in the paper, one of the things they talk about is that the system learned something that’s actually very easy for humans to learn, way later in the game. It’s actually, I think, like, I forgot — shicho, or some specific move to, like, the ladder. The fact that the system took forever to learn something that’s very first for humans, I just — I’m endlessly fascinated by this relationship between humans and computers, and what we can learn from computers and vice versa. And also, what it means for the field, when you can actually add to how people learn to the field of artificial intelligence, machine learning.

Frank: Yeah, with reinforcement learning, one of the challenges has been, sometimes you get into this training epic where there’s no improvement, right? Because you’re just playing these games over and over again. And, like, what is it that causes the next game to be smarter? And this one was, like, in three days, it got incredibly smart. So, yeah, hats off to these guys.

Sonal: Yes. And let’s also add the other takeaway, which is that you can do amazing things with simple architectures, because we’re at a point and a moment right now where you can have 4 TPUs instead of 48.

Steven: For me, the one of the takeaways is, if you can frame your problem with a set of constraints and a set of rules and a fixed set of operations, that’s a very powerful concept that changes — because we’ve been so data centric, people have stopped trying to think of their solutions algorithmically. And it’s entirely possible that there is an algorithm, which is the same thing that I think we would have said a year ago, which is — everybody was so focused on machine learning that traditional algorithmic approaches might have been overlooked.

Sonal: And now…

Steven: And here we are again with proof. But still, now we have to go back and say, “Well, you know, there’s some basic machine learning that can work. There’s some basic algorithmic stuff. But that key is really the set of constraints.”

Frank: Yeah. For people who are interested, I highly recommend Andrej Kaparthy’s YC talk on this. You can search for it. It’s on Google Slides.

Sonal: He works at Google, right?

Frank: The whole talk is basically about where will artificial general intelligence come from? And he basically compares and contrasts this, sort of, rule-based world of Go, to something messy — like how would you build a pick and pack robot for Amazon? And so, if you’re interested in this topic of what’s the difference between rule-based board games, versus the messy real world, it’s a great presentation…

Sonal: That’s great for the generalized intelligence side. Are there any parting messages for entrepreneurs building companies that have very specialized things? Because one thing we have argued in this podcast, including you, Steven, when we did a podcast on building product with machine learning — is that sometimes the best places to play are very specific domain focus, whether or not they have a clearly explicit set of rules or not. So, any thoughts on that?

Frank: Yeah, so one takeaway is, one of the ways we evaluate startups from an investor’s point of view is, have you picked the best techniques to solve the business problem that you’re trying to solve? And I think this paper basically opens up a frontier of exploration that you might not have thought about before. Because, I think if you started an AI-centric company today, you would definitely be on the “get data, get labels, train model, make prediction” path. And this opens up another area you can try and figure out, “Am I solving a business problem where this supervised learning approach is gonna lead me to glory, instead of these — or, this reinforcement learning approach — is going to lead me to glory, as opposed to supervised learning or unsupervised learning?”

Steven: I think [that] the most interesting thing for me in looking at different companies and just talking to founders is just that there’s this world going on of advances and new things all the time. And you could get on the treadmill of always trying to be the newest thing. And, you know, we knew that the next generation of companies, you know, two years ago, we’re just gonna be, you know, whatever you were doing before plus AI. But the thing was, it wasn’t clear that that was always the best solution. And then, you replaced the AI with “now it has to be data and machine learning and labeling.” And so what Frank was saying is just super important to internalize, which is that part of being a founder, and building a new product is knowing that the reason why you’re choosing the technologies you’re choosing — and not just because you think that’s what investors are looking for. Our jobs, on that side, are to actually ferret out, like, who actually has a handle on the problem they’re solving, and a line that they can draw from that problem to their chosen solution path. And that is too often overlooked.

Frank: That’s exactly right. My favorite joke that I tell is — when I ask entrepreneurs, what machine learning algorithms they’re using in their product, when they’ve claimed to be a machine-learning-based startup. The answer I am not looking for is, “I don’t know. I’m sure we’re using the good ones.”

Sonal: Well, on that note, you guys, thank you for joining the “a16z Podcast.”

Steven: Thank you.

• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter
• 

  Frank Chen is an operating partner at a16z where he oversees the Talent x Opportunity Initiative. Prior to TxO, Frank ran the deal and research team at the firm.

  Follow Twitter
• 

  Steven Sinofsky is a board partner at a16z. Prior, he spent 22 years at Microsoft in a variety of positions including leading the Windows division.

  Follow Twitter Website

Beyond the overly simplistic framing of trade as “good” or “bad” — by politicians, by Econ 101 — why is the topic of trade (or rather, economies and people adjusting to trade) so damn hard? A big part of it has to do with not seeing the human side of trade, let alone the big picture across time and place… as is true for many tech innovations, too.

Speaking of: how does the concept of “trade” fit with “innovation”, exactly? They’re both about getting more from less — as well as creating new opportunities — shares Russ Roberts, host of the popular EconTalk podcast (and fellow at Stanford University’s Hoover Institution, PhD in economics). But there’s another very provocative theory at play here — fast-forwarding us from the time of the Industrial Revolution to the 2000s — that could make us rethink the relationship between trade, capital, labor, productivity/economic growth, shares Noah Smith, columnist at Bloomberg View (and former professor of finance at Stony Brook University, PhD in economics).

And where does China come in — and out — of this picture? Put it all together, and maybe, just maybe, it could help explain why we’re investing in labor-saving innovations/ automation more than ever today. Because one thing is for sure, agree both Roberts and Smith — who otherwise argue with each other on this episode of the a16z Podcast (with Sonal Chokshi) — you can’t stop the march of technology. It’s here, it’s coming, and we’re just going to have to meet it, prepare for it, …roll with it.

Show Notes

• Discussion of the history of trade, including the Industrial Revolution [0:44]
• How trade and technological innovation are linked [9:28]
• The impact of trade between rich and poor countries [16:53], and how trade with China affected both economies [26:13]
• Trade deficits, and a debate over their effects [31:17]

Transcript

Sonal: Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal. Today, we’re talking about all things trade and innovation. And it’s a really meaty topic. So we begin by talking about why, beyond the obvious, the topic is so hard, what trade is responsible for when it comes to tech change and jobs. And we cover everything from some really interesting theories of the past, and whether they matter for the present and the future, to the debates around productivity, manufacturing, China, protectionism, and more. So a hell of a lot.

To have this conversation, we have two special guests. Russ Roberts, host of the longstanding and excellent “EconTalk Podcast” and research fellow at Stanford University’s Hoover Institution. And Noah Smith, who did his Ph.D in economics at the University of Michigan, was a finance professor at Stony Brook University, and is now a columnist at “Bloomberg View.” And he’s the first voice you’ll hear.

Causes of the Industrial Revolution

Noah: We’re really talking about the part of trade they don’t teach in Econ 101. We’re talking about the adjustment to trade. And in Econ 101, there’s, pretty much, this unstated assumption that economies adjust very frictionlessly and very easily, and everything goes very smoothly. To be honest, it’s really not easy to tell when that will and won’t happen. And there’s no general theory that economists have about adjustment. Economists often think to the very end and think about equilibrium, and ignore the path for how you get there. But, you know, that can be a little bit like saying, like, World War II is great because it brought us human rights. But the road to get to the UN Human Rights Convention was a little bumpy and…

Sonal: Yeah, the reality is very messy in that way.

Russ: The only thing I would add to that. I think a good Econ 101 teacher talks about the adjustment. I think it’s a bad teacher and a bad economist who says, “Well, trade creates net benefits, so therefore it’s good for the country.” Trade is good on average for the Americans, and it obviously has big distributional effects depending on who you are, what you do. There could be long-term distributional effects, that your skill is no longer valuable. And there could be a short-term adjustment which is, it’s no longer valuable. You’re going to have to find something new to do and it may take you a bit of time to find it. And it takes a very long time.

Noah: Right. Or you may never do it and you may just retire, go on welfare, and then someone younger may then find something better when they get out of school.

Russ: Correct.

Sonal: I don’t want to go into Econ 101, but I’d love us to actually have some building blocks for the conversation, simply because there’s a lack of economics education. If you guys were to, sort of, launch off with what we need to know about trade, like, what would be some of the big ideas that we should start with?

Russ: Well, trade as a political issue goes back shockingly long. I’d say it goes back probably about 800 years.

Sonal: Why not longer?

Russ: Well, around 800 or 900 years ago.

Sonal: I’m pretty sure people have been trading for centuries.

Noah: There was probably someone in Egypt saying, like, “Oh my God, this Mesopotamian grain is going to put us all out of business.”

Sonal: Yeah.

Russ: For sure. But the issue was whether trade was good for the country or not. It’s still the issue. But around 1776, Adam Smith came along and said, “We’ve made some fundamental mistakes in mercantilism.” Mercantilism was this idea that trade could harm you because you’d lose your gold. And Smith pointed out gold is not something you can eat, it’s not really the measure of a nation’s well-being or people’s well-being. And he made the case for the benefits of trade. 

And if we come to the modern times, I think most economists would say trade is important because it allows us to get more from less, just like innovation does. It’s a way to increase productivity. So instead of making things for yourself, you trade for the things that you don’t make as well as somebody else, as cheaply as somebody else. So…

Sonal: There’s actually a really good anecdote that I saw about this on Twitter recently. And I don’t know if it happened recently or not. But, basically, some guy in the last few years decided to make a sandwich from scratch. And it was everything from growing the vegetables, to milking the cow, to flying to the ocean to make salt. And, by the way, he didn’t even include that flight in this price or it would have been more, but it cost him $1,500. You know, that’s way more expensive than, like, $7 which is the average price of a sandwich.

Russ: If you tried to make everything for yourself, true self-sufficiency, you’d probably starve to death. In modern times, though, we worry about, and correctly so, that some of the people who benefit from trade are not necessarily going to be the same people who bear the costs of trade. We make something abroad, it means we’re not buying it here domestically anymore, and those people are going to have to find new ways of working. Trade makes things inexpensive for everybody. That’s great. It also makes it harder for some people to use their skills. The part that I always like to emphasize that people forget, is that when you make things cheaper for everybody, that frees up resources to make new things you wouldn’t have had before. And the last thing, I would argue, is that the biggest benefits from trade take place over longish periods of time. The opportunity for the next generation to come along and use its skills well. But it can be hard on this generation.

Sonal: The one thing economists seem to almost universally agree on is this — I mean, there’s never any universals with economists. But, like, this idea that it’s actually in your best interest as a state, country, whatever, to open up your borders to trade. Politics aside, it seems to be one of the fundamental beliefs.

Noah: Just because economists, you know, express agreement on something does not mean it’s right. But in this case, I think that, broadly, the economists’ consensus is fairly correct. But then there are some important caveats that get lost when we talk about that. And one of the things I think is important to emphasize is our ignorance. We understand some basic things about trade, and then there are probably some things about trade we don’t understand at all. So for example, if you look in history, a lot of people wonder about how trade affected the industrial revolution. As Russ said, trade can look a lot like innovation. In fact, in economic models, the gains from trade will look just like productivity, and there’s a complete equivalence between those in a lot of models. So, basically, if you have a robot to do something cheaply, you know, from the consumer’s perspective, that’s exactly the same as getting a low-paid worker in China to do it cheaply, so trade can be like innovation. But when we look at the really big epic booms in innovation, like the industrial revolution, trade may have some more complex effects on that.

Sonal: Yeah, we actually had economist Joel Mokyr on the podcast talking about the industrial revolution, but it was more about the open trade routine ideas. And how that led to what he called an unprecedented quantum leap in economic and tech progress. And included things like the steam engine and automating fiber looms.

Noah: Well, one of the big theories about the industrial revolution is that a combination of expensive labor in Britain and the Netherlands, and cheap resources from the Western hemisphere — the so-called New World — combined to make it suddenly worth entrepreneurs’ while to start using machine labor instead of humans. And the Romans had all the technology for steam engines, they even did some mining with steam. A Roman inventor had a little steam engine on his desk called the aeolipile. But they never really deployed it en masse. And China had all the technology for power looms and just didn’t…

Sonal: Yeah.

Noah: …put it together and start using it.

Sonal: Even in India too. And so these things go way back. So your point is that this wealth of the New World…

Noah: Right, the cheap capital, essentially, from the New World — cheap resources and capital, combined with expensive labor in Britain and the Netherlands. There is this theory that that’s what sparked the industrial revolution, by causing business owners to finally say, “All right. You know what? Forget you, humans, we’re just going to use a bunch of machines.” And then suddenly they said, “Wait a second, these machines are really good. Maybe we could make slightly better machines.” And tinkered and tinkered. And then you got, suddenly, on this track of continuous improvement and continuous searching for new industrial technologies to make people richer. And, obviously, that is the greatest episode of human flourishing in the world. By far the biggest event. Probably bigger than the agricultural revolution.

Russ: You can almost look at that as we flipped a switch and something just…

Noah: Something happens.

Russ: A lot of things exploded. Our lives are remarkably easier, safer, longer, higher quality in so many dimensions — not just monetary ones, and it’s always important to add that. And yet at the time, of course, the political economy of that revolution made Dickens famous. Gave Dickens something to write about — the costs of that in human toll and human lives — misery, in the short run, was very strong. A lot of people were put out of work by those machines, a lot of people had low wages as a result of those machines. And a lot of people worked with those machines and didn’t make a lot of money, and had a very dispiriting existence. Charlie Chaplin’s “Modern Times” being the best pop culture version of what that experience was like. The artisans’ life, the craftsmanship of previous days, and the meaningfulness that people got from their work suddenly changed. And the point I want to emphasize is that, that response — the anger and fear that that engendered — was very natural, very real, very justified at that point. And I think it’s important, though, to look ahead to future generations and say, even though they were very unhappy at the time and wanted it stopped, those innovations, are they happy now with the way the story turned out? That their grandchildren or great-grandchildren…

Sonal: That they could see. Fast forward…

Russ: …had these remarkable lives.

Sonal: …into the future. I actually remember you talking about this in “The Human Side of Trade,” where, like, 40% of the workforce in the 1900s was in farming, and now it’s, like, 20%. So if you go back in time, that farmer would have been really crying and sad about the lack of opportunity for himself and his kids. But if you brought that farmer into the present, he’d actually cry in happiness, to see his great grandkids have so much more than he did or they did when they were going through the really hard times. But it’s obviously really hard to see that big picture right now.

Russ: It’s hard to do that. And I think it’s important to do that, because economic change is dynamic by definition, and it’s lively, and it’s hard to know what’s going to happen. But often, the longer run impact is very different than the immediate impact.

Trade and innovation

Noah: Absolutely. Not just [in the] long run, but I think it’s important to emphasize the nonlinear impact. The fact that sometimes small things can have big consequences. But now, back to the theory. So, one theory of the industrial revolution is that it happened because, basically, business people started replacing overpriced British and Dutch workers with robots.

Sonal: Yeah.

Noah: The robots of the day. And the question is, what if, in a much smaller way, in the 2000s, we started replacing robots with cheap workers in China? What if we were going the opposite direction? And what if trade can temporarily, under certain circumstances, have a slowing effect on innovation? I mean, it’s obviously not going to stop the industrial revolution from happening, or something like that. We’re not talking about effects that big anymore. But what if trading with, you know, the New World, so to speak, with all its abundant resources, made capital really cheap and caused a tech boom? And what if the, sort of, productivity slowdown that we saw starting in the early to mid-2000s, actually — what if part of that came from relying on outsourced labor in that boom, instead of on better machines?

Sonal: Why is that idea important? Like, what’s the natural implication of that?

Noah: The natural implication of this is that, basically, the more cheap labor you get, the less incentive there is for labor-saving innovation, for entrepreneurs and technologists to invest in labor saving innovation. And I don’t necessarily endorse this theory. It is an idea.

Russ: It’s a clever idea. And I think there’s some truth to it. The reason I think it’s an important idea is that it suggests an explanation for the productivity slowdown that isn’t as alarming as the view that says, “We’ve figured everything out that we can, and everything is going to get worse from here on out.”

Sonal: Yeah. Like the Robert Gordon view of things. Fall of growth, death of innovation.

Noah: Maybe we took a one-decade pause in automation. And maybe now that wages have risen so much in China, and things like that, maybe now we’re going to get back to the task of automation.

Sonal: Innovating. In this conversation, it’s basically machine improvements to save labor. That’s your definition of innovation in this context.

Noah: Right. If it doesn’t save us labor, why are we doing it?

Russ: Right. You would think that the opportunity to use inexpensive labor outside of the United States — a lot of it is from China, but it’s also lots of other places — should still lead to gains to Americans in the form of lower prices through the innovation that’s taking place. So in a way, if you think about what’s the difference between China closing its borders, and we never had this opportunity to trade with China. You’re suggesting we probably would have invested more in capital here that would have raised workers’ productivity here. Ironically, what happened instead, is that workers, you’re saying, got less access to capital, therefore didn’t get those productivity gains. But we, as consumers, still should have gotten the benefits of that in the form of the lower price, because that shouldn’t affect it?

Noah: Exactly. And so, really, under some pretty simple conditions, you can show that as China opens itself up and we choose to outsource to China, the overall aggregate short-term gains will be bigger than they would’ve been if we had done the hard work of trying to invest in new machines. But, if you think about the long-term effects — so that’s talking about what we call, you know, static efficiency or short-term effects. You can show pretty easily that that’s going to be bigger. And that’s what Econ 101 always talks about, static efficiency. When I’m talking about…

Sonal: Wait, what’s static efficiency, exactly?

Noah: Static efficiency just means, like, how much cheap stuff are we getting today? How much productivity are we getting today?

Russ: How much is the pie bigger than…

Noah: How much is the pie bigger?

Russ: …than it could be if we’d not had that opportunity?

Noah: Exactly. The pie today, when we think back to the industrial revolution and the, sort of, long-term consequences. If you buy this theory, then for hundreds of years, entrepreneurs were doing the easy thing of getting a whole bunch of cheap medieval laborers to run their printing presses, and looms, and mines, and whatnot. And they were maximizing their static efficiency, their short-term productivity. But, because they didn’t do the hard work of invention and innovation, they didn’t get those knock-on effects further down the road. And when they finally said, “Okay. Oh, my God. Workers are so expensive we’re going to just spring for a steam engine.” Maybe in the short term, that cost them more. And so they wouldn’t have done it if they hadn’t had to. But in the long-term, that opened up whole new veins of innovation.

Sonal: There’s actually a concrete example that just jumps into my mind about this right now playing out today, which is India’s decision, or discussion around not having driverless cars, because they don’t want to displace the labor that can — they don’t want to take people’s jobs away, essentially. And then my initial reaction is exactly what you’re talking about, which is — but they’re doing this at the cost of then investing in these labor-saving long-term innovations.

Russ: That’s a fantastic example, because as Benedict Evans points out in an “EconTalk” podcast, one of the impacts of, you know, autonomous car introduction is a dramatic reduction in the cost of getting from here to there.

Sonal: Yeah.

Russ: So if you think about a current cab ride. A current cab ride or Uber has a huge labor component, and it also has an insurance component. But the cost of getting from here to there is going to go down. And that means that more people are going to go from here to there. Which means there’s going to be more opportunity and more resources available to do new things. So the worry is, and this is the essential trade political question. The worry is, are those new things going to be useful in employing the people who used to be the drivers of the cabs and the Uber’s? And that is the real question.

I think we’ve politicized trade dramatically in the last few years. Is it just for the benefit [of] corporations, of the top 1%? And I think that’s totally wrong. However, there is a real issue, which is, will the people whose skills have been employed productively in, say, the cab or truck driving business — will they be fruitfully employed, beneficially employed — in whatever comes along? And new things will come along. That, I’m very confident about. What I’m not so confident about, and am concerned about, as, like, everyone should be, is whether the skills of the people who are displaced are going to be available.

Sonal: So we’ve morphed into a discussion now or debate around tech and jobs.

Russ: But it’s the exact same discussion that people should have about trade with China and displacing manufacturing jobs in the United States, or in innovation like the autonomous car displacing drivers.

Sonal: Is it the same as the debates that have played out since the beginning of time, or is it actually different now because technology is accelerating faster? Because a big part of this is the adjustment time.

Noah: The debates are always the same. That doesn’t mean that the underlying truth is the same. And sometimes we confuse this. So for example, anti-war movements were making broadly the same arguments during the Iraq war and during World War II. But those were obviously very different wars.

Sonal: That’s a good analogy. So how does that play out in this question of trade?

Noah: The people who were scared of trade are scared because of things that very immediately affect them.

Sonal: Right.

Noah: When you hear economic debates, almost everyone is talking about what immediately affects them — what they see in their own job, in their own industry.

Sonal: Their day-to-day livelihood or their children. Maybe one generation out at the most.

Noah: Exactly.

Sonal: Yeah.

Noah: And so you see people wondering, “Will I lose my job? Will my business face…”

Sonal: But that’s old.

Noah: “…increased competition?”

Sonal: So I’m still not hearing what’s new. What’s the different underlying truth in this case?

Trade between rich and poor countries

Noah: What’s new is the type of trade, what will happen, who we’re trading with. What we’re trading, what kinds of trade we’re opening up. What the policy changes are, and what kind of other innovation technology is growing that alters this process. Those things can be very different. So, for example, opening up trade — the simple example — opening up trade with a rich country, like a country in Europe or, you know, Japan, for example — is in Econ 101 or any theory going to be very different than opening up trade with a very poor, low-wage country. So…

Russ: Same with immigration.

Noah: Same with immigration.

Russ: High-skilled immigration is going to be very different. Both of which are good for the country as a whole, but have very different effects on particular groups…

Noah: That’s absolutely right.

Russ: …of people within the country, so that the politics are going to respond to that in different ways.

Noah: Yeah. And so the point is that trade issues now, for example — people are talking about TPP and TTIP, which is basically a trade treaty with, you know, Japan and Korea, a trade treaty with Europe. Those are trade with rich countries, and that’s not going to look at all like opening up trade with China did in the 2000s. It is not going to resemble it at all, and yet people treat it as if it’s the same thing. One interesting thing is, if you look at some of the research papers that are coming out now like Autor and his co-authors.

Sonal: Autor et al.

Noah: Autor et al.

Sonal: If I recall, he’s arguing now that some of the benefits of trade with China actually were not as great as he thought they were, and that we could have, in fact, slowed down our process of trading with them. As, kind of, the gist of what the latest paper was.

Noah: Right. So, what’s interesting is that during the ’80s and ’90s, people were very scared of trade with Japan and Germany. You know, Japanese car companies coming in and out-competing our companies, and blah, blah, blah.

Sonal: Yeah.

Noah: And so what I’m saying is that when economists around in the late ’90s and early 2000s went back and looked at those episodes, what they found is that the displaced workers almost all found new jobs in doing very similar things for the same or higher pay. If you look just at manufacturing. It affects more than manufacturing. But if you look just at manufacturing, manufacturing employment really held up very robustly during all of that so-called Japanese and German competition, all throughout the end of the ’90s. You know, very robust, and you had this very smooth adjustment. And people seeing that we’re like, “Okay, safe.”

Then China. First, we had most favored nation trading status, permanent normal trade relations, and then the WTO entry. And what Autor, Dorn, and Hanson found is that the adjustment of the average worker in competition with Chinese workers was much, much slower and worse. And they found that, you know, the average displaced manufacturing worker in the ’80s and ’90s who was displaced by Japanese competition tended to find just as good or better a job. The average manufacturing worker displaced by Chinese competition in the 2000s got a crappy low-paid service job, or went on welfare, or disability — which, in some cases, has acted as a shadow welfare system. So people who drew the lesson from the ’80s and ’90s that the economy always adjusts real quickly, and that trade is essentially safe, didn’t think very carefully about the difference between trading with other rich countries and trading with a labor-rich capital poor country.

Sonal: Like China and India?

Noah: Like China, and maybe like India. Although, there’s some difficulties with infrastructure in India that means that India’s unit labor costs may still be high. So when we talk about cheap labor, one thing to mention is we don’t just mean wages, we actually mean unit labor costs. Which is basically, for a given worker, how much can I produce? And so, China’s labor was cheap not just because Chinese wages were low, but because Chinese wages were low relative to Chinese productivity. Indian productivity is held back by the fact that electricity keeps going out. And it’s very hard to get the products on the roads — the infrastructure sucks. So when you have an Indian factory on the coast, Indian unit labor costs are lower than Chinese unit labor costs. The reasons China was cheap — I mean, cheap wages were part of it, but they were subsidizing energy, they were giving everybody really cheap coal without worrying about the environmental impacts. They had very low labor standards. And they had a bank finance system that essentially fed super cheap capital, super low interest loans, to all these Chinese enterprises.

Sonal: Overall lower unit labor costs.

Noah: Overall lower unit labor costs. And it was so low, and so sudden, and China is just so big. I mean, if we traded with a tiny little poor country, we wouldn’t notice anything, but China is four times the size of our population. And trading with that sudden monster of a labor dump being dumped on our markets had a very deleterious impact on the lives of a lot of the workers that it hit in America.

Russ: So here is the problem I have with that analysis. There’s, obviously, some truth to it. I think the trade shock from China is different from the trade shock from Japan or Mexico. And I think they’re probably right that the adjustment for workers in the 2000s was tougher than the adjustment for workers in the ’80s. That it was much harder. I do think it’s part of a very long trend, though, that goes back to 1945. The aftermath of World War II, manufacturing employment steadily left the United States. It just got a little quicker in the ’80s, and then quicker still in the 2000s as a proportion of total employment. So the question is, for me, what did we learn from this? And is it because they are different kinds of countries?

Sonal: Yeah.

Russ: Is it the speed of the adjustment that we’re struggling with? Or is it a third factor, which I want to put on the table, which is that the labor market — I don’t think works as effectively as it worked in the ’80s and the ’90s. People are having a lot harder time moving to find new work, and we don’t fully understand why. There’s different theories. Is it because the jobs are in the cities and rents in the cities are very expensive? I don’t think we understand why. And if we don’t understand why, we’re going to have a very tough time doing the right kinds of policies to adjust to it.

Sonal: Exactly.

Russ: No matter what, it’s certainly true that we need a better system to help people find new opportunities, right? And give them the skills that they need to find new employment. But we don’t really have a good understanding of what the nature is of the problem we’re trying to fix. And the reason that’s important, and this is, again, where it ties into technology and autonomous cars, artificial intelligence, robots, etc. The worry among different people, and I think it’s a genuine worry, is that in the past, when you were displaced by technology, you’d find a new opportunity that the new technology helped facilitate, because it made things less expensive. But what if the technology takes over everything, and there’s no place left for people’s skills?

Noah: That’s a really important question, but let’s go back to a couple points. Manufacturing employment shrank as a percent of the total workforce since 1945, but total manufacturing employment didn’t.

Russ: We had a growing population.

Noah: And total manufacturing employment has shrunk since 2000 dramatically. So, at the exact same time China entered the WTO, total manufacturing employment in the United States — which had been at a stable level for decades and decades — starts to plunge?

Russ: We also, at the same time, had the introduction of the computer and the internet.

Noah: That was in the ’90s. And the big investment boom for computers was in the ’90s. And that’s when we got word processors, and spreadsheets, and blah, blah, blah. We got broadband in the late ’90s.

Russ: Yeah, we also got the application of those techniques.

Noah: It absolutely could be. What I’m saying is, the common notion that manufacturing employment steadily declined is wrong. What actually happened was that manufacturing declined as a percentage of our economy, because most of the new things we wanted from the new economy — new technologies — were services. Basically, we got a whole bunch of manufactured stuff, and the amount of physical stuff that we bought stopped increasing very fast, but we kept employing about the same number of people, and their productivity increased. But the percentage of manufacturing in the economy went down because our demand for services increased much faster after, you know, the 1960s than our demand for manufactured stuff. 

But it wasn’t until the 2000s that absolute manufacturing employment started to fall, and that manufacturing employment started to fall along with manufacturing productivity slowing down. So you saw a productivity slowdown at the same time as a fall in employment. When you see the trend in manufacturing worker productivity suddenly get flatter, and yet manufacturing employment plunge off a cliff, in absolute terms, not just relative terms, for the first time in many, many decades — you’ve got to wonder. If this unprecedented decrease in the absolute level of manufacturing employment was from automation, why did it coincide with the productivity slowdown? Why wouldn’t we see a productivity speed-up? If the manufacturing employment started plunging off a cliff because of technology, why don’t we see a productivity speed-up in that industry, because instead we see a slowdown?

Russ: The only thing I’m confident of is that it’s tangled together, there’s a bunch of things going on at the same time. I agree that it’s possible that the dominant effect is trade rather than technology. I also would suggest it’s really hard to measure productivity and output in this area, because we’re agglomerating…

Noah: It’s a lot easier in manufacturing than in other industries…

Russ: It is. It doesn’t mean it’s easy.

Noah: …because you can count the actual things.

Russ: Yeah. But that’s not enough. You can’t count the things because you’ve got to aggregate computers, airplanes, bolts, screws, nails, and other things. So you’ve got to use a dollar value. And then all of a sudden, you’ve got issues of how do you aggregate?

Noah: Within industry productivity, you can measure, like, number of planes.

Russ: Right. That’s…

Noah: You’re not aggregating by price there.

Russ: Correct.

Impact of trade with China

Sonal: We’re talking about the manufacturing and the productivity slowdown. But where does this idea of people being able to buy more because of the cheap products that are coming into the market fit in? Because while people may have had a hard time finding jobs, they can buy more with less.

Noah: We do that by trying to adjust for inflation. So, if we have very slow inflation, prices aren’t going up that much relative to wages. What we’ll see is what we call real wage gains. If…

Sonal: You get more money for…

Noah: …you get more stuff.

Sonal: …for your money. Right.

Noah: More money for your buck, then your real wage is going up. It turns out that real wages in the United States have been pretty close to flat. Now, they’ve risen the last couple years, but in the 2000s, real wage gains were very, very anemic. Here is where it gets tricky. There’s many different measures you can use for inflation, so how much should the price of healthcare weigh in that? How much should the price of rent weigh in that versus the price of the TV? It’s very tricky. And you can use some inflation adjustments that show a pretty strongly rising real wage, or that show essentially no wage slowdown in the 2000s.

Sonal: Right.

Noah: Or you can use PCE, inflation shows that one.

Russ: The PCE. But the other problem, of course, is quality. So, weave in something like a TV. A TV today is very different from a TV 15 years ago. Incredibly different from a TV of 30 years ago. The number of things that your phone does and replaces is thousands of dollars’ worth of technology. I have a video camera, I’ve got a still camera. So when the phone gets more expensive in nominal terms, meaning the actual dollars you pay, or a little bit cheaper sometimes, but it does a thousand more things or holds a thousand times more songs…

Sonal: It’s like a category error, right?

Noah: Right.

Russ: …you need to adjust for that and the Bureau of Labor and Statistics isn’t particularly good at adjusting for it. So that’s another caveat you have to put on the real wage issue.

Sonal: But what do you make of this argument about, just going back to the theme of trade in connection to this, and the entry of China and what that did for our economy? When I think of the reverse, that you could, say, limit trade — and this is like the protectionist argument that comes up sometimes. I think about it disproportionately affecting poor people, because they’re the ones who go to Walmart to buy the cheap products on the shelves.

Russ: Well, if you don’t have any job at all, obviously, the fact that prices are cheaper is not so important to you. If you do have a job, even a job that pays less, but if prices are a lot lower, you could still be better off.

Sonal: Right.

Russ: And one way to think about Walmart, it’s a retail outlet for China.

Sonal: That’s exactly how I would think about it.

Russ: The thing that’s remarkable to me is that we are lucky right now to live in a time when the unemployment rate is incredibly low, it’s 4.3.

Sonal: Didn’t we actually just hit a statistic, that we are actually finally have risen on the jobs? I just saw this this week.

Russ: The employment rate, the actual percentage of people who are employed which is not…

Noah: Employment to population ratio, the best labor market indicator.

Russ: The best. Much more important than unemployment.

Noah: Prime age, prime age.

Russ: It’s finally getting better. And you emphasize prime age because a lot of the drop that we’ve seen is actually the aging of the population, not a crisis in the labor market.

Sonal: Like what happened in Japan.

Russ: Right.

Noah: We could use age-adjusted prime age.

Russ: Not going to disagree with the reality that for 10 years, we didn’t do very well. We had a horrible recession that was just a disaster. But now things are pretty good, and yet the perception of how the economy is going is, “It’s awful.”

Noah: A lot of people are just thinking politically. When you ask them, “How is the economy?” They think, “Do I like the president?” I want to say something else. The thing about slow adjustment, about some workers, sort of, permanently losing their livelihood, the stuff about competing with low wage versus high wage countries, the thing about the possible innovation slowdown in automation from a decade of cheap labor. All these caveats, they’re all over. This is all about the past, and this is an argument about the past. The China shock is over. China unit labor costs are now at parity with the United States’ labor costs, and there is nothing on the horizon that looks anything like China. Basically, the China shock was this giant one-time thing. Suddenly, you had a fifth of humanity with very little capital and a huge amount of labor, and high productivity, and all this cheap financed energy, and capital, and stuff — be dumped on the world economy. A fifth of humanity in one decade.

Sonal: Sticker shock, all at once, huge scale.

Noah: Correct. It’s all at once. China’s no longer undervaluing its currency. That was very mercantilist. They were building up their U.S. dollars so, like, modern-day gold. They were doing that in the 2000s. They have stopped now, and have actually been propping up their currency instead of making it artificially cheap. They’re selling off that giant pile of reserves, they’re going into reverse mercantilist mode. So all of this is over. And so this argument is an academic argument for “How should economists think about trade, how should economists study trade, how should economists talk to the public about trade?” It is not an argument about “What should our trade policy be right now?” Trade has gotten less ambiguous a lot in this decade than it was last decade.

Russ: My only comment is we focus a lot on our neighborhood, which is realistic and understandable. But I think it’s important to remember that trade has transformed lives elsewhere.

Trade deficits and their effects

Sonal: We didn’t talk about the argument of asymmetry in trade, and what happens when you have one country that’s being more protectionist and another one that’s not. I’ve heard the argument. I think Paul Krugman made it, or maybe Adam Smith, I don’t remember, but, “Even when you’re open and everyone else isn’t, everyone still benefits.” And that’s the bottom line of this free-trade idea. What happens when it is disproportionate?

Russ: So, in the 1980s, when the debate was over whether we should let Japanese cars come into the United States to compete with American cars, a lot of people said, “We shouldn’t because they don’t let other products of ours into their market.” And my view was, “If they want to harm their own citizens by keeping out American products, that’s their choice. Don’t confuse that with the fact that we benefit when we let in Japanese cars.” There’s a mistake and focus on the value of exports. That, somehow, trade is only fair if we have balanced trade. That we import exactly the amount that they accept of our goods. It misunderstands the fact that trade in goods is not the only thing we do. We trade in money, and assets, in capital surplus, and services as well. And so, America, on average, is going to be better off when we open our borders. Not as much as it would be if other countries joined us and opened theirs, too. People say, “Yeah. But we need to use that as a negotiating tool.” And my view is, it doesn’t work very well historically. So if we say to China, “We’ll practice free trade only when you do.” They’ll say, “Okay, fine. We won’t, and you won’t.” And then we’ll end up in, not just a trade war but possibly a real war.

Sonal: Right.

Russ: So, I think trade is very important for that reason. It builds bonds between countries, even if other countries don’t want to be as open as we are, say, for political reasons. Just one more example. And the Japanese don’t let in rice, foreign rice, because the Japanese farmers are very small and very politically powerful.

Sonal: Yeah, just like the corn farmers in the U.S. I mean, like, high-fructose modified corn starch or corn syrup. Whatever. You know what I’m talking about. 

Russ: We pay a premium for sugar, because we keep out foreign sugar. People in China pay an enormous premium for rice. It’s just pure politics. And the world would be a better place, I believe, if we didn’t listen to that.

Noah: So, that’s generally right. There are a couple caveats.

Russ: Thank you, Noah.

Noah: Thank you, Russ.

Sonal: I like how you guys are fighting by caveating, it’s great.

Noah: Well, caveating is important.

Sonal: It’s very important.

Russ: It’s the economics way of life sort of caveat.

Sonal: It’s actually the “a16z Podcast” way, which is, I love nuance.

Noah: So, to inject some nuance. The idea that trade builds bonds between countries, I believe, is an overrated idea. Because on the eve of World War I, Great Britain and Germany were each other’s largest trading partners. My friend Paul Poast, who is a political science professor, has shown that trade agreements are a great way of cementing alliances. But if you don’t have an alliance, then, you know, it’s not as helpful. The other caveat is that exporting at the company level raises productivity.

Sonal: What do you mean by exporting at the company level? 

Noah: So, when a company enters a foreign market, it experiences a jump in productivity. Obviously, companies that were more productive to start with are going to be more competitive in foreign markets. There’s pretty good evidence that exporting forces companies to up their game in general.

Russ: Trade generally does that, right?

Noah: Trade generally does that. So…

Sonal: By the way, is that as true of services as it is of products?

Russ: It’s harder to export services.

Sonal: Right.

Noah: Services, we do have a lot of service exports, actually.

Russ: We have foreigners who come to America to go to college.

Noah: Things like that.

Russ: That’s an export.

Sonal: Right.

Noah: And our universities are competing with Chinese universities. The point is, if we have a large trade deficit, that’s not necessarily bad. Because, you know, if there’s unbalanced trade, someone has got to run the deficit. But if we have a large trade deficit, it means we’re not exporting as much as we would be if there were balanced trade. And that could be depriving some of our companies of this pressure of international competition that forces them to up their game.

Sonal: It’s kind of similar to the argument you made earlier, actually, about just how it improves innovation overall. It’s actually in the same vein.

Noah: Could. It could be in the same vein. You know, in most economic models, technology falls from the sky like manna from heaven, and companies just take the technology that the magic inventors somewhere invent, and they apply this.

Sonal: It’s like the deus ex machina in, like, literature.

Noah: Yes, it is. There’s an argument to be made that encouraging your companies, and not just your big companies like Boeing, but your small companies, to get out there and sell on the foreign market instead of just, sort of, cozily selling to, like, the neighbors in Ohio.

Sonal: Their limited market. Right.

Noah: But actually try to get out there and sell to people in Bangladesh or wherever.

Sonal: Well, that’s where the internet comes in.

Noah: But I worry that our companies are experiencing lower productivity from lack of exporting. Obviously, in the 2000s, the currency thing with China was part of that. But even then, it was a modest part. I think, really, the case is we have got this giant domestic market, and we run, you know, trade deficits. But we’ve also got this giant domestic market and it’s just really easy for companies to, kind of, slack off.

Russ: Well, I disagree because I think politically that it’s going to end up with politically powerful companies getting more than just the incentive to export.

Noah: I mean, small companies. I mean small companies.

Russ: Even small companies, I just don’t think that should be a policy goal, to have exports for exports’ sake, because I just think it leads to too many problems.

Noah: Japan does a lot of things wrong, but it also does some things right. And it has an export promotion agency called JETRO. If you’re a small Japanese company, you don’t know how to sell stuff to Bangladesh. JETRO will hook you up with local resources, and will try to hook you up with some financing, and things like that to basically nudge you toward these markets. Yeah, I think America could absolutely use something like that.

Russ: Well, we have the EXIM Bank, which mainly works as a subsidy to Boeing, not small companies.

Noah: That’s, yeah, right.

Russ: And they claim that they’re good for small business. And I think the political — what you would design, Noah, and what I might design if I were in agreement with you, would be different than what we’re going to get.

Sonal: So we ended on the note that China had this unprecedented scale, this shock effect, because of the fact that it was a one-time tenure, one-fifth of the world. We’re talking about the past. So what happens next when it comes to — we think about the evolution of technology. We’ve alluded to, you know, automation and what’s happening with jobs. But just more specifically what happens next?

Russ: The easy answer is, we have no idea. But the second answer which is, I think, going to be true no matter what happens is that it’s going to be very hard to do something about it even if we don’t like the outcome. Technology is incredibly powerful right now, innovation is incredibly powerful. We’re going to get autonomous cars in the United States if the technology works out. I don’t think the political power of the taxi cab industry is going to stop it, just like I don’t think the hotels can stop Airbnb. And so I think we should be focusing on the costs of that transition and preparing people…

Sonal: Education, etc.

Russ: …with education and better skills to cope with it.

Noah: I couldn’t say that any better. Russ, I completely agree.

Russ: You guys finally agree completely with each other on this one.

Noah: I mean, on this point, you can’t stop technology. And you can’t suppress it. These things are coming. CRISPR is coming to design your babies someday.

Russ: I was going to say, yeah.

Noah: Driverless cars are coming. We’re going to have to roll with it.

Sonal: Yeah.

Noah: We’re going to have to be flexible. We’re going to need things like continuing education throughout people’s entire life, and not just, you know, teach them stuff and then release them into the workforce. We’re going to need all kinds of things to become more flexible and to roll with these things. Because trade in the 2000s was something we might’ve been able to manage better than we did. But that’s over, and the challenges of the future are completely different than the challenges of the past.

Russ: Well said.

Sonal: Thank you for joining the “a16z Podcast,” guys.

Russ: Thank you.

Noah: Thank you.

• 

  Russ Roberts
• 

  Noah Smith
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

There are the things that you carefully plan when it comes to an IPO — the who (the bankers, the desired institutional investors); the what (the pricing, the allocations); and the when (are we ready? is this a good public business?). But then there are the things that you don’t plan: like the worst financial crisis since the Great Depression… as happened before the OpenTable IPO. There’s even a case study about it.

And so in this episode of the a16z Podcast, we delve into those lessons learned and go behind the scenes with the then-CEO of the company — now general partner Jeff Jordan — and with the then-banker on the deal, J.D. Moriarty (formerly head Managing Director and Head of Equity Capital Markets at Bank of America Merrill Lynch), in conversation with Sonal Chokshi. Is there really such a thing as an ideal timing window?

Beyond the transactional aspects of the IPO, which relationships matter and why? And then how does the art and science of pricing (from the allocations to the “pop”) play here, especially when it comes to taking a long-term view for the company? What are the subtle, non-obvious things entrepreneurs can do — from building a “soft track record” of results to providing the right “guidance” (or rather, communication if not guidance per se) to the market? And finally, who at the company should be involved… and how much should the rest of the company know/ be involved? In many ways, observes Jordan — who got swine flu while on the road to the OpenTable IPO — “your life is not your own” when you’re on the road, literally. But knowing much of this can help smooth the way.

Show Notes

• The history of the OpenTable IPO [0:47]
• A discussion of pricing [10:09] and details around the IPO process behind the scenes [16:20], including mistakes along the way [21:02]
• Key takeaways from the IPO experience and advice for others [24:00]

Transcript

Sonal: Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal. Today we’re doing one of our war stories podcasts, where we have founders, makers, and operators share the story behind the story. And, joining us for this episode, we have a16z general partner, Jeff Jordan, who was president of PayPal at eBay before going to online restaurant reservation network, OpenTable, where, as the CEO, he oversaw the company going public.

We’re gonna talk about all that in this episode, focusing on everything from the relationship-building involved on the road to IPO, and the nuances of pricing and allocations, to the broader market context, and some concrete advice for entrepreneurs. And, last but not least, we have special guest J.D. Moriarty joining this conversation. He’s now SVP of corp dev at LendingTree, but was formerly managing director and head of equity capital markets at Bank of America, Merrill Lynch.

Background on the OpenTable IPO

Jeff: J.D. was the lead banker, the capital markets expert from Merrill Lynch on the OpenTable IPO. He and Harry Wagner of Allen & Company were the key — two keys who basically helped execute a deal in about the worst capital market situation in late…

Sonal: When was that?

Jeff: Late 2008, early 2009. A venture-backed technology firm had not gone public in a couple of years. The concern was that the window was closed, bricked-over, and the only exit path for tech companies was going to be M&A from then on. We ended up pricing at the nadir of the worst financial crisis since the Great Depression.

J.D.: I used to talk about 200 IPOs a year, and the only IPO prior to OpenTable in 2009 was a company called Mead Johnson, so a very defensive company, that type of thing that should go out in 2009 in consumer products.

Jeff.: They made, like, the floor wax or something right?

J.D: Yeah, exactly.

Sonal: Very far from this weird thing called OpenTable, which is not even a product you can physically touch.

J.D.: Correct. And so, you know, people tend to look at when an IPO prices — and you have to recognize that most companies take six to eight months to get there. From our org meeting to kick off the process where the bankers and the management team begin the process of preparation, our pricing was about eight months.

Sonal: I have to ask a really dumb question. Why do you need a bank? Like, why can’t you just directly IPO, bluntly?

J.D.: Actually, at one point, Jeff asked me early in the process, “Hey, why do we need to go see all these investors? Can’t we just do a $35 million IPO to four or five folks?” I think the way to think about the IPO process is, you’re not just doing the IPO. You have to take a two-year view towards, how do we get this to be a stable public company that can grow — and achieve not only the company’s goals, but the goals of the early investors, with regard to monetization over a long period of time? And oftentimes, as this deal showed, those goals are different.

Jeff: It is an interesting observation is — going public does not create liquidity. All the insiders cannot trade when you go public.

Sonal: Why is that?

J.D.: Well, a standard expectation [is] that the new public investors — taking a chance on this new company — is a 180-day lock up. That is kind of a market standard.

Sonal: The lock up period, right.

J.D.: There are certainly exceptions to that. And we can talk about things like the IPO discount, etc. But in order for somebody to take the risk of a newly public company, they expect certain things. Now, there are plenty of IPOs that have secondary shares, don’t misunderstand. But the early investors are locking up for 180 days. There is a period of time when the right way to think about it is, your true monetization is really down the road.

Jeff: Yeah. And so, if when the 180-day lockup expires, all the insiders and all the management team run to the floor of the stock exchange and try to sell all their stock, all the third-party — all the owners will disappear too. Because if the insiders don’t have any confidence in it, then why should I own it?

Sonal: I mean, just to give a little bit of a counterpoint, though. Obviously, market conditions change. If the company hasn’t gone public in, like, 10 years, and you have to get some liquidity out — or you’re a founder who has to give up a little bit of shares in the secondary market in order to, you know, loosen up your…

Jeff: We are in support of that, but if they try to get full liquidity on day 181, the stock price is gonna be, like, $2.

Sonal: Yeah, it’s the reality of how the market works.

J.D.: Yeah. We’re talking about the technicals of, how does that stock get to market? There’s another part of this, which is simply, when do you time the IPO? Fundamentally, we encourage people to — don’t think about the market conditions today. Think about, is this a good public business? And you have to answer that question first. And for most companies, you’re never gonna time the market.

Sonal: You can’t time the market because it’s clearly a long process. But then why do people talk so much about there being certain windows, in which there is an ideal time that, you know — like, a window can open and shut, and this is on a global scale of 10 years, 15 years, etc.

J.D.: This was actually — there’s a case on the OpenTable IPO. It’s at Stanford Business School now, taught in the “Formation of New Ventures” class. Andy Rachleff wrote it. And it basically says, should OpenTable go public now? The analysis said, there are windows in the IPOs. Yeah, they kind of come and go, and the best companies often open windows that were then previously closed.

Sonal: Interesting.

J.D.: So, the best companies can go out whenever they want. The good companies typically wanna wait for — till the investors are feeling good. The mediocre and bad companies wanna get out whenever they can. And what happens — when the window opens, the early people to go out — typically perform very well as public company stocks. And then, as more time goes by, and you get towards the end of a window, the companies that then are going out — kind of rushing, “I gotta get out or I’m going to miss it” — are typically not the highest quality companies, and they tend to underperform the market. One of the things that had us go was, okay, we think we’re a good company and we think we can perform in the market. We wanted to meet bankers ahead of time.

Sonal: And how far in advance did you do that?

J.D.: We started about a year and a half out or something like that.

Sonal: A year and a half?

J.D.: Just surgically, just with the bankers. We kind of orchestrated into one or two conversations. But before we did the formal bake off, where you select your lead bank — and all due respect, Merrill at the time, they were, you know — they were not the top of the pyramid, in terms of tech bankers. So we — because we were the only IPO, we had every bank [that] wanted to do it.

Sonal: So how do you pick?

J.D.: Goldman, Morgan, you know, just down the list…

Sonal: Yes, so how do you pick them?

J.D.: We got to know the people at the firm. And we put particular value in the capital markets function, because that is the function that interacts between the company and the investors. We wanted someone who we thought understood our business well, and we thought could represent our business well to investors.

Sonal: And by investors, just for clarity, you mean investors like institutional investors?

J.D.: These are institutional investors who invest in public securities. Typically, the largest ones are mutual funds, managing billions and, you know, billions of dollars. We were looking for tentpole investors who would go for a while, and we’d meet with them every six months or so. And they got to know the business. They got to know us. They got to — we’d developed a soft track record. Because they’d say at the first meeting, “What are you gonna do in revenue this year?” “Oh, we’re gonna do $70 million.” Then you come back six months later, “What did you do?” And we said, “We did $80 million.”

Sonal: Oh, you mean, like a soft track record of delivering results. It’s almost like quarterly reportings.

J.D.: Of performance, yeah.

Sonal: But they’re, like, informal — verbally.

J.D.: Yeah. Am I the kind of person who overhypes and under delivers? Or do I under hype and over deliver? Or do I tell them what’s good about the business and what’s bad about the business?

Sonal: That is such a golden nugget, because it’s invisible to the world. When you see the outcome — the process behind the outcome is invisible, which is the whole reason we’re doing this. I didn’t know that. Why does that relationship that the capital markets expertise matters so much in the lead up to the IPO?

Jeff: We wanted it to be not a black box. One of the things other CEOs had told me who had done IPOs when I reached out, it’s like — somehow, you know, you do this roadshow, you get to the pricing meeting. They say, okay, we recommend the price is this, and then the shares just magically disappear. And we really cared about who got the shares. So, we wanted to have a vote in who got it, who got the shares.

And, because it was such a tiny offering, we wanted to concentrate the shares in that shortlist at a much higher level than what’s typical at the pricing meeting. You know, we — J.D. shared the spreadsheet, and we’re like, “No, no, no, we have to give these guys 10 times more.” And he’s going, “No, no, no.”

Sonal: Wait, so who voted in the — it was like the, you know, the bank, you guys thought…

J.D.: Well, typically, to Merrill’s credit, typically, the bank pretty much decides. They engaged in a dialogue with us. And that’s probably the biggest thing where we met in the middle, said, “Okay, I understand your rationale.” So, we had this very constructive dialogue around that. In many IPOs, that dialogue does not happen.

Sonal: That is such an artful, behind-the-scenes orchestration.

J.D.: It’s all a hangover from the ’99, 2000 period, when those allocations were truly a black box. And somebody said to me at one point, is there any innovation in the IPO market? And I said, the biggest change over the last 10 years is that it’s become more transparent. And that is more the norm today — that there’s genuine conversation around it. Now, I’ve seen the other side of it, which is a management team says, “No, it’s gonna be like this.” Ultimately, their vote is the vote that matters.

But I’ve seen scenarios where they make mistakes there too. What the OpenTable team did well is actually invest in not just the bankers, but actually the investors. The thought process was, the better they know our business. These are the people who are gonna have skin in the game and really own enough of our stock.

Sonal: Hold the tent up.

J.D.: And know where the business can go and hold the tent up in a difficult time. And so, Jeff essentially invested in that process, and I think it was critical. One of the debates we had was size of IPO. 

Sonal: Like, the amount of the initial public offering.

Jeff: The proceeds, yeah. And so we spend a lot of time doing analytics for companies on how big does your IPO need to be? What does your market cap need to be? How big do the proceeds need to be? And if you — if you just, sort of, look, it doesn’t pass the common sense test, right? Why does some portfolio manager at Fidelity, who manages billions and billions of dollars, invest in the OpenTable IPO, when the initial proceeds are only gonna be $37 million?

Now, ultimately, because it went well, we ended up raising just shy of $70 million in the IPO. And then, in September, we ended up doing a follow-on transaction that was $210 million. But the point was, why is it worth it to Fidelity, Morgan Stanley investment managers, T. Rowe Price.

Sonal: Yeah, so why is that? I wanna know.

J.D.: Because if you — if you invest the time to let them see where the business can go over time, they’re gonna leg into their position over time. You know, if we had been — we were in a lousy market. One expression we always use is, “In difficult times, our investor clients focus on the things they own, not those things that we wanna show them.”

Sonal: What do you mean by that?

J.D.: Meaning, new ideas. So, it’s just a risk curve issue. And Jeff made the point about great companies being able to go out in any market. Good and okay companies need to pay more attention to the investor risk curve.

The question of pricing

Sonal: Back to the notion of pricing. We had Lawrence Levy, who is a former CFO of Pixar on this podcast, and he’s the one who helped Steve Jobs take Pixar public. And one of the things that he talked about — how it was the biggest fight between him and Steve. And the reason was because, of course, he wanted to go high — because he wanted a big-ass IPO, like Netscape at the time. And he was on the heels of that. And Lawrence was like, “No, no, you wanna deliver some returns for the investors.” And there was sort of this dance back and forth. How did you guys sort of do that dance?

Jeff: Yeah. A lot of discussion.

Sonal: Is that a euphemism for fighting?

Jeff: No, no. There’s a lot of discussion. You know, our at IPO market cap was $450 million, roughly. Raising proceeds of just under $70 million.

Sonal: The IPO size was $70 million.

Jeff: Now, I’ll be the first to admit, did it trade, kind of, too well? Yes.

Sonal: So why is it bad if it trades too well?

Jeff: There’s too much of a pop.

J.D.: It means the company didn’t get as much money as they could have if they had a crystal ball and knew what…

Sonal: Right, because the whole point is to get capital to continue growing and building the business, I get it.

J.D.: Correct. You don’t wanna leave a lot of money on the table. Now to be clear, when you end up floating a small amount of the business, you kind of compound this problem.

Sonal: Why is that?

J.D.: Go back to the point around the Fidelity’s and T. Rowe’s needing larger position sizes — they recognize that the two events that they care about are the distribution of shares at IPO, the allocations which we went back and forth on, and the first day of trading. And then these stocks become very, very illiquid.

Sonal: They’re in the public market, how do they become illiquid?

J.D.: Because the float is only $70 million.

Jeff: And we convinced people that it was an interesting stock to buy and hold. That meant we had no daily trading volume.

J.D.: So, if the stock was trading around $30 a share, there were days when it was trading, literally, 2,000 shares. 2,500 shares.

Sonal: Not many people moving money.

J.D.: Correct. And so, you can get these huge gaps. So it did trade “too well.” That’s a balance that we’re always trying to strike.

Sonal: Yeah, so from your perspective, Jeff.

Jeff: Yeah, I know. So, when we — the original documents had a $12 to $14 price range. I think we updated it to $16 to $18 over the course of the roadshow, because the roadshow was going well. The first two investors said, “I want a full allocation.” So, it quickly became a hot IPO. We ended up being oversubscribed, 20 to 1, 25 to 1, something like that. So, we probably could have run it up into the mid-20s easily.

Sonal: But you guys priced it at…

Jeff: We talked to the market to 22, and we priced at 20. And most management teams — board CEOs are gonna grasp for that last dollar. I think the OpenTable team collectively was very thoughtful about the fact that, this is just the IPO. I care about the next two years.

Sonal: Did you guys — tell me the truth. Did you guys have, like, a magic number in your head before you start those pricing discussions? Like, did you think in your head, you know what, when I go to sleep at night, I want $25 when this thing goes on the market?

J.D.: No, we didn’t, you didn’t. Part of our strategy was, we’re gonna do a teeny, little IPO. And then if it went well, we were gonna do a pretty big secondary. And so, the company was much more focused on “make the secondary successful” than it was “make the IPO successful.” Part of making the secondary offering successful was, you needed a couple of deep pocket people in the IPO, even though it was a teeny, little IPO. So, one of our leading shareholders ended up being whittled down off of Fidelity.

And so, when we say, “Will, invest out of your $10 billion — whatever it is — fund, $4 million. And he’s like, ‘I don’t have the time to read your earnings release at that level.’” But we convinced him to come in, because then, in the secondary, he was able to back up the truck, and he got what he wanted, which was a larger ownership allocation. His IPO allocation was, what — 5% of 70, so $4 million, some number like that.

Jeff: Yeah, and one of the things that made the add-on so much easier is because everybody knew that we could have priced well above $20. When we priced at $20 that, I think, built some real goodwill between the management team and the investors. 

Sonal: That you guys are willing to be thoughtful about the long term.

J.D.: And we ended up optimizing for who got the shares, not the price they got the shares at. And I would do that again in a second. I’d advise management teams to do it like crazy, because as a CEO managing a public company, you don’t want people who are in and out on momentum, hot money — because you’re spending, then, all your time literally marketing to new investors, “Please buy my shares.”

There are multiple reasons to want a small — from my perspective, to want a small handful of tentpole investors who buy and hold your stock. One is, they buy and hold. The other is, it just makes your life easier. I mean just…

Sonal: Because you’re only, like, talking to a pool of four to five people?

J.D.: Yeah, I’ve got a handful of owners, and most of them — I actually said, “How do you want me to work with you? You own a lot of my shares. Do you want me to call you after every earnings call?” And they’re like, “No. I’ll listen to the call.” Almost all of them were just like, “Nope, I’ll reach out if I need anything. Thank you very much.”

Jeff: In the first year and a half, there was a period where we dealt with the momentum crowd coming into the stock.

J.D.: Late.

Jeff.: And it was challenging.

J.D: That sucked.

Jeff.: Because we essentially went from, you know, the projected — keep in mind, when we’d gone public, in 2009, the projected top line growth in the business was 20%, from an analyst perspective, just under 16% to 20% depending on what the analyst — it was always a high margin business. And then it was when you went to 40% top line and 40% margin, that’s when every momentum investor came in. And so, that was one of the things that I think became more challenging to manage.

Sonal: And momentum investors, you mean, like hedge fund people?

Jeff: Not just hedge funds, but in many cases, it is. That’s a broad stroke. Ultimately, they’re investors who just care that you’re gonna beat the quarter. 

Sonal: Yeah.

J.D.: I mean, it was so interesting, because there were a handful of investors, before the IPO, who were tracking the business. So, I was told that Fidelity — Will does not do IPO pitch meetings. Will walks into the meeting and spent the whole 60 minutes there. Dennis Lynch at Morgan Stanley does not do IPO meetings. Dennis was early. He was waiting for us when we got there. You’ve got those who you’re like, “Okay, they’ve done their homework, they get it, network effects, everything else.” Dennis can tell you what three private companies he wants an IPO allocation in today, for five years from now.

Sonal: Because they have a strategy, they think about it.

Jeff: They have a strategy. The other guys, you’ve just blown away six quarters and said, “Oh, my god, I need to latch on to that sucker.”

Sonal: Yeah, get me into that.

Jeff: It’s like, “Where were you when it was $20? You’re buying at $110 now.” They’re the guy that will jump in, but they also jump out. That’s when stocks freefall.

Behind-the-scenes IPO process

Sonal: Tell me about any behind-the-scenes fights or discussions that you’ve had with your team. Like, were there disagreements? I mean, you might — you’re saying this, but were there parts where you guys were like, “We can’t agree on the pricing that these guys are discussing with us. We can’t agree on the timing.”

J.D.: Not a ton. The board gets involved at a few steps along the way. One is, I involve them in the selection of bankers. They were there.

Sonal: Is that a best practice, that you advise people should actually involve their board in that?

J.D.: I did. We did a bake-off. We had, like, six people, and we tried to do “wisdom of the crowd.” No one was allowed to say who they liked and didn’t like, and we got a ranking one to six. And it was unanimous. We did a subgroup that got — went deeper than the rest of the board, the IPO committee. Then the board also gets involved in things like, okay — do you launch the roadshow? And what’s the price? Almost all of the decisions — the process is being run by the management team. And so, in our case, it was CFO, Matt Roberts, and myself. And we insulated the entire rest of the company from it.

Sonal: Wait, so you did not involve the rest of the company? It’s just you guys?

J.D.: No, they’re out there back in San Francisco building the business.

Sonal: Building product.

J.D.: We’re spending two and a half weeks running around the country.

Sonal: Is that typical? Is it the CEO and the CFO that you typically want in the room?

Jeff: Yeah. And in fact, one of the mistakes that we see companies make periodically is one, building employee expectations towards an IPO too early. And two, involving too many people. If you think about it, one of the things that larger companies, private equity-backed companies tend to do well, is they value the option value, they get themselves ready to go. But guess what? They’ve also got more resources at the company to do that.

Sonal: Wait, can you break down what you mean by option value? That’s a very loaded — those are two very loaded phrases in our business.

Jeff: Sorry, the preparation, the preparation phase. And with venture-backed companies, I think you have to be mindful — and the boards are mindful of the fact — of the distraction that you can create through the IPO process. And so, what Jeff and Matt did was say, “You know what, this is going to be borne by us. Everybody else, do their job.” Everybody else was focused on doing their job and managing the business. Go back to when we were talking about the odds of it being a lousy market, that we might say, “You know what, this is not our year to go.” We’re there. So, it would have been a huge distraction.

Sonal: One of the things we haven’t talked about is that, you know, in a lot of cases, the IPOs involve novel technologies that are not familiar to the market, and you’re essentially selling a new way of doing things. Now, it’s very familiar to us, to actually book our reservations online. But how do you think about involving other key, like, technical people to help sort of educate, or is that the CEO or the CFO? Don’t you feel frustrated as a founder, that my CFO can’t represent this that well?

J.D.: I had a very good CFO. Matt Roberts did a fantastic job. He actually was the one who orchestrated almost all the IPO till the roadshow. But if your CFO can’t tell the story, you need a new CFO.

Sonal: Well, that’s why I’m asking, you’re honestly, when I think of a CFO — no offense to all the CFOs out there — I think of number people who are just sitting there with, like, spreadsheets.

J.D.: Different CFOs can have different styles. The investor has to trust them. And then that’s what the investor is looking at. “Is the CFO telling me the truth, know what’s going on in the business?” And how does that play?

Sonal: Is there one thing that you’re like — I want every CFO to have this, from both of your perspectives?

J.D.: Integrity, attention to detail.

Sonal: That too.

Jeff: That’s true.

Sonal: The market wins that too. My mom with her, like, 15 Snapchat shares wants that too. I mean, we all want that.

Jeff: We often ran into management teams that wanted to have too many people on the road. Three people is, sort of, the outer number, and…

Sonal: Who is the third, by the way, when it is not the CFO and the CEO?

Jeff: To your point about — it depends on the business. To your point around technology. Periodically, if it’s a highly technical business, you might suggest you have that person there for Q&A, but you don’t want to have the distraction. You wanna have dialogue. Most investors expect CEO, CFO, dialogue. So when you get to the CFO question, we can tell which teams are gonna need a lot of preparation, and it’s seldom both CEO and CFO. And then we just — we hit him with questions. These are the types of questions you’re gonna…

J.D.: It was the CEO, in my case. He just doesn’t wanna say it loud here. You’re in a different city every night. You’re doing, like, seven meetings a day, hour-long meetings a day. And the thing they stress more than anything is — give exactly the same presentation, and answer every question exactly the same, because of regulation, FD, fair disclosure. Literally, they said no, no, no, don’t be playing around with giving that slide two different ways. You give that slide one way.

Sonal: You’re like a robot.

Jeff: We did it 42 times in, like, two weeks.

Sonal: Oh my god.

J.D.: All in different cities.

Jeff: You get really tired of hearing your own voice.

Sonal: Yeah, I can imagine.

Jeff: And if you don’t, I’m not sure I wanna work with you.

J.D.: No, it’s unbelievable. One really helpful thing is, the OpenTable core customer included bankers living in New York, Boston. Earlier in my career at eBay, none of those owners would use eBay.

Sonal: Interesting.

J.D.: Unless they happen to be collecting something because, you know, time is much more valuable to me than money, and e-Bay was a cost-saving thing.

Sonal: It strikes me that that’s actually one of the challenges, because OpenTable, then, is a consumer business. It’s one of the challenges of enterprise-facing businesses, and especially SaaS businesses, where the financial model is also not as familiar for people to talk about. 

So, I wanna talk about the bumps in the road now, and the unexpected things that happen. I mean, there’s a lot you’re doing in the road up to the IPO — a lot of prep, clearly. But despite all your hard work, unexpected shit happens.

J.D.: It happens in every one. I wasn’t at eBay at the time, but I believe Amazon launched their auction competitor on — during either the IPO in the secondary, and Yahoo launched their auction competitor on the other one.

Sonal: Oh man.

J.D.: We had two big ones. One is, we got our obligatory patent troll lawsuit. That happened while we were on the road. They wait until you’re on the road, point of maximum leverage.

Sonal: Of course, they do.

J.D.: And file.

Sonal: They can get their money, get you out of the way.

J.D.: So, I get the call, you probably, you are saying, “Oh, by the way, you just got served.” You’re like, no. So that was one. The other one was a little more self-inflicted. And it turned out, no one had done an IPO for a long time, including the SCC, our accountants, and our attorneys. And our attorneys, at the last minute, update the filing, you know, they keep — it’s very formal. They update the filing the night before, and the SCC gets it, and they say this is approved, and you’re ready to sell the next morning. So, after a bottle of wine that night, when you have the price.

Sonal: And you’re, like, relaxing.

J.D.: We’re trading the next morning and I wake up a little early for — like, 3:00 a.m. Yeah, I mean, you’re just wired. Go to the gym. I’m working out, I open up my smartphone and see …

Sonal: Was it a Blackberry at the time?

J.D.: It probably was a Blackberry at the time, because this was 2009. “Offering on hold.”

Sonal: Why? What happened?

J.D.: Turned out the attorneys had — when they did the last turn, had attached the wrong attachments. The SCC approved something…

Sonal: Human error.

J.D.: …that we actually knew was erroneous. And so, if we started trading on an erroneous document, there’s a chance the SCC can say, “No, no, no, you have to unwind all those trades. Go start over.” Now, you’re just like, it’s in every newspaper. The business section in America. We’re going public today. Now we’re not going public.

Sonal: Well, what did you guys do?

J.D.: Our attorneys sort of start — all the attorneys on the deal, just start trying to get the SCC on the phone. And so, we ended up delaying the opening, finally right as the market opened the SCC, “Oh no, you’re blessed again.” And so, then started trading an hour or two later.

Jeff: But we did have a delayed open. We had a significantly delayed open. Not unlike Facebook’s delayed open, just a different outcome.

Sonal: So just, like, 10:00 a.m. instead of 7:00 a.m. kind of a thing.

Jeff: Yeah. If you’re on the New York Stock Exchange, you’ll typically open closer to the 9:30 start, and on NASDAQ, they have a — they always have somewhat delayed windows. We were very delayed.

J.D.: We were very delayed.

Sonal: And that is time — a case where time is literally money. It’s, like, ticking away if you’re not…

J.D.: Yeah. Now you know I’m obsessed with Hamilton.

Sonal: We both are.

J.D.: One song says, he walks the length of the city. They had me show up about a half-hour after trading has started, so I don’t walk into a potential disaster. That’s, like, four or five miles, just through the city. It starts trading well. He goes, you’re in good shape. You can go home. And so, I walked back.

Sonal: Wow, you know what I mean.

J.D.: And then I walked across. And then I was just like, oh my god.

Sonal: What about that whole “ring the bell” thing? Didn’t you guys wanna, like, be there when they’re doing that whole thing?

J.D.: We didn’t end up having it the same — we had delayed.

Jeff: Which, by the way, is the weirdest thing, because it’s a soundstage on Times Square.

Key takeaways and advice

Sonal: That’s amazing. Okay, wrap up and takeaways, relationships matter, timing matters. But, while I understand your earlier point, that you can’t time the market itself, the context — the broader environment — does matter. And how do you, sort of, look at back then — that was 2009, and now, 2017? It’s eight years later? What are some of your views on how IPOs have changed given this context?

J.D.: So, a couple of the things that are big takeaways from the open team have actually been somewhat formalized in the market. And so, one of my big takeaways from this transaction was, what the team did well was invest in the process, be ready to go. They valued the option value of being ready. They invested in that, and then were able to respond to an open window, essentially. The other thing that Jeff highlighted was spending time with the public investors, long before that 45 minutes to an hour-long meeting when you’re on the road.

Well, post the JOBS Act, that second part has been somewhat formalized. You can do that more easily today. It’s called “testing the waters” meetings. Now, some management teams probably place too much value on it. To your point, you weren’t going and meeting with a cast of thousands. You were meeting with a narrow group of believers.

Jeff.: A half dozen. A half dozen.

J.D.: All right. And so, what I tell people is, beyond a certain number you hit diminishing returns. It is particularly helpful for a unique business that you don’t think you can get a full appreciation for in that one-hour meeting. And so, it’s a business-to-business discussion, but have a discussion with your bankers around whether the “testing the waters” meetings have value on a relative scale for you. But that’s something that the market has enabled with the JOBS Act. Layer that into your timing equation.

I think the other piece of advice I’ve given people is, time the IPO for your business and your team — your team, including your board and investors. Don’t time it around the market. Time it around the right time for your business. Think about the scale that you need to be at to be a public company. One of the questions we get is, you know, what market cap is too small? Well, below certain thresholds, we just shrink the number of investors who will buy the deal. And that’s not a good leverage thing for the company.

Sonal: But there are small-cap IPOs out there that are okay?

J.D.: There certainly are.

Sonal: In fact, that seems to be a growing trend in some ways.

J.D.: Yeah, there certainly are. I think that you have to think about — how unique is the business? If there are four public companies that give an investor the same exposure, and three of them are of decent market cap, and you’re going to be the very small one, you better offer something different. OpenTable was certainly unique, and thus, way more leeway from the market.

Jeff: There’s also, I mean, you can go public too early. You can go public too late. And if you want a multiple, you wanna be a growth stock.

Sonal: Why does that matter to be a growth stock?

Jeff: You get a different fundamental valuation and a different set of investors who are willing. If your growing investors can say, “Boy, if they keep that up for five years, look at that envisioned future. That’s wonderful.” You’re not growing, they look at it and say, like, “It ain’t gonna get any better than this.”

Sonal: And it does seem like the best technology companies are like that.

Jeff: Oh, yeah, it’s — you want to have the perception you’re a growth company, which actually means you have to be delivering growth results. And typically, growth rates decline over time. We were in the teens — year over year growth rate, according to the analysts’ models — when we went out. That’s not really a strong growth company. Then we increased growth in the 30%, 40%. That was a growth company. So if you wait too long — but too early, if you’re not ready to be a public company, you can — you know, your business is unpredictable, you know, a bunch of things there. So there is this element of, okay, the biggest timing is from the company’s perspective, not from the market’s perspective.

Sonal: That’s a really great shift in mindset. So, just one quick thing then. You mentioned results. And that is obviously, like, the quarterly results, the earnings calls, and the whole song and dance that goes around that. How do you manage that?

Jeff: So, well, two thoughts, one is on the timing of going out. When you go out, you don’t want to miss the first x quarters — [don’t know] how many x is — but you wanna be highly confident you can exceed the expectations that your investors have. So, I usually counsel CEOs, before they go out, have something in your back pocket. We had two or three initiatives that we tested at small scale, that we knew were gonna work and add to the business. So, I was highly confident.

Sonal: But isn’t there a tension, though, that — of course, you wanna make your numbers and I get that’s important for the market confidence in you as a company. But it’s also very frustrating, because — a criticism people say about IPOs is that, then, you’re now wedded to this ridiculous quarterly measurement of innovation.

Jeff: I think that’s whether you will allow yourself to be or not. The pressure is there to conform to investor expectations. You know, “Oh, my god, I’m gonna miss their number.” That’s the most oft-cited reason for not wanting to be a public company. Look at Jeff Bezos. He’s run the business exactly as he wanted. He told investors exactly how he’s gonna run it. He’s been completely consistent with that. And some of his investors bought in his IPO. They’ve stayed with him, what? 20 years now. And so…

Sonal: I actually heard some statistic just yesterday, because of its anniversary, that some of the people who got the early IPO, they might have only spent, like, $100, and it’s now worth $64,000.

Jeff: So, part of it is, what investors did you recruit? And then how do you communicate with them? So, we had an interesting early question. Do we give guidance? Do you just say, “Next quarter, we think we’re gonna do revenue of x and earnings of y.”

Sonal: Isn’t that what analysts do?

Jeff: Well, analysts do that. But often the company gives a range, and that helps the analysts cue in on the range. If the company doesn’t give it, you’re leaving the analysts to come up with their own numbers. I called — tough question — [should] I give guidance. I called the three largest holders, and I said, “Should I give guidance?” All three said no. I go, “Why?”

Sonal: Why?

Jeff: And they go, “Well, we want you to do what’s right in the strategic long-term interests of the business. If you give guidance, there’s gonna be pressure for you not to do what might be right, with new learning.” And it helped that our business was highly predictable from outside. You know, it’s not one of these, like, did we close the last deal on the last day of the quarter? You know, the diners are being seated by the millions. And so, the law of large numbers kicked in. It was very predictable.

J.D.: With great metrics. Like, you guys invested in explaining the metrics that matter to you as a management team. Periodically, we hear people get this mantra of no guidance, and they interpret it as no communication. They’re two very different things. Right? You were not signing up to a quarterly number. But it was fine because you were giving some much transparency as to what drives the business.

Jeff: Right, they can build their model.

J.D.: They can build their model.

Sonal: Okay, so any last parting thoughts?

Jeff: Taking OpenTable public was one of the most interesting things I have done in my business career. And part of it was I — turns out, in my career, I’d never done a financing. So, for my first financing to be taking OpenTable public, in May 2009, at the depth of the financial crisis.

Sonal: The worst time.

J.D.: It was an amazing learning experience, and there’s a lot of emotion into it. I think I was more exhausted — I ended up with the swine flu at the end of the process.

Sonal: Swine flu. I haven’t heard that phrase in a while. That totally ages that time again.

Jeff: It was the height of the craze on that too. Not only is the whole economy coming to an end, but we’re all gonna die. And so we knew it was a problem. We’re walking around with bottles of Purell. But we shook 400 hands. Your life is not your own.

Sonal: Thank you for joining us in this podcast.

Jeff: Thank you. Thank you, J.D.

J.D.: Thanks, Jeff.

Jeff: It’s good to see you.

• 

  J.D. Moriarty
• 

  Jeff Jordan is a managing partner at Andreessen Horowitz. He was previously CEO and then executive chairman of OpenTable.

  Follow Twitter
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter