Today’s episode is a conversation about four big trends in the tech world. Any one of these trends would be notable on its own, but we cover all four in this hallway-style chat, as a16z General Partner Chris Dixon talks with Sep Kamvar (professor of Media Arts and Sciences at MIT and now cofounder of cryptocurrency platform Celo); and Elad Gil (investor and the cofounder of health technology company Color Genomics, and formerly at Twitter and Google).

This is a wide-ranging survey of some of the major shifts in technology right now, but it’s really a meta-story of how innovation happens, which is most definitely not in a straight line.

So here are the trends they cover:

*crypto (of course);
*AI and machine learning (including GPT-3 – you can also listen to our explainer episode on what’s hype/what’s real there on our show 16 Minutes);
*full stack startups (which Chris first wrote about in 2014);
*and collaborative web/collaborative enterprise/ social (including RTC or real-time communication within the browser), which is where the conversation begins.

Show Notes

• WebRTC and WebGL and the move toward collaboration [1:23]
• The rise of crypto [5:30] and how new technologies improve over time [8:54]
• The current state of AI, with a special focus on GPT-3 [13:57] and how more advanced AI technology may be able to replicate itself [17:25]
• Discussion of full-stack startups [21:33] and why there aren’t more of them in the market [25:28]

Transcript

The next big platform shift

Chris: Elad, you and I have been talking about this. And I know you’re very excited about it, this kind of this feeling that there’s a new stack of web infrastructure, things like video and audio, collaborative video and audio, rather, we sort of have the infrastructure now that it works in a way that it hadn’t in the past. And that’s unlocking a whole new wave of interesting applications.

Elad: People are always looking for the next platform and what the next big platform shift is. And I think it kind of may have snuck up on all of us in the form of WebRTC and WebGL and then related API companies providing sound or other things that then built on top by many other companies. And I think this shift is substantiating itself in two different ways and I almost call it the collaborative web and then separately, the collaborative enterprise.

And if you look back 10 years, people kept talking about during the first social wave, everybody kept talking about how there’s going to be a social enterprise and how every SaaS product was going to be more social and collaborative and that largely failed. And it feels like that shift is finally happening in part due to things like WebGL. You see Figma, for example, is the first really strong example of a WebGL-enabled application, allowing you to collaborate in real-time with other people.

In parallel, WebRTC is really allowing for really interesting concurrent sessions around video. And so, you’re starting to see that in terms of a lot of products being built around virtual office rooms, virtual conference rooms. And I really do think this is the moment where collaboration is finally being built into the enterprise world and enterprise products. And then in parallel, WebGL and WebRTC really seem to be enabling really interesting social experiments right now in terms of new social products.

You have really amazing video and audio quality. So the time lag is gone so you can do things like Clubhouse. We see lots of interesting video experimentation. So you can see almost like degraded forms of VR or other things happening in browser. So, I just think now is a really exciting time of innovation around this new web stack.

Chris: And to your point about sneaking up on us, we’ve obviously had, you know, the ability to have conference calls, group audio for, you know, decades, right? Like, the fact that they’re (now) so low latency and you’ve got like the visual representation of the room means. To me, it’s like, if you remember the old days in the conference calls how you always have people talking over each other partly because of, whatever, 300-millisecond delay?

It’s remarkable how the conversation switches from person to person. It’s the latency. I mean, we’ve all now experienced this with Zoom, right? Like the fact that it doesn’t stutter, the fact that, you know…or very rarely does, like, it’s somehow kind of crossed over this point of good enough.

Elad: We’re finally hitting the point now where in terms of video quality and the ability to stream concurrently across multiple users and in terms of audio quality, we’re hitting that point where the web infrastructure is really supporting the ability to have extremely low latency.

Chris: When you call it a new platform…we’ve cited a few examples. But when you say platform, that means you think there will be thousands of examples or do you think it’s gonna be a whole new wave that goes 5 to 10 years?

Elad: I think like any “platform,” there are going to be a handful of things that really matter that will really be the important things on it. And then a lot of things will be experiments that fail or don’t work. And I don’t know 10 years from now what’s going to be the main set of applications. I just think it is a shift that enables a bunch of new applications to be built, particularly either social or collaborative enterprise.

One example that I think is worth noting in terms of what’s coming due to WebRTC is it’s quite possible that if you look at virtual reality or VR, the predominant use case in the near term may actually shift to the browser. And so, I think right now, in order to experience VR, you need a headset. You need, in some cases, client software, etc.

And so, there’s more obstacles and hurdles to be able to just participate. And I think one of the things I found really interesting about WebRTC and WebGL is the ability to suddenly create VR-like experiences where you just drop any URL and you can show up. And so, the big question in my mind is Oculus almost like the desktop computer versus mobile devices, where the desktop really helps you do powerful tasks but you can do a lot on your phone and it’s sort of the mainstream use case for most of the internet today. So, I think that’s another thing that we’ll see if it happens or doesn’t happen over the next decade. But that may be one interesting long-term trend to watch relative to WebRTC and WebGL.

Crypto, and patterns of innovation adoption

Chris: So, let’s talk about the next trend, crypto. We’re all involved in this. Elad, you invest in crypto. Sep, you’ve co-founded a company, Celo, in crypto. Obviously, I spend most of my time investing in crypto. So, can you tell us a little about why you’re excited about it and the stuff you’re working on in Celo?

Sep: I’ll start off with kind of a general principle that I think is true for all of the technologies that we’re talking about. There are certain class of technologies that increase the expressive range of a certain medium. And when you increase the expressive range of a medium, a lot of things pop up that were not possible before because you now are playing in a new design space.

The historical example that I always love to point to is in the 1800s, the invention of the metal ferrule in painting — the little piece between the paintbrush and the paintbrush handle — and the collapsible easel. Those two things together allowed people to A, bring their paintings outside, and B, start to paint with a new brushstroke that allowed them to quickly dab paint onto the canvas. And those two ended up kind of giving rise to a form of painting that we now know as Impressionism.

And so, it’s interesting to think about that. Impressionism was a result of technological advances in painting. And you see that same thing with the web and the internet in general. There were technological advances in the medium of text. And so, all of a sudden, people could send text more quickly. Anybody could be a broadcaster. You could start putting text together with code to create different things. And that vastly increased the expressive range of text in a way that led to all of these things that you could not predict in advance.

So, for example, in ’94 and ’95 when the web was starting to become popular, one could not imagine that, “Oh, well, one day I’ll be able to press a button and order my groceries on this and have my groceries come to me,” you know? And so, I think those are really interesting from a broad-brush technological point of view.

Why I’m excited about crypto is that crypto does this for money. It increases the expressive range of the technology that we know as money. And that I think will follow very similar to the internet. You know, at the beginning of the internet, you saw it allowed people to pass messages more quickly to one another across a distance in a way that was just qualitatively different than fax.

And that is like the first thing that you started seeing with crypto and it has direct implications to things like remittances or banking the unbanked. But then on top of that, the second implication of the web was that anybody could become a broadcaster. I mean, with YouTube, anybody could have their own TV station. And in the context of crypto, you have the same democratization but in financial services. And so you see this kind of rise in decentralized finance or open finance.

And then third, most exciting, is it allows money to become programmable in the same way that the internet allowed text to become programmable. And that, I think…I mean, we’re seeing some early things today. But that’s, I think, the aspect that we’re still the earliest and it has the most legs. And it is the most powerful and the most difficult to predict at this stage since we’re in such an early phase.

Chris: My framework for this is when there’s a really big breakthrough technology, there’s two stages. And the first stage, you do things you already did but do them better. And the second stage, you do new things you never could do before. And this goes back to the collaborative web stuff we were talking about before. Like, in the first stage, you know, we’re gonna do better video conferencing, right, better audio conferencing and that will probably be a wave that lasts a few years.

And then at some point, people will start to figure out this is a whole new set of things we’ve never done before. Like, the analogy on the web, right, is the first era in the ’90s, people were just kind of putting websites up. They were basically one way. They were brochures and magazines. But then it took another decade to realize there’s things you can just do that you could never do before like social networking, right? It’s multi-way medium, not a one-way medium, right?

It’s similar to my understanding of the history of film. When film started off, you know, they filmed plays, right? And then they realized you could do all these new kind of film-native things, right? And I think crypto will be the same thing. And you hear…the mistake people make is they say, “Oh, great, you can lower payment fees. You can send cross-border payments.” And all of that is true but that’s only phase one, right? Phase two is things we can’t think of, we can’t even imagine.

It’s funny. If you go back and you look at all the ads for mobile phones, like, for 10 years, Nokia and all these folks, they were all trying to convince people to use mobile phones and there’s always stocks, weather, email. There’s literally I think no person in the history of that field that predicted, you know, half the things that we’re using today. So, for me, I think that framework kind of applies whenever there’s a really big breakthrough technology. It just takes a long time to really explore the new design space it was on.

Sep: And, you know, I think one of the reasons for that is a lot of times, the things that are new arise from the things that are old just at scale, at quantity, you know? And that’s actually really interesting because it helps give a framework for predicting things. So you could imagine, for example, blogs were predictable from zines before the internet, you know? But it would be qualitatively different because then you imagine what happens if there’s like thousands and thousands of zines and anybody could access those zines and so on?

And so, then, that kind of starts the creative process going.

Chris: And then, I’ve been directly involved in this, in the infrastructure stuff, people were working on it but it was frankly a little academic until recently. And so, the fact that the applications have taken off so much, and it’s made this scaling problem like a really, really urgent issue. I think it will dramatically accelerate the pace of innovation on the infrastructure side, right? It’s no longer academic, it’s now a very practical problem and there’s real customers and people willing to pay money.

And, you know, the same feedback with…you’ve seen, I think, throughout the history of computing where the app developers on the first iPhone start pushing it to the limit. And that pushes Apple to, you know, go faster and the chip guys to go faster and the whole thing. And then you get that beautiful flywheel that drives everything forward.

Sep: And this is something that’s been very much on our minds as we’ve been developing Celo. So, basically, kind of when we started Celo, the conversation that we were having was, the blockchain reminded us that money is just a technology. And, of course, money has always been a technology. It’s just hard to remember that it’s a technology because its features haven’t changed very much for the past 300 years. But as a technology, its features can change and as a widely used technology, its features have an impact on the society that uses them.

So, I remember when the internet was first getting popular, people were like, “Whoa, you could imagine putting the whole encyclopedia on the internet.” And that was true, but it underestimated the true potentiality of the internet, which was that the encyclopedia would be part of a much richer, much bigger information ecology.

And so, similarly, I see the same thing happening in money, in value. National currencies will continue to exist and continue to be important, but there will also be local currencies, regional currencies, global reference currencies, store-of-value currencies, medium-of-exchange currencies, functional currencies, all interoperating with one another in a rich ecology, not dissimilar to the internet.

We now are starting to have the technology to implement these ideas at scale. But to do a number of these things right, we needed some form of stabilization of cryptocurrency. We needed some methods around identity. We needed advances in light client, and so on. And so, that helped guide the infrastructure that we’re building to enable this.

Chris: It’s going to be an exciting year in crypto. Celo has launched and is continuing to roll things out and a whole bunch of other exciting crypto projects. And so, sort of all of the things that were kind of hatched back in 2017, in ’16, ’17, ’18, are kind of finally all coming out now and it should be really exciting.

Elad: It just seems like that next wave is starting up again too in terms of incrementally new things. Like YFI (Yearn Finance) I feel like just came out of nowhere, for me at least. And so I think that we’re going to see renewed enthusiasm, I think, in crypto in the reasonable near term.

AI, GPT-3, and ‘hyper-evolutionary’ new forms

Chris: Let’s talk a little about AI, sort of the other…it’s amazing right now. I feel like any one of these things would be a major tech trend and we have all of them going on at the same time. So, AI, I don’t personally work on it as a day job but follow it, I guess, as a hobbyist. The big news being GPT-3, which is an algorithm out of OpenAI, which has just shown kind of remarkable results with natural language processing.

And from what we can tell, this is not going to be slowing down. Today the more computers you throw at these kind of neural networks, the smarter they get. And at least at the moment, these systems continue to scale at a pretty healthy rate. So we should see kind of more and more really interesting stuff. Elad, I think you’ve followed this area pretty closely. How are you feeling about it?

Elad: I think GPT-3 is almost like the starting shot for a whole new interesting era in natural language processing or natural language understanding that’s going to take a decade to play out. And I think the historical antecedents or analogs are, back in 2012, there was something known as AlexNet from this guy, Ilya Krzyzewski, which was really the starting shot for machine vision in terms of a shift where that was the first time where you really saw a big step up in performance for a while and that’s really led to everything from face recognition on the iPhone to machine vision in pharma.

Similarly, in 2013, Google switched to recurrent neural networks for speech recognition and then later really did a lot of interesting things in deep reinforcement learning. And that ended up becoming a multi-year precursor to what became things like Amazon Alexa or Echo or a lot of the really good speech recognition technologies we have. And now in 2020, I think similarly, GPT-3 is a natural language analog to these two other key moments in machine learning-based understanding of vision, speech, and now natural language.

I actually think this may be one of the biggest shifts because if you think of how much of the world’s information is embedded in text or how much we communicate in text, this is really the big revolution. And that includes things like enterprise document processing. If you move to natural language, you can start thinking about smart data entry. All the robotic process automation suddenly becomes automated. You can effectively have APIs, in some sense, almost self-construct on top of text in really interesting ways.

There’s things that are very tactical. For example, in your email inbox, all the replies should be auto-generated and then you should just be able to go through and approve them as a person. We’re not there yet, again. It’s a 10-year journey. But, you know, we’ll see things like that. We’ll see legal documents just auto-marked up relative to what your company would normally do. Companies like Clarity are working on early versions of that. If you’re an author and have writer’s block, maybe automatically, you get prompted for three or four different next paragraphs to kick off how you should think about it.

Or in the long run, maybe there’s a whole class of auto fan fiction. So, you know, you really love the novel “Twilight,” and 100 different versions of “Twilight” are spawned. So you don’t have to wait for somebody to come up with “Fifty Shades of Grey.” It just auto-generates, you know, multiple different interesting, you know, fanfic stuff.

On the gaming side, I think you’ll have non-player roles, NPCs, that seem like real people. In health care, maybe you have a mental health specialist who’s really just a robot.

I think this is a really exciting shift and it’s going to take a long time to play out but the technology is finally starting to show hints. Just like in 2012, AlexNet showed hints of what could happen in machine vision. And in 2013, Google showed what could start happening in speech recognition. It feels like this is one of those steps. And so I think it’s significant in terms of a starting shot, although, I think it’s going to take a lot of time to play out.

Sep: I’m really excited about the translation opportunities, in particular the opportunities to translate English to machine understandable code.

Chris: They’ve actually had demos of this with GPT-3, right? Where you describe something and it would actually write the code for you. I haven’t personally tried it but it seems like they’re not canned demos. It really does kind of work.

Sep: And, you know, it’s really straightforward to do that in the context of data structures. You could imagine translating a sentence into a data structure. And it’s not a far step from doing that to natural language querying of SQL. And then it’s not a far step from that to auto-generating code. And so, that’s super exciting to me because you can imagine, there are certain things that are straightforward to build if you know how to program. And they should be straightforward to build if you don’t but it takes kind of ad hoc interfacing to do. Creating a new ERC-20 token, for example, is a pretty straightforward programming task that I can see that someone could use machine translation using any of these technologies, but GPT-3 in particular, to start translating human text to machine text.

Elad: To your point, I think the second that machines can really write and edit code and can spawn instances of themselves and self-replicate, at that point, I think we’re really shifting from a technology to a life form. And I think at that point, you know, we really have this hyper-evolutionary new form of life that’s self-replicating, self-editing. And, you know, one of the interesting things is people always think that a true AGI or self-intelligent agent will come out of Google or Facebook or one of the major companies.

To put these threads together, one could argue maybe where it’s going to really emerge is on the blockchain where you have these really interesting human incentives and competition around something of real value. So you have sort of an optimization metric that’s very crisp when you’re competing to effectively complete financial transactions or contracts and they’re going to get more and more complicated. And so, I think the merger of these two areas will someday happen and it’s going to be fascinating to watch in terms of whether you have this sort of emergent system of self-replicating, self-editing code with strong financial incentives built into it.

If you look at the biology side of things, that replication plus mutability plus selection is really what drove the emergence of intelligence, right? And so, really, the selective function is you need to have a large enough number of different beings or entities. You need them to be able to change at some rate so that they start adapting to their environment they’re being selected for, and then you need that selective pressure.

And when you start having machines be able to edit themselves and to write themselves and replicate themselves at scale, you’re both expanding the number of potential entities that are evolving. But you’re also upping the clock rate. You’re not waiting for a person to write something and test it and then iterate on it and then test it again and try and understand it and theorize and then write more code. You just have systems that are replicating and changing themselves.

And imagine if as a human, you could edit your own DNA and change certain features and experiment with that very rapidly. That’s what’s going to happen in the world of code. And so, I think it’s a long time away, but once code can write itself, I think that’s really when things kick-off for the emergence of a true AGI-based life form.

Chris: There’s no reason this couldn’t be applied for any kind of symbolic system. So for a mathematician, you know, the computer suggests five different proofs. Scientists, the computers suggests five different theories or interpretations or models or whatever it might be. And maybe in the near term, it works alongside a human. Maybe at some point, the machine gets so good, it doesn’t need that. And it’s probably, going back to the framework I was suggesting earlier, the stuff we’re describing falls in the category of doing existing things better. There will probably be crazy new things that we can’t even imagine right now but some developer or entrepreneur will come up with.

Elad: The analog I’ve heard or the analogy I’ve heard for GPT-3 is it’s kind of the clever student who didn’t really study for the exam and half the time kind of bullshits it and half the time knows it. And to your point, GPT-3 can write the next paragraph. The question is, what does GPT-20 look like? GPT-50? As we iterate on these systems, you suddenly have the thing that can really write the fan fiction novel for “Twilight.” So it’ll be really fascinating to watch.

Full-stack startups

Chris: So the last thing we’re going to talk about is what some people call full-stack startups, which is sort of a new way to build startups. Sep, it’s a concept you were interested in. But essentially, the idea is whereas in the old days, software startups mostly stuck to just building software, more and more entrepreneurs are building companies that are sort of software-enabled but also build core capabilities in other areas.

So, just as an example, in fintech, it used to be that the only way you would go to market, you’d build software and try to sell it to a bank or an existing financial institution. Now, more and more, you have these things, like Chime is an example, of an online bank that just sort of bypasses Citibank and goes directly to consumers. It’s an app. You can download it.

Robinhood is another good example. Instead of building software and selling it to Schwab, they just built software and built an app and went directly to the public, right? And this is happening as sort of a new design pattern for startup organizations that’s, I think, letting startups penetrate more and more deeply into industries that had previously kind of resisted software innovation. Sep, I know it’s a topic you’re interested in.

Sep: For the audience, Chris wrote a blog post called “Full-Stack Startups” back in 2014, which is, I’d say, a must-read. It’s one of the most concise and articulate descriptions of this phenomenon that I’ve ever read. I think, basically, kind of full-stack startups were later to emerge than pure software startups for a variety of reasons. Mostly because there was a fair amount of low-hanging fruit in software itself and it is harder to do a full-stack startup because you basically have to start two companies at the same time.

I mean, if you’re starting a full-stack construction company, you have to start a construction company and a software company at the same time. And it’s hard enough just to start either. And full disclosure, I am a co-founder of Mosaic, which is a full-stack construction company. So I’m biased here. But once you’re able to do that, if you’re able to do that, then it allows something really powerful, which is it allows you to write software not just for existing processes, but it allows you to innovate on process at the same time as you innovate on software. And very specifically, it allows you to innovate on process in the way that software enables.

And so, in the same way for crypto, software increases the expressive range of a whole range of things. And that expressive range allows new processes for things like building houses or selling eyeglasses or so on. And it’s really helpful to be able to have a really tight loop between changing the process itself, which is not inherently software-based but new software allows you to do that and then to iterate on the software itself. And so, that kind of opens up an area of innovation that is really difficult to do with either side of the stack alone.

Elad: It seems like there’s a lot of other places where that approach that you mentioned, Sep, really applies. I mean, a company I co-founded that, you know, I haven’t really been operationally involved with for many years is Color Genomics and it’s doing a large proportion of COVID testing in a number of different markets. And a lot of the value, I think, of what the company does on top of just running a vertically integrated lab and other software around it, is all the virtualized care delivery and all the patient interactions, doctor interactions, etc., beyond just, “Hey, can you run a better lab?”

And so I think, to your point, that vertical integration has made a huge difference for Color as an example. And similarly in real estate, not just Mosaic, but it seems like Opendoor, which is literally going in and repainting the interiors of houses as it buys them and things like that and is layering on mortgage and title and everything else of the home purchasing process. It seems like there’s just an enormous amount of innovation in terms of the ability to build something that’s full-stack.

Sep: You know, I mean, it’s interesting. I’ve had a similar question, a little more general from, Chris, the time you wrote that blog post, which was…”Why is there not a flowering of full-stack companies in the same way that there’s a flowering whenever there’s a clear possibility of innovation?” You’re not seeing as many full-stack startups as you are seeing with crypto, for example. I’ve come to two reasons. I imagine there’s more.

The first is that it is really difficult in either context, either in the startup context because it requires kind of an expertise in two very disparate areas as a startup. And I think the second thing is I think it’s tantalizing to take a big industry and, say, dabble on some technology and it will become a full-stack company. But I think you have to have a specific point of view around what the technology is and a real innovation in that technology. And so, I think for those two reasons, I think it’s just like the rewards are great but the difficulty is hard.

Chris: Netflix is a really interesting example, right? So Reed Hastings, obviously a genius, but his prior company to Netflix was Purify, which is a debugger. I mean, it was a very, very technical product. He’s very much a computer scientist, you know? And then, he built…you know, now Netflix is doing all this original content, is becoming more and more dominant in the movie industry. I don’t know…you know, this pattern of having a technologist figure out the other industry, in that case, Hollywood, like, I don’t know if there are examples of the opposite happening of the Hollywood people figuring out the technologies part, you know?

Sep: Yeah. And there’s few examples of Netflix too.

Chris: Sure. And it’s so hard. And that company… I mean, that’s an amazing story, just independently. They had to pivot multiple times while being public. So, you know, and there’s just so few people…

Sep: Right. And he’s a remarkable entrepreneur.

Chris: Yeah. Maybe they’re just so few people like him and Elon Musk and it’s such a hard thing to do. And, you know, raising…it requires a ton of capital and decades of work. So, I think that might be why there’s not more. It’s just really hard.

Sep: For us, the way we were able to do it is Salman, my co-founder, he has a Ph.D. in computer science from MIT but he also grew up in a construction family. So he had deep expertise in construction and deep expertise in computer science both from a young age.

Elad: I think it’s kind of notable or interesting that most of the examples I can think of of really successful full-stack startups are second-time founders. So, with Mosaic, you know, your co-founder, obviously, had an amazing background in terms of family construction and everything else. But the flip of it is you, Sep, had already started, you know, companies before. Elon Musk had two successful outcomes before. Reed Hastings had a successful outcome before. So, it almost feels like you need a stable financial base plus enough know-how in terms of building a company to begin with so that you can take on this extra challenge of doing a second piece of it, you know, to Sep’s point around needing to build two companies at once.

Chris: That’s a good point. Thanks, Sep and Elad. That was awesome.

• 

  Elad Gil
• 

  Sep Kamvar
• 

  Chris Dixon is a general partner at a16z, where he leads the crypto/ web3 funds. Previously, Chris was cofounder & CEO of startups SiteAdvisor and Hunch (acquired by eBay); and an early blogger at cdixon.org.

  Follow Twitter

It seems like investors are especially obsessed with the psychology of decision making — high stakes, after all — but all kinds of decisions, whether in life or business — like dating, product management, what to eat or watch on Netflix — are an “investment portfolio” of decisions… even if you sometimes feel like you’re making one big decision at a time (like, say, marriage or what product to develop next or who to hire).

Obviously, not all decisions are equal; in fact, sometimes we don’t even have to spend any time deciding. So how do we know which decisions to apply a robust decision process too, which ones not to? What are the strategies, mindsets, tools to help us decide? How can we operationalize a good decision process and decision hygiene into our teams and organizations? After all, we’re tribal creatures — our opinions are infectious (for better and for worse) — so how do we convey vs. convince, and not necessarily agree but inform to decide? Especially given common pitfalls (resulting, hindsight bias, etc.), and “the paradox of experience”, including even (and more so) winning vs. losing.

Decision expert (and leading poker player) Annie Duke comes back on the a16z Podcast — after our first conversation with her for Thinking in Bets, which focused mainly on WHY our decision making gets so frustrated — to talk about her new book, which picks up where the last left off, on HOW to Decide: Simple Tools for Better Choices. In conversation with a16z managing partner Jeff Jordan (and former CEO of OpenTable and former GM of eBay among other things) — so, from all sides of investing, operating, life — Annie shares tips for decision makers of all kinds making decisions under uncertainty… really, all of us.

Show Notes

• Cognitive biases that affect decision-making [3:05] and tools for overcoming them [6:08]
• Disadvantages of the traditional “pros vs. cons” list [11:44]
• How long decisions should take [14:43], and how decision “hygiene” can streamline the process [20:14]
• Making decisions within groups [24:47] and shortening feedback loops [31:14]
• Considering optionality reversibility [36:08] and hedging bets through decision-stacking [40:31]

Transcript

Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal. Today, we have another one of our early book launch episodes for a new book coming out next week, by expert decision strategist — and leading World Series of Poker player — Annie Duke. You can catch the podcast we did with her a couple years ago, for the paperback release of her first book, “Thinking in Bets.” That episode was with me and Mark interviewing Annie, and was titled “Innovating in Bets” — as is perhaps also one of the signature themes of this podcast. But in this episode, we talk about her new book, “How to Decide,” which picks up where the last book left off. And the discussion that follows covers lots of useful strategies, tools, and mindsets for helping all kinds of people and organizations decide under conditions of uncertainty.

Annie is interviewed by a16z managing partner, Jeff Jordan, who was previously CEO and then executive chairman of OpenTable, former GM of eBay North America, and much more. They begin by quickly covering common pitfalls in decision-making, then share specific tools not to do and to do, including how to operationalize good decision hygiene into teams. And when to spend time deciding or not, especially when not all decisions are equal and some may seem bigger and more impactful — whether it’s investing in life decisions, like getting married, or business decisions, such as what product to invest in, or what strategy to pursue, or what market or what investment. As a reminder, none of the following is investment advice, nor is it a solicitation for investment in any of our funds. Please be sure to read a16z.com/disclosures for more important information.

Jeff: So, Annie, as the author of one of my favorite books, what motivated you to do a sequel — your new book, “How to Decide: Simple Tools for Making Better Choices”? “How to Decide”?

Annie: So, when I think about what “Thinking in Bets” was about, it was really the way that our decision-making gets frustrated by this, kind of, discorrelation between decision quality and outcome quality. And then toward the end of that book, I was — kind of, a little bit of an exploration about how you might become a better decision maker given the uncertainty, but it was mostly a “why” book. And so, this is really trying to lay out for people — how do you actually create a really solid and high-quality decision process that’s going to do two things? One is, get a pretty good view on the luck, which you need to do. You need to be able to see it for what it is. Obviously, you can’t control it, but you can see it. 

But then the other thing, and I think this was something that was really fun — I got to really dig deep into this problem of hidden information, that when we’re making decisions, we just don’t know a lot, because we’re not omniscient. And we also aren’t time travelers. And so, I got to actually do this really deep exploration into how you might actually, really, improve the quality of the beliefs that you have, that are going to inform your decisions. Which was a topic I covered a tiny bit in “Thinking in Bets,” but here we do like a super deep dive.

Jeff: It is a super deep dive. And why I love your books is — it’s so germane to what we do in our day job, which is [to] make decisions under extreme uncertainty. So, to recap, [talk about] why trying to learn from experience can go sideways.

Common decision-making pitfalls

Annie: Sure. So, you know, both of my books, kind of, start a little bit in the same place and then they diverge from each other. But I think that’s because it’s the most important place to start. What I talk about, at the beginning of this book, is what I call the paradox of experience — which is, obviously — experience is necessary in order to become a better decision maker. You do stuff, the world gives you feedback. You do more stuff, the world gives you feedback. Hopefully, along the way, you’re becoming a better decision maker, given that feedback.

The problem is that any individual experience that we might have can actually frustrate that process. We can learn some pretty bad lessons when we take any individual piece of feedback that we might get. So experience, while necessary for learning, also is one of the main ingredients that makes us worse decision makers. And it really just, kind of, comes from this problem, that in the aggregate, if you get 10,000 coin flips, we can say something spectacular about the quality of our decisions and what we should learn or not learn from them.

But that’s not the way that our brains process information. Our brains process information sequentially, one at a time. And because we’re, sort of, getting these outcomes one at a time and we’re just taking really big lessons from something that’s really just one data point. And the two main ways that that frustration occurs is because of resulting, which obviously I cover quite a bit in “Thinking in Bets,” where we use the quality of an outcome to derive the quality of the underlying decision. You can run red lights and get through just fine. And you can run green lights and get in accidents. So, these things actually are correlated at one, but with resulting, we act like they are. And then the other problem is hindsight bias. We aren’t really good at, sort of, reconstructing our state of knowledge at the time that we made a decision. And so, once we know the outcome, you not only kind of view that as inevitable, but we’ll also, sort of, think we knew that that outcome was going to occur — none of which are true. So, those two things combined are really problematic.

Jeff: You had this beautiful imagery of decision forestry, which resonated with me.

Annie: I sort of think of them as cognitive illusions. What those illusions are creating for us is the idea to say — it’s the only outcome that could have occurred. In reality, though, what we know is that, at the moment that we make a decision, there’s all sorts of different ways that the future could turn out. When we’re at the moment of a decision, we can see all those branches of the tree, where I become a fireperson, or I become a poker player, or an academic, or whatever, you know — sort of, imagining all the different ways that the future could unfold. But then after the future unfolds as it does, we take a cognitive chainsaw to that whole tree, and we just start to lop off the branches that we happen not to observe. In other words, we, sort of, forget about all the counterfactual worlds. And we end up thinking that there was only this — only this one branch that could have happened, because we sort of chainsawed everything else away. We sort of forget that there were other paths that could have occurred.

Jeff: How do you keep the forestry from lopping off the branches? As you started turning to how, you started with some really useful tools.

Tools for analyzing options

Annie: So, there’s two tools that you could do when you’re thinking about that, actually three. The first has to do with trying to reconstruct what the actual state of knowledge that you were in. When you think about — what did I know beforehand, and what did I know afterwards, you can now start to sort of reasonably see — what was the information that was informing the decision at the time? When you actually go through this process, you’ll spot, “No, wait a minute, that was something that revealed itself after the fact.” That’s one thing that can be very helpful.

Another thing that can be very helpful is to actually go through this process of thinking about this two-by-two matrix of the relationship between decision quality and outcome quality. So, there’s a quadrant, which is — good decision, good outcome, which you can think of as like an earned reward. Good decision, bad outcome — that would be bad luck. Bad decision, good outcome — dumb luck. Bad decision, bad outcome — I guess would be, like, your just desserts. When you’re thinking about any outcome that you’ve had in your life, if you do that over time, what you’re going to see is that you’re going to have certain patterns about which quadrant you’re really filling in a lot. So, if you’re seeing that you’re really only putting things into, like — good, good, bad, bad — you need to start seeing how luck is influencing you.

And then the other thing you want to do is just start thinking about particularly the good, good quadrant. Because we are asymmetrically willing to go and try to find some luck in there. Let me explain what I mean. So, if you have a bad outcome, you already feel bad. You’re sad because you lost. And it’s, kind of, nice to go in and deconstruct that, and analyze process, and really look at the quality of the decision that led to that outcome. Because if you find some bad luck in there, you get a little relief.

Jeff: You, kind of, get off the hook.

Annie: Right. It’s like a door out of the room. Luck is giving me a way out of this. So, we’re actually pretty eager to go around and explore those bad outcomes. What we’re not so eager to do, though, is when we have a good outcome — to apply the exact same process. To actually spend some time in there thinking about, “Well, you know, what was luck or was there a better way?” And the reason why we don’t want to look at that is because we feel pretty good. If you find out that you won because of luck, that’s a door that you actually don’t want to have open for you. 

So, I actually put a lot of focus, when I’m thinking about using this tool, of really digging into that one quadrant. And what you can see is, in order to actually be thinking about which quadrant that fits in, you have to actually apply this other tool — which you can do in retrospect — which is actually to do some exploration of, like, what are the other things that could have happened? Because if you don’t understand those counterfactuals, it’s very hard to actually appropriately place any outcome into the right quadrant. So, I have tools in the book which will help you, sort of, reconstruct these things retroactively.

Jeff: It’s kind of interesting. The investment community often tries to capture the thinking at the time through the investment memo. Which then, you know, records, okay, you know — these are the potential outcomes that we can envision, here are the probabilities of the different outcomes. And in total, we’re willing to make this bet, even though there are some outcomes that are pretty unattractive, to say the least.

Annie: And that, absolutely — if you think about a knowledge tracker, that’s what you’re doing. It’s like you’re trying to reconstruct an investment memo. It’s better than nothing. But what you really, kind of, want to be doing is — doing this stuff prospectively. You want to have some sort of record of not only what you thought at the time, but also exactly what you said. Like, what are the ways that we think this could turn out? Like, what are the payoffs of each of those possibilities? How probable do we think those are? So, you can actually look at, generally, two things — what’s the expected value, what’s my downside risk. And then you can, obviously, compare options to each other. What I think is actually really important, though, about thinking about this, like, evidentiary record — that you’d like to create at the time of the decision, as opposed to [trying] to reconstruct, is that it’s not actually an extra step.

Like, people talk about decision journals, which feels like work. Because it feels like an extra step where you’ve done the decision, and now you’re trying to record everything. The fact is that a really great decision process is going to produce this evidentiary record naturally. And obviously we’d prefer to have that, because what the evidentiary record is giving you — what that investment memo is supposed to give you — is, sort of, what your expectations of the world are. Not just like, do I think I’m going to win or lose at this probability, but also, like, what do we think is going to be true with the world in general? What I find in my work is that when people lose, they’ll do these process dives. The problem is when there’s a big win, they’re like, “We won.”

Jeff: Yep, exactly. When an investment goes bad, you do spend time trying to say, “Okay. What can I learn? What can I do differently?” And then when it goes well, you just spike the ball in the end zone and do a dance.

Annie: And we really are just, like, spiking the ball. But there’s so much to be learned from the wins as well, and I would argue, actually, more. Particularly, by the way, when the <inaudible>, it’s like, there’s going to be — in a lot of ways, there’s more to be learned from the wins than the losses, right? Because the thing is, like, you know, you can win for all sorts of reasons that you didn’t expect. And yet we spend a lot more time in our decision process, exploring the losses that were for reasons that we expected — than the wins that might’ve been reasons that were unexpected. Maybe we could have cleaned up the process, or there was information that we’re missing that we could have applied, so on and so forth. We’re, kind of, losing a lot of the learning time. We’re not being very efficient when we do that. And the other problem with that is actually that that has downstream effects that are quite bad. I’m going to do things that are very consensus. So, I’m going to want everybody to agree with me.

Jeff: Yeah. That resonates a lot. So, you take on using a pro-con sheet. And it was funny. I was cleaning up offices a couple years ago, and I found sheets in different places, and aggregated my career decisions. And, you know, I came to the conclusion that they were pretty much worthless. And so, you come to the same conclusion in the book. Why are pro-con sheets worthless?

Annie: So, let me just say, a pro-con list is actually a decision tool. And if you have a choice between that and nothing, I think a pro-con list is very slightly better than nothing. But here are the problems with the pro-con list. The first is that it’s flat. It lacks any dimension. It’s like a side-by-side list — here are the pros, here are the cons. And I don’t really understand how you would weigh one side against the other without adding some dimension to that list. And that dimension would be two things. One is, how bad? What’s the magnitude? The other dimension that’s missing, which is terrible, is probability. 

So, in that sense, I’d rather just use the decision tree. And for an option that I’m considering, I want to just think about what are the reasonable possibilities, what are the payoffs for those, and what are the probabilities of those things occurring? And then I can add that dimension back in. Without that dimension, it’s not a great tool for comparing one option to another — because, again, I can’t calculate, like — any kind of weighted average here. If, like, I’m choosing between two colleges, is the one with more pros, like — am I supposed to not go there? I really, kind of, don’t know, because I don’t have this dimension.

And then the third problem, which I think is actually the most dire, is that what we’re really trying to do is to reduce the effect of cognitive bias. Pros and cons lists actually amplify all of that stuff. It’s, kind of, a tool of the inside view. And let me just say for people listening — I imagine some people saying, “No, when I go to make a pros and cons list, I haven’t decided yet.” I have news for you. The minute you start thinking about a problem, you’ve already started deciding. You know, regardless of whether you’ve made that explicit or not, you’ve already started to get yourself to a conclusion. And now when you go to do a pros and cons list, it’s just going to amplify the conclusion that you already want to get to. So, I think it’s just not a very good tool.

Jeff: My worst career decision, by a mile, was joining a company called real.com, right at the beginning of the internet era. It was being purchased by Hollywood Entertainment, which ran the Hollywood Video stores. And it was a bad decision. I <inaudible> in a year, I got scars. But when I went and saw the pros and cons, the pros were aspirational and the cons were delusional. I clearly had decided before I started the list.

Annie: Yes, exactly. When we start to use something that feels objective, like a pros and cons list, we get that feeling of like — well, now I can have confidence that it’s a really good decision. So, one of the things that I’m very wary of — is that I think that there’s certain things that can come into a decision process that feel like it’s certifying the process. So, we end up with this combo of a decision that isn’t really better, but that we feel is much more certified.

How long a decision should take

Jeff: I love the tools you described using the decision trees. The prospective gathering of information. Then you took your “how” into an interesting direction. And I really enjoyed the part on spending your decision time wisely. <Oh.> So, it’s a book about, you know, making great decisions — and then you start talking about all the decisions that you shouldn’t apply it to.

Annie: <laughter> I know. So, I spend the first six chapters really, kind of, laying out what a pretty robust decision process would look like. And then I, sort of, take a hard left and I say, okay — so now that you know, mostly you shouldn’t be doing that. Which I know sounds a little bit odd, but it’s this meta skill of understanding that obviously you can’t take infinite time to make decisions, because opportunities expire, and you’re losing the ability to do stuff in between. And so, we want to really think about what types of decisions merit taking time, and what types of decisions merit going fast. And it just turns out that most of the decisions that you’re going to make on a daily basis are ones that you should be going fast on — much faster than you actually do. And in some ways, I think that people, sort of, have it reversed.

Jeff: Throw out a couple examples, because that’s where it really came alive to me.

Annie: Okay. So, let me ask you this. What’s your guess — obviously pre-pandemic — what’s your guess on the average amount of time that an adult in America takes on — what to watch on Netflix, what to wear each day — I mean, at the moment it’s sweatpants, but, you know, we’ll ignore that — and what to eat?

Jeff: If you’re my mother-in-law, she used to spend a half hour every time we went to a restaurant.

Annie: So, like, she’s not even that much of an outlier. If you add it all up, over the course of a year, the average adult is spending between six and seven work weeks — like, literally, on just those three decisions. I’m sure she’s looking at the menu, and then it’s quizzing like all the waitstaff, and asking everybody else at the table what they’re going to order — like, try to go back to the chef, looking on Yelp. So, here’s my question for you. Let’s say that we ate a meal together, and you were trying to decide between two dishes. Like, what are two dishes that you would have a hard time deciding between?

Jeff: Fish and a good veggie stew.

Annie: Okay, okay. So, you’re trying to decide between those two things. If you’re [your] mother-in-law, you’re quizzing everybody. So, let’s imagine that you ordered the veggie stew and it came back. And let’s imagine you got this bad outcome, where the food was really yucky and you didn’t even finish it because it was so gross. So, now let’s imagine it’s a year later and I say, “Hey, Jeff, how are you feeling right now, happy or sad? So, you remember that horrible veggie stew you had a year ago, how much of an impact did it have on your happiness today?”

Jeff: Zero.

Annie: Zero. Okay. So, let’s imagine I catch up with you in a month and I say, “Hey, Jeff, feeling happy or sad right now? Do you remember that horrible veggie stew you had like a month ago? How much of an effect on your happiness did it have today?”

Jeff: None.

Annie: None. What if I catch you a week later, by the way?

Jeff: None. Now, if it had been the fish that had been bad, like, a week maybe…

Annie: Maybe, but not the veggie stew. <laughter> Okay. So, what I just walked through with you is something I call the happiness test. I use happiness, generally, as just a proxy for, are you reaching your goals? Because we’re generally happier when we’re reaching our goals. So, you can substitute any goal that you have in there. And this is a way for us to figure out how fast we could go. Because, basically, the shorter the amount of time in which your answer to the question is — “Did it affect your happiness at all?” — is no, the faster you can go. Why? Because there’s a tradeoff between time and accuracy.

So, in general — not always, but in general — the more time we take with a decision — and there’s more time for us to, like, map these things out, and actually calculate, like, expected values, and figure out what the volatility might be. Or gather information, get more data, all of those things. Generally with time, we should be increasing our accuracy. So that’s why we can speed up — I’m assuming no food poisoning here — that when we look at the worst of those outcomes, that it has no effect, it’s neither here nor there. Which means that — we can take on the risk of saying, “I’m going to spend less time, because I’m willing to risk the fact that I might increase the probability of the worst outcomes, because it doesn’t really matter to me.”

Jeff: Then you make another point — that you can repeat the decision, next day at the restaurant — and order the fish instead of the tasteless stew.

Annie: That’s the other thing that you can look at, which is — when you have these low-impact decisions that are quickly cycling, and they repeat very quickly — so that’s, like, what to watch on Netflix, what to wear, what to order at a restaurant — we should go really fast for two reasons. One is you’re going to get another crack at it in like four hours. And then the other is that — one of the things that we actually don’t know well, although we think we do — is, like, our own preferences. We’ve all had that experience of having a goal, achieving it, and realizing that wasn’t really what we wanted in the first place. And then there are certain types of decisions where it’s just really helpful to, sort of, get some feedback from the world.

So, when we can actually cycle these decisions really quickly — and I’m not really too worried about, like, making sure I’m making the best possible decision in terms of accuracy. What I’d rather do is get a lot of cracks — get a lot of at-bats — so that the world can start giving me information back more quickly, and I can start cycling that feedback a lot faster. Then I’m going to build much better models of the world. And what my own preferences are, and what my own goal goals are, and what my own values are, and what works and doesn’t work. Such that when I do actually make a decision that really matters, my models of the world are going to be more accurate — by having just, sort of, like, done a whole bunch of stuff really fast and not really cared whether I won or lost.

Practicing decision “hygiene”

Jeff: That makes perfect sense. Now, one of the chapters that I loved was decision hygiene. I found this book fascinating from the perspective — as both an investor and a former operator. I mean, an investor, it’s obvious — you’re making two or three investments a year. You’re seeing, you know, hundreds of companies. How do you decide? But as an operator, there are a few decisions you make each year that are super, super important. Particularly, the ones that I used to labor over was — okay, you have to commit. You have to invest your product resources — your most valuable asset, your engineers — into specific deliverables. You know, is it going to be A, B, or C? And that’s the most important decision I made all year, other than possibly people decisions. Explain a little bit — how you can maintain great hygiene. It resonated in both my professional experiences in a really significant way.

Annie: I have to say, like, the decision hygiene stuff — and the ideas of predicting these intermediating states of the world — apply so much in a startup environment. Because obviously, kind of, the nature of a startup is that you do have very little information, and you’re making pretty big bets on a future that, by definition, is going to be somewhat contrarian. So, making sure that you don’t get in this, kind of, group think. Like, I’m not saying don’t believe in yourself, of course — but this is actually a way to have more belief in yourself. Because the quality of the decisions that are going to come out of a good decision hygiene process are going to be so much better. And that becomes much more important in a situation where we are at a paucity of information. And then it starts to actually close feedback loops more quickly for you, which also increases the quality of your models and information. So, I actually can’t think of a place where this is more important than in a startup environment.

So, let me just start, kind of, the premise — why you need some decision hygiene. I don’t have control over luck. What I can do is, I can make decisions that reduce the probability of a bad outcome. You know, even if I make a decision that’s only going to have a bad outcome 5% of the time, I shall observe it 5% of the time. And luck is what is determining when I observe that bad outcome. So that’s kind of one side of the puzzle. The other side of the puzzle has to do with how you construct your decision process. What do you think your goals are? What do you think your options are? What do you think your resources are? What do you think those possibilities are for any given option you’re considering? What do you think the probabilities of those things occurring are?

Basically, your whole process is built on this foundation. Like, that whole house is sitting on top of a foundation — which is your beliefs. And by beliefs, I don’t mean things like religious beliefs. I mean, just like — what are your models of the world? How do you think the world operates? What are the facts that you have? What’s the knowledge that you have? And that foundation that that whole process is sitting on has two problems. One is that a bunch of the things we believe are inaccurate, so it’s like cracks in the foundation. And the other is that we don’t know very much. So, it’s like a flimsy foundation. The solutions to both problems are the same — which is that we need to start to explore that universe of stuff that we don’t know. That’s where we run into new information that helps us beef up our foundation. And it’s also where we happen to run into corrective information —  things that can correct the inaccuracies in the things that we believe.

The other thing that helps us, too — when we were talking before about the pros-and-cons list that gets you, kind of, caught in your own cognitive bias — is to realize that like a lot of the cure to those kinds of problems is to get other people’s perspectives. So, two people can be looking at the exact same data, and they can come to very different conclusions about the data. That’s what a market is. It’s different perspectives colliding. All right. So, having set that stage, one of the best things you can do for your decision-making is finding out what other people know and what their perspectives are on the problems that you’re considering. The problem is that without really good decision hygiene, you’re not actually going to be able to execute on that properly. So, let’s figure out — how do we get this into a team setting? Basically, human beings are very tribal, and we’d like to, sort of, agree with each other more than we actually do. And our opinions are really, actually, infectious. So, in order for you to know that you disagree with me, what is the thing that you need to know from me first?

Jeff: Well, what do you think?

Annie: Right, exactly. And this is where we get into this huge problem in interpersonal communication. When people ask for feedback, pretty much 100% of the time, they tell the person what they believe first. I’m thinking about a particular sales strategy or whatever. And I will lay out for you, not just the information that you need, but I also tell you my opinions on that.

Jeff: “Give me your unbiased opinion,” right. Now that I biased you hugely, right.

Annie: Right, exactly.

Jeff: So, the reason your decision hygiene point, maybe, was so interesting to me — is you called out one of my tools that I used as an operator, which was quarantining in group settings. I found, at OpenTable, that if I walked in and had, you know — put a strategic choice on the table, there was one-and-a-half people in there who would drive the discussion, and their opinion would always carry the day. <Yep.> So, I developed a tool where I would pre — on very important, big-time, strategic decisions, I would ask everyone to send me their lists of prioritizations. And then I would aggregate them, and then feed that back to the group — to heighten the contradictions, essentially. The quiet person who didn’t really want to put a contrary point of view, and spar with the other person. All of a sudden, the data is on the table because they quarantined the gathering of it. And then I found the conversation was so much better than just throwing it open and having, you know, the charismatic, loquacious, opinionated person, carry the day every time.

Annie: I could quarantine my opinion. But as soon as someone else talks, as you just so nicely put — it’s like everybody else is infected anyway. I’m just a really huge fan of pre-work. Figure out what it is that you’re trying to get feedback about, give everybody the same information. And then actually elicit those responses. Now, the more specific, the better. So, I like them to rate it, right? Give me on a scale of zero to five. Because then I can find out, like — Jeff is a four and, you know, Annie’s a two. And maybe Jeff and Annie need to have a conversation, because it turns out that there’s quite a bit of dispersion of opinion there. What this allows you to do is — first of all, it actually disciplines your decision process, because you have to think about what are the things that matter to this decision that I’m trying to elicit opinions about. And let me be clear, it’s not that I don’t think people should provide rationales. I think those are actually quite important. It’s just that they need to have something that’s much more precise. It’s like a point estimate, because I need to see where the dispersion is and then let them give the rationale.

Jeff: I used to give a hypothetical budget. You have a million dollars. Here are the 12 ideas you can invest behind. Deploy your budget. And each person would deploy. And then all of a sudden, you’ve got something that’s really powerful. And you’ve got — oh, you loved this idea, and you hated this idea. Let’s discuss the idea.

Annie: Right, which I love. Exactly. So, you can actually see that they disagree with each other, or see that they do agree with each other. It also makes you actually think about — what are the component parts of this decision that really matter? You can start to actually create for yourself — almost, like, a little bit like a checklist — but here are the things that we need to pay attention to, and that I actually need to get the feedback on. So, what Kahneman would call these are mediating judgments. You’re thinking about what are the mediating judgments for any broader category that you might be judging on. And that helps you to really discipline the decision process. You then bring that together in one doc, and you sort it into — here are areas of agreement, here are areas where there’s some dispersion.

People get to read that prior to coming into the meeting. So, they’ve actually seen, sort of, the full slate now, of what the different opinions are in the group. This does really great things for your meetings. It makes them much more efficient, much more productive. You’re not surfacing all that stuff in the room, which just takes a long time. <Absolutely.> And by the way, you’re not going to surface all of it anyway, so that’s bad. 

But the other thing is that now you can come in and you can say, here are areas where we generally agree — yay us — but let’s not talk so much about the fact that we agree. Which is what happens in a lot of meetings, where you’ll say something, Jeff, and then I’ll go, “I agree with Jeff, and let me tell you why.” And then somebody else is like, “Yes. And I have more color to add to that,” because everybody, sort of, wants credit for that idea. But we don’t care now, because we already found out we all agree.

Yay. Yes. Right. There was this round. Cool. Right? But now it turns out that Annie thinks the earth is flat over here. Okay. So, now what are we going to do? Jeff thinks the earth’s round, Annie thinks the earth’s flat. And that’s where you really want to be focusing your time — on places where there’s dispersion. And you want to focus that time in a way where it’s not about convincing anybody of your opinion. It’s about just informing the group. And then if anybody, sort of, agrees with you, I’ll say, “Hey, you know, Sonal, you also agree that the earth is round. Is there anything you want to add to that?” So, you’ll get to say your piece. And then, “Annie, you believe the earth is flat. Is there something you didn’t understand?” Now, notice, in no way is anybody saying, “You’re wrong,” or “You haven’t thought about it this way,” or whatever. It’s, I get to tell you, “Here’s something I don’t understand.” And then we, sort of, get to the point where I say, “Okay. Explain your position.”

There’s really amazing things that come out of that process. Thing number one is you get much more comfortable with the idea that everybody doesn’t have to agree. Number two is people have different mental models. And so, you get to expose everybody to those different perspectives, and the different facts people are bringing to the table. So, the whole group becomes more informed, which is awesome. The third thing is that the person who is conveying their position becomes more informed. Why? Because in the process of having to defend why I believe the earth is round, I discovered that I actually can’t explain that very well. So, maybe I have to go google some stuff or look it up. And there’s going to be good stuff that comes out of that, because I’m going to be more likely to actually moderate — because I’m, sort of, poking around in my knowledge a little bit.

And then the last thing, I think, that comes out of this, that’s really good — is that once you get into this idea of “convey” versus “convince,” you realize that you don’t need to agree to decide. You need to inform to decide. And that the idea that all of you would be on equal agreement about whether you should do something or not is completely absurd, because we don’t have to — because that’s the whole point. If you thought that that was the goal, why do you have more than one person on the team?

Jeff: Yeah. You want a diversity of opinion. <Right.> And if you don’t tease out the different opinions, then you make an inferior decision. I actually thought this was one of my management secrets. Like, you just outed it in your soon-to-be bestselling book.

Annie: Yeah. So, actually, what’s interesting about that problem, I think that teams often act like a pros and cons list where…

Jeff: Interesting, yeah.

Annie: …we have the intuition that more heads are better than one. So, when we bring more heads into a decision, you have this decision that feels much more certified. But what we know is that when you allow people to make these decisions in, sort of, committee style, like in a team meeting — that the decision quality often isn’t better. And there’s lots and lots of science that shows this.

Shortening feedback loops

Jeff: So, one of the things in venture that is often cited as a challenge in decision-making is [that] the feedback loops can be forever. What’s your take on that — feedback loop in decision-making?

Annie: Yeah. So, basically my take is that there’s actually no such thing as a long feedback loop. Which I know sounds weird, right? Because, obviously, you’re saying, like, we invested in a company — we find out how it exits, like, 10 years from now. Isn’t that a really long feedback loop? But the thing is, I mean — going back to this idea that when you make a decision, it’s a prediction of the future — it’s not like you’re just predicting what the exit is going to be. You’re predicting a whole bunch of intermediating states of the world. And that might be just like, for example — like, what is the arc of the ability to attract talent for this particular founder? Just, like, as an example, right? You know, obviously, is it going to fund at the next round. I mean…

Jeff: It’s a good example, the funding management team.

Annie: Right. If you knew for a fact that they weren’t going to be able to hire a good team, you won’t invest in them. So, it’s really good to, sort of, make predictions about those things, and make them probabilistically. Because as you’re making these types of forecasts now, over the course — in a much faster time period, you’re starting to see — when we say that there’s a 60% chance that this intermediating state of the world is going to exist, does it absolutely exist 60% of the time? 

Because in the end, the thing that’s so important to understand is that you are saying that you’re an expert at the market that you’re investing in. So, you want to be explicit about the things — those predictions that you’re making about that market, both near term and far term — so that you don’t have to wait around 10 years. Because the thing is, you’re going to have to make another investment in between. You can’t just make the investment, wait 10 years, get your feedback, and then make another investment. And now, if you’re actually being explicit in the way that you’re thinking about those things, you can actually create much tighter feedback loops.

Jeff: It’s just aggregating to get it into a set of milestones?

Annie: Right. There’s no reason you can’t do that out in the world. One of the knocks that people will say about poker is, “Oh, but you get really fast feedback and so, pooh-pooh on you.” And I’m like, well, yeah — except that it’s really just a compressed version. There’s the end of the hand, which is what you’re thinking about. But in between the start of the hand and the end of the hand, there’s all sorts of predictions that I’m making in between.

Jeff: I’ve been an investor for nine years. The feedback loop is 10. I’ve made 35, 40 decisions. If I deferred any learning to the end, it would be pretty wasteful. And there’s another psychological thing we fight — the phrase is, “Your lemons ripen first.” So, if your company goes for 10 years, there’s a pretty good probability it’s going to have a good outcome. <Right.> But the ones that die after, you know —  can’t raise the next round, can’t have the management team. That’s when your negative outcomes manifest before your positive outcomes. And psychologically, you have to manage through that.

Annie: Yeah. So, this actually, I think, gives you a tool to be able to do that, because you have no secondary way to be right.

Jeff: Yeah, it does.

Annie: Like, how am I doing in terms of, like, calibrating around how likely I think this company is to fail? You know, in what ways is it going to fail? What does that actually look like? The other thing that comes from that is that — when you make yourself, sort of, break this into its component parts, when you actually force yourself to do that — I think it actually improves the knowledge that goes into it. Because you have to start thinking about — what are the things that I know, what are the things that I can find out? What are the perspectives that I could consider? What are the mental models that I could apply that will help me with this prediction? Because it is now recorded — it is part of that evidentiary record, which we’ve already said is incredibly important — that allows you to have that look back. And because you know that you’re accountable to it, I think it actually improves the accuracy of the original decision. Because it makes you be more fox-like rather than hedgehog-like, because you know that there’s going to be a look back.

Basically, fox-like thinking is looking at the world from all sorts of different perspectives — applying lots and lots of different mental models to the same problem to try to get to your answer. And hedgehog is like — you approach the world through your one big idea. So, you could think about, like, in investing — you have like one big thesis, instead of looking at it from all sorts of different angles. Generally, what you find is that fox-like thinking is generally going to win the day. And this is something like Phil Tetlock — I’m sure a lot of people are familiar with “Superforecasting” — talks a lot about. So, apart from the fact that you can speed up the learning cycle, I think it actually improves the decision in the moment — the knowledge that at some point someone’s going to look back at it.

Jeff: Yeah. I think that’s absolutely true, and it’s a good tool. And we may start implementing that at the firm really soon. You know, as investors, we have the — we get the benefit of being able to make a basket of decisions, you know, diversification. A lot of the people making decisions are making, like, one decision. What is the impact of optionality? How do you deal with, you know, that one decision?

Annie: So, first of all, here’s the secret. Your decisions are a portfolio, because you make many of them in your life. And I understand, one decision — like this particular product decision. But that’s actually, kind of, like a false segregation, because you’re, kind of, working across different decisions. But I do understand that some decisions you’re making feel like they’re much higher impact. Like, when we go back to the happiness test. Obviously, like, when you’re, sort of, putting your eggs in one product basket, this is something that if it goes wrong, it’s going to have a very big effect on your ability to achieve your long-term goals. But that doesn’t mean that you can’t think about, “How can we just, sort of, move fast? And then, how would we then apply this to making a higher quality decision about something like that?”

So, one of the things that we want to think about besides impact, when we’re considering how fast we can go, is optionality. Which is really just — if we’re on a particular route, how easy is it for us to exit? Can we get off the route? Because obviously when we choose a particular option, we’re foregoing other options. And there’s obviously opportunity costs to those — to not choosing those options. And what we’re doing is we’re saying, “This action compared to others is going to work out better for me, a higher percentage of the time, than other options that I might choose.” But we know that after you choose something, sometimes new stuff reveals itself, or the world tells you some things — that maybe this isn’t a road that you want to be on. So, then the question just is, how easy is it for me to get off the road?

So, one of the things that we want to look at is what people call type one or type two decisions — or Jeff Bezos says two-way door, or one-way door decisions. That when you have a two-way door, when it’s easy for you to quit — and either go back and choose an option that you previously rejected, or choose a new option that you hadn’t previously considered — that we can go faster. Because really, it’s a way to mitigate the downside, right? If I’m kind of on a bad route, I can at least get off and try to figure out how to get onto another route. So that would be like going on a date — super quittable. I can leave in the middle if I want. Getting married — a little harder, less quittable, right? So, you know, taking a few classes online — much easier to, sort of, quit than, like, actually committing to a particular college. Or renting, more quittable than buying.

Jeff: By the way, it turns out doing online classes and going to college is now the same thing.

Annie: It is. My children will tell you that. That is so true. But the more quittable something is, the faster we can go — because when we can quit, obviously that mitigates the effect of observing the downside outcomes. The other thing we can do is actually think about portfolio theory, but for decisions that we don’t think of as investments — even though all decisions are investments. Which is, sometimes we don’t need to choose among them. So, you can date more than one person at once, right? I actually don’t need to choose between these two options. I could actually do both at once. And then I can, kind of, figure out which one’s working better. And, you know, we did this with, like, A/B testing in marketing. That happens in software development — where you’re, sort of, trying to decide between two features. And you develop them in parallel, and you test them with one set of users, and another set of users are seeing different features.

Jeff: A number of businesses do business locally. You’ll have restaurants in San Francisco and LA — delivers groceries in San Antonio. You can charge — you can have different pricing approaches in the different markets and just learn. I mean, no one in San Antonio is going to know what you did in Montpelier, Vermont. So, try it out and you learn and learn and learn, then you go national.

Annie: Exactly. When we can do things in parallel, obviously we’re also better off. And then the other thing is sometimes you have an option that isn’t quittable, but you can still quit it because you can negate it. So that would be, like — let’s say that I’m invested in a stock, and it’s totally illiquid — have no ability to sell it. If I could find a stock that’s perfectly negatively correlated with the stock that I own, and I buy that in an equal amount, I’ve now solved my problem. So, I’ve quit it even though it wasn’t liquid. That’s just hedging. So, if you can find something that’s, kind of, negatively correlated with the first thing, then you can actually go faster. So that, you have to think about in advance, right? This thing is pretty illiquid. It’s gonna be hard for me to exit. Is there something where if new information reveals itself, I can, kind of, just negate that decision? And if you can do that, then you can also go faster.

Decision-stacking

Annie: So, now that we’ve, sort of, understood, like — there’s the impact of the decision, and then we have this optionality thing — like, can you quit, can you hedge? We can now get to this idea of decision stacking. Which helps us when we have to make this big bet — is to say, what are the things that I can do before — that are going to help me to gather information? So that when I do have to make that big bet that’s going to be hard to reverse, my model of the world is going to be better. So, how can I start to use this idea of making some little low-impact decisions, just to kind of see what’s going on — to do some things in parallel? I can blunt it in order to start building better models of the world, so that when I do actually put this out into the world, then I know something more about the market.

So, when you know that you’re going to have one of those on the horizon — I mean, they normally don’t just, like, hit you by surprise. So like, “Oh crap, I’ve got this decision to make!” It’s just really good to try to stack these other types of decisions in front of it. Because when you do actually have to make that decision tree — when you are actually trying to figure out like what the user uptake of something is going to be or, you know, whatever — what people are willing to pay for something —your model is just going to be so much stronger for having thought about what are the things that I could do in front of that really big decision.

Jeff: De-risking. You know, trying to get all these little nuggets of directional information to give you higher confidence in the really big decisions.

Annie: Yeah. And you can even apply this in, like, all sorts of different places. But, you know, the classic thing is dating before you marry. One of the things that I find is that when people aren’t, like, 90% sure that it’s the right path, that they’re pretty reticent to actually execute on it. But, you know, we have to make lots of decisions where we’re 60%. And by the way, when we estimate ourselves to be 60% on something, we’re overestimating that — because we’re just deciding under uncertainty. It’s just, kind of, how it is. We don’t have a lot of information. So, once you have an option that appears to be significantly better than the other ones, you just have to do a final step — which is to say to yourself, “Is there some information that I could find out that would cause me to flip this option in relation to the other options that I have under consideration?” And now it just becomes really simple. If the answer is yes, you can just say, “Can I afford to go get it?” You might not be able to because of time or money. And if the answer is yes, I can afford to go get it, go get it. If the answer is no, look — this is the state that we’re always making decisions under. I don’t have a time machine. My decision-making would be much better if I had a time machine. Sadly, I have none.

Jeff: That’s the next book, the time machine.

Annie: The time machine, right. I know, right, exactly.

Jeff: This has been a fascinating session. Thank you for spending the time with us on the “a16z Podcast,” to paraphrase Sonal.

Annie: I am so grateful to have gotten to come on and to get to discuss this stuff with you. I had so much fun.

Jeff: I’ve been looking forward to this conversation for quite a while.

Annie: No. I’m so excited, because I did get delayed a little bit due to a small misprint.

Jeff: That wasn’t a small misprint. That was a big misprint. And now, I have an eBay collector’s item, which I’m the perfect person to know how to monetize.

Annie: Yeah, right. So, for people who don’t know, is that — books get printed in, sort of, 20-page sections that get bound together. And really, a lot do with COVID — one section got printed twice, and one was totally missing.

Jeff: I was just questioning my mental facilities while reading, because I was…

Annie: But don’t worry. It’s been repaired.

Jeff: Excellent.

Annie: October 13th, when the book is out, you will get an appropriate copy.

Jeff: That will be awesome. I can’t wait.

• 

  Annie Duke
• 

  Jeff Jordan is a managing partner at Andreessen Horowitz. He was previously CEO and then executive chairman of OpenTable.

  Follow Twitter

Millennials and Gen Z have been hard-hit by the one-two punch of the 2008 and 2020 financial crises. That experience has radically shaped their approach to finances and their mindset around credit and debt. This episode explores how fintech founders are now designing products tailored to the financial challenges of younger consumers, from managing and avoiding student loans to building credit to saving and budgeting apps.

Historically, students have largely been overlooked by traditional banks. Due to a combination of economic forces, predatory lending practices, and uninformed decisions, millennials have more outstanding student loans—and owe more money—than any prior generation. According to a poll released this week by the data intelligence company Morning Consult, just 46 percent of millennials believe their student debt was worth attending college.

Amira Yahyaoui wants to change that. She’s the founder and CEO of Mos, a platform that allows students to apply for every government college financial aid program with a single application. In this episode, Amira joins host Lauren Murrow and a16z fintech partners Anish Acharya and Seema Amble to discuss how fintech can cut through bureaucracy, downsize student debt, and optimize—and ultimately automate—consumers’ financial futures from an early age.

Show Notes

• The student debt problem and how technology can help [1:29]
• Why millennials and Gen Z have different expectations around finance [6:43], how traditional banks have responded [11:18], and how fintech apps are targeting students [13:10]
• Apps that help students learn financial responsibility, as well as a discussion of alternatives to traditional four-year college degrees [17:56]
• What fintech founders should consider when designing products for Gen Z [23:38]
• What banking may look like in the future [25:15]

Transcript

The college debt problem

Amira Yahyaoui: We decided to tackle the problem of paying for college and, more importantly, accessing higher education. So we think that money should not be the reason you decide to go or not. We want to make it free and we want to make it accessible. And we’re hacking the system to make it that way.

Lauren Murrow: In that financial aid is such a maze and such an obvious pain point, why haven’t others come in and tried to compete on that front?

Amira: Helping students go to college without debt is not a new idea. But honestly, the major reason is no one really wanted to build it. Most of those who build solutions either went to college with scholarships, or were able to pay for it, or didn’t go. And when you didn’t feel the pain, it’s hard to want to solve it.

So a lot of noisy people in Silicon Valley, they will tell you, “Okay, the college degree is not necessary.” But then you look at their LinkedIn profile and they all went to Stanford. So a thing that you see, especially in Silicon Valley, is that those who are anti-college think if you can afford it, you go, but the rest of the population doesn’t need it. But that’s a big misunderstanding of why people go to college and why people need an education.

The second thing is millennial fintech was a lot about lending. And that was, at the time, how companies like SoFi and others were created. Which is: loan providers are horrible, so we are going to be better loan providers. And millennials really loved it. But today, if you talk to Gen Z, it’s like, “Why should I pay that much? Why should I screw up my future with a $150,000 loan?”

Seema Amble: I think what we also see is that students and parents alike don’t really know how complicated the process of finding funding is until they actually get to that point, navigating the FAFSA process, as well as the private loan process. People don’t know where to turn, what to apply for, even the difference between a federal loan and a private loan. And that process is unnecessarily complicated, if we’re trying to get people educated.

Amira: Absolutely. The number one reason why people don’t access their rights in the world is bureaucracy. You have to spend your day on government websites with 1998 designs and no API and PDFs on horrible bureaucratic stuff.

Lauren: We talked about how companies like SoFi targeted students with lending. But traditionally, why haven’t banks courted students in the past? Why haven’t they specifically designed tools for this demographic?

Anish Acharya: I think that they did for a little while. You know, I signed up for a credit card in university and I got a T-shirt. And I think there was a lot of that credit card marketing happening in college campuses. And that, of course, was outlawed under Dodd Frank, because what they realized was, hey, your “free T-shirt” actually costs like $500 in credit card interest. And once that went away, banks really didn’t have a way to make money off of students. And because they’ve traditionally been so short-term oriented, they pulled out of the market altogether.

Seema: And I think the other point, too, is they’re losing that touch point on the student side. So, before, the banks were getting subsidized by the government to provide student loans. And that got removed; in 2010, the federal family educational loan program ended. And so we’ve seen the government providing an increasing share of the student loan market.

Lauren: How big is this market? Amira, do you have a sense of scale?

Amira: So the U.S. government gives around $1 trillion dollars a year of aid. But in the student financial aid part alone, the total amount is $135 billion. And these $135 billion dollars are cut into very small checks—just an infinite number of applications.

Anish: I feel like the magic of this country is that it’s so loosely coordinated that all of the best ideas just sort of, through market forces, percolate to the top. But anytime you have to do something that’s highly coordinated—like, let’s find one way to access all of these programs—it’s enormously challenging. And look, the same things happened with COVID, right? The countries that were able to create an enormously coordinated response have been really successful. And the countries that are intentionally very fragmented have been less successful.

Amira: Absolutely. And if you think about financial aid, it’s the same thing. Why does a student need to spend every weekend filing forms and entering their name 15 times? Assuming that someone should deserve the right to have a higher education should not depend on how good you are at filling out stupid forms. And that’s the absurdity of it.

Gen Z’s expectations around finance

Lauren: I think part of it is changing of consumer expectations. This generation has embraced mobile banking and fintech in a way that previous generations did not. And they grew up in the wake of the last financial crisis, so perhaps they don’t have the same unwavering trust in institutions as their parents did. They’re often more leery of debt and credit, as we see younger consumers preferring debit to credit cards. And they grew up with the internet. So all this seems to set the stage for the rise of fintech serving this particular market.

Amira: How do you expect the most educated, the most intelligent, but also the most cynical generation to just trust without facts? There is an incredible, beautiful change in consumer behavior, which is probably the biggest bullshit detector of all consumers. So if you are trying to trick the user in some flow or some ad, that just doesn’t work anymore.

Seema: I think you’re touching on a really important point, which is trust and transparency, which we’re seeing not only in student lending startups, but in the new generation of consumer fintech startups overall. Millennials and Gen Z really need to believe that financial institutions are on their side and trying to help them navigate this, rather than all the gotchas and fees and fines to make money off of you.

Anish: I think another interesting point is we live in a generally high-trust society, where people are willing to try new things. If you go to many other countries in the world—India is one that Seema and I have studied, for example—it’s not a high-trust society, which is why you see these massive conglomerates. They do like 25 different things: they make cars and sell cement and they offer financing. If you look at how trust is changing, though, trust is actually eroding in institutions and legacy providers of financial services. And trust in software and technology startups is really increasing. So I think it’s the best time to be building a company like this.

Lauren: What are the specific consumer expectations that we’re seeing out of the younger generation? And how do you see fintech companies addressing that in product?

Anish: I think that the old model of “hey, here’s a free t-shirt if you take my credit card,” you know, banks’ focus on owning a transaction is going out the window. I’m glad that Amira brought up SoFi because I think that while they did a lot of interesting things, they brought the old mentality of treating every opportunity to interact with students as a transaction. And, there’s only so far you can get in arbitraging the loan APRs that are being given to students that went to Harvard and Stanford. And I think that we’ve seen the results of that. [SoFi] built a big company but has struggled to build additional financial services because giving someone a better loan is great, but probably not enough to have a lifelong relationship with them.

So I think that the expectation now is to have a deeper, more meaningful relationship. And look, I think consumers understand that companies have to make money, which is why models like subscriptions or even just charging consumers directly make a ton more sense. Consumers are actually smart, smarter than maybe we’ve given them credit for in the past. So I think there’s a higher expectation for the longevity of the relationship. There’s a higher expectation of value that’s delivered. But there’s also a higher willingness to pay, all of which is really good news for startups.

Seema: I also think if you build trust early on with the student that can go a long way. The process of paying for college, navigating FAFSA and scholarships and grants was not always seen as a business. The whole student loan process was really seen as being more on the not-for-profit side, rather the profit side. And to Anish’s point earlier, personal loans and credit cards for students were really just thought of as something that a big top 10 bank would end up offering on campus or a local credit union would offer. And now fintech has the opportunity to weave something that was traditionally not-for-profit, and create a real product around it, and improve the students’ experience, not just from a financial perspective.

And that really presents an interesting proposition in the sense that if you help students find money to pay for college, you can probably build a longer relationship with them over time and offer more financial services. But you’ve unlocked something that they would have an incredible amount of pain getting to themselves.

Amira: And I think there is also a behavior that is very different, which is that usually companies try to win more than their users, right? And Mos is the opposite actually. The user wins more than the company And that is a different way of thinking. And I think that is also what is hard to understand for a bank that just wants to win more than you.

I mean, I’ve been with bankers in the room when I explain what I do and I remember one banker who looked at me and said, “Economically, it doesn’t make sense.” And I’m like, it actually makes total sense! The bank of the future will be a bank that is able to transform $1,000 dollars in to $5,000 dollars, because a 1.2% gain is not interesting for this generation.

Anish: It’s also interesting because banks’ whole business has always been two things: taking deposits and making loans. So even if you assume that they were able to do this it would just be very unusual for them to do it because it’s not their business.

Seema: Because their mindset is: how do I create a transactional account or a financial product and sell it to you, rather than, how do we actually build a tool, or a product, or a service, that is actually solving the pain point first? Rather than being around the dollars and cents.

Lauren: And now, of course, there’s a slew of fintech companies that are specifically targeting students. They’re tackling student debt, how to manage it, and more recently, how to avoid it. We’re seeing more around savings, particularly saving for tuition. There’s budgeting apps targeted to students, helping them establish credit early. And then of course, there’s lending apps with terms specifically tailored to students.

How fintech is targeting students

So one point I want go back to is the importance of building that customer relationship early. I think more fintech companies are recognizing that. And so we’re seeing companies like Greenlight or GoHenry or Step that are encouraging kids and high schoolers to start saving early. Can we talk a bit about that trend?

Anish: I think that navigating the long-term relationship is a real thing. You don’t ever want students or individuals to feel like they’re in the kiddie pool of financial services. And when you graduate from college, a lot of things in your life change, you want to start being an adult. And I think you have to design the product, as well as the brand, to be one that actually can have that longevity. You don’t want it to feel like, hey, this is a constraint my parents put on me, versus, hey, this is something that’s actually going to enable me for the future.

Amira: I actually think that if you are thinking about keeping a long-term relationship, starting too early is not the solution. I think you should start the first year of adulthood.

Seema: And I think getting people early means just getting them at the critical juncture, which is, when people are applying for college they’re taking that critical step of managing their own financial independence. And so I think this is a really interesting time to build trust when they relied on probably their parents, and they’re taking control of that.

Lauren: And the data backs that up, that a little less than half of college students report having any credit cards, and among those, about 60 percent got their first card when they were around 18 or younger, which suggests that many of them are starting their credit card experiences around the same time that they transition into higher education. But we’re also seeing increased parental dependence. As of July, 52 percent of 18 to 29-year-olds were living at home with their parents, according to the Pew Research Center. That’s a rate higher than the Great Depression. In addition, 6 in 10 adults are relying on financial help from their parents. How does that impact the theory that companies should be trying to reach out to these consumers and capture them at this transition moment when, in fact, many of them are perhaps not reaching that transition moment at the same point that past generations did?

Seema: I think one thing to point out here is that, even if you applied for all the federal loans and scholarships available to you, there’s a good chance that you won’t get the full amount and there’ll still be a gap. And you’ll have to fund that gap, usually through a private loan. Usually you go through a bank, for example, to get it. But that requires a cosigner. Like, who’s the cosigner going to be on that loan? Probably your parents. And so you end up falling back on your parents if you can, in many situations. And the cost of living is going up. And so I think that’s part of the reason that you see them still reliant on their parents.

Anish: In many other countries, kids depend on their parents for a longer period of time. So I think that if it leads to a better long-term financial outlook for the individuals, then it’s not necessarily a bad thing.

Amira: What I think is going to happen is that they will be more responsible financially, just because financial decisions are not made alone. Actually the fact that a lot of those young adults now live with their parents I think will be very interesting in terms of how their financials will go in the future. I think Gen Z is going to surprise many, many, many people with how little credit they will be using in the future. I think they’ve seen their parents crushed by debt. They are super aware about the consumerism and all of that.

Anish: While I think that you’re right, there may be less of a functional need to get credit, I do think that it’s an important sort of emotional and psychological milestone to start to establish that credit. So look, while I think they may take less credit, I still think that they’ll probably participate in the system by establishing credit, at least in a lightweight way.

Seema: And I think establishing credit isn’t just about: okay, now I have a credit score that’s in the prime segment, and now I can take on more debt. It’s also preserving optionality, which I think isn’t something that you totally understand when you’re 18. So, you know, you need to build a credit score if you one day want to buy a home—and maybe Gen Z doesn’t—but you have that optionality to do so. Or for an auto loan or any of these big purchases that a lot of times when you’re 18, you’re not really thinking about.

And we’re seeing, on the product side, a number of products that that are helping students think about how to build credit. Because I think that’s not something that students think about. Things like: the account I’ve had opened the longest is actually what’s driving my credit score. And how do you build a good credit score? And so these are all things that you have to learn, which is an opportunity for fintech.

New paradigms and alternatives

Lauren: I like that you brought up that idea of financial responsibility. There are many fintech companies that are designing budgeting apps specifically with students in mind, which are offering and things like ways to track their spending or setting goals and challenges, and sometimes peer comparisons on spending habits that are anonymized. So it’s interesting to see for those that perhaps don’t have that parental backstop, there are also companies that are seeking to tackle this idea of financial responsibility, and saving, and credit, in a way that appeals to a younger demographic.

Anish: It’s funny, we’ve talked a lot about this. I think to some extent, yes, budgeting is important, but it feels like the much bigger lever is ensuring that students are getting access to every single dollar of financial aid that’s available to them. I think we should actually focus on bigger ways and bigger moves to actually assist these people.

Lauren: If the existing budgeting apps are insufficient and you want to think bigger, where are those areas that you see opportunity?

Anish: I’ll give you one point that’s really piqued my interest. Just as 20 years ago, blogger and other technologies made every individual into a publisher, perhaps what’s happening today is that every individual’s becoming an investor. And look, I think that the next generation is much more savvy and is going to participate in investing at a higher rate. And I think instead of talking about controlling costs, let’s talk about increasing “revenue.” And there are a lot of ways to do that that I think are interesting.

Seema: Yeah, I think there’s still a lot more to be done on the employment side.

Lauren: That’s a great point, Seema, in that rising college seniors are now facing the worst job market in modern history.

Seema: Traditionally, that was like, oh, I’m going to go work at a library or the coffee shop. Now, we’ve seen an explosion of online tutoring, for example, in COVID. I think we’ve seen a lot of platforms pop up, but I don’t know if they’ve necessarily connected with students specifically. To Anish’s point, it’s a lot around: how do you increase the amount of revenue for the student, in addition to the loans, so that they can become more entrepreneurial?

I think you’re also seeing a change in how education is being offered, especially in COVID, where kids aren’t able to actually go to school in many cases. But it’ll be interesting to see what traditional education looks like, and also the necessity to pay for traditional education versus am I going to go through a coding boot camp or what we would traditionally call vocational training, but really skills based. And I think alongside of that, we’re also seeing things like ISAs pop up. An ISA is an income sharing agreement. Essentially, the student borrows money from the institution to fund their education. In exchange, they pay a percentage of their salary over time after graduation. And it’s up to a certain cap, so if you end up making a lot of money it doesn’t scale with it.

Anish: I think the important point here, though, is it’s a false dichotomy to say, hey, either you deserve to go to college or you don’t and you should go to coding boot camp. I think consumer choice has to be the number one consideration. And I do think there’ll be a barbelling that happens, which is there’ll be a bunch of students that want the brand or the experience of going to a Stanford, Harvard, and they’ll do that. There’ll be a bunch of students who want either the life experience of going to a small liberal arts college or perhaps the sort of vocational training of going to something that’s much more coding boot camp oriented. And I think the schools in the middle are going to be the ones that struggle, because consumer preference is going to clarify around choosing those options.

Amira: I agree. I think we are in the Stone Age of disrupting colleges. No student who has a choice between Stanford and another solution will pick the other solution. We’re very, very, very far from that.

Lauren: I think a big part of that we have to acknowledge is, particularly this year, the widening gap between income levels and tuition costs, as many tuition rates remain the same. It’s not just a dichotomy of “I want the experience of college” or “I don’t want to be on Zoom.” I think there are also very real financial considerations that these companies are taking to task.

Student debt in the U.S. now totals more than $1.6 trillion. And 7 in 10 college seniors are graduating with debt. So I do think that Gen Z, in particular, views their parents and, to some extent, millennials as a cautionary tale.

Amira: Also, in the future I think they will be pretty critical of companies that are just selling debt all the time to them in a pushy way, and making it super easy, and making it super accessible.

Anish: Yeah, I think there’s also an interesting example of a principal agent problem here, which is the government has income-based repayment plans for student loans, but the channel for activating them is through servicers of student loans. And, you know, servicers of student loans don’t really have an incentive to move people on to plans like this. So even when the government has the best of intentions, their channel to their “customer,” their citizen, is very inefficiently delivering the information.

Amira: Yeah, absolutely. I mean, getting student loans through the government is the least bad solution, right? But the solution is to get an education without paying out of pocket and, most importantly, without taking loans. Before considering any loan, even the best loan possible, they need to look at all the other solutions they can use to pay for college. And there is a ton of free money out there.

Advice for fintech founders

Lauren: So if we’re building more tools for this increasingly savvy, increasingly digital generation of consumers, what should founders consider when they’re designing financial products for them?

Anish: A transparent business model. Being clear about how you’re making money and making sure that your incentives are aligned with your customer in the short and long term.

Seema: I think one company that’s done a good job of connecting with younger generation has been Cash App, not only on transparency around the fees, but on the marketing side. Both in terms of product—so they offer things like fractional stocks, new products that are on the market and not available necessarily other ways—but then they do the $Cashtag and the Cash drop on Twitter. And it’s an inherently social experience in a way that other financial products just haven’t been able to reach young consumers.

Amira: I would also add it has to really work, not marginally work. If you are building a product that is 5 percent better than what others have built, you’ll get nobody. There are so many copies of every single app out there, especially in fintech. Just the number of new banks—they all look the same. You will not make somebody switch for 10 percent better. With this generation, you really need to build something that is worth the attention span and as I said, the bullshit detector. We only sell to students and we need to build the product that they want to buy.

The future of banking

Lauren: As more fintech startups design with this mobile-first consumer with high expectations in mind, do you think this will have a long-term impact on traditional financial services? We’re seeing a bit of that already.

Anish: Not really. Of course we should have more digitization. Yes, of course we should have more innovation. The challenge for banks is not that they lack talent, it’s the things that would be most “innovative” or create the most utility for their customers are things that are unfortunately bad for their short term revenue. So I think there’s a huge structural disincentive for them to compete directly. And I think what happens as a result is that banks end up being further commoditized, where they are just loan providers who are indistinguishable, and companies like Mos end up being the intermediary and owning all the economics of helping students navigate refi, etc.

Seema: And, and owning the customer relationship.

Anish: 100 percent.

Lauren: So all of this indicates this revived urgency, if we haven’t already seen it, for companies that help the younger generation manage their finances, save earlier, avoid and pay down debt, build credit, afford real estate, all this. When we’re talking about designing for Gen Z, or for students, where do you still see opportunity? What’s next?

Amira: I believe that the banks of the future will not look at all like the backs of today. Having your money on even a very well-designed app and basically earning some perks here and there, it’s not worth it.

Lauren: What is your vision for that bank of the future? What will it look like?

Amira: The bank of the future will not only manage your money or help you save, but it makes your money multiply. You come in, you’re a student, and your bank just applies you to everything to fill out your account so that you can pay for college. So I really think that those services should be part of whatever is like your money management solution.

Seema: And I think that involves also bringing together a lot of your accounts. We’re investors in Tally, which helps you manage your credit card debt. But being able to manage all your debt across sources, or even your income streams. Wealthfront just launched a product, as well, on the investment side. But this concept of automation of your personal finances is very much still in its nascency.

Amira: Another one is about accessing the job market. Today, colleges help you for your first job. And in the future every person will have 20 jobs during their lifespan. How you get trained for all of them is going to be something interesting.

Seema: And I think a lot of that starts when, people are students or even probably even younger, in high school. But this idea of not just taking one job, but you’re glomming together a variety of, you know, a tutoring job, a passion economy project, from an early age.

Anish: Also, just no longer having just one job for life. That’s over, you know? And people are going have multiple jobs in their career and maybe multiple jobs at the same time, and they won’t necessarily have to congregate in the same city centers. The way that we work and the way that we make money is going to change dramatically.

Seema: I spoke on the employment side, but then also on the education side. Even if four-year education may continue in certain ways, I think people will add on their own forms of education. And so looking for ways to pay for education is going to have to adapt. It’ll generally just be more modular.

Anish: I agree. The other thing that I think is interesting is, how has leverage changed over human history? The most historical model of exerting leverage was through labor—having a lot of people working for you. The second was through capital, which is “the rich get richer.” And I think the new model for leverage is software. You know, perhaps in the last generation, you could only own an investment property if you had achieved a certain degree of wealth. But now with the fractionalization of things like investing in real estate, almost anyone can participate in the economics of an investment property. So I’m really bullish on software making things that were previously only available to people that had a lot of capital available to all people. And that’s why I think the mentality of hustle culture and “everyone’s a founder” and all of that is a hugely positive thing and very trend-aligned with what’s happening in technology.

Lauren: Thank you all for joining us on The a16z Podcast.

Anish: Thank you, Amira. Thank you, Seema.

Seema: It was great to join you guys.

Amira: Thank you so much.

• 

  Amira Yahyaoui
• 

  Anish Acharya is a general partner at a16z. Prior to joining the firm, he served as a GM at Credit Karma. He also founded SocialDeck (acquired by Google) and Snowball (acquired by Credit Karma).

  Follow Twitter
• 

  Seema Amble is a deal partner at a16z where she focuses on fintech companies. Prior to joining the firm, she worked at Goldman Sachs and LeapFrog Investments.

  Follow Twitter
• 

  Lauren Murrow is an editor at Future. She oversees posts, podcasts, & special projects for a16z's consumer and fintech teams. Previously, she was a senior editor at WIRED, where she edited op-eds and features.

  Follow Twitter

There is no spoon. Or rather, “There is no such thing as ‘data’, there’s just frozen models”, argues Peter Wang, the co-founder and CEO of Anaconda — who also created the PyData conferences and grew the early data science community there, while on the frontlines of trying to make Python useful for business analytics. He views both models and data as fluid, more like metaphysics than typical data management… Or perhaps it’s that when it comes to data, those with a physics background just better appreciate the mind-bending complexity and challenges of reining in the natural world, and therefore get the unique challenges of AI/ML development, observes a16z general partner Martin Casado — whose first job after college involved computational physics simulation and high-performance computing in Python at Lawrence Livermore National Laboratory. (Wang, meanwhile, graduated in physics.)

But this not just a philosophical question — the answer has real implications for the margins, organizational structures, and building of AI/ML businesses. Especially as we’re in a tricky time of transition, where customers don’t even know what they’re asking for, yet are looking for AI/ML help or know it’s the future. So what does this all mean for the software value chain; for open source collaboration and commodification; and for the future of software businesses? After all, it’s not written in stone that “All information systems must be deconstructed into hardware, and software, and data” and that “software must have these margins”… Will there be a new type of company?

image: Pawel Loj / Wikimedia Commons

Show Notes

• Discussion of various data management tools [1:44] and whether new tools are needed [5:52]
• Software vs. hardware [10:00] and a discussion of what data is [13:04]
• Managing the inherent complexities in data [14:22] and the backgrounds of the hosts [16:47]
• Different company types that are trying to rein in data complexity [22:00], and a vision of a new company built on AI/ML workflows [32:17]
• Advice for companies in the AI/ML space [38:37]

Transcript

Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal. For this week’s episode, we have one of our hallway-style conversations. And this one is literally like eavesdropping in on a debate and discussion that actually started as a Twitter thread debate and discussion — all around the question of whether and how data and AI/ML (machine learning) companies are different than software companies, and what that means for the future of software businesses. Our guest even questions our view of software eating the world — or rather, asks what happens when software is everywhere? What comes next? 

Our guest is Peter Wang, the co-founder and CEO of Anaconda, who also leads our Open Source and Community Innovation group — as well as created the PyData community and conferences, and has devoted a lot of time and energy to growing the data science community there. And he’s in conversation with a16z general partner Martin Casado, who’s written a lot about the evolution of software businesses, the new age of data, and especially AI/ML economics. You can find those pieces at a16z.com/mleconomics.

The two dive into a number of themes throughout this conversation, ranging from open source and crowdsource innovation, and the messy ways that innovation really plays out — to what it means when you move from hardware to software to data and AI/ML — abstracting something that is not just complicated, but actually complex. And then, they touch briefly on what it means practically in building a new type of company, as well as the evolving role of data scientists. But the conversation begins with their shared vantage points in coming from physics, which is relevant here since these new kinds of businesses and products involve a process of experimenting, much as with physics.

The best tools to run AI/ML

Martin: Both you and I come from the physics, computational physics background, and we both, kind of, been pushed into this data, AI/ML data science — and I don’t know if that is coincidence, or if we have an affinity for that. Before we get into that, though, there’s  kind of a competing view of the world, which basically says, “SQL can do everything.” And it’s funny, we spent a lot of time actually looking at the data science, or the data landscape, and it feels like there’s two kinds of worlds. There’s, like, the data warehouse maximalists, which like — will stick all data in the data warehouse, and then we’re gonna do SQL. And then we’re gonna have some extensions to SQL, like you see popping up in, like, BigQuery, or whatever, and that can do everything that needs to be done. And oh, by the way — if someone’s using Python and R, all they’re really doing is basic regressions. And so we can just make that a simple extension, and we’re done.

And then there’s the other view of the world, which I like to call the Hadoop refugees, which is like — actually, we do hardcore computation, and we need R and Python because the stuff we do is very sophisticated. I mean, I know you’re squarely on one side of those. But I wonder, like, do you think there’s a convergence that happens? Do these stay two worlds? Does one become irrelevant? Like, what happens there?

Peter: Just because you oppose extremism doesn’t make you an extremist, right? I would say data warehouse maximalists are extremists.

Martin: <laughter> Fair enough. Yeah.

Peter: And I see a heterogeneous world. It’s the old yarn about, I guess, I don’t know — there’s so many variants of this. But Alan Perlis, a great computer scientist, has some really great quotes about — some irreverancies about these kinds of things. But I would say that to the idea that everything can be expressed in SQL, it’s like — which SQL? With how many extensions? Because at the end of the day — and with how many like extensions upon extensions, and Multicorn on your Postgre actually running a Python kernel. Yeah. I guess you’re doing a SQL, but you’re running a Python script, you know, so that’s not really — it doesn’t count. 

And frankly, a lot of stuff runs Access and VBA in this world. VBA isn’t SQL. I think if you choose to look at the world through a particular lens, you can choose to count everything else as residuals and rounding errors, but if you take off those lenses, you see a much more diverse landscape. And I think that’s where, for me, I see the space for SQL, and I understand the reasons why — it has evolved into a particular kind of animal. Like the shark is still the best predatory fish in the ocean, but it’s not the biggest predator in the world.

And I think there’s something about that, that if you’re in the ocean, you’re gonna basically [be] shark-like if you’re gonna eat a lot of fish. So if you’re in that business data analytics world, especially because a lot of business data looks like fish — it’s evolved to look like food for the sharks. So that’s kind of the way it is. But what Hadoop opened up back in 2012 — I called it the Hadoop battering ram. I said, “Listen, we’re not gonna win the Hadoop game. We’ll let the Hadoop vendors go and fight against the Teradatas, and the Oracles, and all the classical data warehouse guys. Let them do that thing. Once it battered down the door, we’re gonna come flooding in with all sorts of heterogeneous approaches to data science, data analytics — things that are hard to ask in SQL.”

And moreover, there’s a term I use, which I don’t hear used very often. Now, obviously, you’ve heard the term shadow IT, which is used quite a bit, but there’s a shadow data management — that’s a far, far more insidious and dangerous problem. When I was at a large investment bank, they had a million-dollar Oracle database sitting somewhere, and it was too slow to actually run the analytics they needed. And so what they did is they had an instance of this Oracle database, it costs a million bucks, and what they did is the only query they ran was a bit full table dump into a CSV. And then they took that CSV, and they did everything else with it. And it was Python scripts. It was some random Java crap. It was a bunch of other stuff. And it was sort of like, so if you’re a data manager — if you’re, like, in the data management practice, you say, “Wow, we just have another big old million-dollar instance stood up. Our data management techniques are great.” It’s a, what do you call it, a Potemkin village, I guess, right? But then when you actually go, and you ask the developers, “Hey, where’s the source data for this stuff? Where’s prod data coming from?” Like, “Oh, yeah, this file share backslash-backslash something or the other, or you know, that file.” I’m like, “That file? What about that database?” “Don’t touch the database. It’s too brittle.” Right?

So there’s this kind of stuff going on, and everybody listening to this knows what I’m talking about. That shadow data management is absolutely a pernicious problem, and data science is just eating it alive. Because to ask the question you want to ask, you have to integrate datasets together. Master data management is about siloization, normalization, and all this kind of stuff.

Martin: You’ve hit to the segue,, too. I just think it is so germane to what we’re here to talk about, which is — there’s clearly problem domains, which SQL is totally fine for, right?

Peter: Yep.

Martin: And you can only get the problem domains, which is just not good. I mean, like any sort of hardcore statistics is just not very good for. And the point of us being on this podcast is actually to talk about, okay, like, “Listen, we’re seeing kind of new types of companies and new types of workloads, and they’re around kind of processing data.” And I totally hear you, that this shadow data management is a real issue. And you can make an argument why that exists is not because people are stupid, or they don’t know how to do good workflows. It’s like, literally, we don’t have the tooling to deal with data in the right way. 

One macro question that I have that I would love to hash out with you is, are we seeing a fundamental shift in workload that requires a fundamentally new set of tools and a fundamentally new type of company? Or is this just more of a transition where we can kind of put into service the old tools? And I just want to be a little bit more specific, which is — in the past, you had your toolkit of systems approaches, and you have a software system, and you’d kind of pull them out and apply it to the problem, and SQL is one of them. And we kind of understood how those software systems behaved. And we kind of understood how the company is supposed to run and behave.

You know, as an investor looking at a lot of data companies, they just don’t look the same. The types of tools they use, the type of operational practice they use. And the one that you pointed out was a great one, which is — now data becomes so primitive that you want to actually apply, like, software techniques to, in a way, but we don’t have the tools to do that. And then we’ve written posts about [how] margin structures look a lot different, the way you build your company [is] different. And so, just — do you think this mess is because data scientists don’t have formal CS trainings? Or do you think this is an entirely different problem domain, and we should actually look at what the future looks like for that, and development tools, etc.? This is like the heart of what we’re talking about.

Peter: This is absolutely the heart. And I will try to start from the top, which is this concept that every baby or every child is born — and they’re raising it, they think their child is normal, right? They think of, like, your child is like the normal thing. So you have developers coming online in the late 2000s, let’s say, and they think this is the world. Even me as a professional starting in ’99, right, it’s like, “Well, this is just what there is.” The more you start researching history and looking back, you’re like, you know what? We’re just building in this industry — we just layer. It’s frozen accident, on top of frozen accident, on top of frozen accident. Very, very few times do people make principled intentional revolutionary shifts, right?

Martin: Totally. Yeah.

Peter: You basically Band-Aid a substrate. Okay? So starting from the top, what I would say is that, there is no law — there was nothing carved in stone that Moses brought down from the mountain that said, “All information systems must be deconstructed into hardware and software and data.” There’s no such thing. It was information systems, full stop. The fact that we had different cost structures for innovation in hardware, versus software, versus networking, and so forth — that has led to different rates of innovation, different places, things like that. And so when a business steps in and says, “Okay. What’s on the shelf that I can use to accelerate my business processes?” Then it makes sense, because this thing, that thing, the other. Like, when you buy a car, you buy the car, and then you put CDs in the car. You don’t go buy a car with a CD pre-spec, right?

Martin: Is there the exception of technical innovation in certain areas? So for example, like — we now know how to build systems that extract very useful information out of data pretty simply. That didn’t really work in the late ’90s. Like I remember, the whole first neural network, like, genetic programming.

Peter: Oh, yeah. Right. Right. Yeah. Yep.

Martin: The asthma of the late 90s. I did a number of projects on that that didn’t really work. They actually work now. So you could also argue that the technical landscape [has] changed. It’s not just been a macroeconomic issue on the company.

Peter: Yeah. I mean, ornithopters work if you can flap hard enough, right? It doesn’t necessarily mean it’s the right architecture. <laughter> And it depends on the density of air. Ornithopters might work great [on] Mars, but not on Earth. Right? Propellers work better on Earth. Right? Well, with internal combustion engines, and etc., etc. But the point is that, yes, you’re right. I guess my point could be said thusly. There is a multi-dimensional optimization surface we should be thinking about, not just the optimization surface of software, or a data architecture, or data management, and things like that. I mean, yes, someone did software-defined networking, and you know that better than anybody.

Software vs. hardware

Martin: But here’s what’s interesting to me, which is if you build a hardware company, the tools you use, the money that you need to raise, the innovation pace, is defined one way. And if you do a software company, it’s actually defined quite differently. Although you still use, like, a lot of the same practices, it’s still engineering. You can still modularize. It’s not clear to me that as soon as you move to data, you’re in the same domain. 

Software, to me, feels like an engineering problem that you can modularize — you can build interfaces, you’re building it from the ground up, you control all the primitives. Data feels like science. It’s like you’re trying to reign [in] the complexity of the physical world. Right? It’s one thing to, like, build a house — building a very complex building is very hard, and we had to do all this design practice and the other, but we got the skyscraper. That’s very different than understanding the cosmos, because the cosmos is so complex, and you don’t understand what it is, and you don’t have a blueprint. And data companies are defining the cosmos more than building the skyscrapers. Does that make sense?

Peter: You hit it on its head. I’ll just back up and comment on one thing relative to the hardware and software. Hardware is frozen software to some extent, but the pace of — oh, how to put it? Because hardware is expensive and slow, and has been, at least historically — the industry has a much more robust view towards standards. Now here’s the thing — because you have standards, now you have a binary, bullshit proof, “Does it work, or does it not work?” kind of thing. Okay. That then reflects and changes, then, kind of what you need to do. 

Software — what it does, it makes mistakes in hardware expensive, because there is an inter-subjective reality beyond any particular vendor about what is a mistake. In software, because it moves so fast, it’s too fast running to build specs and hard specs and say, “Did you meet this performance spec you said we’re gonna do?” No one cares about that. Software is just so fast and loose. It’s like jazz. I mean, so — because it moves fast. And there’s not a — you can’t put that thing in. Then the price of making a mistake in software is almost completely subsumed or lost. And so it’s cheap to make mistakes in software because the cost is invisible.

Martin: 100%. However, the actual engineering practices aren’t that different, as far as, like — I mean, you’re absolutely right, like, formal verification is much more important in hardware, but it still feels like engineering to me. You know exactly where you’re going. You have a roadmap. You build an engineering team around that. Data is different.

Peter: Data is different.

Martin: You don’t have a roadmap. Like it is the universe that you’re trying to like, you just infect inside out.

Peter: In fact, this is the exact critique. You’re absolutely right. When you talk about what you do in software and hardware companies, you are trying to manage complexity, for the most part. You get something, but the thing that always screws you — I figure, every kind of engineering is trying to achieve some kind of lift while finding some kind of drag. Right? And in the case of software or hardware engineering, usually, it’s achieving performance or something like that, or some scale of computation, while minimizing complexity — and having manageable errors and things like that. Okay. So that’s those things. But it’s very goal-oriented.

Martin: Yeah. Building to a goal. It’s one thing to say, like, “I’m gonna build this complex system, which you can basically describe, do mock-ups for any destination.” That’s very different than saying, “Extract insight out of this.”

Peter: That’s right. The great John Tukey said, “There’s two kinds of data analysis. There’s confirmatory — kind of, reporting mode — and there’s exploratory.” And the thing you’re talking about, the reason why data smells — and data practices smell like science, is because there is no such thing as data. All data is just frozen models. Right?

Martin: Totally. 100%.

Peter: Every single data set comes from a sensor, even a picture. Everyone thinks, “Oh, well, I took a picture.” Right? That’s just raw data. No, it’s not. There’s a Bayer matrix. There’s a log transform. There’s a gamma correction. And, fundamentally, there’s an exposure time, which is a temporal sampling domain. So there’s all of these things. There is no such thing as data. There’s just frozen models. And where businesses get screwed up is when they treat data management as, sort of, this goal-oriented siloization — it’s a static artifact, and it is artifact management. It’s almost like a — sort of ad hoc library process. And that’s not the same as the kind of data thinking — or the way when you think about data in an ML/AI sort of world. Because in that world, we see that models and data are both fluid. It’s much more from a meta — not to get too metaphysical, but it’s more of a process-oriented metaphysics. It’s much more temporal-oriented than the static views that current data management practice has. And that’s why I think the SQL database extremists are not going to win this particular round.

Martin: So, I’m a systems guy. Right? Like, I did my Ph.D in computer systems. And in systems, we have five tricks. It’s like virtualization, caching, you know — like, we literally have five or six tricks that we throw at every single problem. And you can build amazingly complex systems with these things. Like, you know, we understand distribution, we understand consensus. And so while a piece of software like Google is very complex, it actually can be reduced into subproblems that we know answers to, and then you know, we can — so I would say, like, the relative complexity — the relative entropy of a software system — is finite. It’s not clear to me if you’re trying to use data to run a system that the entropy is as finite.

Peter: Well, yeah.

Martin: Meaning you don’t control nature. I mean, what do we use data for? We use data for pricing. We use data for fraud detection. We use data for calculating wait times. Okay. So what are the inputs from these things? These things — it’s like people’s behavior. Like there’s so much entropy in all of us. It’s like the weather. It’s like this…

Peter: It’s hugely lossy, right?

Martin: Well, it’s these classically chaotic, high entropy systems. And so one of my theses is — and I’ll just have to test this on you, is that building a software system is a relatively low entropy exercise because you’re dealing with primitives that you understand and you’re engineering it. Where actually trying to deal with data, you’re reining in so much entropy, and you’re trying to extract it. That ultimately is why we end up with different companies, because it’s just much, much harder to, like, deal with that much complexity.

Peter: Yeah. Well, that makes a lot of sense. And the Cynefin framework talks about the difference between complex and complicated and chaotic. Right?

Martin: Yes. Yes. Yes. Sure.

Peter: Right. And so complicated. And I think the pithiest way to say this is —  complicated means that you can take it apart, understand the bits, and put it back together again. Complex means that you cannot do that. Right? So a fine Swiss watch is complicated. A cockroach is complex. And so I think when you talk about computer systems — because I’m not a systems guy like you are — but one of the best things that I’ve heard about it is that everyone thinks — what is the quote? Everyone thinks distributed computing is about space, but really it’s about time. What is the time horizon in which we can define a unit of atomicity? What is the time to coherence? Right, etc., etc. And so it’s always a space-time trade-off.

And I’m sorry, I’m making this look so like, into the physics world, but I see it that way because it’s a natural flex for me. In fact, I wanted to major in computer science, but my dad — who was a physicist — he said, “Look, son, if you become a computer programmer, if you go into computer science, you’re gonna become a programmer, and you’re just gonna build tools. If you’re a scientist, though, you’re gonna be the one using those tools to make an impact.”

So I majored in physics. But then, as soon as I got out of physics, it was ’99. And I’m like, “All my friends are getting, like — they’re getting starting bonuses, and they’re getting jobs, and they’re worse programmers than me.” And so I ended up joining a computer graphics startup. And that’s when I started using Python, was in ’99. I realized that I could script a bunch of C++ much better with Python than with broken template support in Visual Studios. It was God awful.

Martin: I came to networking by way of computational physics. Actually, when I was a computational physicist — I was a computer scientist doing computational simulation at Lawrence Livermore National Lab. That was my first job after undergrad. I was a huge Numeric user, because that was the only way to do high-performance computing in Python — and from what I understand, that became Anaconda. I would love it if you would kind of give the history of that project.

History of programming tools

Peter: So in ’99, it was Jim Hugunin — and I think there’s some others that I might be forgetting — can be credited with working on some of the early matrix stuff. And then Jim Hugunin worked on Numeric, and they realized that the operator overloading in Python would allow you to do something that looked a bit like MATLAB. You know, like, it’s okay — it looks like you’re right back to code. And it’s like, “Hey, this hack kind of works.” And also, Python’s C level extensibility meant that they could build a little tight C library that would be fast. So you’re writing the scripting thing, that little syntax looked like MATLAB, but it ran at basically C speed, which is really important.

So then, it turns out, though, that some of the features they built — the Space Telescope Science Institute folks, the ones who run the Hubble telescope — they had some other ideas about what they wanna do with this library, and Numeric wasn’t quite flexible enough, or some other stuff. But they created an alternative matrix library called NumArray. And NumArray had, like, fancy indexing. NumArray had a few other things. And so the ecosystem in the early 2000s — when I first got my first paid job doing Python, [it] was 2004 and I was doing consulting on Python, and SciPy, and all that stuff. And there was still a split between NumArray and Numeric. Or, in fact, most of the libraries that were trying to build on top of this stuff — they built a compatibility layer called Numerix, which would then flexibly import sub-symbols from these different libraries depending on what you’re trying to — it was terrible.

Martin: The wild and wooly days of early Python.

Peter: You know, it’s a mess. Crowdsource innovation is always a mess, but the result is still nice, because what happens is you end up getting somebody like Travis Oliphant — who comes along in 2005 and says, “This is a mess, and this is slowing down innovation because everyone has to do the work twice. We got to make it work with NumArray and with Numeric, and we can’t make forward progress.” So he spent a year of his life into making — just coding and designing, and he made a really nice thing, and he called it NumPy.

And he came out with it in, like, end of 2005, 2006 timeframe. And then the world rejoiced. And I was like, “Oh my God, this is great. This is the unification we needed.” You know, at the SciPy conference in Pasadena the following year, we gave him an award. Anyway, that’s what happened in the mid-2000s. And then, many years later, then in [the] 2010 timeframe, he actually joined the company I was at, Enthought, and then we had many happy days there, doing a lot of scientific computing, consulting. Which is fun for science nerds, but a niche area. Right?

But then we started getting contracts and consulting inquiries from hedge funds, and from banks, and investment banks, and things like that. And by the end of the 2000s, I’m walking to the floor of, like, JPMorgan, Bank of America, and they have thousands of people relying on SciPy and NumPy to run advanced models. You had coders sitting next to traders, like on the energy desk, and you’re like, “This guy is asking me really deep questions about SciPy. He’s really trying to do stuff with this.” So I had this insight that — I think Python is ready to go into the mainstream, like, business analytic space. And that’s not just MATLAB that it could be taking market share from, but maybe SAS. So at the same time big data was starting to crest at that time, or peak — and I realized that people wanna do more than just ask SQL questions of their big data. And in fact, when I went to the first Strata, in 2011, all of the vendors on the show floor were selling many different flavors of Hadoop. SQL integrations, faster Hadoop, etc., etc.

But then, when you go to the tutorials, every single data science tutorial was teaching Python and R, but there’s no Python vendor. And also, Python is kind of janky for some of the stuff. It doesn’t play with Java very well. Python and R were both second class citizens in the Hadoop world. So I said, “You know, I think there’s something here.” And that’s why I started the company. We started as Continuum Analytics in 2012. And it was Python for business data analytics, Python for data science. That’s what led to that. Anyway, that was a long, sort of, exposition. But to your question about the history of all of this — how this came [about] — but I think that when you talk about software systems, it’s actually very interesting. We build software systems thinking they’re merely Lego bricks — that we make relatively homogenous, or homogeneous. Or, well-structured studs are spaced this way, they’re this big and this tall — and then we can stack them together, and boom, now you have a bigger Lego.

But in reality, when you look at any real software language in modern software systems, there’s complexity to it — more than the complication. And that’s where your worst bugs lie. You know, like, you have some npm module that pulls in some other crap, and that interferes with some other crap, and it tries to install this other thing on your system — and now you have complexity beyond the complication. So I think the practice of software is bedeviled by the fact that it actually is playing, at this point, with so much complication that it basically appears complex to our human minds.

Tactics for dealing with data complexity

Martin: Barbara Liskov has my favorite Turing Award acceptance speech ever, and if you haven’t heard it, you have to hear it. And it’s basically about modularity and computer science. And it’s how you can take big problems and make them small problems. Like engineering with modularity — you can rein in complexity. So you have a complicated system, but I think you can actually manage the complexity. I’ll give you an example on the data side where that’s not the case. There are natural systems that are self-similar. By self-similar, it means that they retain the same stochastic properties no matter what zoom level.

So, unlike a software system, if you’ve reduced it down to a method, you’ve got, you know, a fairly simple abstraction. There are some natural systems like, say, coastlines — that it doesn’t matter at what level you look at it, they still are, like, super complex. So one thesis is like, yes, software systems can be complex, but like, they’re more complicated in that you can modularize and focus on things. That’s not necessarily the case with data. Data is as complex as the natural world. Again, like, you don’t have control over the weather, and the weather is self-similar. And no matter what zoom level you look at it, it still maintains the same stochastic problems. It’s not like data. You don’t have the tools necessarily to reduce the complexity to something that is merely complicated like you do with software.

Peter: Right. So the question then in the data practice world, then — let’s just keep it at that level, then, which I think is a great place to be talking about it — to which point do you stop? What is your optimization criterion? Right, because all engineering is a trade-off. So for the amount of effort you want to put in, how well do you need to understand that coastline? If you’re trying to target a guided missile into a window of a building, you don’t need to map the coastline down to a millimeter, right? So on and so forth. So I think that when you get to data, you recognize that, it really, ultimately — if you actually want to get all the value out of it, you’ve got to loop it around into the overall OODA loop of your business — the observe, orient, decide, act loop —and actually take action with it and correct and zoom into the appropriate level.

Real-world implications for businesses

Martin: I think this is kind of what this all boils down to. So now the question is, let’s say that you’re building a company — that instead of the goal of the company is building a modular software system, [it’s] reining the complexity of data, which we’re seeing more and more companies do. What does that mean to deal with that much complexity? So what you just mentioned is, well, okay, maybe you look at, like, the different zoom level — or maybe you’ve got, like, a full feedback system, or whatever. But before we even get to how you do this, I would like to either agree or disagree that the companies trying to rein in that complexity are different.

Peter: I completely agree with that. The companies that actually understand even the problem they need to solve, they have a better chance of solving the problem. Because it’s actually very much like cloud computing. It used to be — how do I build the software on the basis of the computational resource I have access to? Well, once you have ability to access essentially limitless computation, you’ve got to ask about, “Well, what is it I would need to build? What do I really wanna do, right?” So I think with data, it’s a similar thing, where you say, “Well, you can put in for any <inaudible>. You can put in more money and get more texture, more resolution on your predictions.”

Martin: Exactly.

Peter: Where do you stop?

Martin: Exactly. Exactly. Right.

Peter: And stop is, like — I can only convince this CEO to hire three data scientists? So that’s where we stop? Is this what three scientists can do? I think that’s how a lot of people are winging it right now, but the interesting thing with the hedge funds — you look at them is — they understand this. Like some people say, “You know what, we’re not gonna work at the microstructure level. We’re just not gonna do that because there’s a few big players that play the high-frequency stuff. We’re gonna leave that out. We’re gonna do kind of longer-term stuff and do bigger strategies — some, you know, longer-term strategies.” So they self-select into zones where they believe they have the observational capacity and connect that to execution capacity.

Again, it’s about the OODA loop. They believe they can run a coherent loop. Data is important in all of that, but more importantly — is keeping track of the model, because it’s not just processing data anymore. At some point, it’s also going to be modifying the systems that are then producing that data. Right? It’s a loop. And the most effective companies — it has to be that the data processing is part of both the inference and the execution step. Right?

And one that was the most shocking to me, honestly, in the last 10 years I’ve been doing this — so many businesses — big businesses, at the heart of a lot of really important parts of the business — the models are very old. They’re very stale. They iterate very slowly. And it’s a massively human-intensive task with VPs and PowerPoints, and everything else to get revs on models. And then you go to the, like, hedge funds, and it’s like, “No, we hire engineers.” They come in, and they code MATLAB, and they’re trading $100 grand the first week. Right? That’s different. That’s a very different view of the OODA loop.

And, you know, I think in our Twitter exchange, this is where I said — all companies are gonna have to look like hedge funds. Because in a world where you can have essentially unbounded observational capabilities — you can be a logistic startup, and you can basically get data as good as FedEx or anybody else doing logistics. You could be — you can do whatever. There’s a great leveling field with regard to the sensory capabilities. There’s a great leveler with regard to cloud computing capabilities. You don’t need to go hire 100 sysadmins just to go and rack a bunch of servers. You can just turn on some things.

So with that being said, you can now have extremely low footprint, fast-moving companies that are just there to run the OODA loop, and to have extremely explicit intentional sense-making around the modeling. And for them, data, then — it’s sort of like the difference between a fish — the way a fish sees water, versus somebody holding water to ladle. Right? You don’t even think about the data because you’re just swimming in it. Right? Obviously, you understand data.

Martin: Yeah. So this is like the silly VC observation. The silly VC observation is if you look at a software company that doesn’t have to deal with the complexity of data, they tend to have relatively high margins, say 70% to 80%. And the reason is, is because they’re building skyscrapers, and then they sell those skyscrapers, and the team needed to build a skyscraper is relatively fixed — and then you can sell as many of those as you want. That’s kind of the software model.

When we look at companies that are reining in the complexity of data, and that’s how they extract value, the more people you put to rein in that data, the better your results are. And so now your incentive [is] to, like, have more and more people try and work on that data over time. So I think the structure of a hedge fund is — we hire more people to work on the data, we can potentially get more money. Just because they’re actually reining the complexity of that data. But in the software world, all of that complexity is basically going into the margins — yet, depending on who the buyer is, you can’t increase the top line in the same way. 

So let’s say I’m gonna sell five copies of my software, right? Now, if I sell five copies of my software, people are buying the software. They’re not buying the results of the data. Like, maybe they’ll like my software better because it’s more accurate or less accurate, but the number of people working on the data doesn’t directly drive the amount of software that gets built. And so now you have this existential margin issue, which is — you want to increase the number of people working on the data. Labeling it, cleaning it — because you can always get some improvement.

Peter: Right. Here’s the question. If we think about — in the software space, you have software vendors and buyers. And the theory of a software vendor, again — going back on our history, there used to just be computer companies. And then Bill Gates was like, “Hey, stop pirating my crap. Pay for my software, because software is a thing. It’s not just your long-haired hippies copying each other’s Unix code. Like software is a thing, right? You need to pay me for it.” Letter to Hobbyists 1970, whatever it was, or something like that. But he did that at the beginning of the PC era. And the PC era basically said, “Well, here’s a set of standards.” Here’s x86. The x86 ISA. Here’s EISA, and BUS, and your peripherals, and networking, and all this other crap. And so you have a set of standards that in the space — oh, actually this recent blog post that I think you — I don’t know if you wrote, but you promoted. The narrow waist of TCP/IP and the…

Martin: Oh, yeah, that was me, me, and Ali. That’s an old networking guys look at crypto.

Peter: The point is, you know, a lot of these things rhyme with each other. When you have standards, what they do is they reduce the cost of innovation, and they increase the innovation surface. The PC era was such a gigantic — it’s such a gigantic leveler, that allowed the era of software to thrive. But again, Moses didn’t have a third tablet that said, “There must be software-hardware divided.” And that software must always have these kinds of margins. We’re now entering into an era where people are considering the entire stack of what an information system is. And so, when you look at that, there’s no reason at all, why — if I’m an end-user, customer, buyer — why should X percentage of my alpha, or my margin, or surplus, if you wanna talk about capital and all that stuff — why should this percentage of my surplus go to all accrue — broadly across all these companies — broadly accrue into just one software vendor? Because if I insource it in-house — the technology — and I have the FTEs, all of the residual value stays within the boundaries of my firm.

And this is what a hedge fund does. In fact, when I go and try to sell to hedge funds, they don’t generally buy software. They use our open source. They like to get consulting services and ask questions. They’re very high-end users of our open source stuff. But they basically say, “Why should I share anything?” Like, they’ll buy a database, they’ll buy some things that they perceive to be truly infrastructure and truly commodity. Anything above that, if there’s a chance of it contributing deeply in a generative way — not a decomposable way, but in a generative way to their alpha — they’re gonna keep it in-house. It’s proprietary.

I was at a dinner with a CTO of a hedge fund. And he’s like, “Tell me why I should care about open-source.” I’m like, because they had [an] internal, like, crappy version of pandas — and I was trying to give him the story of like, “Look, if you just use pandas, you would basically leverage all of the — you basically have cost amortization of innovation for you,” right, “and it’s not differentiating value for you to have your own little tabular data structure.” People think that open-source is winning, or has won. I think the fact that open source is commoditizing all this stuff means that software itself — the value chain is collapsing. And so, right now, open-source is a movement. I think, unfortunately, it’s confused. There’s sort of this Stallmanesque religious aspect to it almost. And then there’s something deeply beautiful about crowdsource innovation, and legit community collaborative innovation, that’s really important. And we’re almost losing that because everyone’s like, “Oh, but open-source has won now.”

I think that’s a mystery of the situation. And it’s a thing I keep tweeting about, because I’m saddened by the loss of that thread of the principle. Why do we do open-source? Why do we do crowdsource innovation? So anyway, it’s that conversation. I think software companies do look different because they have thrived in an era of relative — the substrate they’ve sat on is pretty flat. And now we’re entering a space where performance matters a great deal, where the information systems are integrated again. Software is only one component of a whole integrated information system. And because of that, now it’s no longer, like — I can sell just one piece of software across 1000 companies and just harvest all of this margin.

Companies built on AI/ML workflows

Martin: So here’s my mental model on these things. Let’s imagine that you have two companies, Company A and Company B. So Company A, they’re building a system, and all the properties of that system are gonna be defined as software. And so they’ve got a roadmap, and then they build the software over a period of time. That’s Company A. Let’s say Company B — let’s say, actually, they’re gonna use just all off the shelf, kind of, AI/ML workflow, but they’re not actually really writing software. It’s all about getting the models to be predictive. And so the entire company is around cleaning data, labeling data, training the models. Right? They’re very, very different, because the complexity of the second one is just far, far greater. And I would say, defensibility of the second one is far, far greater just because of the nature of data. And so it feels to me there’s almost like an emergence of a new type of company.

Peter: Absolutely. Yeah.

Martin: Where the organization, the margins, the go to market — everything is being dictated by the fact that they’re processing data, rather than writing software primarily. I think we’re all still trying to understand what that second class of company looks like.

Peter: Yeah. One of my pitches is that by harnessing the power of open-source to commoditize, to do the disruption on a lot of classical data processing systems, we would basically be one of the last great software companies, and be one of the first great AI companies. The margin doesn’t come from how well you do the software bit. And so, I think that’s the big news. I mean, maybe I have a bit of a controversial view on this. But I think that the era of software being the dominant part of the stack — I know, you know, Marc Andreessen likes to say, “Software is eating the world.” It is eating the world. But it’s a ruminant at this point, right? It’s not the most efficient digester of the value.

And so, look, you benefit from chlorophyll, even though you’re not a plant — you just eat a lot of plants. <laughter> But I think in the era of — I mean, if we’re gonna, kind of, to go to the — complex systems thinking, right? In the era of data abundance, the people who can build models, refined models, and execute on them, fastest are the ones that are going to win. They’re the chaos agents in the ecosystem. So, look, we still live in a world of plants. But there’s a beautiful infographic I saw the other day, which is how much biomass is on the earth. Most of it is plants. And then you got, like, this little bit is animals. And there’s a little bit there’s, like — this little bit is mammals, and there’s, like, this little bit is humans. I think that in the world order to come, there’s still gonna be, of course, hardware and software companies, so on and so forth. But I think the margins where you really wanna look for the growth is gonna be those people who are moving like animals, and not just claiming a spot. “I’m gonna go here, grow my leaves.” You can still catch some sunlight, but your optionality — I mean, you know, business is war, your optionality is reduced. And the companies that can move fastest among these different places, those are the animals, and that’s going to be running faster OODA loops.

Martin: I would love to talk about how this impacts the actual business. I’m not sure there’s a huge change on go-to-market, except for the fact that there’s two types of these kinds of AI/ML companies. There’s the infrastructure companies, which basically build the tools to use AI/ML. And that standard — that looks like a standard software infrastructure company. Like, it’d be, like, a data company or something like that depending on your point. And then there’s those that use data science AI/ML to tackle problems in the real world. And in those, it’s kind of interesting, because you end up not building a software company, but more of a farming company or an agricultural company. And so, you’re not selling to core IT right? So they just tend to look very different than typical software problems because they’re selling to a different constituency.

Peter: They’re not software problems. The software is a means to an end, not the end unto itself.

Martin: And this is particularly germane to AI/ML, because it allows us to solve problems that typically software hasn’t been good at solving in the past. Like, it allows us to solve vision problems better than we’ve been able to do it before. Audio processing problems better than we’ve been doing it before. It’s kind of like the best way to interoperate with the physical world. And so now we’re off, like, building these companies that solve these kinds of real-world problems. And you just have different looking companies to do that because, again, you’re selling to the person that inspects the HVAC system. You’re selling to the person that is the farmer. You’re selling to the person that does manage the forest.

I think one thing for the very high level — and, like, anybody creating a company in this space needs to think through is the following, which is — if you’re building just the infrastructure, just the tooling, and the nuts and bolts, you look like a software company, and somebody else deals with the actual AI/ML application. And that’s fine. But let’s say that you yourself are ingesting the data, cleaning the data, labeling the data — there’s a lot of variable costs to do that. Like, every customer may have a new data set. And what happens is this impacts the margins of your business, like, it looks like you have lower margins, because, for every customer, you’ve got all of this work to do. And so I think you’d need to make a decision early on whether — do you want to be the one that’s doing that work, because that’s something you can actually offload to the customer.

So let’s say you go to a new customer and say, “Listen, we’re gonna take all of your data, we’re gonna clean your data, we’re gonna create your models, and we’re gonna solve your problems.” And in that case, you internalize all of that. And as far as your organization, you need to know that this is basically a services arm. Another option is you can say, “Customer, we’re gonna give you all these tools, but you’re gonna have to bring in your own data, you’re gonna have to hire people to label it, you’re gonna have to learn to tune your models. And we’ll help you with all of that, but you’re the one that’s gonna go ahead and sink that cost.” And so you have to think very deeply of how you structure your company relative to the variable headcount — like, the headcount that has to grow per customer, because that seems to be the big difference that we see for these AI/ML companies, and the typical software company.

Peter: Yeah. I think it’s hard to do one of these companies right now because we are in a transitional time. A lot of the customers don’t even know what they’re asking for, and they’re kind of looking for that help. And even now, people recognize it as a growth area, and where the future’s headed, so they wanna spend some money on it. But, absolutely right, the amount of work you have to do per customer starts looking a lot like a services play. And there’s a reason why a lot of companies, when you really look inside the skeleton— like, why I think I called it the skeleton buried in ARR. You see a lot…

Martin: <laughter> Totally.

Advice for companies in the AI/ML space

Peter: Eric von Hippel has a great book around democratizing innovation. And he says, “Even when we have a space in which a product is possible, products usually only cover 60% to 70% of the end-user need. The end-user still has to do.” And he’s not talking about software. He’s talking about people like, you know, welding things onto the side of their tractor. He’s talking about, in general, the customer has this thing they need to do. When it comes to the AI/ML application areas, it’s a lot more than just 30%, and it has to be customized per customer site. 

So I think for businesses right now, in this transition, it’s super hard not to end up looking — if you’re doing a good job for your customers, it’s hard not to look like you’re doing a services play. Now, that being said, there are, I think, viable strategies through this. Which is that you can specialize in an area and domain and say, “Look, we’re gonna come in and work on your data set. But we have our own reference model we’ve built.”

Martin: That’s exactly right. That’s exactly right.

Peter: And now we can benchmark you against that. We can bring some of our own magic juice into this. So now the thing that is generalizable across or product-izable across a thing — maybe it’s only for that sector, but the thing that’s generalizable is not just the software, it’s actually more defensible than the software.

Martin: I just wanna very quickly put a fine point on this. There’s two things that you brought up that are very important to realize. The first one is, we are in a transition. So customers don’t even know what it means to, like, label data and clean data. Maybe in five years, you can go to a customer and say, “We’ve got all the tooling for you, but you’re responsible for managing the data,” and therefore, you offload the cost. It’s just today. You just don’t have enough education in the market to do that. They don’t have data scientists, etc., etc. And so I think in order to get the market into that transition, the startups have to do that. Like, you have to build out that basically — services arm. The second point you made is actually, I think, the critical one is — there actually is some commonality in verticals. And so you can reduce that margin by sharing as much as possible, but it does require customers to share data, or at least share models. And that’s sometimes a tough conversation with the customers.

Peter: Well, it’s not just sharing models. I mean, there are deeper and interesting, more leveraged plays to be made. For instance, you go into a sector, and you realize, “Oh, all of these people are doing their own craptacular things. These are their limited budgets, and their data sets are broken this way — but holy crap, there’s this other vendor over here with this data set. I can go and negotiate an exclusivity with that vendor. And now I’m the only one that can bring that kind of model lift into this particular sector. So there’s a lot of that 1800s-style, like, homesteading to be done in this space. So I think it’s more than just the “Let me average <inaudible> Central Limit Theorem everybody in this industry.” There’s some really cool things to be done.

Martin: So the first thing companies need to figure out is what type of a company they are. Many are very confused about that. You need to know are — you a software company and you’re building tooling, or are you a company where the majority of the complexity of the company is around data. And by the way, many companies started as software companies and end up as data companies, and then they’ve structured things incorrectly. So let’s say that you’ve come to the answer to that, and you’ve figured out you’re a data company. Once that happens, you need to understand that often companies that are extracting value from data — there’s a lot of complexity per customer in order to do that. And you need to structure your company the correct way, which is like — just realize it may be hard to scale, just realize you’re gonna have different processes around the actual data. Or come up with a strategy to offload that to the customer.

Now, the reality is, because the market is so immature, it’s unlikely the customer is gonna be able to do a lot of that, but it’s something that you can, over time, train the market to do and do that transition. But I think this is the big sticking point with many <inaudible>. They think they’re software companies. They end up being data companies. They didn’t build the organizations to deal with that intra-complexity. It’s coming down in the margins. Everybody is kind of confused. And so I think just a little bit of self-awareness and a little bit of planning go a really long way in this space.

Peter: But it requires a very different — many West Coast firms have the thesis that to do a really great tech startup, you need at least a tech founder somewhere in there, because they kind of see where things are going. For a really good AI startup, you need to have machine learning people at that leadership level because they know what it means. They know why a single data set can be a billion dollars, or swing a billion-dollar deal. The difference between a software engineer and, like, a data scientist is that — software, you generally know what the inputs are, or the types of inputs, and your goal is to construct a system that, given these inputs, produces these sets of outputs. So you have very nice, clean definitions around correctness, for the most part.

With data science, there’s unfortunately not that. You can have a piece of code, and for some sets of values, it’s correct. Other sets of values, it still produces a result, but those results are wrong. And a function’s correctness is dependent on values. This is the key thing that differentiates all of data science — from machine learning — from classical software engineering. Classical software engineering, it’s like, we’ve got our test data set, we’ve got our prod data set. It works in test, it’s gonna work in prod, right? That’s not how data science machine learning works at all. In data science, machine learning, the correctness of a function is value-dependent, and also performance-dependent — and the performance also value-dependent.

So now you have this intertwined synthesis of a data, and a modeling, and a computation problem that cannot be decomposed into orthogonal vectors, right? That’s the difficulty of this. What I think is that in 5, 10 years time, every company that is actually still in existence and doing well has to, essentially, have synthesized and brought a synthesis in of their data capacity, their data modeling capacity, the model build, and computation — the hardest thing is appropriate computation — and economical fashion to suit their needs.

So the word I like to use for this is cybernetics. I mean, we are right now in between the software era and the cybernetic era, and I think we will get to a cybernetic future. And cybernetic, by the way — you know, it comes from the same word as Kubernetes, right? It means governor. It means a theory of action and control. So businesses have to see computation really moving its way up. Data modeling process has to move all the way up to the very tippy-top of the business. That synthesis will happen, it will have to happen. And that’s what the selection pressure is in the business world. I don’t know exactly the path we’ll take to get there. In the transitional time, businesses who want to basically get in ahead of the curve, they’ve got to have very clear thinking at the leadership level. And they must have a very clear understanding with their investors about what they’re gonna look like as they chase the marlin, because it’s gonna take a little while.

So I think that’s the trick right now, is that you’ve got to find founding teams or leadership teams that have a solid understanding of software — of what software is and isn’t, of where the value is in the software activity. And of where the value is in the data and data modeling activities. In a time of fog, you’ve got to have very, very clear-headed thinking about that sort of thing. But ultimately that synthesis must be what comes.

Martin: Thank you.

Peter: Thank you so much.

• 

  Peter Wang
• 

  Martin Casado is a general partner at a16z where he invests in enterprise companies. Prior, he was cofounder and CTO of Nicira (acquired by VMware) and is the creator of the software defined networking movement.

  Follow Twitter

We’re living in an unprecedented era of online collaboration, coordination, and creation. All kinds of people are coming together — whether in an open source project or company, an R&D initiative, a department in a company, a club or special interest group, even a group of friends and family — around some shared interest or activity. But the word “members” is faceless, and doesn’t help us really understand, support (and better design for) these communities.

So in this special book launch episode of the a16z Podcast, Nadia Eghbal — author of the new book Working in Public: The Making and Maintenance of Open Source Software published by Stripe Press — shares with a16z editor in chief Sonal Chokshi the latest research and insights from years of studying the health of open source communities (for Ford Foundation), working in developer experience (at GitHub), researching the economics and production of software (at Protocol Labs), and now focusing on writer experience at Substack.

Eghbal offers a new taxonomy of communities — including newer phenomena such as “stadiums” of open source developers, other creators, and really, influencers — who are performing their work in massive spaces where the work is public (and not necessarily participatory). So what lessons of open source communities do and don’t apply to the passion economy and creator communities? How does the evolution of online communities — really, social networks — shift the focus to reputation and status as a service? And what if working in public is also about sharing in private, given the “dark forest theory of the internet”, the growing desire for more “high-shared context” groups and spaces (including even podcasts and newsletters)? All this and more in this episode.

Show Notes

• What “open source” means [1:56], types of communities [4:17], and how they control growth [7:19]
• The modular nature of open source platforms [10:16] and the ideological framework driving open source software [12:48]
• Further discussion of managing growth and the creator’s time [15:12]
• Open source contributors who create their own brands [20:10]
• Discussion of platforms that are abandoned [22:34]
• Subscription models and building an audience [26:51]
• Platforms that are deliberately outside the mainstream [31:27] and their relation to newsletters, email lists, other semi-private spaces [34:43]
• Crisis of the commons and how it relates to online platforms [36:37]
• Guidelines for community managers, platforms, and communication tools [42:15]

Transcript

Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal, and I’m super excited to do one of our special book launch episodes, for the new book coming out just this week — “Working in Public: The Making and Maintenance of Open Source Software” by Nadia Eghbal, and published by Stripe Press. The topic actually applies to all kinds of communities and groups coming together, whether it’s an open source project, an R&D initiative of a department in a company, a club or a special interest group — even a group of friends and family, because it’s all about how people come together to coordinate and collaborate around some shared interest or activity — whether participatory or not, whether code or content. 

And so, one theme we also pulled the threads on in this episode is about how the learnings of open source communities do and don’t apply to the passion economy and creator communities as well. Nadia has long been immersed in studying the health of communities, including getting funding from the Ford Foundation to study open source, then worked at GitHub in developer experience, then did research at Protocol Labs, and is now focused on writer experience at Substack.

For longtime listeners of the “a16z Podcast,” I’ve actually had her on the show years ago — along with Mikeal Rogers of Protocol Labs, then of the Node.js Foundation — where we talked about the changing culture of open source. You can find that episode on our site. But in this wide-ranging hallway style episode, Nadia and I cover everything from types of communities, social networks, and the evolution of being online. And, ironically, while the book is called “Working in Public,” we also talk about the emergence of private spaces, as well as the tragedy of big public commons — and how to counter the tragedy of commons, which is why I believe everyone should read this book. Because there’s a dearth of literature out there for the era of unprecedented online collaboration, creation, and consumption that we’re in. 

We end with some quick practical advice for community managers, platforms, and leaders, but we begin by quickly defining open source in this context, with a really useful taxonomy for categorizing communities.

Defining “open source”

Nadia: You know, early on, I was just like, “Oh, I really hate this term.” And I just wish we could have gone with something else, like public software or whatever.

Sonal: Ooh, I love that.

Nadia: I love it, too. Unfortunately, it’s pretty hard to change terms that everyone else agrees on.

Sonal: Yes, I know this firsthand.

Nadia: Yeah, I mean, I personally find the term kind of intimidating. And I don’t know, it doesn’t sound exciting when I say the term open source. But it really does just refer to the distribution side of code. The existence of open source licenses made it very easy for anyone to use and modify and republish someone else’s code, then put it in their own software. But it doesn’t really say a whole lot about how open source is actually produced. And so I make this analogy in the book — which is actually an analogy I borrowed from my friend Devon.

Sonal: Devon Zuegel — she’s hosted a couple of podcasts for me. I love her.

Nadia: Yes. And she says something like, “The term open source doesn’t mean anything, any more than the term company does.” It’s like, yes, we kind of get what a company is. But there are so many different kinds of business models for different types of companies — and so, similar is open source. Saying something is open source tells you a little bit about how the code might be used, but doesn’t really say anything about how they’re actually being made. Someone has to continue taking care of it.

Sonal: One of my favorite parts of the book is how you actually outline different types of communities. You call it classifying project types, but it’s really, to me, how people organize and, like — essentially social networks, really. So why don’t you break down that taxonomy. And, by the way, the reason I’m asking is, because when I think of the arc and history of open source, the concept that comes to mind for everybody is that classic book by Eric Raymond, which is “The Cathedral and the Bazaar,” and I think that framing has too long framed our discussion of open source, and frankly any online community.

Nadia: Yeah, everyone sort of has this maybe, like, general understanding of what community is — like, there’s a bunch of, like, members, and they’re kind of organized around some common interests, or reason for spending time together. In that highest level definition of a community, there’s an underlying assumption that, like, all members are sort of similar. And just the term “members” sort of, like, washes over the underlying dynamics between those different members.

And so, what I started by doing was saying, “Okay, there is a difference between — at minimum, in open source — people who are contributing to open source, and people who are using open source.” So I try to sort of separate out users and contributors and say, okay — in some open source projects — or, as you said, really communities in general — some communities have high contributor growth, and some communities have high user growth. And then there’s sort of, like, different permutations of that.

Types of open source communities

Sonal: It’s like federations, clubs, stadiums — and I forgot the fourth — toys. And tell me what those are. So I think it’s really useful to start with your taxonomy of federations and onward.

Nadia: So federations are like the really big open source projects we might be thinking of — like Linux or Wikipedia — where you have a lot of people who are contributing to the project, and you have a lot of people that are using that project. But there are enough people that are working on the creation of that project that, like, it does form its own sort of contributor community. By contrast, clubs have a lot of people who are participating in its creation, but they don’t have as many people that are using it. And so that product that’s kind of focused on a niche interest — the example I like to give is Astro Pi, which is a Python library for astrophysicists.

Sonal: Right. It’s high contributor, because they’re incredibly interested in that, but very low user growth — because how many people in the world are really interested in that intersection?

Nadia: Exactly. And toys I sort of mentioned in passing, and they’re probably the least interesting thing to talk about. That’s where you have both low user and low contributor growth. So that might just be, like, a personal project that I’m tinkering around, no one else is really looking at it. They’re sort of waiting in the wings before they become one of the other types of communities.

And then the fourth model is stadiums. And this is the one I think is most interesting and most overlooked, because it’s kind of a newer phenomenon. And so this is a situation where you have one or maybe a couple of contributors, and then you’re making something for, like, a very large audience of users. And so you can imagine someone’s standing in the middle of a stadium — there’s this imbalance where, in this case, the developer is feeling a lot of inbound requests, a lot of comments, issues, pull requests — just a lot of needs from their users.

But there aren’t that many people who are actually able to help. Contrast it to a federation, where you imagine something like Linux — is extremely widely used, but there’s also a very mature and well developed ecosystem of contributors to support it. But a big part of this book is taking the time to stop and look a little bit more deeply at — what is that giant audience of faceless users, and are there interesting dynamics happening there that actually make this look more like a community. Where, like, a stadium is actually a legitimate type of community that stands alongside the clubs and the federations. We just haven’t really taken the time to understand it before.

Sonal: I love that, Nadia. What I found fascinating about stadiums is — you’re essentially describing — and I think about this as someone who cares about content and social, is a rise of an influencer. This is no different than influencer economies, in many ways, where you have sort of, like, a star contributor and then, like, a bunch of people kind of in this stadium — literally, in your analogy, watching them. And you even say that it’s this shift — and I think you’re quoting someone — to facing the stage, versus facing each other. So when you have this person who’s on the stage, and they’re, like, the primary contributor — and let’s just say creator, because we’re essentially also talking about creator economies here. <Yes.> You made this distinction that they may be intertwined and influenced by their community around them, but they’re not actually “doing peer production” in the classic collaborative way of the first eras of open source. So can you explain that a little bit, and tell me a little bit more about why that’s happening?

Nadia: Definitely, it does have this broader application to what’s happening to individual creators on all these different social platforms today. Most open source projects, we can probably say, used to be clubs — where just, like, not a lot of people were using open source in its earliest days. You kind of had this group of weird developers who just loved using it and maintaining it for their own purpose. And then, eventually, we kind of hit this point where open source became so popular that tons of people were, kind of, discovering all these projects and using the code. And I would attribute that in large part to the creation of a platform — GitHub — which kind of united all these projects together and made them discoverable in one place, with a much more standard way of contributing and discovering — and just, like, thinking about what is open source. 

For a lot of people, GitHub is basically synonymous with open source. Another useful parallel trend here is just that — open source projects started becoming a lot smaller, due to just platform effects of different languages having these package managers that made it really easy to find and discover and use lots of different libraries. And so, now these projects are smaller. They have one developer at the helm, but they have 10x, 100x, 1,000x more users that are coming in.

And so, suddenly you go from having these clubs where everyone kind of, like, knows each other — if you’re using it, you’re kind of expected that you will be contributing back if you need something, instead of asking someone to do it for you. Suddenly you have, like, all these outsiders that are, like, flooding into a project and using it. At the same time, you also have some portion of those users who are now coming into the project and asking for things. And they don’t have the same background that the core developers or creators or maintainers do. They’re just sort of, like, asking for things and leaving. 

A useful analogy here might be thinking about a small town that was largely undiscovered, was not connected to a highway — and then once it became part of a highway system, then you suddenly have all these tourists who are now stopping by some cute little town. And suddenly, it changes the nature of the entire town. Because, I mean, in some tourist towns, you can have — more than half the population is actually tourists, and not even local residents. It’s either, “I’m just gonna completely close off and do my own thing, or have to, like, welcome everyone.” And those are kind of, like, the two extremes that I often hear about when they’re trying to, like, think about, “How do I manage this volume?”

And so, what I’m sort of trying to suggest is there’s something in the middle there — where it’s okay to make things and share them in public. But it doesn’t mean that everyone has to participate. And that’s a theme that I really tried to push on in this book — is that something being public does not mean that it has to be participatory.

Sonal: You actually shared a great analogy in your book, where it’s like the Twitter user — like, an early Twitter user, before they become kind of famous or big. They’re very good about, like, responding to people. They’re building their community. It’s very, like, peer-to-peer. And then there’s a point where some of them become even more influential. And they’re so overwhelmed by mentions and replies and questions that they can’t even remotely respond to any of it — let alone little of it. So I thought that was a very useful analogy for thinking about that. Because one can also evolve over time. 

Modularity of open source platforms

So you mentioned that there’s this, kind of, increasing packages — where people can kind of take things and combine them. And this really stood out to me, because one of my absolute favorite themes when I think of, sort of, meta themes for innovation, and how people change the world, and how people change things — is modularity. And I have this, kind of, joke of, like, “Modularize all the things.” Let’s talk about why that modularity matters. And the example that we both know is modularity in the form of, like, the node package manager. Our mutual friend, Mikeal Rogers, ran much of the work in the Node.js community. Let’s talk about how that shift has mattered.

Nadia: So, on the one end you have very monolithic software, where if you change one thing, it has a lot of patience for changing other things. Software products that look like this tend to be a lot more thoughtful and deliberate and slow about what they actually want to accept as a contribution, or changes that they want to make. Because the whole thing is tightly coupled — but also, sort of, brittle in that way. And so it has just a very different implication for, like, how many contributions we actually accept — how much can we actually change things?

What happened when open source became a little bit more modularized — which is probably best exemplified, as he said, in the story of npm and JavaScript — is that now instead of having this tightly coupled code — you can imagine, like, a tower made out of Lego blocks. Where you can remove one of those blocks, and, like, the rest of the tower still stands. So, it completely just, sort of, changes how we think about a single piece of software. And that, like, instead of having to think about the major implications of changes between the different parts of the code, you can actually say, “Hey, I’m gonna grab, like, lots of different components from different types of developers. I want this person’s library and this other person’s library” — and just, like, fine-tune it to look exactly how you want. And as a result, it enabled a lot of new creation in open source.

Sonal: In the crypto world, the community and the team here loves to talk about the composability of open source projects. Composability being the idea that you can take these building blocks — you mentioned Legos — and that’s really important, because people are combining, remixing, and reusing. And it’s kind of a buzzword, but I use it on the podcast — I’m gonna stop being ashamed of it, which is — combinatorial innovation. And it’s very “primordial soup.” Like, you get all these ingredients, and then it leads to this combining and recombining and evolution, and the Cambrian explosion. I’m just throwing [out] a ton of buzzwords there. So, that’s why the modularity matters. So now, can we talk for a minute about what it means from a project point of view? When open source goes from big to this small kind of — collectives of people that may come together, what are the implications of that?

Nadia: So, if you talk to free software activists from, let’s say, the ’80s or the late ’90s…

Sonal: I used to edit one of them — Richard Stallman. And he would call that “Libre.”

Nadia: Yeah, people that are really focused on the sort of, like, ideological implications of open source or free software — if you talk to them, you’ll find that — or I, at least, found — that a lot of them are really concerned about the liberation and protection and longevity of the code itself. Like, freedom is not referring to any freedom of developers. It’s referring to freedom of the code. <Right.> But if you kind of come down to, like, a world where things are a lot more modularized, suddenly the focus shifts from the code to the people who are behind it — because now every piece of code is much smaller and more trivial. 

There are very well-known developers, especially in the world of JavaScript, where that really encourages a lot of this, sort of, style of development. There are very well-known developers who make hundreds or thousands of popular npm modules — which are each their own separate project, but they’re — each one’s very small. And so, suddenly it kind of becomes more about the person behind it. A useful parallel here might be thinking about the impact of tweets versus blogs, where a blog post is this, like, lengthier thing, and a blog post kind of stands alone as this beautiful piece of literature or whatever.

But then, like, if you’re really into using Twitter, like, you might tweet like 100 things in a day — and one tweet might go viral, but, like, you have so many more that come up, right? And so, it kind of just becomes, like, about the person tweeting. It’s not about, like, “Oh, he wrote that amazing tweet six years ago that I often revisit.” Like, that’s not really what it’s all about. And so, I think — to sort of summarize this — I think this shift towards modularization also helps drive why we’re seeing more interest in reputation-based and status-based economies. <Yes.> Because it just, like, wasn’t the factor before. It was all about the code. Now, it’s all about the people.

Sonal: Mikeal Rogers and I actually wrote a piece about this when I was at Wired. It was called “The GitHub Revolution.” And this was like in early 2013. And, basically, the fundamental point is that GitHub inverted the model from project to person, and then identity came [into] the picture. But to your point, when you have these modularized packages, and individuals who are very tied to that, it does become about the person. But now, on the social side, if reputation and the person is at the center — not just the code — what does that mean for how these groups organize? And what does it mean for how they manage and how they collaborate?

Managing growth and the creator’s time

Nadia: Yes, there are absolutely different implications for how these different types of communities can and should think about organizing, and how they think about growing and maintaining over time. The currency that I’ve settled on was focusing on a producer’s attention as a limited resource. So we all talk about the attention economy, but the attention economy tends to refer to a consumer’s limited attention. But we don’t often talk about a producer’s limited attention. So, like, a creator only has a finite amount of time as well. If we’re thinking about creators and not these, like, big distributed communities now, the creator is kind of, like, on their own, and their attention is not gonna scale by themselves. 

The first line that I would draw is between clubs and federations, which are dealing with an abundance of attention, because they can be high contributor growth. And then stadiums that are dealing with dearth of attention, I guess you could say, because their contributor size is not growing significantly, but their number of users is growing.

Sonal: Right. And just again to emphasize, you’re talking about the attention of the contributor and the creator?

Nadia: Yes. And the ones that are probably most interesting to talk about today are the difference between clubs — which have high contributor growth and low user growth — and stadiums, which have high user growth and low contributor growth. And so, one of the things that previous online community literature focuses a lot on — and especially also in open source — are governance processes. And governance is probably more useful and important to talk about in the context of larger contributor communities, because these are coordination problems, right? Like, you have multiple members with a stake in the community who are all coming in with their own interests. And you’re looking to figure out, like, how do we all best work together?

On the creator’s side, there’s probably another version of these processes that need to be developed for stadiums that’s not really about governance, in the same way, because you usually only have one or a couple people that are at the helm. It’s more about the relationship between that creator and their audience and, like, “How do I interface with my audience? How do I make them feel heard? How do I utilize people that might be willing to help or pitch in?” So there’s a lot of just, like, different kinds of strategies they can think about around, like, how do I — given my limited amount of tension, like, how do we make sure that stuff continues to get done.

Sonal: Right. To pull on a couple of other threads there — does this mean that these relationships even have to be persistent? I want to hear your thoughts on that, because we talk about these very stable federations that have been around for decades. But one thing that I find very appealing — and might be a bug to you, but I think is a feature — is that some of these things seem like they don’t have to be persistent and can maybe be very ephemeral, when you have that kind of small modular setup.

Nadia: I absolutely think the relationship between creators and their audience becomes a lot more ephemeral. And we should almost be, like, leading into that design, right?

Sonal: Yes. I really strongly believe this.

Nadia: Yeah. And so, like, there have been these terms that have existed in open source for a while — the idea of, say, like, a casual contributor — to distinguish between someone who’s kind of dropping in and making one contribution, versus someone who’s a more, like, active or present community writer.

Sonal: Right. Didn’t we even call them — I think in our last episode — they’re drive-by contributors, right? <laughter>

Nadia: Yes. Drive-by contributors, casual contributors. And so these are the people that are not coming in with a pro-social attitude. One thing I did find in my research is that folks that come in as these more active contributors making substantial contributions — a lot of them do come in displaying pro-social attitudes from the beginning.

Sonal: Ah, interesting.

Nadia: Yeah. So they are coming in saying — they’re looking for a community that they want to be a part of, and they want to help out. So, like, one behavior you might see that’s different about an active contributor versus a casual one is someone coming in and, like, answering someone else’s question, instead of opening an issue saying, “Fix my thing.” It’s, like, two very different kinds of behavior, right? Like, one, you’re trying to help someone else — and, one, you’re asking for help. Like, “I want to get something out of this. I want to get my contribution merged. I have a question that I need answered,” whatever. They’re coming in with some sort of personal interest.

Sonal: By the way, you also use the word parasocial in your book, which I had to look up because I didn’t even know that was a thing.

Nadia: I actually think, like, parasocial is a great way to just describe what kinds of community these stadiums essentially are — which, it basically just means, like, one-sided communities. Where, like, one side of the audience has a deeper, more perceived intimate relationship with the creator than the creator does to them.

Sonal: That’s very similar to podcasting.

Nadia: It is very similar. If a creator were to treat every single fan that they met or every single person in their audience as someone that they’re gonna develop a deep and long-lasting relationship — like, that’s just exhausting. It’s completely impossible. But if they say, “Okay, like, we are gonna just meet this one time. Like, how can I make sure that this person feels fulfilled, or whatever, and I manage this without giving too much of myself?” And so, yeah, like, these interactions are more ephemeral. And we can, sort of, design around that where, like — “Here are a bunch of, like, self-serve resources.” Or we can encourage users to help each other, instead of always turning to the creator for help. And so all these other, sort of, like, supporting satellite communities can thrive and flourish on their own without needing an involvement from the creator.

Open source creators and branding

Sonal: What do you think of something like “The Ringer,” where you have someone like Bill Simmons — the analogy here is, he’s a hotshot coder — but, really, he’s like, a creator. He did “Grantland,” and then he went out on his own and did “The Ringer.” And then within that, he built a constellation of brands underneath his parent brand. It’s both bundling, and also, like — just constellation communities. Do you have thoughts on how that works? And how that might play out in the open source world as well?

Nadia: Well, I guess there is a version of that that happens in open source, which is — you have this broader language ecosystem. I’ll keep coming back to JavaScript as the best extreme to demonstrate this. And we can drill even further into JavaScript — let’s say like the React ecosystem. And within React, there are a bunch of associated projects that a React developer might use. And so when we think about who is a contributor to that project, like — yes, you could look at who has actually made contributions to some specific subproject. But you could also say, “Well, who’s contributing to, like, React more generally?”

And so taking, like, webpack — or something that is a subproject that a React developer might expect to use — someone might have never contributed to Webpack before. But if they’re well known as a developer in some other part of the React ecosystem, then they already have a little bit of currency and a little bit of reputation if they were to try to come in and open a pull request, or make a contribution. And so I don’t know exactly what the analogies are between that and sort of, like, subscription bundling, or what that can look like.

But one thing might just be that when we think about — what would it look like to have more subscription-type support for open source developers — which GitHub Sponsors, Open Collective, there are examples of this already — we might think a little bit more about, well — it’s not just this one project that this developer works on, but they work on this ecosystem more generally. And so, maybe similarly — the way that, like, a writer might have started with, like, one type of newsletter, and then, like, they join forces with another one — and then, suddenly, we’re sort of supporting this entire bundle of people that are working on a similar theme. You can imagine that happening with open source developers, where they’re no longer just tied to, like, one specific project, but it’s like, “I support your development work more generally.”

One of the more obvious examples, I guess, I could point to is Sindre Sorhus, who has done pretty — like, thousands of mostly npm-related projects. <Right.> But he’s sort of, like, his own mysterious entity. It’s not really about any one specific thing that he does. <Right.> He’s just, like, a very generative person. And he is supported through sponsorships.

Sonal: I’m gonna ask you a crazy question. This is a thing I’ve been very fascinated by for a long time. I tried pitching this at Wired — by this idea of, like, digital suicide. Taylor Lorenz writes these beautiful pieces about, like, Instagram, and all these various communities online, etc. And I’m also fascinated by this phenomenon of all these, like, teens creating multiple accounts and multiple identities on their Instagrams. And then they also abandon them, which is something I love — this idea of this kind of abandoned wasteland of digital identities and places, because it feels like the real world to me. That there are places that are ghost towns, and places that have been lost in the sands of time, for better and worse. Do you have thoughts on how that may or may not apply to open source, because not only do these things not have to be persistent — they can be ephemeral. Is it okay that they die, or that they even have — up front — a calculated, kind of, end point?

Nadia: Oh, this is where software gets really interesting — and, I think, different from most other forms of creation. Because if someone creates an Instagram account that gets really, really popular, and everyone’s following it, and then eventually — suddenly — this person goes dark, and we never get another post of them — a lot of people will be sad about it. People might create, like, spin-off accounts, and tribute to that original account, whatever. But, like, the world doesn’t actually, like, break and shut down. 

If a maintainer has a product that is wildly popular, and they’re just sort of, like, over it, and they disappear — and this does happen often — that code is still — if it’s popular, is being used by a bunch of other people. And, like, code changes over time. It doesn’t need upkeep and maintenance. Intrinsic motivation helps a lot with — on the creation side of things in the very beginning. If something becomes really popular, then you start getting these more extrinsic rewards, like reputational benefits, or status, or whatever. But a lot of stuff is sort of front loaded.

And so, if you’re talking about maintaining a software project into perpetuity, after a while, you’re already known as the creator of that thing — there’s not really any additional benefit. But you still need to keep maintaining, and, in fact, sometimes those maintenance costs get even higher over time. And so, that’s why I think it helps make the case for — we need to find other reasons to keep people wanting to maintain stuff or make it easy for them to step down. Because intrinsic motivation really only takes you so far. And so, if something happens, where they actually, like, need some changes to be made or need updates to be made to the project, and the maintainer is nowhere in sight — and nobody else has the ability to, like, make commits or contribute to the project, like — this actually can create, like, real problems for software.

Sonal: You’re basically describing software as more of a living, breathing organism, actually, in that context. <Yes.> You either want to evolve it and keep it going, and generationally it can evolve into something else and have offspring, etc. But it’s a different thing than when you just have, like, an abandoned site or, like, an abandoned farm somewhere.

Nadia: And this is why, like, from the beginning, the ability to fork code or basically, like, copy the exact repository somewhere else — has become — it was, like, a very important part of it early on to say, like — someone can always take the copy of this code and make their own version of it somewhere else. Unfortunately, this comes back to this dependency issue today, where — yes, in theory, you can fork a project. And in practice, there might be a lot of other things — other software libraries, other pieces of software — that are pointing to that specific project. And so, if you fork it, you now have to somehow convince all those projects to start pointing to your new project. 

And so, it is this challenge with open source where sometimes, like, a maintainer disappears and is nowhere to be found. But, like, they still need to keep doing things to the code and to the project. Forking is not always an answer — an easy answer to that. It’s still about telling everyone like, “Hey, come over here. Use this.” And that’s actually why I think this concept of maintenance, that is maybe easier to see in software, actually really applies to every creator today. <Yes!> Because a lot of people go through this experience of, you know — you did one thing that might have gotten you this huge, like, seed initial audience or whatever.

Sonal: It made you internet famous too, for lack of a better phrase. <crosstalk>

Nadia: You can have that moment, but then you know, you have to continue creating things — otherwise, people are gonna stop paying attention to you. Much like writing code, it’s not enough to just sort of, like, publish it once and be like, “I’m done. Like, I’m never gonna touch this thing again.” If you are trying to build this reputation over time — and some people will say, “I had one viral video on TikTok, and that’s it. Like, I’m never doing anything else again.” But if you’re trying to be, like, a TikTok creator, you’re gonna have to start making more hits over time. And so your reputation is itself this thing that requires maintenance in order to stay relevant.

Building an audience

Sonal: It is literally one of my favorite parts of your book, because it reminds me of the theses that we’ve talked about at our firm, too, around the passion economy. Here, the artifact is code, but it can be any activity that’s being coordinated, quite frankly, in your framework of your book — which is why I really believe, again, that this book is applicable to everybody. Open source is almost a misnomer, because people think it means code. And it really means everything. It’s like any kind of creation and consumption, frankly — but what you’re really saying is a maintainer is not just a coder, it’s a creator. And they’re maintaining their content in this world, or whatever they’re creating — which I think is incredibly powerful. 

And what’s really powerful about that is — then you think about sort of the related business models for that. Like, when I think of the example of what subscription and SaaS, Software as a Service, did for the world of on-prem software and how people used to sell software — and you had, like, the suited person do this big multimillion dollar deal install, never see them again. SaaS changed the game for everybody in companies because you had to consistently earn their dollars every month, but in a way that was a wonderfully sticky, stable relationship too, like you were mutually dependent.

Nadia: Yes, this is, I think, a giant red arrow pointing at why subscription models are gonna only become more and more interesting in the very near future, because they do take into account this need for, like, ongoing development. There are ongoing costs associated, and you have to earn that, as you said, over time. And so, they’re capable of sort of capturing both your existing value — the value that you have accrued thus far — and also speaking towards the future value that you might create, because when you subscribe to someone or someone’s thing, you’re saying, “Like, I expect there is gonna be more stuff being created in the future.”

Sonal: And this goes back to the phrase, and something you said in your book, and that’s sort of theme for me — you can be transactional, but be in a very high sustained relationship, because it’s a repeated game — which is what subscription is. I think that’s super fascinating. So what do you think the implications are, then, for people who change clubs. They go to a different — they create a new stadium. Like, how has this new passion economy and model evolved?

Nadia: I think this kind of comes back to the value of platforms. And they’re a distribution power. So, I mean, in theory, in the past — without having a platform, which is essentially just a stage for creators. That is always gonna exist, is always there for the creator — without that, if you wanted to go off and, like, start something new somewhere else, it’s really, really hard. Because how are you gonna direct anyone to your new thing all the way over there? <Right.> It’s like building a house but not building a road to the house. 

Whereas, like, platforms have this very important role that they play for creators, where if you want to do something new, you have an audience that you’re building on there that you can use to seed whatever your new ideas. Platforms make it so much easier than you could have in the past. I mean, this is also — so I work at Substack, and this is also why I and everyone that I work with believe really strongly in the power of an email list, because an email list is something that you own. And if you want to do something new with it, if you want to do something totally different, like — you have an audience that is sort of built in and that you can take around with you wherever you want. But even if you don’t have an email list, like, having a Twitter following or having an Instagram following, or whatever, gives you that sort of, like, seed money to do something else.

Sonal: Basically, you’re saying that you have the distribution because your audience travels with you. And that’s an important currency because you don’t have to start from scratch every time. That does go to your other point as well, that the reputation is the key and the currency there. And that’s where status — and you talk about this in your book, and Eugene Wei’s thesis about “Status as a Service” comes in.

Nadia: Eugene’s thesis came out, thankfully, while I was writing my book, and it was very helpful, because I was like, “Okay, now I have more vocabulary to explain the things that I mean, that I’ve been struggling with.” I think actually this framing of status economies helps explain some of the shortcomings of GitHub thus far, because there isn’t, sort of, a meaningful way to measure someone’s status — or just have a clear picture of what someone does on the platform, or what kind of developer they are. You can look at any one specific project, and you can see how popular it is, you can see how many stars that it has. But if you go to a developer’s profile, it’s not super clear what they’re known for. You can technically follow a developer on GitHub, but it doesn’t really mean anything — not at all the way that it does on Twitter or something like that.

And so, I think if you talk to well-known developers, or developers that have these larger followings, they’ll probably tell you that they keep their audience on Twitter or somewhere else. And GitHub serves a little bit more of this utility function, as Eugene said. Where if a platform fails to provide this sort of status benefit, then it basically becomes a utility. They will continue to develop the social and status aspects of their platform. But right now it really is much more of a utility, I think.

Sonal: So, you mentioned the power of a platform. And you’ve been using this analogy of, like, cities, and highways, and connecting houses, and connecting a small shop or a small village to a highway — and what that does for people. What about the opposite, when people go off the grid, essentially, and go outside our purview into these sort of private, dark social places — whether it’s WhatsApp groups or Telegram groups, or private stadiums, private groups. And you mentioned in the book — and I saw Yancey Strickler’s tweet about this when he did it on Twitter — he’s a former co-founder, CEO of Kickstarter. He draws the analogy of the dark forest. The reason that we can’t communicate with aliens is because the world is so vast, and the only way people can protect themselves is by being in this dark forest, where there are these vast spaces of separation. So you’re not in this vast — you’re not actually in what is commonly referred to as a public commons. You’re actually very isolated.

Nadia: The only thing I would maybe add there to the dark forest concept or metaphor, is this idea of hostility — that we are all, actually, surrounded. There are all these other people out there. If we’re sitting here wondering, “Where are all the aliens?” They’re there, but the theory being that we’re all trying to stay out of each other’s way and not be detected because…

Sonal: Destruction will be the result.

Nadia: Yes, it’s not a good thing to meet anyone else as curious as we are.

Sonal: The dark forest comes from the idea of the Fermi paradox, I believe — and I’m a big fan of “The Three-Body Problem” trilogy. They have the wall facer — he’s the one who figures this out. So I thought that was a super interesting analogy. Tell me a little bit more about your thoughts about the dark force theory of the internet, and how that applies here. What happens when people go off platform?

Nadia: So Yancey Strickler’s comment about this. And I think basically a lot of people are observing that, okay — we started with these really big social platforms that have grown to become really big — so, the Facebooks and the Twitters and the Instagrams and YouTubes of the world. These are sort of, like, the biggest stages possible. And so, the analogy to what’s happening on the very public web right now is that everyone’s still talking, it’s just sort of, like, we’re kind of moving to these little corners, without fear of being attacked or jumped on somewhere where all context has otherwise collapsed.

Sonal: I’ll say one more thing, because I’m a big fan of the work of the sociologist Ronald Burt — and he talked about this concept of structural holes, where you can have, like, clusters of activity and networks. And I came across this because when I used to work at Xerox PARC, we used to talk a lot about the innovation that would happen when different fields would collide. And it’s because you have these containers — these clubs, these stadiums — of people who have strong ties, but then these really interesting things can happen with what are the weak ties, and then the structural holes in the network. So, if you map these out as, like, a universe of clusters, imagine what’s possible when you can actually bridge some of those structural holes across communities. <Mmm.>  Like, your book made me think about that, actually. I wonder if that’s where the future is going. Is a bundle maybe that? Who knows what’s happening there? I mean, we’re only at the beginning of it.

Nadia: Yeah, I mentioned this quote in the book, but Kevin Systrom said in an interview in 2018, I believe, that social media is in this pre-Newtonian age where we know that it works, but we don’t know how it works.

Sonal: Ah, I love that.

Nadia: I just think that’s really perfect.

Sonal: It is. It’s perfect for the time we’re in. And it’s perfect for why your book is so relevant.

Nadia: This is where I think the model of clubs versus stadiums becomes really useful. For a long time, everyone was really focused on, like, the highly public aspects of the social web. But people are now starting to look at the semi-private web and these quieter spaces. The biggest parallel trends that I’m seeing right now — like, one is seeing this formation of these creator-oriented communities that look like stadiums on the big public stage — in, say, like Twitter or whatever. But then you see this other emergence of, like, group chats. And group chats have become this really — I mean, [they] have always existed and [have] kind of become a much bigger thing in recent years, <Yeah.> partly because people are looking for a relief from this high, heavy public space. And those map really well to clubs, where you aren’t trying to add a lot of users to your messenger app. You’re trying to just keep it to, like, six of your closest friends. In most cases, we’d say that you’re, like, actively suppressing user growth. But contributor growth is high, where you’re totally down to chat with your friends and that little group.

So, those map really well to the clubs that I sort of identified here. Whereas stadiums apply to both these, like — creator communities are happening in very public platforms but I think can also help us understand why things like podcasts and newsletters are having such a great moment in the sun right now. Because they’re designed for that one-sided, parasocial type of community. <Mmhmm.> 

Where if, you know, we’re recording a podcast right now, it’s just a conversation between me and you. And, yes, hopefully, thousands of people will be listening to it later. But that we’re, sort of, like, doing this in public — meaning that we’re publishing our conversation — but we’re not actively interacting with the audience that might be listening to us. And similarly, with a newsletter, I can write this long-form post and share my thoughts in a higher-context situation. I assume or hope that most of the people subscribing have some context for who I am. And then I can, kind of, send it out, and people can read it on their own time. It’s not the same thing as when I tweet something out, and then literally anybody with an internet connection… <Right.> I made a public tweet — can see it and respond to it and pass it around and do whatever they want with it.

Crisis of the commons

Sonal: And so, to summarize, the clubs are the projects — the spaces with high contributor growth and low user growth, like these private messaging groups. The stadiums are, like, the projects with a low contributor growth and high user growth, like these newsletters and podcasts. I really think, Nadia, one of the best things about your book is this framework of the federations, the club, the stadiums, the toys — because you dehomogenize this phrase “open source and community.” And then it, correspondingly, gives people frameworks for what that means for how you build, support, nurture that. 

So, I’m now gonna switch to asking you some practical questions about that. Platforms are having a moment right now, for better or worse. It’s one of the reasons that we also are very excited about crypto and talking about communities. And I want to talk about the tragedy of commons and the work of Elinor Ostrom, who is definitely having a moment. Right now you and a lot of other people I know have been citing her work. One of our former partners, Jesse Walden, wrote a post about cooperatives as an analogy for crypto networks. And he cited some of the conditions that she cites in governing the commons — and then you yourself summarize the conditions. Can you, A, tell me what those are — B, tell me why you think this is important, and then help me connect the dots for how that matters practically?

Nadia: Sure. So, Elinor Ostrom was a researcher who became well known for her work around trying to understand why the tragedy of the commons occurs, and how we might avoid it or move around it. Tragedy of the commons just sort of being that — if everyone has access to a shared resource. You can imagine a fishery or a forest — anyone can cut down wood in the forest. But if everyone does that, and just, kind of, does what they want for themselves, then eventually that forest is gonna be depleted unless it is managed in some shape or form. 

And so, tragedy of the commons is this concept from ages ago that is, maybe, one possible outcome of the commons. But it’s almost like when people talk about the commons, they always talk about tragedy of the commons — as though you can’t have, like, a non-tragedy of the commons. And so, Elinor Ostrom is basically looking at — what are situations where commons are being sustainably self-managed. And she did decades of research looking at these, like, fisheries and forests, and just, like, different examples of commons, and then documenting what she found and summarizing them into principles for — if you are in the situation where you have this shared resource, how can you manage it without everyone just sort of taking for themselves.

And so, I talked about her conditions in the book a bit, and the ones that I’ll point to that are most relevant for this conversation are — this idea that in order to have a well-managed commons, you do need to draw boundaries around membership. It needs to be clear who is allowed to appropriate from the commons and who isn’t. And then with that, there are all these implications of, well, what does it mean to be a member of the commons? A couple of things that I’ll highlight are, one, this idea that you have high context for your interactions with other people that are also members.

Sonal: Yep, that creates trust. That’s what creates trust. It’s just like in a company. They say the best advice you can give to any team or any fast-growing group is the more shared context, the more trust you have — because you can do more shortcuts together in your work.

Nadia: That’s right. High context, high trust is a really important implication of having these clear membership boundaries. And then the other thing I’ll point to is the idea of having a low discount rate — which is just saying that if you’re a member of this community, you expect to be around for a while.

Sonal: Sorry, what do you mean by low discount rate?

Nadia: Low discount rate is just this idea that if you’re invested in the community for a long period of time, you’re not planning on hopping in, saying something rude — if this applies to online communities — and then just, like, hopping out and disappearing. You’re like, “I’m stuck here. I need to, like, actually learn how to work with everybody else.”

Sonal: Right. It’s actually, kind of, like, skin in the game.

Nadia: Yes. In order for a commons to function in this healthy way, you need to have these underlying conditions of high context, of having skin in the game, of having clearly defined membership, among a bunch of other things that I won’t get into here. Her work is finding, I think, renewed appeal right now — especially because people are trying to answer these questions in open source and in online communities elsewhere, of just, like — how can communities self-manage and not implode over time? 

There’s so much that is relevant about her work to today. I think it mostly applies, though, to the concept of clubs — clubs basically being this commons, where everyone has a stake in what they’re creating. If we think about a stadium — a creators’ community that is on a very public social platform — the whole concept of the commons kind of breaks down, right? Like, I mean, if I’m tweeting in public, anybody can read what I’m saying. And until recently, as Twitter’s now making it possible for people to limit comments on their tweets, and things like that — but for the most part, like, anybody can just, like, comment on my tweet and jump in.

And so, understanding, I think, both what her theory of the commons was — and why it doesn’t really apply to today — can help answer some of these questions about — is it okay to have common threads that are entirely open to everyone? What are the problems that might arise from that? And then what can we do to actually limit interactions from outsiders, so that the people that are most involved or have most skin in the game can actually get stuff done? 

And it’s a difficult thing to talk about, because it can be taken as gatekeeping or trying to keep other people from participating. It’s just, like, a touchy subject. I can’t say that we should just close off the boundaries entirely. And I think this gets to, again, the idea of — you can have things that are public but not participatory. It’s okay to make software that anyone can use. That doesn’t mean that everybody who uses your software can also participate in its production. So, it’s really just about finding kind of, like, a middle ground there.

Sonal: You have so many great analogies in your book — that sometimes this is more like directing air traffic, given the flood of abundance we have on our internet today. So, on that front, I’m gonna ask you just a couple of quick questions on the practical front. Let me do this lightning round style. What is one key piece of advice you might have for community managers?

Advice for platform management

Nadia: For community managers — first thought is, just know what kind of community you’re in charge of. Which is where I think it’s helpful to have a set of different models in your mind of — are you actively trying to bring in lots more contributors? It’s okay if you’re not. Some communities do better on contributor retention, and less so on contributor growth. And that’s totally fine. Or is it the kind of thing where there is a lot of work that needs to be done? And do you think you stand a chance of recruiting more people? Then go recruit more people. 

It’s fine to have a community that isn’t super high growth but is stable. It’s fine to have a community that is extremely high growth, where you’re trying to bring in lots of different kinds of members and make it this really bustling, kind of, federation-style community. It’s fine to have just one person that is, sort of, standing up in front of a crowd. That is a community in its own form, but it just requires different sorts of strategies to figure out how to manage it.

Sonal: You’re basically saying — know the difference of whether you have a club or a stadium. And, by the way, you quoted that person talking about the Newtonian phase. Who knows? There might be times when you can have both in one place, so that can change. And then how about advice for platforms?

Nadia: For platforms, I would say — take your creators seriously and the responsibility and the relationship that you have to them. And what I mean by that is that platforms are really the only place to create these closed status economies that enable creators to continue doing their work for however long they want, and to open up all these amazing opportunities for creators. And sometimes that doesn’t directly happen on your platform — as in, like, maybe it’s not that they can raise money directly on your platform, but it is important to make their status legible to others so that they can take that clout and that reputation and actually, like, shop it around to turn it into other opportunities.

Sonal: I might even add that crypto is great for that, because that’s where you can actually port some of your currency — your reputation currency — and prominence in a way that — like, in blockchains. And then, finally, advice for leadership and/or communication tools. Because we’ve talked a lot about — I think a lot of times people make the mistake of talking a lot about collaboration and coordination. But they don’t often talk about the communication part of things. And this is particularly heightened in our remote world. So, any advice you have on the communication tools side and then anything for the leadership side?

Nadia: More so the leadership side. On the leadership side, I think it’s, again, about knowing your community and not being afraid to be decisive. A lot of communities that I’ve looked at have suffered in one direction or the other of — either being overly deferential to their community and trying to treat it like this pure democracy, when really the community’s size or shape is just so unwieldy that that’s not really possible. And so it is okay to say, “These are the decisions that are being made. And we don’t have to make this — bring this to a vote every time we want to decide what we want to do.”

And maybe also on the flip side, that depending what type of community you are overseeing, there are ways to bring those active voices and contributors into leadership and encourage more people to participate. But, again, it depends on whether you’re on this, like, high growth side or low growth side. On the communication tools side, I think this idea that separating the ideas of public and participatory is just gonna lead to a lot of really interesting things happening in the near future. Just getting playful with the idea that a community does not mean that everyone is participating at equal volume and, you know, shouting at each other — because we’ve seen with, like, every social platform, that gets pretty hard at scale.

And so, like, as we’re creating new things today, it’s fun to think about the opportunity we have in front of us to actually design from scratch about — in thinking, like, how would we have these sort of scaled social interactions? And so disambiguating the idea of public and participatory can just lead to really fruitful new ways of communicating.

Sonal: This reminds me — one of my favorite quotes from Questlove. This is from his book, “Creative Quest.” He basically writes that when you make work, you are the creator, but also the eventual audience — which I think is such a powerful idea. There’s, like, so many different ways to interpret that. What I love about your book is that it’s not a grand theory of everything. It ties together lots of different themes together in a really meaningful way. But you can also tease them apart regardless of your vantage point, whether you’re a creator, open source, business — however, you might define a company or any form of coordination and collaboration.

I also appreciate, given that you were kind of newer to the community — compared to, sort of, the first, early generations — that you don’t bring this sort of chip of nostalgia, and, sort of, come at it from a very first principles approach, and just sort of really bring all your insights together. So I just want to thank you for “Working in Public: The Making and Maintenance of Open Source Software.” I’m gonna add my own personal subhead, which is — and many, many other kinds of orgs. So don’t just not read it if you don’t think it’s about you, because open source is everyone. Thank you for joining the “a16z Podcast.”

Nadia: Thank you. It was a pleasure to be here.

• 

  Nadia Eghbal is an independent researcher exploring how the internet enables creators. Previously she focused on the production of open source software and worked at Substack. She wrote the book Working in Public.

  Follow Twitter Website
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

In this episode of the a16z bio Journal Club, bio deal team partner Judy Savitskaya and Lauren Richardson discuss research that aims to enhance the efficiency of photosynthesis and carbon fixation. These two processes are used by plants and other phototrophs (like algae) to convert light energy and carbon dioxide from the air into organic matter. The pathways took millions of years to evolve, but can scientists use advances in biochemistry and synthetic biology to increase their efficiency?

The two articles discussed were both published in the journal Science and are both from the lab of Tobias Erb at the Max Planck Institute for Terrestrial Microbiology. The first article, published in 2016 develops a synthetic pathway for the fixation of carbon dioxide in vitro. The second article, which was published in May 2020, combines this synthetic carbon fixation pathway with the natural photosynthetic pathway isolated from spinach to create an artificial chloroplast.

This combination of natural and synthetic components to improve the efficiency of these pathways has a number of potential applications, including in engineering our crops to grow faster. We discuss these exciting applications, how evolution has restricted the efficiency of carbon fixation and how these engineered solutions get around that problem, and the use of microfluidics for vastly improved experimental design.

“A synthetic pathway for the fixation of carbon dioxide in vitro” in Science (November 2016), by Thomas Schwander, Lennart Schada von Borzyskowski, Simon Burgener, Niña Socorro Cortina, Tobias J. Erb

“Light-powered CO2 fixation in a chloroplast mimic with natural and synthetic parts” in Science (May 2020), by Tarryn E. Miller, Thomas Beneyton, Thomas Schwander, Christoph Diehl, Mathias Girault, Richard McLean, Tanguy Chotel, Peter Claus, Niña Socorro Cortina, Jean-Christophe Baret, Tobias J. Erb

a16z bio Journal Club (part of the a16z Podcast), curates and covers recent advances from the scientific literature — what papers we’re reading, and why they matter from our perspective at the intersection of biology & technology. You can find all these episodes at a16z.com/journalclub.

Show Notes

• Why natural processes around carbon fixation can be slow and inefficient [2:23]
• Possibilities around improving plant performance, capturing more carbon dioxide [5:37]
• Three key advances of this research [7:15], and the steps needed to bring it into the real world [13:08]
• How this research could lead to the creation of artificial cells and other improvements over natural biology [14:54]

Transcript

Lauren: Hello, I’m Lauren Richardson and this is the “a16z Bio Journal Club.” This is our podcast where we cover recent scientific advances, why they matter, and how to take them from proof of principle to practice. In today’s episode, I’m talking with bio deal team partner, Judy Savitskaya, a resident expert in all things synthetic biology. We cover recent research that seeks to improve the processes of photosynthesis and carbon fixation, and how these advances could one day be used to improve crop growth and carbon sequestration in plants.

First, a quick biochem refresher. During photosynthesis, also known as the light cycle, light energy is captured by chlorophyll and then passed through a series of reactions to the energy-rich chemical co-factors ATP and NADPH. These co-factors are then used by the carbon fixation cycle, or dark cycle, to drive the capture and conversion of carbon dioxide into more complex carbon molecules like glucose. Plants and other phototrophs use these two processes to turn sunlight and carbon dioxide from the air into organic matter. These are hugely powerful processes that have generated essentially all the organic matter on earth, from the wooden trees to our own bodies. But these processes also aren’t perfect, and scientists have for decades been trying to make them more efficient.

The two articles that we discuss today were both published in the journal “Science” and are both from the lab of Tobias Erb at the Max Planck Institute for Terrestrial Microbiology. The first article, published in 2016, develops a synthetic pathway for the fixation of carbon dioxide in vitro. The second article, which was published in May, combines this synthetic carbon fixation pathway with the natural photosynthetic pathway isolated from spinach to create a synthetic chloroplast. This combination of natural and synthetic components to improve the efficiency of these pathways has a number of potential applications, including engineering our crops to grow faster. Judy and I discuss these exciting applications, how evolution has restricted the efficiency of carbon fixation (and how these bioengineered solutions get around that problem), and the use of microfluidics for vastly improved experimental design. But first, we start with a discussion of why the dark cycle, this process of carbon fixation, is not as efficient as it could be.

Limitations of natural carbon fixation

The key thing here is that the dark phase has this great limiting step, which is this enzyme known as RuBisCO. It is just super slow. And that’s the first enzyme in the pathway that binds carbon dioxide.

Judy: Poor old RuBisCO — when I imagine it, it’s like an old man enzyme with, like, a long white beard and it makes a lot of mistakes and it goes really slow — but it evolved really early on, and then was a key requirement for these organisms to live. Furthermore, RuBisCO makes a lot of mistakes, which is that it often subs in oxygen molecules for carbon dioxide molecules. So there’s a huge body of work trying to evolve RuBisCO to be better, but as it stands, our plants are stuck with this really old enzyme that is not as efficient as it could be.

Lauren: Yeah. Instead of evolving RuBisCO, it seems like plants have evolved kind of everything around it, so there’s all different classes of plants that have modified to support the slow cycling of RuBisCO, and to be efficacious in different environments and to limit the error, as you call it, of RuBisCO, which is also known as photorespiration.

Judy: It’s kind of crazy that, like, rather than this enzyme evolving to be better, there’s entire mechanical systems involved to, like, open these pores in the plant cells to be able to let in more or less oxygen at different times of the day, and it’s this highly complex thing that has evolved to make up for the just poor efficiency of one enzyme. The Tobias Erb lab developed essentially a synthetic Calvin cycle, so it’s a different method for fixing CO2 into some sort of carbon-containing substance.

Lauren: I say dark cycle, you say Calvin cycle.

Judy: Fun factoid is that it’s actually the Calvin-Benson-Bassham cycle, but Bassham doesn’t want his name included because he thinks it’s a disservice to all the students that worked with him on the project, so he has requested that it be called the Calvin-Benson cycle.

Lauren: In the 2016 article that you mentioned, the authors developed this very cool synthetic pathway for CO2 fixation that did not use RuBisCO. Instead, it used a combination of 17 different enzymes from nine different organisms that could do this dark phase half the reaction 10 times faster than the plant version that does rely on RuBisCO. And they called this the CETCH cycle or the C-E-T-C-H cycle.

Judy: In the previous paper, they sort of cheated by adding in these enzymes that would just produce NADPH and ATP as starting points for their synthetic carbon fixation cycle so that they can kickstart part of the experiment that they really cared about. In this new paper, what they’re doing is adding in a module to create that NADPH and that ATP that is light-driven. So it doesn’t require the experimentalist to add in these enzymes or to add in the substrates for these enzymes.

Lauren: Yeah. What they’re doing here is they’re linking the light cycle, so the photosynthetic element to the dark cycle, the carbon fixation part. So the goal is to have this own self-sustaining reaction because that’s what plants are. So let’s talk about the implications of this research.

Real-world implications

Judy: The biggest and most interesting implication here is that you could use some of the insights from these papers to upgrade how plants perform. And the idea is to basically counteract some of the evolutionary pressures that were present when we weren’t using these plants for crops, or to sort of make up for some of the inefficiencies of natural selection — like, for example, RuBisCO being a bad enzyme. This entirely new cycle for doing carbon fixation could really dramatically increase the rate of carbon fixation and the rate of growth for plants that we use as crops.

Lauren: These synthetic chloroplasts that they created are actually more efficient than natural chloroplasts, and that’s because they don’t have RuBisCO, which is slow. And they also don’t suffer from photorespiration, which is that wasteful process we were talking about, where RuBisCO uses oxygen instead of carbon dioxide. And, in most plants, they waste about 25% of their energy from photosynthesis on photorespiration. So there’s this way in which you could kind of get around the photorespiration problem with something like these synthetic chloroplasts.

Judy: When we think about, on a global scale, the carbon cycle, and if we’re concerned about release of too much carbon into the atmosphere, there’s sort of an interesting class of solutions here, which is to increase the rate at which our crops pull carbon dioxide out of the atmosphere, and that kills two birds with one stone. One is that it increases your efficiency of food production, and at the same time, you’re removing more carbon dioxide from the air. You’re actually using it for something useful.

Three key advances

Lauren: Yeah, that’s possibly a very elegant solution. Let’s dig into these methods and results now. So in plants, photosynthesis happens in chloroplasts, and chloroplasts contain an internal membrane structure called thylakoid membranes which contain chlorophyll, the molecule that actually is able to capture light energy and convert it into energy that the plant can use. And all the other enzymes in the pathway that are needed to go from light energy to ATP and NADPH, which are these energy-storing molecules.

So the way I see it, there were three key advances in this paper. The first was extracting these membranes from spinach that contained the enzymes for the light cycle, and getting that into a functional unit; then linking it to this synthetic CETCH cycle — this synthetic carbon fixation pathway that they’ve created — and then the third was to use microfluidics to really optimize and integrate these two cycles together so that there was this self-sustaining basically synthetic chloroplast.

Judy: I mean, I think it’s cool that they’re able to show you can get this thylakoid membrane module, separate it from the rest of a chloroplast, which is integrated complex, large organelle. They can just take this one piece of it, and then it works like the black box you would expect it to. There was one change they had to make, which was to add exogenous ferredoxin, which is like the one component of this, sort of, electron transfer process that is not attached to the thylakoid membrane. Other than that, it kind of just transferred wholesale into this in vitro context and worked. So I’m sure there’s, like, lots of experiments here that were failures that we’re not seeing, or that are, like, buried in the very, very large supplemental materials for this paper, but it’s really impressive that they were able to basically show the function of this module in vitro without all the bells and whistles surrounding it from the natural organism.

Lauren: So next, they linked these thylakoid membranes, the part that’s performing photosynthesis, to the synthetic CETCH cycle. What do you think about this fusion of the natural and synthetic components? Because that’s what they’re — basically they’re doing here. They’ve got the natural photosynthesis machinery, and then they’ve got the synthetic dark cycle machinery.

Judy: Yeah, it’s interesting because it’s sort of, like, demonstrating that we understand half of it, right? So there’s this — there’s two approaches to understanding the parts of a system. There’s the bottom-up and top-down. So if you understand all of the components of some enzymatic pathway, you should be able to add them all in, one at a time, purified, and then recapitulate the behavior of the full pathway. So that’s sort of what they’ve done with their first paper with the CETCH cycle, and then there’s a different way that biologists understand nature, which is by breaking it down. So you start with, like, this is how the organism works — and then take away pieces until you figure out what’s like the set of things that is necessary to do a certain reaction. And this is kind of cool because it’s a fusion of both of those worlds.

Lauren: Yeah. I think there’s something interesting, and the rate-limiting step is this RuBisCO, that’s part of the dark phase. It makes sense to tinker with that element, but you don’t have to reinvent the photosynthesis arm, the part that is working. You can appreciate, kind of, the beauty that nature has already provided and use that in combination with the things you want to change.

Judy: Yeah. That’s a really good point, actually. I hadn’t thought of that, but this really suggests that you can move this CETCH cycle that they’ve engineered into an organism that already has that thylakoid membrane piece intact, and you should expect them to just work together well.

Lauren: So, and the third aspect of the paper, they’re using microfluidics to integrate the thylakoid membranes with the CETCH cycle, and to create these basically artificial chloroplasts. So talk to me about what they did with the microfluidics, and what the benefits of using microfluidics for this approach are.

Judy: Yeah. The real benefits of droplet-based experiments is that you can do many of them at once. So the idea here was to create lots of these little droplets, so that each one can contain a different experiment with a slightly different version of the CETCH cycle, or a different ratio of these components that they’re putting together. And they used color-based barcoding, so they could tell what reaction was happening in a given droplet by changing the amount of these different dyes that they added in. The idea is to basically be able to do many experiments in parallel and look at them in one go.

Lauren: So basically, it’s a way to multiplex the experimental design.

Judy: Yeah, that’s — that’s a perfect way to say it. There was this interesting figure at the end where they showed that they get more production of glycolate, so sort of, like, output of their process in the droplets than they do in bulk solution, given the same amount of chlorophyll to start with.

Lauren: My understanding was that it’s all about the right amount of cofactor regeneration, so ATP and NADPH regeneration from the thylakoid membranes to support the optimal functioning of the CETCH cycle. And then, do you think the inherent next step is using microfluidics? Would they be able to, kind of, dose in the exact amount for optimal production?

Judy: Yeah. I mean, they’ve got 17 enzymes to play with, so that’s, like, a lot of parameters that you can modify, and then you can change the levels of each of those enzymes. So this microfluidic tool gives them the opportunity to test, like, at very high levels of multiplexing how to optimize this cycle and optimize its interaction with the thylakoid membrane.

Introducing new processes to plants

Lauren: I’m wondering how many steps do you think there are between this work and, like, what they’ve achieved now, and actually getting that into plants?

Judy: That’s actually a really interesting question, because they’ve shown that this synthetic, like, hodgepodge enzyme set works in vitro. That does not mean that it’s going to work in vivo at all. So the first thing is to put this into some really simple organism that’s easy to engineer, like an algae. And the idea here is that you would use the natural thylakoid membranes activity from that organism, but then it would express the enzymes from this different CETCH cycle instead of the natural Calvin cycle, and what you’d need to do is a ton of optimization. I’m not going to sugarcoat it. So is this on the horizon? Probably not. I think the microfluidic experiments that they have are going to be helpful because if they can start with, sort of, extracts of this algae, put it into these microfluidic experiments, and then do their multiplexing there, they can do many more experiments at once, but there is still going to be a big jump from that to the actual organism.

Lauren: Yeah. It’s kind of like the benefit of the CETCH cycle was that they could use all of these different enzymes from all of these different organisms and create this brand new pathway, which was so neat in vitro, but that creates a whole host of new problems for that in vitro to in vivo switch.

Judy: Yeah, absolutely. I mean, I think that’s actually where a lot of, like, the interesting insights into biology come from is, like, we understand how the system works in isolation. We put it into the context of the cell and suddenly everything breaks, and so now the question is like, why did it break? So lots of cool biology coming from trying to transfer this work, but I would not expect, you know, next year to see a paper where this cycle is fully functioning.

Possible future applications

Lauren: The authors of this paper really blew my mind with the last paragraph of their discussion where they talk about using these synthetic chloroplasts in combination with other life characteristics such as self-repair and reproduction in the idea of basically creating a fully artificial cell. When you start thinking about fully artificial or synthetic cells, you know, that makes you think about fully artificial or synthetic tissues, and that kind of scales up to a fully synthetic organism — having the ability to synthetically harness the light from the sun, carbon dioxide from the air, and turn this into, you know, a designer metabolic pathway that could fuel a synthetic life force — is very exciting to me and just kind of wild to think about.

Judy: I love the term synthetic life force. If you think about the cell and all of its functions as a graph, like in the classical computer science sense of the word graph, it is like a super complex structure with, like, many interacting nodes and it’s, like, very hard to get your head around it. How could you ever build that from scratch and make it self-sustaining? But, like, this is a really big piece of that — generating energy, making it happen without an external agent putting in that — putting in new molecules. Like, that could — that could handle, like, a very large portion of the graph that is necessary to make life work. I will say what this gets you is that you don’t have to feed sugar, right?

Lauren: There’s definitely something about the, like, independence of it, though. Like, there’s, you know, providing sugar or feeding it, versus being able to create those energy-storing molecules de novo, which can then be turned into mass, or broken down again as sources of energy. I think it’s really interesting to think about, like, what are the essential processes that you would need to create a fully self-sustaining, independent-of-human-support system that is lifelike in this way, you know, based on biology and not, you know, a robot that we build in the lab.

Judy: But also, like, how do we define lifelike? Just because we metabolize the particular chemistry that we use to do that, that’s just one instantiation. It’s kind of like what happened to result from evolution and then, like, stick because it’s really hard to evolve out of this, maybe even local minimum, maybe not global minimum in terms of how good the processes are. So yeah, I think let’s definitely push on synthetic cells. I think it makes a lot of sense to start with, like, things that look like existing biology, but, like, why stop there? Why not go to something that’s sort of hybrid or exploits entirely new chemistry that we’ve never seen?

Lauren: Yeah. And this kind of can even get us back to what we were talking about at the beginning which was, like, how bad RuBisCO is as an enzyme. RuBisCO originally evolved in environments where there was not a lot of oxygen, so it was before the great oxygenation of the atmosphere. And so, this problem with substituting oxygen for carbon dioxide just wasn’t a thing when it first evolved. And as oxygen increased in the atmosphere, it had to start making trade-offs between the specificity of whether it chose oxygen or carbon dioxide, and its efficiency — so it could be more efficient, but then it would incorporate oxygen more often, versus it could be more specific but then it would be even slower. So if you’re designing a system de novo, is there a way to bypass some of the evolutionarily inherited tradeoffs and make something that’s just more finely tuned to the situation that you want to design?

Judy: So evolution is kind of always lagging behind how the world is changing, which is exactly why RuBisCO is evolved for a world that we no longer live in — but humans can adapt much faster. That’s, like, this interesting philosophical idea that people will say evolution has infinite creativity — like we could never, you know, think up the things that evolution has created. And I think that’s true to some extent, but evolution is fundamentally limited to the designs that are within a certain distance of the designs that are out there in nature today. You’re not going to get a really huge rapid change in an organism just because it wouldn’t survive that sort of transition period. So there’s all of these transitions that evolution can’t pass through, but we can as humans. So I actually think, like, in a lot of ways, human creativity can go way beyond what evolution has made, and I think there’s, like, a ton of opportunity here.

Lauren: Yeah. I don’t think it’s necessarily about being better than evolution. It’s learning from evolution and seeing all the different ways that evolution has functioned and then kind of taking, you know, the best of the best <Plan matching…> and our own — yeah, our own knowledge. And, you know, what AI will be able to provide to us is, like, even beyond our own knowledge is, like, new ways of looking at these problems and these solutions and, like, being able to input them in completely creative ways that, you know, evolution hasn’t found yet and neither have people.

Thank you, Judy, for joining me on “Journal Club” this week. To sum up, we are excited about this work, as it demonstrates that you can improve the process of carbon fixation and link it to the natural photosynthesis machinery from plants. This bioengineering solution could be applied to our crops to improve growth efficiency and carbon dioxide sequestration. That’s it for “Journal Club” this week. You can find all these episodes at a16z.com/journalclub. Thanks for listening.

• 

  Judy Savitskaya is a deal partner at a16z where she focuses on bio companies. Previously, she worked on synthetic biology research at UC Berkeley and was a computational modeling and neuronal networks researcher.

  Follow Twitter
• 

  Lauren Richardson

Many SaaS startups often find their initial product market fit with small and medium businesses (SMBs) and then move up market. Because it’s easier to move up rather than down market, this go-to-market motion has often given SaaS startups the advantage against incumbents, who already have the added price and complexity of traditional enterprise selling.

But moving up market comes with its own challenges.

In this episode, David Ulevitch and our newest enterprise general partner Kristina Shen look at the SaaS go-to-upmarket motion. They cover why it’s easier to move up market than down, when to make the move, and how to price for it, including why so many founders underprice, deciding on free versus paid trials, and navigating the transition to larger accounts.

Show Notes

• Fundamentals of the up-market approach [0:51]
• Questions around timing [3:00], how the up-market strategy developed [5:09], and what the future may hold due to remote work [6:29]
• Discussion of pricing approaches [7:49] and mistakes firms make around pricing [14:57]
• Important metrics to consider [16:39] and a discussion of free vs. paid trial pricing [17:50]

Transcript

Das: Hi, and welcome to the a16z podcast. I’m Das, and in this episode, I talk SaaS go-to-market with David Ulevitch and our newest enterprise general partner Kristina Shen. The first half of the podcast looks at how remote work impacts the SaaS go-to-market and what the smartest founders are doing to survive the current crisis. The second half covers pricing approaches and strategy, including how to think about free versus paid trials and navigating the transition to larger accounts. But we start with why it’s easier to move upmarket than down… and the advantage that gives a SaaS startup against incumbents.

Up-market fundamentals

David: If you have a cohort of customers that are paying you $10,000 a year for your product, you’re going to find a customer that self-selects and is willing to pay $100,000 a year. Once you get one of those, your organization will figure out how you sell to, how you satisfy and support, customers at that price point and that size.

But it’s really hard for a company that sells up market to move down market, because they’ve already baked in all that expensive, heavy lifting sales motion. And so as you go down market with a lower price point, usually, you can’t actually support it.

Das: Does that mean that it’s easier for a company to do this go-to-market if they’re a new startup as opposed to if they’re a pre-existing SaaS?

Kristina: It’s culturally very, very hard to give a product away for free that you’re already charging for. It feels like you’re eating away at your own potential revenue when you do it. So most people who try it end up pulling back very quickly.

David: This is actually one of the key reasons why the bottoms up SaaS motion is just so competitive, and compelling, and so destructive against the traditional sales-driven Saas motion. If you have that great product and people are choosing to use it, it’s very hard for somebody with a sales-driven motion, and all the cost that’s loaded into that, to be able to compete against it.

If you have that great product and people are choosing to use it, it’s very hard for somebody with a sales-driven motion, and all the cost that’s loaded into that, to be able to compete against it. —David Ulevitch

There are so many markets where initially, we would look at companies and say, “Oh, well, this couldn’t possibly be bottoms up. It has to be sold to the CIO. It has to be sold to the CSO or the CFO.” But in almost every case we’ve been wrong, and there has been a bottoms up motion.

The canonical example is Slack. It’s crazy that Slack is a bottoms up company, because you’re talking about corporate messaging, and how could you ever have a messaging solution that only a few people might be using, that only a team might be using? But now it’s just, “Oh, yeah, some people started using it, and then more people started using it, and then everyone had Slack.”

Kristina: I think another classic example is Dropbox versus Box. Both started as bottoms up businesses, try before you buy. But Box quickly found, “Hey, I’d rather sell to IT.” And Dropbox said, “Hey, we’ve got a great freemium motion going.” And they catalyzed their business around referrals and giving away free storage and shared storage in a way that really helped drive their bottoms up business.

Questions of timing

Das: It’s a big leap to go from selling to smaller customers to larger customers. How have you seen SaaS companies know or get the timing right on that? Especially since it does seem like that’s really related to scaling your sales force?

Kristina: Don’t try to go from a 100-person company to a 20,000-person company. Start targeting early adopters, maybe they’re late stage pre-IPO companies, then newly IPO’d companies. Starting in tech tends to be a little bit easier because they tend to be early adopters.

Going vertical by vertical can be a great strategy as well. Targeting one customer who might be branded in that space, can help brand yourself in that category. And then all their competitors will also want your product if you do a good job. A lot of times people will dedicate a sales rep to each vertical, so that they become really, really knowledgeable in that space, and also build their own brand and reputation and know who are the right customers to target.

Das: So right now, you’ve got a lot more people working remote. Does this move to remote work mean that on-premise software is dying? And is it accelerating the move to software as a service?

The old school way of doing SaaS go-to-market is bundle everything together, make the pricing super complex. New modern SaaS pricing is keep it simple, keep it tied to value, and make sure you’re solving one thing really well. —Kristina Shen

Kristina: This remote work and working from home is only going to catalyze more of the conversion from on-premise over to cloud and SaaS. In general, software spend declines 20% during an economic downturn. This happened in ’08, this happened in ’01. But when we look at the last downturn in ’08, SaaS spend actually, for public companies, increased, on average, 10%, which means there’s a 30% spread, which really shows us that there was a huge catalyst from people moving on-premise to SaaS.

David: And as people work remote, the ability to use SaaS tools is much easier than having to VPN back into your corporate network. We’ve been seeing that, inside sales teams have been doing larger and larger deals, essentially moving up market on the inside, without having to engage with field sales teams. In fact, a lot of the new SaaS companies today rather than building out a field team, they have a hybrid team, where people are working and closing deals on the inside and if they had to go out and meet with a customer, they would do that. But by and large, most of it was happening over the phone, over email, and over videoconferencing.

And all the deals now, by definition, are gonna be done remote because people can’t go visit their customers in person.

Das: So with bottoms up, did user behavior and buyer behavior change, so the go-to-market evolved? Or did the go-to-market evolve and then you saw user and buyer behavior change? I’m curious with this move to remote work. Is that going to trigger more changes or has the go-to-market enabled that change in user behavior, even though we see that change coming because of a lot of forces outside of the market?

Kristina: I definitely think they are interrelated. But I do think it was a user change that catalyzed everything. We decided that we preferred better software, and we tried a couple products. We were able to purchase off our credit card. And then IT and procurement eventually said, “Wow, everyone’s buying these already, I might as well get a company license and a company deal so I’m not paying as much.”

While obviously software vendors had to offer the products that could be self-served, users started to realize they had the power, they wanted to use better software, they paid with their credit cards. And now software vendors are forced to change their go-to-market to actually suit that use case.

Das: If that’s the case that when user behavior has changed, it’s tended to be the catalyzing force of bigger changes in the go-to-market, what are some of the changes you foresee for SaaS because the world has changed to this new reality of remote work and more distributed teams?

The influence of remote work

David: We’re in a very uncertain economic environment right now. And a couple of things will become very clear over the next 3 to 9 to 15 months — you’re going to find out which SaaS products are absolutely essential to helping a business operate and run, and which ones were just nice to have and may not get renewed.

I think on the customer, buying side, you’re very likely to see people push back on big annual commitments and prefer to go month-to-month where they can. Or you’ll see more incentives from SaaS startups to offer discounts for annual contracts. You’re going to see people that might sign an annual contract, but they may not want to pay upfront. They may prefer to meter the cash out ratably over the term of the contract.

And as companies had empowered and allowed budget authority to be pushed down in organizations, you’re gonna see that budget authority get pulled back, more scrutiny on spending, and likely a lot of SaaS products not get renewed that turned out to not be essential.

Kristina: I think the smartest founders are making sure they have the runway to continue to exist. And they’re doing that in a couple of ways. They’re preserving cash, and they are making sure that their existing customers are super, super happy, because retaining your customers is so important in this environment. And they’re making sure that they have efficient or profitable customer acquisition. Don’t spend valuable dollars acquiring customers. But acquire customers efficiently that will add to a great existing customer base.

The smartest founders are preserving cash, and making sure that their existing customers are happy, because retaining your customers is so important in this environment. —Kristina Shen

Pricing approaches

Das: To go into pricing and packaging for SaaS for a moment, what are some of the different pricing approaches that you see SaaS companies taking?

Kristina: The old school way of doing SaaS go-to-market is bundle everything together, make the pricing super complex, so you don’t actually understand what you’re paying for. You’re forced to purchase it because you need one component of the product.

New modern SaaS pricing is keep it simple, keep it tied to value, and make sure you’re solving one thing really, really well.

David: You want to make it easy for your customers to give you money. And if your customers don’t understand your pricing, that’s a huge red flag. Sometimes founders will try to over engineer their pricing model.

Kristina: We talk a lot about everything has to be 10X better than the alternatives. But it’s much easier to be 10X better when you solve one thing very, very well, and then have simple pricing around it.

I think the most common that most people know about is PEPM or per employee per month, where you’re charging basically for every single seat.

Another really common model is the freemium model. So, think about a Dropbox, or an Asana, or a Skype, where it’s trigger based. You try the product for free, but when you hit a certain amount of storage, or a certain amount of users, then it converts over to paid.

And then you also have a time trial, where you get the full experience of the product for some limited time period. And then you’re asked if you want to continue using the product to pay.

And then there’s pay as go, and particularly, pay as you go as a usage model. So, Slack will say, “Hey, if your users aren’t actually using the product this month, we won’t actually charge you for it.”

David: The example that Kristina made about Slack and users, everybody understands what a user is, and if they’re using the product, they pay for it, and if they’re not using it, they don’t pay for it. That’s a very friendly way to make it easy for your customers to give you money. If Slack came up with a pricing model that was like based on number of messages, or number of API integration calls, the customer would have no idea what that means.

Kristina: There’s also the consumption model. So Twilio only charges you for every SMS text or phone call that you make on the platform any given month. And so they make money or lose money as your usage goes. The pricing is very aligned to your productivity.

David: Generally, those are for products where the usage only goes in one direction. If you think of a company like Databricks, where they’re charging for storage, or Amazon’s S3 service, it is very aligned with the customer, but it also strategically aligns with the business because they know the switching cost is very high, the churn is very low. And generally, in those businesses, you’re only going to store more data, so they can charge based on usage or volume of data.

Kristina: Recently, there’s been a huge trend of payment as a revenue. It’s particularly common in vertical markets where SaaS companies are adding payments as a revenue in addition to their employee or subscription revenue. If you look at Shopify, for example, more than 50% of their revenue is actually payment revenue. They’re making money every single time you purchase something off one of their shopping cart websites.

Das: When you’re working with a founder or a SaaS startup, how have you seen them find the right pricing model for their product, for their market?

Kristina: Step one is just talk to a lot of customers. Try to figure out what is the market pricing for possible alternatives or competitors, understand their pain points and their willingness to pay. And just throw a price out there, because you have to have a starting point in order to actually test and iterate. Particularly in the SMB, or the bottoms up business, you can test and iterate pretty quickly because you have so many data points.

David: I always tell founders, step one is to just go out there and talk to customers. Step two is just double your prices. I don’t think there’s ever been a great company with a great product that’s fallen apart because their pricing was wrong. But a lot of SaaS startup founders really under price, and you don’t want to find out two or three years later that you were 200% underpriced.

A very common thing that SaaS companies do, they’ll have the basic package that either is free or low cost, that you can just sign up online for. They’ll have a middle package where they share some pricing, and then they’ll have the enterprise package where you have to contact sales to find out more. And that way they don’t actually have to show the pricing for that third package. And that gives the salespeople the flexibility to adjust pricing on a per deal basis.

Das: When you’re working with companies, why are they underpricing their products?

David: I think it’s psychological. People need to price on value, and they don’t know how much value they’re delivering relative to “Oh, it only cost me $100 a month to provide this service, so I just need to charge $200.” But if it turns out you’re saving your customer $50,000 a year, then you’re wildly underpriced.

You have to remember that SaaS is essentially a proxy for outsourced IT. You’re spending money on a SaaS service to not pay to develop something internally, or to have to pay IT to support something that’s more complex on-prem. Software is much cheaper than people, and so generally, the price point can be much higher.

People need to price on value, and you have to remember that SaaS is essentially a proxy for outsourced IT. You’re spending money on a SaaS service to not pay to develop or support something internally. —David Ulevitch

Kristina: And the other thing is your value increases over time. You’re delivering more features, more products, you understand the customer better. It’s the beauty of the SaaS model and cloud model that you can iterate and push code immediately, and the customer immediately sees value. A lot of times people have the same price point from the first customer sold to three years later and the 200th customer. Quite frankly, you’ve delivered so much value along the way that your price point should have gone up.

The other thing I’ll say is a lot of people discount per seat pricing a lot as they move up market. We tend to tell people that the best validation of your product having great product market fit is your ability to hold your price point. So while there is some natural discounting on a per seat basis because people do deserve some volume discounting, I would say try to resist that as much as possible.

Das: Especially for a technical founder, it’s so tempting to get in there and fiddle with these knobs. How do you know when it is time to experiment with your pricing and packaging?

David: If you’re looking at your business and you see that you are doing more deals, and they’re closing faster, you should raise your pricing. And you pay attention to how long it takes to close deals and whether the number of deals is staying consistent as you do that. And, at some point, you’re going to find out when you’re losing deals on price.

I think a moment where companies have to plan ahead to avoid having to course correct is after they roll out massive pricing and packaging changes, which are pretty natural as companies move up market. But how they navigate that transition to larger accounts, and how they either bring along or move away from those smaller, earlier customers who got them to where they are, tends to be really important because they can get a lot of noise on Twitter, they can get a lot of blowback from their customers.

So Zendesk is a company where they rolled out a major packaging change. And when they rolled it out, they hadn’t planned on grandfathering in their early customers. They got a lot of pushback, and very quickly, they put out a blog post and said, “We hear what you’re saying, we appreciate you building the business that we’ve become today. We do need to have a package for the future. But all the people that have been customers so far will be grandfathered in for at least a period of time into the old model.”

Kristina: If you iterate pricing constantly, you don’t really have this problem because your customers will be used to pricing changes. You normally pair them with new features, and it all kind of works out. But if you have to go through a big grandfather change, I tend to lean towards treating your early customers really, really well. They adopted when you weren’t a big company yet. They probably co-built the product with you in many ways. And so, it’s great to get more dollars out of your customer base, but treat your early customers well.

 A lot of people discount per seat pricing a lot as they move up market. The best validation of your product having great product market fit is your ability to hold your price point. —Kristina Shen

Common pricing mistakes

Das: Are there any other failure modes that you see startups really falling into around pricing and packaging or any common mistakes that they make?

David: I think a lot of founders don’t always map out the cost or model of their pricing and their product relative to their cost of actually doing sales and marketing and customer acquisition.

Kristina: Inside sales is so popular in Silicon Valley. When you’re selling more to an SMB or mid-market type customer, the expectation is that you’re educating and helping the prospective customer over the phone. And so, you’re not expected to be as high touch.

But 5K is almost the minimum price point you need to sell to the SMB with an inside sales team in order to pay for the outbound costs and all the conversions, because there is typically a team that sits around the quota carrying rep. And so, price matching — how much your price point is compared to what your go-to-market motion is — matters a lot.

Other big failure modes that I see, people guess the ramp time of a sales rep wrong. And ramp time really ties to the segment of customer you’re selling into. It tends be that if you’re selling into the enterprise, the ramp time for sales reps, because sales cycles are so long, tend to be much longer as well. They could be six months plus, could be a year.

While if you’re selling more into SMB or mid-market, the ramp time to get a rep up and running can be much shorter, three to six months. Because the sales cycles are shorter, they just iterate much faster, and they ramp up much more quickly.

David: The other thing that people have to understand is that sales velocity is a really important component to figuring out how many reps you should be hiring, whether they should be inside reps or field reps. If it takes you 90 days to close a deal, that can’t be a $5,000 a year deal, that has to be a $50,000 or even $150,000 a year deal.

Das: Kristina, I know you’ve done a lot of work with metrics. So how do those play in?

Kristina: Probably the one way to sum it all together is how many months does it take to pay back customer acquisition cost.

Very commonly within the SaaS world, we talk about a 12-month CAC payback. We typically want to see for every dollar you spend on sales and marketing, you get a dollar back within a year. That means you can tweak the inputs any way you want. Let’s say that doing paid acquisition is really effective for you. Then, you can spend proportionally more on paid acquisition and less on sales reps. Vice versa, if you have a great inbound engine, you actually can hire a lot more sales reps and spend more on sales headcount.

With all formulas, it’s a guide rail, so if you have customers that retain really, really well, let’s say you’re selling to the enterprise, and you’ve got a 90% or 95% annual retention rate, then your CAC payback could be between 12 and 24 months. But let’s say you’re selling to the SMB and churn is 2% or 3% monthly, which ends up being like 80% to 90% annual retention. Then, because your customer is less sticky, I would recommend looking at a CAC payback of 6 to 12 months.

Free vs. paid trial pricing

Das: How should you think about doing a free trial versus a paid trial?

David: On the one hand, the bottoms up motion where people can try essentially a full version of a product before they buy it is extremely powerful. On the other hand, I’ve started to try to think about how I advise companies, when they are thinking about a free trial for something that might cost $100,000 or $200,000 a year? Do we do a paid pilot that has some sort of contractual obligation that if we meet then turns into a commercial engagement?

Kristina: I do think the beauty of the bottoms up business is that you can get people to try the entire experience of the product for free, and they fall in love with it, and a certain percentage will convert. And that works really, really well for products that can self-serve.

When you start moving up market to more complex products, the challenge with trials is it takes work to actually implement the product, whether it be integrations, IT has to give access, etc. You lose that self-serve ability, which is so amazing in the trial.

And so, I tend to be more in the camp of paid trials, if it costs you money to actually deploy the trial. And when you’re selling to bigger customers, they associate value when they have to pay. Once a customer has to pay you, then they feel a need to make the project successful and thus they will onboard, schedule things, give you data and access.

David: If you can get to a point where you get the customer to do that paid pilot, such that the only difference between a pilot and an actual customer is just the signing of a contract, that’s very powerful.

Now, that does force you to have a really good pre-sales motion to make sure that you can deliver on the promise you’ve made your customers. When companies don’t have a great product, and they paper over it with professional services and sales engineering and post-sales support, that paid pilot thing doesn’t work because the experience isn’t good enough. So, it really is incumbent on the SaaS company that does a paid pilot to make sure that they are able to deliver on that experience.

Kristina: And one emerging trend recently is people signing an annual contract with a one or three month out, as a replacement to the paid pilot. Because it’s the best of both worlds, the SaaS company that’s selling the product gets a higher level of commitment. And the customer gets the optionality of opting out in the same way as a trial without any clawback.

It really comes down to where procurement falls. Sometimes procurement is at the beginning of that decision, which makes it more like an annual contract. Sometimes procurement is at the one or three month opt-out period, which means the customer already has a great experience, loves the product, and it is an easier way to convert procurements to actually sign on…

David: And that is a really good segue into renewals. I always tell founders, you might have this subscription business, but it’s not a recurring revenue business until the second year when the revenue actually recurs.

I think you really have the first three months to get a customer up and running and happy. And if they’re not, you then have about three months to fix it. And if all that works out, then the remaining six months of the contract can be focused on upsell and expansion.

It’s not a recurring revenue business until the revenue actually recurs. You have 3 months to get a customer happy. And if they’re not, you have 3 month to fix it. Then you can have 6 months focused on upsell & expansion. —David Ulevitch

Das: Awesome. Thank you, Kristina. Thank you, David.

Kristina: Thanks so much for having us. This was fun.

David: Yeah, a lot of fun, great topics, and our favorite thing to talk about.

• 

  David Ulevitch is a general partner at a16z where he invests in enterprise and SaaS companies. Prior to joining the firm, he was the founder and CEO of OpenDNS (acquired by Cisco).

  Follow Twitter
• 

  Kristina Shen is a general partner at a16z where she invests in enterprise and SaaS companies. Prior to joining the firm, she worked at Bessemer Venture Partners, JMI Equity, Goldman Sachs, and Credit Suisse.

  Follow Twitter
• 

  Das Rush

It used to be that the only way for humanity to grow — and progress — was through destroying the environment. Sure, the Industrial Revolution brought about the growth of our economies, our population, our prosperity; but it also led to our extracting more resources from the planet, more pollution, and some nightmarish human conditions as well. But is this interplay between the two — of human growth vs. environment, of protection vs. destruction — really a zero-sum game? Even if it were true in history, is it true today? How about for developing economies around the world today — do they have to go through an extractive phase first before entering a protective one… or can they skip that phase altogether through better technology (the way they leapt to mobile)?

And if capitalism is not responsible for environmental degradation, than who or what is? Where does technology come in, and where doesn’t it — if you believe we already have the answers to saving the environment? Marc Andreessen and Sonal Chokshi interview MIT economist Andrew McAfee about all this and more, given his new book, More from Less: The Surprising Story of How We Learned to Prosper Using Fewer Resources — and What Happens Next.

So what does happen next? From nuclear power to dematerialization to Tesla and the next cleantech revolution (or not), this episode of the a16z Podcast brings a different perspective to an important discussion around taking care of our planet… and also ensuring human progress through the spread of human capital and technology.

image: Kevin Gill / Flickr

Show Notes

• Overview of progress from the Industrial Revolution to today [1:22]
• Definition of capitalism [5:47] and why it promotes efficient use of resources [11:22]
• The benefits of clean nuclear energy [17:19]
• Advantages of cap and trade to curb carbon emissions [19:00]
• Debate over the future of resource use [27:21] and Fuller’s predictions about dematerialization [30:35]
• New technologies needed to continue dematerialization trends [36:40]

Transcript

Sonal: Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal. And I’m excited to do another one of our co-hosted episodes with Marc Andreessen, who joins me in interviewing MIT economist Andrew McAfee — who we’ve actually had on the podcast a couple years ago on a great episode with his coauthor, Erik Brynjolfsson, on their book, “Machine, Platform, Crowd.” But Andy’s new book takes a very different turn from that previous series of books — and focus, on the beam of bits — to focusing on atoms, the physical world. Basically the environment. It’s called “More from Less: The Surprising Story of How We Learned to Prosper Using Fewer Resources and What Happens Next.” And I think it’s a really important book, contributing to the important dialogue we’re having right now on taking care of our planet, and also of taking care of human progress — especially because these two don’t have to be a zero-sum game of the two in conflict with each other.

So, what does it take to go from that narrative of extraction and destruction to one of protection and progress? So, in this episode, we cover everything from what capitalism’s role is in all this — including what it is and isn’t — to the global environment, including China and India. And throughout, we discuss the technology, from energy use and types of energy, to dematerialization — and, surprisingly, the idea of that well before software was even invented. Stay tuned for that bit. But we quickly begin with the technology and effects of the Industrial Revolution.

From the Industrial Revolution to today

Andrew: The industrial era kicked off with the Industrial Revolution, and the James Watt steam engine, and all those other technologies — was this period of amazing human growth. The growth of our economies, growth of our prosperity, growth of our population. And that was great in a sense, but it really did feel like there was a trade-off between improving the human condition and improving the state of the world. And in the industrial era, if you looked at the evidence, you could make a pretty strong case that we were increasing our growth at the expense of the planet that we all lived on. We took more resources from the earth every year. We chopped down more trees. We cleared more cropland. We took more fossil fuels out of the earth. We polluted more. We either domesticated animals or drove them to the brink of extinction. And the reason I decided to write “More from Less” is, I don’t think that’s true anymore.

The evidence supports the idea that in the richest countries — and I’ve got the best data for America — that that trade-off between the human condition and the state of the world is actually in the rearview mirror. Because in almost all the ways that we could care about — improving the human condition — we’re taking fewer resources from our planet, we’re polluting it less. Some of the animals that we pushed to the brink are coming back. I didn’t hear that story being told. And so hence the book.

Sonal: So one thought that struck me in looking at the example of the Industrial Revolution, which everyone points to as the greatest story of progress — you point out that it ended slavery but increased child labor?

Andrew: There were some pretty nightmarish situations early in the industrial period. There really were factories full of kids under the age of 10, working 14-hour days.

Sonal: Yeah.

Andrew: And some of these kids weren’t even sent there by their parents. They were orphans. And this was what we decided to do. I consider that a moral mistake, and different than what kids were doing on farms before. But in most rich countries, slavery ended early in the industrial era. Child labor ended before the 20th century. But we didn’t start dealing with pollution and species that we pushed to the brink of extinction until much, much later than that. So we kind of looked after ourselves first and then the rest of the planet afterward.

Sonal: So, Andy, I want to probe the conflation of capitalism and extraction of resources when it doesn’t actually have to necessarily be that way. But one stat in particular that struck me on that front is that research emerged showing that the U.S. GDP was closely intertwined with energy consumption. You talk about this in your book. Clearly, there’s something about more energy consumption tied with the success of an economy.

Andrew: If you draw a graph of the U.S. economy — the real GDP of the U.S. — from 1800 to 1970, and then you add one more line to that graph, which is total energy consumption per year from 1800 to 1970 — those lines are really hard to tell apart. They sit right on top of each other. And there’s this whole stream of research that turned into an assumption — that if you told me what your energy use per capita was, I would tell you what your GDP per capita was or the state of advancement of your society. And we almost use those two things as proxies for each other. One of the super weird things is that that relationship has completely broken down in the United States — where, again, I know the evidence really well. Total U.S. energy consumption has been basically flat since at least the end of the Great Recession, and maybe even before that started.

Now, in the old-fashioned way of looking at things, you say, “Oh my God, there was this massive recession.” Absolutely not, it grew like crazy. But we’ve divorced energy use from growing the economy. And one of the broad points I make in the book is — that story is very broad. We’ve divorced most other kinds of using up atoms — using materials — from our prosperity growth. And that relationship is not unique to America. It exists elsewhere, and it will spread as we spread capitalism and technology. 

One of the things I have fun with in the book was trying to defuse tension, because there are a lot of audiences where if you say capitalism, they start throwing rotten tomatoes at you. They just can’t hear the word. It’s so triggering. So one thing I tried to do is say, like, “What do I mean by capitalism?” And I don’t mean cronyism. I don’t mean corporatism. I don’t mean regulatory capture, or financialization. These are all real things. These are all perversions of actual capitalism.

Sonal: Yeah, I hate that capitalism gets a bad rap. And while we may argue for a better form of capitalism, can you just break it down and sort of tease apart the myth from the facts when it comes to, like, what is capitalism? I think sometimes people are using different labels for different things quite honestly.

Andrew: Yeah. And let’s be clear on what we’re cheerleading about. Capitalism is the best way the earth has ever come up with to get goods and services into the hands of people. Now, that’s a really important thing for a society to do, if you don’t want your people to starve or die of exposure. And when I talk about that, I mean a few pretty specific things. First is that private companies are responsible primarily for producing those goods and services. It’s not the government. It’s not individual craftsmen or artisans or anything. Number two, they use prices that are not centrally set or controlled. And prices convey a huge amount of information about abundance and scarcity, and where you should allocate your attention.

So, we really need prices to be floating around in an economy. We need your property rights and your contracts to be respected by a working court system that believes in protecting those things. So that, if you’re an upstart, if you have a good idea — either the government or some big powerful company, or some billionaire, can’t just come and take it from you without compensating you and without your agreement on that stuff. One of the most important phrases for capitalism is voluntary exchange. You can’t force me to sign a contract. You can’t make me buy a product or forbid me from buying a product. You can’t stop me from moving to another state. So you just have this — it’s free-flowing. But there are these hard and fast constraints and rules about it. If you get those things right, the goods and services will become abundant to people.

One of the things I loved writing the book was that Adam Smith nailed all of these topics in 1776. And yet here we are almost 250 years later, and we’re arguing about things that he kind of put to rest a long time ago. He said you need actual competition, not cronyism, for the benefits of capitalism to accrue. Amen to that.

Marc: He actually went further. He actually called business people the enemies of capitalism.

Sonal: Why?

Andrew: <laughter> He’s got that famous quote that, “Men of trade seldom meet together, even for merriment, except it winds up in a conspiracy against the public.”

Marc: Yeah, he basically argued, it’s like what modern libertarians are actually arguing, which is basically to the extent that business people, like, start to get involved in political policy. They try to rig the political system in their favor. And then that trips the line between so-called capitalism and corporatism. Then politically, that’s sort of the distinction between being pro-business and being pro-market.

Andrew: Or being pro-incumbent and pro-market.

Marc: Right, right, exactly. And what you want is you want to be pro-market. This is what we run into in our business. You know, because we launched these new companies that don’t have any political power whatsoever. And they go into these industries that in some cases are heavily dominated by incumbents. And invariably what you find is an intertwining of the incumbents with the regulatory system, often under the color of consumer protection. But it actually turns out, what’s happened is the incumbents have rigged the system. They’ve rigged the politics for their own preservation, and the hypocrisy gets exposed in the form of, like — you just have a product that’s just obviously better. And then the captured regulatory state comes to try to shut it down to protect incumbents.

Andrew: Well, my favorite one of that was — for a while, I think France or Paris had the requirement that a limo had to go back to its home station for 15 minutes before picking up another customer. Why on earth would that be?

Marc: That’s right? Well, this is always the absurdity of would you really rather, like, stand out in the rain with your arm up, seriously? Right? And by the way, would you really rather have a system in which the driver is able to, like, eyeball you in the street and decide to not give you a ride?

Sonal: I was about to say it because what people don’t talk about is the disproportionate impact on people who don’t look like they’re someone who you want to give a ride to. And now you can get a ride anywhere by anybody.

Marc: Yeah, exactly. But then there’s this risk that, you know, you become the thing that you hate, you know — which is always a danger.

Andrew: We also need to acknowledge there are problems that capitalism itself doesn’t solve. People getting left behind, inequality of some kinds of opportunity, the lack of a safety net, pollution, species loss. Absolutely, these are well understood — sometimes called market failures — and we need to be thoughtful about those things. But, again, Adam Smith, I don’t think he talked about species loss and extinctions, but he got these things right in 1776. And it kind of frustrates me that there’s still this big Marxist hangover going on, where people willfully or ignorantly misunderstand this economic system that we have.

Marc: So in the 20th century, were capitalist systems or communist systems worse for the environment?

Andrew: Oh my God, there’s no comparison. I think the single saddest and most tragic story that I learned when I was researching the book is about the Soviet whaling industry. The Soviets signed up for all the treaties to sharply limit the whale hunts. And then they ignored the treaties that they signed, which is bad enough — and they killed about 200,000 additional whales over the decades before they finally stopped. The crazy part of the story is why they killed 200,000 additional whales. And the answer is for no good reason at all. They didn’t eat the meat. They didn’t need the blubber, because they were already self-sufficient in oil. The only reason they did it was because they had Stalinist five-year plans for growth in the fisheries industry. And whales weigh a lot. And if you kill lots of whales, you grow your fisheries industry.

There’s this heartbreaking story about the guy that was in charge of the fisheries industry. And he was such a pro at executing Stalinist five-year plans, he was named Hero of the Soviet Union. And one of the Soviet scientists went to him at some point and said, “We have to stop this. There will be no more whales for our children to see.” And his reply was, “Our descendants will not be the ones to fire me from my job.” <Damn.> So, you know, we can talk — the capitalist systems, we made pollution mistakes, yes, and we corrected them. What closed communist systems did was keep making mistakes under cover of darkness for no good reason.

Capitalism’s effect on emissions

Marc: I mean, this is very relevant to current events, right? One of the things that is very common, obviously, in the United States right now, is the theory that capitalism is responsible for environmental degradation — and unless we convert to a socialist system immediately, like, the environment is doomed. And, therefore, the very clear assumption and statement is that shifting to a socialist command and control system will lead to better environmental outcomes. That’s a very common theme right now. Like, how do you address that in present times?

Andrew: Yeah. And it’s a tiny bit hidden, right? Because the people who make that argument — I hear them railing against capitalism and saying we need to take better care of the planet via some alternative. And then they get kind of vague about what that alternative is. But I think they’re all either dodging the fact, or lowballing that they want central planning. They want a command and control economy. And let’s call that what it is. It’s something between socialism and communism. And the thing that we need to keep in mind is that the capitalists — the free societies of the West — were the ones that dealt with their pollution problem earliest and best. And what I consider the great triumph of the environmentalist movement that, you know, kicked off around Earth Day, was that “we, the people” demanded that we stop having polluted air and dirty water and things like that. And we got it via things like the Clean Water Act, the Clean Air Act, the Marine Mammal Protection Act. These were landmark pieces of legislation.

The single most important thing that happened after the legislation was passed was we got clever about how to reduce pollution levels. The story of cap and trade for reducing particulate emissions from power plants and reducing sulfur dioxide is such a fantastic story, because we put together this coalition of environmentalists and conservative economists. And we put in place a market system for trading pollution — which sounds weird and bad, except that it has cratered our levels of SO2 and other particulate pollution, and done it for about one-fifth of the originally estimated cost. It was just this extremely efficient thing to do. So the notion that capitalist systems have no way of dealing with increasing pollution is just dead flat wrong.

Sonal: When I was reading the book, one thing that struck me was — do you think that with developing countries today, like India, China. I mean, one would argue they’re more developed than fully developing, however you define it —that they even have to go through an extractive phase first, in their first phase of figuring out how to use their resources? Like, I guess my question is why couldn’t they leapfrog this extractive phase, and just go right to a more practical phase when it comes to the acceleration of technology? Do you think that extractive phase has to happen?

Andrew: It’s pretty clear to me that America and the UK, and I think most other super-rich countries, are past peak stuff. If you weighed our economy year after year, it would weigh less year after year. India and China and Bangladesh are not yet at peak stuff. But they will get to that point much earlier in their GDP per capita trajectory. Because, you know, Nigeria is not going to lay an extensive copper telephone network across the country. They’re not going to build as many coal plants per capita as we did, because that’s just economically inefficient to do. I’ll be surprised if the Chinese have as many private cars per capita as we did earlier in our history, because it’s really impractical to have that heavy, expensive asset sit idle 95% of the time.

So I do think that this technologically very sophisticated economy is going to get countries through this resource transition much earlier than we went through it.

Marc: So one of the things that’s so striking — carbon emissions, right, in the U.S. are falling. And are you telling me they’re starting to fall in certain parts of Europe as well?

Andrew: Yeah, the EU, in general, has been on a shallow downward trend.

Marc: Yeah, you know, there’s lots of advances being made in energy-efficient, you know, technologies of all kinds. And so one would imagine, like, this will continue. Let’s take the strong advocates for dramatic action at their word that we’re gonna run into real trouble globally. How do you not progress from there, to believing we have to take a very different approach from a foreign policy standpoint — in particular towards China and India — potentially up to and including coercive actions? Because if you look at the graph of global emissions growth, it’s very clearly two, like, gigantic examples.

Andrew: So we’re going to invade them to make them reduce their carbon emissions? I don’t see how that plays out. Let me give you a couple of softer ways, because I think there are a couple important ones. One is, they gave the Nobel Prize to Bill Nordhaus last year for his work about how to deal with global warming and the notion of a carbon dividend. When Nordhaus proposed his carbon tax — and I like the phrase carbon dividend better, because it’s not a tax where the government keeps the money, you pass through the government directly to people and give them a carbon dividend, hopefully skewed a little bit toward lower-income people. As part of that, you also do what’s called a border adjustment, where you look at all the imports into the country, and if they come from high-carbon sources, you tax them — just like you would if they were made in this country with high-carbon sources.

I think that’s a really strong incentive for our main trading partners. And China’s probably exhibit A here, to start literally cleaning up their act in this regard. The other thing is, you know, we have one source of power — we have one way to generate power that is scalable, clean, somewhat economical, and not intermittent. And it’s called nuclear. And there are a couple of countries like France and Sweden that have cheap electricity and the cleanest power in Europe. And we’re running away from — and the rest of the world. I find this completely perverse. Why not put together an international coalition — and along with that, an international patent bank — so that it’s cheaper to produce the new generation of nuclear reactor. I’m pretty sure that will get the cost down to the point where it becomes an economic no-brainer, even for low-income countries, to start transitioning into a clean energy environment.

I would do both of those things way before I would try to coerce other countries into changing their energy profile, or doing it in a way that would slow down their growth or impoverish their people.

Marc: So I’m glad you brought up nuclear. I was gonna ask you that. So many groups, just, like, flatly roll out nuclear as an option. So what’s going on there? And, like, what’s the way through that?

Andrew: I honestly don’t know the answer. Why are they so stridently anti-nuclear? There’s probably a bundle of things going on. One is because of everything from Hiroshima and Nagasaki, to Godzilla, to Three Mile Island and Fukushima and Chernobyl. I mean, I just finished watching their “Chernobyl” miniseries on HBO. So I have this kind of visceral “ick” reaction to the idea of super widespread nuclear power. But I think our homework is always not to trust that initial “ick” and to go look at the evidence. And when you actually look at the evidence and look at the issues, I don’t know how you come away [as] anything except a nuclear advocate. And we worry about things like nuclear waste, and we should worry about nuclear waste.

But we don’t then say, well, how much harm is caused by the pollution from other kinds of power generation? Worldwide, there are clearly hundreds of thousands of deaths a year from people breathing coal dust, and people breathing the emissions from coal plants. So the death toll — it’s not even close. And this is backed up by very good research published in “The Lancet” and elsewhere. There’s a nice article in “Our World in Data” about relative safety levels and death rates from different kinds of power. You walk away from that nuclear’s biggest cheerleader. So I don’t quite know why the reaction is so strident and visceral and negative. All I can say — it is not based on evidence. And I’m starting to see a coalition forming that pushes back against that to say, “We’re getting this deeply wrong on an important issue.”

Success of cap and trade

Sonal: Okay, so you were talking about cap and trade. What made that so successful compared to other attempts? Obviously, there’s a market-based mechanism, but give me more details.

Andrew: Cap and trade — the basic idea is [to] make pollution expensive. Attach a cost to it. In other words, put it inside the market. Pollution doesn’t naturally have a price. And when that’s the case, no matter what the press release says, businesses have a strong incentive to go pollute, if it’s free. Okay, put a price on it. And then here’s the brilliance of cap and trade — allow companies to buy and sell that pollution, or more specifically, that right to pollute with each other. So if I’m super dirty, and I can’t clean up quickly, I’ve got to buy the right to pollute. But I’m willing to buy that right if it’s cheaper than the cost of me cleaning myself up.

Sonal: Right.

Andrew: Somebody will sell me that right and make some cash, because they’re already really clean, and they don’t need that right. So this was a line of economics research that got started with legendary Nobel Prize-winning economist Ronald Coase, and descendants of his ideas got put into practice early in the Reagan administration with the help of, like, the Environmental Defense Fund. So this beautiful alliance formed to say, “Hey, let’s try this market-based thing for dealing with pollution.” They overcame whatever reluctance was there from the incumbents, again, and they did it. And then the research is pretty clear that we can just look at what happened to particulate emission from these kinds of plants. America’s skies are just 90-plus percent cleaner than they were when that legislation was passed. And the cost of doing it is a fraction of the original estimate of that.

So there’s a reason for these kinds of crazy fans of markets for getting things done — they work. And when you can put things like pollution in a market, and you do this with cap and trade and carbon dividends and things like that, these are the most efficient ways to deal with the problem.

Marc: Don’t China and India have to sign up for the same thing?

Andrew: One of the problems with carbon is that the harms from it are not local, and they’re not immediate. So maybe the fast-growing, high-carbon countries right now will choose to ignore it for a while longer. We have a couple mechanisms to get them to not do that. And like I said, if you do a border adjustment for the high-carbon products that we import, that’s a really strong incentive to do things better — if we can make it cheaper for them to be green. And, personally, I think nuclear and, you know — a patent banker, cheap technologies around nuclear — is the path to do that. We clearly have to help the currently low-income world get rich on a lower carbon trajectory than they’re on right now. That’s different than saying that they can’t use more energy year after year. I’m not gonna deny them that right to prosperity.

Sonal: Exactly.

Andrew: But we really want them to get cleaner quicker. I think we have tools to do that. And I don’t think that the Chinese and the Indians are indifferent to the longer-term health of the planet. I really don’t believe that.

Sonal: I mean, they’re living with it in the physical way. Everyone there is facing it and experiencing it in a very real way. And we had this podcast a few years ago with Evan Osnos at “The New Yorker.” We were talking about China. One of my favorite things that he talked about is how, because of the growth of the middle class in China, that there is now a huge cohort of people demanding a better environment — precisely because of the market dynamics.

Andrew: Not just that — getting a better environment. So I found this great research that I put in “More from Less.” A very good economist looked at what happened when China finally got serious about urban air pollution. And the reason they got serious about it was — people were leaving the cities even if they didn’t have government permission to do it. People were leaving because their kids were just clearly getting sick and going to have stunted lives. So China took action. And they brought down their country-wide particulate pollution by 30% in 4 years. And they did it with these draconian means, but they did it. And it took us in the United States 12 years to get that same 30% reduction.

One of the points I make in the book is — democracies are probably more receptive to the will of their people. But there are interesting exceptions in both directions. And China was clearly receptive to the will of its people not to choke off their children with pollution.

Sonal: Right, right. I read a ton of Chinese sci-fi, and it’s literally — the recurring theme is basically about the end of the world and, like, [the] environment. But, Andy…

Andrew: Is that right? That’s cool.

Sonal: Yeah, it’s a really big theme, and you have to read a lot of different Chinese science fiction authors to see this, but that’s basically my genre this year. One thing I want to ask you. I understand from the market dynamics point of view why cap and trade was such a successful idea and example, and it’s been proven out. But why couldn’t a government have simply mandated, like — we will just simply put a limit on this. Draconian measures like China did. Why would that not be as effective?

Andrew: Sometimes we did. That’s how we actually brought down CFC emissions so drastically. We just mandated that they be reduced by X percent over time until they got down to close to zero. The reason that worked is that there’s a relatively small number of industries, a relatively small number of companies, and a relatively small number of products that used chlorofluorocarbons. And to be a little bit more cynical, the other reason that ban worked was — somebody eventually whispered to the incumbent companies, “The CFCs you’re making out right now, they’re off-patent. The new generation of coolants, and propellants, and whatnot — those can be under patent. Those can be a big revenue source for you.” And so they finally got industry on their side.

Fiat can work. You know, for example, it is just flat [out] illegal to dump waste at sea in America. We just did that via fiat. We didn’t put a price on it. You cannot hunt animals in national parks. You cannot hunt deer or duck outside their seasons. So sometimes you want to do things by fiat. But I kind of think if you can put it in a market mechanism, and it’s appropriate to do that, I think you’ll get better solutions quicker. Maybe that’s not right. But I’ve got this deep faith in markets — once you put things in them and price them to deal with that price in a very fast way. If you change a business’ cost structure quickly — man, businesses will run from that increased cost like gazelles run when they smell a lion. It’s just amazing how quickly it’ll happen.

Marc: I will tease — or torture Andy a little bit. And that is, you’re probably well aware — support for market-based systems like cap and trade have collapsed.

Andrew: One of the points that I bring up in the book is that sometimes the crazy side of the argument wins. And I think the crazy is winning on nuclear these days. I think the crazy is winning on GMOs. I think the crazy is winning on vaccines in way too many communities. So, you know, as much as I love evidence and trying to think through things, we better be very good communicators about our solutions because the crazy can win.

Sonal: Can we quickly talk on GMOs and the myths and misconceptions around GMOs? Why did you think it was important to talk about GMOs in your book?

Andrew: The reason I thought GMOs were important to include in this book is, they are great ways to help us tread more lightly on the planet. The crop yields will go up. You can grow them in different places. As climate change happens, you’re going to need plants that are hardier — can survive heatwaves and droughts and things like that. The GMO toolkit is our best toolkit for accomplishing those things right now. And yet, it’s stridently opposed by governments and all kinds of groups around the world. And even the EU itself, in addition to the National Academies of Science, and just about every country that you can think of — has reviewed the evidence, and they’ve all come down and said there is no evidence that GMO crops are less safe for the environment, or for humanity, than conventional techniques.

We can get past the point of saying, “Well, it remains to be seen.” No, we need to go do these things. And the reason I get exercised about this, is when I look at things like golden rice — which is this strain of rice that has beta carotene injected into it via GMO techniques, so that you provide vitamin —is it A? It’s a vitamin A deficiency, happens to babies who are weaned on rice gruel, and it leads to blindness. And that deficiency is responsible for about a million deaths a year around the world. Great, you’re anti-GMO? Honestly, that volume of deaths, that’s on you.

Marc: So you discuss in the book, a very famous — at the time, I guess, in the ’70s and ’80s — a very famous debate between two, at the time, very accomplished people — Julian Simon and Paul Ehrlich. And it’s largely been forgotten, but it’s a highly relevant — and maybe even more relevant today than it was at the time. And maybe you could describe their debate and the famous bet.

Debates over resource use

Andrew: Yeah, my favorite bet of all time was the bet between these two guys. Julian Simon pushed back against the dominant narrative, around the time of Earth Day, which was that growth will come to a bad end — that we cannot keep this headlong, uncontrolled market-based growth for a bunch of reasons. Primary of which was — there’ll become too many of us, the earth will not be able to feed everybody, and we’re going to crash into a massive famine. And the prime exponent of that view was Paul Ehrlich, who still is at Stanford, and wrote a book called “The Population Bomb,” where he essentially said, “Look, nothing we can do will prevent hundreds of millions of people from starving in the years ahead. But if we do things like forced population control, and we take control of the means of production, we might be able to stave off the worst things that could happen.” And one of the things I learned was that Simon agreed with that, and wrote things about population control. Then he switched his view, in this wonderful instance of intellectual honesty and humility. And he said, “Wait a minute, we keep on not seeing famines happen, resource crises. We just don’t see these things. Instead, the evidence shows that most things are getting better.”

And he got laughed out of a lot of rooms, and Ehrlich kept on putting out this gloom and doom, “stripping the planet” narrative. And, finally, Simon challenged him to a bet. And Simon said, “Pick any time period of at least a year, and pick any bundle of resources that you want. And at the end of the time period, if the resources are more expensive in real terms than they are now — which kind of means they’re more scarce than they are now, I’ll pay you the difference. If they’re cheaper, you pay me the difference.” I think this probably appeared like a sucker’s bet to Ehrlich. He picked five resources — tungsten, tin, chromium, copper, and I forget the fifth one. And he said, “All right, let’s put a 10-year period on the bet.”

By 1990, the real prices of all five of those things had fallen. The price of the total portfolio had declined by more than half. And Ehrlich mailed Simon a check to acknowledge that he’d lost the bet — didn’t talk about it very much, didn’t attach any kind of note to that check. So I love that episode so much. And I’m trying to do round two of that. I’m using the Long Bets website, which is part of the Long Now Foundation, started by Stewart Brand and others. And I’m offering bets. I’m saying, for example, that no matter what — I’m saying that resources are going to become more affordable. I’m agreeing with Simon on that, but I don’t stop there. I say, in 10 years from now, I bet we’re gonna use less total energy — not per capita, but total energy America-wide in 10 years, after a decade of continued economic growth.

That’s how confident I am in the one-two punch of capitalism and tech progress to take costs out of the system. And, you know, energy and resources cost money. That’s just my reasoning. If you think I’m wrong, step on up. With Long Bets, you both put the money up front. You designate a charity that will get it at the end, and we’ll see what happens.

Marc: So there’s two historical figures in the book who are heroes of mine.

Andrew: Julian Simon and…

Marc: And Bucky Fuller. He came up with this idea, and I think you say it was 1927?

Andrew: Yeah, the ’20s.

Marc: Maybe just explain his idea. Because that was a remarkable insight at a time when there was probably no actual logical foundation to expect what he was saying.

Andrew: So Fuller was this crazy polymath. And he popularized, for example, the geodesic dome — that’s kind of what he’s best known for today, I think — which is the structure that can bear a great deal of weight and very heavy loads while weighing very, very little. And Fuller thought that we would see more and more examples. And there were plenty of opportunities to do that kind of thing all around the economy. And I found, you know, this crazy book that he wrote in the ’20s. And he said, “Look, I did a bunch of calculations.” And he said, “I thought it might be possible to satisfy all of our wants and needs, essentially while using less stuff, while using fewer materials.” And he called the process ephemeralization, making things more ephemeral. That’s a real mouthful to say. So we use the phrase dematerialization more often now. But Fuller was the guy who said, “Gang, we can do this,” in the 1920s, which is crazy.

Sonal: That’s so crazy. That’s pre-software.

Marc: The economy in those days — it was what Joel Mokyr calls the “wheat and steel economy.” That was during the era where GDP first became an economic metric. And it was literally like tonnage.

Andrew: We were weighing things, right?

Marc: It was, like, how much you weigh your output, right? In tons.

Andrew: Yeah. And then we started counting dollars instead, and that was a huge innovation. So the fact that Fuller came up with that that early, is just this weird intellectual shooting star.

Marc: So if I recall correctly, and maybe I’ve made this up in my own head — but I think that one of the lines he used was, “Ephemeralization is the process of making more and more with less and less.” But then he added a line, he said, “Until eventually we are making everything with nothing.”

Andrew: I think he did go that far. He also said, in 1927 — he said, “It’s the number one economic surprise of world man.”

Marc: Right.

Andrew: And so here we are, you know, 90 years later, and it’s still surprising to people.

Sonal: So, one thing that just blew my mind, because I had not actually read that or known that. How could he come up with that in 1920? This is before software even existed. Like, what would give him — because I understand now, Marc, in 2009, when you wrote “Software is Eating the World” — like, I could see someone making that [claim] now. What gave him the chutzpah to say that in 1920? Like, that’s insane.

Andrew: I have no earthly idea. And I don’t think we would have got to this resource turning point — I don’t think we would have achieved absolute dematerialization — without the digital world, without the computer.

Sonal: Yeah.

Andrew: I think software is giving us back the world because it’s letting us slim, swap, optimize, and evaporate our resource use. And I don’t know how we would have got there in a world where we’re still using slide rulers and file cabinets. Maybe we would have. But in my multiverse, we don’t get there in the universes that don’t have the digital revolution.

Sonal: A lot of people when they talk about dematerialization, they talk about it very literally — like, you’re replacing an object, a hard object, with something — its software counterpart. But just make it clear, it’s actually even deeper than that, because when you do think about ride-sharing, and all these entire economies that are growing off the mobile phone — that is what enables the end of ownership. When you think about the fact that today kitchens can be delivering food to you, that is the thing that changes the shape of cities, etc. I think a lot of times when people talk about dematerialization, they take it very literally as, like — the one-on-one replacement of something physical with something digital, and it’s actually bigger than that. It’s like, a whole services economy and reshaping things.

Andrew: Yeah. I talk about these four different vectors for dematerialization. You know, trimming out how much aluminum is in an aluminum can — that’s slimming it down. Swapping out one resource for another, that’s when rare-earths gets expensive. We walk away from them. Optimizing — using the load factor for airlines — has increased from the mid-50s percent to 80% now. You’re just making better use of these resource-intensive assets that you have — and then evaporate, replace it by nothing at all. The smartphone has made me not print out maps or print out film anymore. We have these different vectors for dematerialization to happen. And the point that I make in the book is — they’re happening in obvious ways, in subtle ways, in big ways and small ways, in the foreground and in the background in every industry. Simply because stuff costs money, competition makes you want to save money. And the digital toolkit offers you these great opportunities to do that. I think the story is just that simple. And if that’s true, it’s not about to end.

Marc: So, if you take Fuller’s thought and your thought to their logical extremes, how close can you get — ultimately, someday — to making everything with nothing? Like, if we’re sitting here 50, 100, 200 years from now — like, what are the prospects for being able to take physical inputs out, you know, either 99.99% reduced or taken out entirely from many of the things we’ll be consuming?

Andrew: That depends on how many of us there are, primarily, I think — but I think we can go a lot further down the dematerialization curve than we are right now. It’s not crazy at all to imagine that, you know — let’s say in 2100, that we’re primarily an urban species. We live in these densely populated cities that are, you know, a lot closer to Singapore than Delhi, for example. We’re growing a lot of our food in very vertical, energy-intensive environments. When we need to build a new building, we’re just recycling the steel and the metal that we used for the previous generation of buildings. We’re already doing that a lot right now. And, you know, grow our textiles in weird vats with Petri dishes of bacteria or something. That’s no longer crazy to think about. Will we be getting our protein from living animals, you know, or from scaled-up Petri dishes in 2100?

Sonal: Lab-grown meat, yeah.

Andrew: And who knows about staple crops, if we’ll need cropland for that? But I’m for damn sure that we’re gonna need a much, much smaller acreage of cropland for all of humanity in 2100 than we do right now. So I don’t know, I don’t have a good way to guesstimate where those lower floors are. They’re a lot lower than they are right now. And I really think that — let’s take 2100 as the year — we’ll be this species that occupies a very small physical footprint on the planet without depriving ourselves. And then we go into nature, kind of because it’s cool and because we want to, as opposed to because we need to strip it to satisfy our growth.

Technology developments needed

Marc: I have a question about R&D — the role of research and development in, kind of, delivering on the dream that you’re talking about. You know, because obviously, everything you’re talking about is sort of dependent on future development of advanced technology and creation of new knowledge. The last, like, 20 years — I would say, there’s been basically, like, two dramatic events in energy-related R&D in the U.S. And one is this incredibly positive outcome with respect to fracking and liquid natural gas. There’s been all kinds of positives to come out of that. And even in the energy industry, a lot of experts were shocked [by] how well that stuff has worked. The curves are amazing, because it’s like —  energy production in the U.S. [is] falling, falling, falling, falling, falling — and then, all of a sudden, it just, like, takes off like a rocket ship. Right? When like, nobody was expecting it.

Andrew: To the surprise of everybody.

Marc: Yeah. So that was the good news surprise. The bad news surprise was, you know, Silicon Valley embarked on a very big push to do so-called cleantech/greentech, particularly between 2010, 2012. There’s a huge push. And there were a lot of extremely smart and accomplished people here in the valley who thought that this was the new frontier for American technology, for venture capital and — you know, with obvious, you know, both huge potential positive benefits for the world but also, you know, a huge opportunity to build new businesses.

And I think there were hundreds and hundreds, possibly even thousands of companies, and a very large amount of money and effort — and a lot of people put a lot of work into this, that the results were extremely disappointing on a number of fronts. I mean, there were, maybe, a few isolated cases of success. I mean, one might say we got Tesla and SpaceX out of that, right? In which case, you know, fair enough. But even beyond that, companies had a much harder time developing and/or commercializing those technologies, or just ended up in dire straits that people didn’t expect.

So I’m very curious of your assessment of, like, what went wrong in the Silicon Valley cleantech/greentech adventure. And what should we learn from that, you know, both, like, as an industry and as, like, a world? If we’re going to try that kind of thing again, like — if we’re going to try to double down on R&D here, like, what lessons should we learn from that in terms of how to do it better?

Andrew: I only know it from a great distance. Here’s a super naive way to look at it. If we think about solar, solar has become dominated by China, primarily, because it’s a flavor of manufacturing that they were already pretty good at. And it’s just a scale economies game. <Exactly.> And they’re quite good at scaling up huge factories and turning out, whether it’s a liquid crystal display or a photovoltaic panel. So I think that’s just very, very tough competition. The other thing that I do believe about solar and wind is that they have a place in the energy portfolio, absolutely. But Germany’s experience with trying to become much more reliant on renewables has not gone very well at all, for a couple of reasons — a deep one of which is — it’s dark sometimes, and it’s not windy a lot of times. We have this very serious problem of intermittency with those renewables.

So, they have to be backstopped with something. And if you turn off your nuclear stations — if you decommission them, like Germany is doing — you get backstop in their case with some very dirty coal-powered plants. So they’ve kind of got the worst of both worlds. Their electricity prices are really high, and their carbon emissions, per unit of energy, are really high. You look next door at France, which is very nuclear, and you see neither of those two problems happening. So I think at the individual competition level, going up against China in a scale game is really, really tough. And I think there are some policy mistakes that can make that situation worse. Does that play at all with your experience?

Marc: I think those are definitely big components. You may know the sort of appendix to that whole saga was, yeah — so there was a huge push for solar panels, including some very advanced — we actually have — here in the conference room, we actually have an old cylinder solar panel, one I keep around just because it’s such a great story. It’s the cylindrical solar panel, right? That would have a huge advantage that it could basically follow the sun. <Tracking.> You could track the sun. The only problem with it was it ended up being a 4x-worse value proposition — price-performance value proposition — than conventional solar panels, all in. That was one of the train wrecks out here that actually took down the whole U.S. government DOE program to fund cleantech.

But the kicker on the whole solar thing is — okay, as you said, it became a mass manufacturing game. And so it kind of became, like, memory chips in the ’80s. It lent itself to the Chinese ecosystem, which is able to do mass manufacturing at scale.

Andrew: Quickly and well.

Marc: Right, right. Exactly. Right. Exactly. And so the Chinese have been able to undercut a lot of their American competitors. The kicker to that is the pro-environment administration then reacted to that by putting tariffs on imports of Chinese solar panels — therefore making it cost-ineffective for Americans to deploy solar panels that otherwise would have been much cheaper.

Andrew: So tariffs are — with the possible exception of a border adjustment tariff, because we got to bring down carbon, right? Tariffs are just — Econ 101 bad idea.

Marc: Well, it went beyond, though, just this specific mechanism. It was more an expression of values on the part of the United States government, which is — in theory we care about the environment. In practice, like, we’re more worried about, like, other things. And so we’ll trade off the environment.

Andrew: Yeah. So, you know, the mantra is all — should be, “Let markets work to develop the goods and services and let free trade happen.” And that’s where prosperity will come from and innovation.

Sonal: For me, I was in the thick of that, because we were at the heart of this whole cleantech movement. When I was at PARC, we had a huge investment in photovoltaics. It was my first big white paper. My question is, why can’t it just be just a timing thing, like everything else. Like, it was just too early, the wrong time, the ecosystem wasn’t built out for balance, the system components and services and everything else, the subsidy models were wrong. Because I actually hope that we can get some R&D to the future with cleantech.

Andrew: We are getting cost declines with solar and wind. The price — the installed price, and then the price per unit of energy once they’re installed, is going down, you know, at a really attractive rate. So it’s not that we’re failing with these things. What I was trying to point out earlier is there are just some basic problems with that style of energy — especially because we’re not getting the battery revolution. And the battery nerds that I talk to say, “Look, there’s an energy density limit here.” So you’re pushing up against some physics. And it’s not that we can’t do anything about it, or that we should stop research. Of course, we should continue that going. But you got to backstop it with something.

Sonal: With some portfolio. Right?

Andrew: And that something, in my view, should also be clean. It should be nuclear. And then let’s let the battle rage for which is the cake and which is the icing. I kind of think nuclear is going to be the cake. And we’ll have a little solar and wind icing if we get it right. But maybe I’ll be wrong about that. Well, I just don’t want us to keep putting, you know, huge amounts of carbon in the air to generate electricity. We don’t need to do that.

Marc: So, this is where — I don’t think environmentalism, for the most, part is actually about the environment. I think it’s about something else. And the reason I say that is because, exactly to Andrew’s point, I think we actually have the answers. I think we have the answers, and I think they’re nuclear — which is just, like, in practice an incredibly safe technology, contrary to what everybody believes.

Andrew: Plus one to that.

Marc: And then, I think, look — like, it goes back to the tariff thing. Like, let the Chinese build solar panels. Let them ride the manufacturing cost curve down, and, like, buy their solar panels.

Andrew: Plus one to that.

Marc: Right. And we have two magic technologies. Like, we have the box that generates power by splitting the atom, and we have the sheet that converts sunlight for free. And both of those are, like, incredibly modern production techniques for nuclear and solar. Like, it would just be, like — spectacular what you could do. If you engineered new nuclear plants today from scratch — like, properly, with the technologies. Most of the functional nuclear plants in the West today are, like, on average they’re — are there any younger than, like, 30 or 40 years old? The average is gonna be over that?

Andrew: I think that’s right. I don’t know when the last new one we built was, but it’s been a while.

Marc: And so, if we took current technology and did that, there are some really amazing ideas of things that we can do. I don’t think this has anything to do with the environment.

Sonal: Well, what I find hopeful, though, about what you just said is that we have the answers. And that’s really important. And so a lot of these things come down to market and other dynamics — regulatory, politics, all of that. So it’s not a technological limit, which I find very helpful.

Andrew: It’s also not a policy mystery anymore. We have these essentially magic technologies, where we should be stepping on the accelerator with them super hard if we really wanted to clean up the planet and stop polluting it with greenhouse gases. If we wanted a policy toolkit to reduce carbon, we have it. It’s worked for other kinds of pollution in the past. Carbon is not mysterious. <Yep.> It’s just comparatively politically difficult. <Right.> Now, I think some parts of the world will be more clear-headed than others. And I hope somebody else will show us the way, and their evidence is going to become unignorable at some point.

Sonal: I also just want to make one pitch for the iPhone moment in cleantech, which I know people think can be very much of a long shot. But I think a lot of technology waves do have their major iPhone moment, where there is the technological tip that then drives everyone else to make cheaper versions of that thing later on, once there is this desire and demand and pull and draw to have the thing. And I actually have to say one thing that I did find promising about Tesla, and their move into solar for the home — and battery as sort of this backdoor, this Trojan horse — that the car is a Trojan horse to actually powering your home — that is a very powerful idea. And over time, who knows where that can go, but…

Marc: I will say one thing that’s just from a consumer psychology standpoint. You know, Elon making electric cars sexy?

Sonal: Yeah, that changed the game.

Marc: That’s a big deal.

Sonal: It was way better than Leonardo DiCaprio driving a Prius, which is what I drive.

Andrew: That is absolutely a big deal, and, Sonal, to what you said, I’m thrilled that there are people willing to make some pretty risky bets on things. On the technology and the innovation front, I agree with Marc — we have some magic bullets. I’m going to mix my metaphors. We need lots of other shots on goal, right? And the innovation and the entrepreneurship ecosystem are a way for us to get more shots on goal. Hallelujah.

Sonal: Yeah, and I’ll just say one last thing on that. One of the things that I find really fascinating is that there is this phase with a lot of technologies where there is that very down moment, where things go down, it seems like it’s dead. And in fact, the thing is being built out under the very surface, and you don’t realize that’s happening. And so to me, the death of the cleantech boom is actually promising because — Marc, you alluded to this —but it did fund — Elon Musk rode those subsidies to fund Tesla in the early days. And so who knows what can happen next?

Andrew: I still think there’s a big place for government R&D. Again, more shots on goal, more attention to this — crazy ideas. And the reason Paul Romer won the Nobel Prize last year was he said, “Economies grow on ideas. Human Capital is the gating factor for increasing our growth and prosperity.” Let’s get more human capital out there.

Sonal: Well, Andy, thank you so much for joining the “a16z Podcast.” Your new book, out October 8 — “More from Less: The Surprising Story of How We Learned to Prosper Using Fewer Resources and What Happens Next.” Thank you for joining “a16z Podcast.”

Andrew: Sonal and Marc, thank you for having me. This has been a blast.

Marc: Thanks, Andy.

• 

  Andrew McAfee
• 

  Marc Andreessen is a cofounder and general partner at a16z. Marc co-created the highly influential Mosaic internet browser and cofounded Netscape.

  Follow Twitter
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

Since the startup (and founder) journey doesn’t go neatly linear from technical to product to sales, tightening one knob (whether engineering or marketing or pricing & packaging) creates slack in one of the other knobs, which demands turning to yet another knob. So how do you know what knob to focus on and when? How do you build the right team for the right play and at the right time?

It all depends on “What time is it”: where are you on the journey, and where do you want to go… In this episode of the a16z Podcast, general partner David Ulevitch (in conversation with Sonal Chokshi) shares hard-earned lessons on these top-of-mind questions for founders; as well as advice on other tricky topics, such as pricing and packaging, balancing between product visionary vs. product manager, how to manage your own time (and psychology!) as your company grows, and more. Much of this is based on his own up-and-down, inside-outside, big-small-big-small, long journey as CEO (and CTO) for the company he co-founded, OpenDNS.

The company was later acquired by Cisco after it pivoted from consumer to enterprise. Speaking of, what are the latest shifts and nuances in selling and buying enterprise products, beyond the phrase “consumerization of enterprise”? Or beyond the cliché of “design thinking” — how does one go beyond user experience and beyond things like fun gifs (which are pronounced, ahem, “jifs”) to focusing on the whole customer experience, and earning the right to be complicated? All this and more in this episode… plus the magic 5 words that will help any CEO (and anyone, really).

Show Notes

• Advice for founders, including the skills needed at various stages of growth [1:51]
• Improving customer interactions [6:13] and the importance of creating value early [8:12]
• How Slack moved into companies from the bottom up, and making software useful [11:24]
• Discussion of packaging [14:15] and how to make packaging and pricing decisions [18:11]
• Adding leadership as a company grows [19:57], including a product manager [25:30]
• The origin story of OpenDNS [30:44] and how it pivoted to a new business model [33:57]
• David’s shifts in roles within OpenDNS, and a discussion of the role of a CEO [37:22]
• Learning to listen [41:42] and David’s background in anthropology [43:57]

Transcript

Sonal: Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal, and I’m here today with David Ulevitch, one of our new general partners who covers all things enterprise. But honestly: that can mean so many different things to different people, so we briefly discuss what “enterprise” products really mean today, for entrepreneurs, companies, and users — especially given the latest shifts driving SaaS beyond the cliché of “consumerization of the enterprise”.

We also cover specific advice on the topics of:

• pricing and packaging;
• how to balance being a product visionary with being a product manager;
• when and how to scale out and hire your leadership team, and how do you know that’s working or not; plus
• how to best manage your own time — and your own psychology! — as a leader while doing all this.

For context: David founded OpenDNS, where he went through a rough period in going from CEO to CTO in 2009 and then back to CEO again, in a company that itself pivoted, from consumer to enterprise. We discuss how did he make a comeback; it’s not like he changed instantly overnight, so what and how were the lessons learned? In 2015, Cisco later acquired OpenDNS, and David then ran their security business, where he also led the acquisition of three companies before coming here. So he’s seen startups from all sides: from being acquired to being the acquiror; from small to big to small to big again; from on the inside and from on the outside.

But the real theme of this episode is the journey many founders make — from technical to product to sales CEO — and while we end with the story of OpenDNS and the most important lessons learned there, we begin with what is the one piece of advice David has for founders?

Advice for founders

David: So I think that there’s not one — you know, I’m not a big fan of platitudes, where you just say one thing and that applies to everyone — because there’s never one thing that makes the difference between success and failure.

As a founder, you’re generally at different stages of the company-building journey. Sometimes you’re a technical CEO, trying to build a product, to make sure it actually is feasible; then you are constantly in the market with customers, doing customer discovery, making sure that you are solving the problems you’re trying to solve. You end up becoming sort of a product CEO, making sure that you have product-market fit; then you end up becoming a sales CEO, on the enterprise side, where you’re trying to generate revenue and figure out how you can go acquire more customers. And then if that works out, you become this really general sort of manager or go-to-market CEO, and you’re thinking about how do you scale and accelerate a business.

And so, I like to think about those as different journeys, where there’s the right decisions for the right time — and really trying to help founders understand, what time is it, and where are they in that journey, and where do they want to go?

Sonal: Is it possible to be all four at once, or is it really tied to the stage of the company?

David: Generally, you want to be more focused than less focused. The reason you’re probably also not all those things at the same time, is a company is a set of these interconnected little knobs, and you can never just optimize one and then forget it and not come back to it. You end up going to another knob. If you fixed pricing  and packaging, then–

Sonal: It’s like a control panel.

David: Yeah, it’s like a control panel. Then you’re going to move to demand-gen (demand generation) to try to increase the top of the funnel. And if you fix that, then you want to make sure that your SDRs (sales development representatives) and salespeople are converting marketing leads into qualified leads. Then you’re closing, then you’re doing customer success. And once you tighten one of those knobs, it just creates slack in one of the other knobs. You might switch those hats from time to time, but I think you’re rarely going to be wearing more than one of those hats at the same time.

Sonal: So I have to ask, since we’re going with this theme of let’s not do the platitudes — a lot of people say “it’s about the customer,” and “the customer journey,” and “understanding the customer.” Honestly, when I hear that I’m like, they [customers] can pull you in a million different directions, you don’t even know what to do. Especially if you’re a technical founder, you don’t know who to sell to; there’s a bit of a chicken-egg. So how do you figure out how to sell to the right customer?

David: It’s okay for a company in the early stages not to know exactly who they want to go after, but they do have to understand the consequences of the customers that they’re targeting.

I think we’re living, today, in one of the best times to be an enterprise software startup. And, to me, one of the reasons is because so many companies today are SaaS and subscription-software companies where there’s a recurring revenue component — it’s better for the customer. Because they know that the customer experience is gonna be good… or they’ll stop paying for the subscription, right?

Sonal: It’s a repeat business; it’s not a one-time sale.

David: It’s a repeat business, right. I never think about the first sale when I look at a business, I always think about, what are you going to do in year two, year three, to make sure you renew the account, to grow the account? It’s actually a way of peeling back the onion to figure out how confident are you on where your product is today?

Because if you say, “Oh, we’re only doing three-year contracts”, well, is that because it’s really hard to implement and tough to get customers onboarded and so you need runway to get them happy? Or is it because you think you’re overselling your capabilities set, and you just don’t want a customer to figure it out within a year and not renew.

But if the startup I’m talking to says, “Oh, well, we did a couple three-year contracts, but we realized that we were priced really low, and so now, if a customer wants a three-year contract, we’re actually going to charge them more on the out-years.” Well, that tells me your product is really good, and getting better.

Sonal: That’s fascinating.

David: Another way to think about that is that what used to just be a product experience is now much more of a customer lifecycle experience: It starts before you sell, building evangelists; then there’s the onboarding part; then there’s making sure the customer is really happy. How do you market to your existing customers to make sure they’re getting full utilization of your product?

That customer lifecycle makes it much easier for a startup that’s getting started to really identify who is the target customer. And then thinking about does that actually map to the business I want to build: Is it the big Fortune 500, the Global 5000, is it SMBs (small and mid-sized businesses)? Because it has all these downstream effects.

It’s okay for a company in the early stages not to know exactly who they want to go after, but they do have to understand the consequences of the customers that they’re targeting

A lot of startups will come in here — and in my first six months here, I’ve now met with over 200 companies — and a lot of them have this ambition to go after SMBs. And one of the cool things about SaaS is that SaaS can take something that in the olden days of enterprise computing, you’d have to buy the biggest server, the biggest box, to get the best solution. But with cloud and with SaaS applications, you can now have the power to get the massively great CRM system, the massively great HR system…

Sonal: I like to think about it as SaaS is very democratizing, actually.

David: SaaS is totally democratizing.

Sonal: Because it enables smaller- and medium-sized companies to have access to big company resources. They don’t have the in-house engineers, but they can essentially “as-a-service it in” to their company.

David: Absolutely. Small companies have unlimited compute. They have unlimited storage. They have unlimited bandwidth now.

Creating value for customers

So when I meet with startups, they often want to ambitiously and altruistically go satisfy this pain for SMBs. But it turns out that the reality is, if you want to charge a higher price point, if you want to pay an expensive sales force, then you’re going to realize that your average deal sizes have to be higher. If I really want to go after a target market where the price point is going to be lower, then I have to think about bottoms-up sales, about self-serve offerings.

I love when I see startups that think not just about who they want to go after, but then they build that into their whole customer experience model — marketing programs, pricing and packaging, renewals, sales — and the whole business model.

I mean, nowadays, think about how many emails you get where it reminds you of a new feature that you may not even have known existed. For instance, I’m using an email product called Superhuman, and every week, I basically get an email from the team saying, “Did you know that you could use this functionality? If you press [command] ⌘ + I, it’ll automatically route someone to BCC in the reply. Or you just press ⌘ + C, it will copy the whole email; you don’t have to select it first.”

Sonal: Actually, I saw this awesome tweet from Patrick Collison, the CEO of Stripe, where he said, I feel like Rahul Vohra, the CEO of Superhuman, is essentially inventing new user-experience interaction paradigms that will eventually cascade into other products. Much like Steve Jobs did with letting us learn new behaviors like how to touch a phone.

David: Patrick’s tweet was right on. In Rahul’s weekly marketing email to existing users, it helps teach me these new things that they’ve unlocked. They become very intuitive, but you still have to learn about them.

One of the best things about building a company today is it’s easier than ever to get close to customers, to constantly get iterative, real-time feedback: both from an analytical standpoint and from customer surveys, NPS scores, all these kinds of things. You have all this telemetry through SaaS products where you actually see how people use your product.

We’re living, today, in one of the best times to be an enterprise software startup… You have all this telemetry through SaaS, where you actually see how people use your product

But the second thing that’s happened is, people talk about this bottoms-up SaaS motion, but it’s not always just that. It’s really about making sure they understand that there are evangelists in the company that you have to win over, before you’re going to get sign off from the CFO, the CIO, the CSO, whoever he or she is that makes a decision — you’re going to have to get some champions underneath that person to be your evangelist internally.

Sonal: So this is a little counterintuitive too, though, because the other piece of advice I’ve often heard from folks is that the #1 mistake a lot of consumerization-of-enterprise type of founders make is that they go TOO heavy on bottoms up, to the point of ignoring the importance of top-down sales. What’s your view on that?

David: I like to frame it a little bit differently. When people talk about the consumerization of enterprise, enterprise customers today are being bombarded by so many different vendors, their attention span is so limited. I think when people say consumer, they mean “easy”. They don’t really mean “consumer”.

The value proposition that I’m hearing when I talk to customers is that the time-to-value needs to be short. It’s actually two parts: The first is, I want value almost immediately, sometimes even before I pay for it. I want a trial, or I want to get up and running on my own, and then I’ll talk to a sales person. The time-to-value can either be t-minus zero days (negative time), or it has to be very short from hours to days.

I always encourage founders to think about the first hour, the first day experience, the first week experience, the first month experience.

The second part is that enterprise software can get quite complex. Zoom is in the news because they went public, and Zoom’s a great example of something where they earned the right to be more complicated.

Sonal: Wait, let’s pause on that for a minute: Earn the right to be more complicated.

David: So this goes hand-in-hand with a short time-to-value. The short time-to-value gets you in the door. We know that you and I could download Zoom on our phones and be in a video conference call, but now we’re like, “Well, wait a minute, maybe all of our conference rooms should have Zoom. Maybe we should integrate Zoom with our Google Calendar.” You’ve earned the right to do that complexity because you’ve already proven so much value. And not only that, the value you get by doing the integration with G-Suite, or by adding some cameras to your conference room so that you can have Zoom-rooms — that complexity is commensurate with the value you’re getting.

When people say [that] “consumerization of enterprise” means it has to be easy or simple, that’s not quite what it is. To me, it’s two things: It’s a short time-to-value; and then the complexity curve is commensurate with the value proposition.

Sonal: So then I wanna ask you more about what needs to go into that “time-to-value”. So, let’s be a little bit more specific.

I mean, I get the point that it’s incredibly competitive, so you’ve got to differentiate fast and show the value. But what are the things that drive that? Is it a great… like, a cute little JIF-y, <chuckles> that, you know, that jumps out at you and makes cute — like, a Clippy-type of thing? I mean, what is it?

David: Did you say a JIF-y, like a GIF?

Sonal: Ugh… are we gonna rumble?

David: Wait, I know how we can do this. <laughs>

Sonal: Are we gonna just stop and, like, end this?

David: I can’t have you trolling me.

Sonal: We can’t be… I am not trolling you! I’m one of the people who calls JIFs, not GIFs (I hate that). You know there’s a world of people that think they should be J!

David: Hold on… I don’t know if I would have agreed to this podcast if I knew you called them JIFs.

Sonal: It’s like nails scratching on a chalkboard. You’re literally right now in my ears. It’s like someone’s poking pins in it right now.

David: How have you been so successful in your career this whole time calling them JIFs??

Sonal: I actually feel like I kind of hate you right now, to be honest.

David: Aw, this is amazing… <laughs> Oh, my goodness. <laughs>

Sonal: You don’t have any friends that pronounce “GIF”, “JIF”?!!

David: I don’t think so. Like, you’re it.

Sonal: Chris Dixon, is he a friend of yours?

David: He is.

Sonal: So I’m outing him on the podcast, because I’m not gonna go down in this ship alone. He’s another one.

David: You might have to edit that part out, because I don’t know if that can be out there… There’s, like, this dissonance in my brain, because you and him are so smart, but you also call it a JIF. <laughs>

I don’t know what to do right now… <Sonal giggles> All right.

Making products useful

Sonal: Okay, so, Slack, for instance, they did a lot of really creative things. I remember I was at Wired, and the product that we used was HipChat. And the thing that eventually got me into Slack was the fact that you could do all these GIFs (whatever;) — <David: yup> you could do more fun things.

David: And they have the integrations, other things could drive information into your Slack channel. And that was not something that happened to HipChat for a long time.

Sonal: That’s right, like Google Documents and…

David: That’s right, Dropbox files. Even automated updates, like, if you’re a developer, when somebody would do a push to production, it could notify people inside the Slack channel.

Sonal: Right. But now that’s not a case where IT has to decide the integrations.

David: That’s right. And they made it easy for individual users. As long as you could use Google Auth to authenticate, anybody could basically set up a Slack channel inside their organization.

After a while, IT says, “Hey, wait a minute. We have all these teams that are chatting on this thing, they’re doing integrations, files are being shared. We need to have a little bit more visibility, a little bit more access control.” And for security and compliance reasons, it became an enterprise sale that went wall-to-wall. It’s now already entrenched in the organization. There’s already integrations happening with some of the developer tools and workflows, and at that point, they’ve earned the right to be more complicated.

Sonal: I’ve noticed this resurgence — and I don’t know if it’s just like a zeitgeist thing or anecdotal evidence — of design-focused startups, precisely because of the thing you’re saying, because that’s one of the ways to instantly differentiate.

David: “Design thinking” is sort of another way of saying…

Sonal: Oh, I hate that phrase.

David: I know.

Sonal: Talk about the platitude-of-all-platitudes. That phrase drives me fucking up a wall.

David: So here’s a better way to frame it. <Sonal: Okay> Cuz I also don’t like that phrase. <Sonal: Yes> Is it’s really about that extension of the product experience, and really taking that more holistic approach. It’s not just about the UI, it’s not even just about the user experience of a particular workflow, it’s about that whole customer experience.

We’re actually entering a period of time where more and more people in the workforce are digital natives, and they want to be power users. You know, why isn’t there an equivalent Microsoft Excel on the web? Like, Google Sheets is not Excel. The current state of collaborative tools in SaaS apps is just so weak, and they don’t let you be a power user.

What used to just be a product experience is now much more of a customer lifecycle experience

Sonal: It’s also, I think, ignoring the realities of organizations today.

David: Totally.

Sonal: Which used to be so siloed. And now you have people collaborating cross-functionally in different ways…

David: You could argue though, Google Docs did create a multiplayer mode where you could have collaborative editing, but it was just such a garbage experience from a functionality standpoint, that…

Sonal: It was an afterthought. It wasn’t baked in natively. That’s, basically, my rule of thumb for all of this: If it’s an add-on, it’s not important.

David: I would say Google G-Suite is an add-on too. Google should just shut down G-Suite altogether <chuckles>, even though the whole of Silicon Valley would go crazy. I mean, they’re a rounding error in their business, they’re a rounding error to productivity, versus what Microsoft has. I don’t think they’ll do that, but strategically, it’s just so unimportant for them.

Sonal: Right.

David: What I would say, though, is that I like software that is easy to use, that has that short time-to-value, but that also allows me to be a power user if I want to be.

And in fact, as an investor, when I talk to companies, I always try to figure out what is their pricing and packaging strategy.

Packaging and pricing

Sonal: So tell me, what is packaging, actually?

David: So, packaging is usually (I mean, it can be a bunch of things) — but to me, packaging is: What set of features are you going to put into an offering to a customer?

I always try to think that you want to make it easy for your customer to give you money — like that is a foundational principle for me — packages are a way to do that. We’ve all been to the restaurant where the a la carte menu is all over the place, but sometimes restaurants just say, “Well, here’s the three options: Comes with one of these appetizers, you get this main course, and you get this dessert.” If you want to make things easier for people to give you money, generally, people come up with packages. And the friction is removed to becoming a buyer.

In the SaaS world, sometimes there might be a tier that says, you’re going to get the full functionality of the product, but you’re not going to get archiving and logging and all this detailed reporting and analytics. It allows the company that maybe doesn’t want to spend as much or isn’t as big to get the full functionality of your product, but then there’s a hurdle. When I think about packaging, usually there’s a key product milestone that happens that forces somebody to jump to the next tier.

Sonal: Interesting. What do you mean? Give me an example of that.

David: Well, sign-on is a good one. Lots of SaaS offerings let you create accounts and use a product, but if you want to tie it to your Okta directory or some other directory service, you’re going to have to jump to a much more expensive tier. But, generally, the customers that have to jump to that tier are more enterprise companies. They have a directory service. They have a single sign-on service. They might want two-factor authentication with tokens. The security person in me doesn’t love that one being a tier because I always think you want all your customers to be secure. But there are other tiers, like compliance. If you’re in a regulated industry, you might not just be satisfied with 30 days of logging. You might need 365 days of logging. You might need to be able to export your logs to another data store.

When people say consumerization of enterprise just means it has to be easy or simple, that’s not quite what it is. It’s two things: a short time-to-value, and the complexity curve is commensurate with the value proposition

Sonal: So far, if I heard that as an entrepreneur, though, I would assume that all packages are tiered. Are there un-tiered packages where it’s just a different combo that’s kind of horizontal?

David: You know, I don’t think I’ve seen that. Generally, it’s much more of a ladder — <Sonal: That’s what I was wondering> Where the next package includes everything in the previous package. And I think that, while there’s usually a number of features that get unlocked when you go to the next package, to me, there’s always one that has that forcing function.

In fact, when I think of packaging, it’s often a way to segment your customer base. Because you’re going to say, we know SMB mid-market (under 1,000-employee companies), they’re going to be in this package. Everything we do, the product manager on that package is thinking about those features, thinking about our persona. And then the next package, the person is saying, wait a minute, I want to go after the 1,000 to 10,000-employee company, and this is what they need: This is how I communicate with them; this might be how I do webinars to them; this is how I’m going to do pricing that more fits their model. Maybe you can’t do a three-year contract if you’re on the low-end product.

All these things are puts and takes that reflect where is the product, who is the customer you’re targeting, and then how do you want to market and create demand with that audience?

With packaging, usually there’s a key product milestone that happens that forces somebody to jump to the next tier

Sonal: Is there a balance or a rule of thumb — I’m sure it must vary by business — in what the ideal number of packages are; or, how many customer segments you should be trying to reach as a startup?

David: Well, I think fewer is better because focus is key.

Sonal: “Less is more”?

David: Yeah, less is more. Time is always the most valuable currency in an individual’s life, in a company’s life. Aligning all that time behind the most important — putting more wood behind fewer arrows — is much more important.

Generally: two packages, three packages. When you make it too complicated for the customer to figure it out, that creates friction to the sales cycle. <Right.> Now, with that said, one thing that startups often do is they share their pricing publicly on the site. And the engineer in all of us, the pragmatic person in all of us, we’re like, well, of course, we want to share pricing because as customers we hate not knowing the price. But as products get much more nuanced, and organizations are buyers, you actually don’t know what your pricing discovery looks like — you’re better off not sharing your pricing.

Sonal: Okay.

David: One way you know you have a great product is when your salespeople are the ones demanding you remove the pricing. Because that means that they’re telling you–

Sonal: You can get more money.

David: You can get more money! Maybe you’re a technical CEO who’s becoming a product CEO who’s becoming a sales CEO; if you’re listening, you’re going to be like, “Wait a minute. They’re telling me we’re leaving money on the table.” That’s generally a very strong signal.

Sonal: I have a stage question on this, though, because if you think about the definition of “startup”, a startup, by definition, is a business under a high condition of uncertainty compared to a more established business — I wouldn’t even peg it to a particular size.

Given that, a startup is an experiment; and the product, you can run multiple experiments at the same time… We’ve heard of the famous pivot — you know, the dreaded “P”-word — there’s all these different flavors of this. How do you run multiple experiments, and also strike a balance with focusing the pricing-and-packaging strategy?

David: Well, that is the art of running a business…

Sonal: Not a science!

David: Yeah, and everything is multivariate. But you can — if you have smart people paying attention to the numbers, paying attention to the data, collecting the analytics, and giving yourself enough time to collect that data — the worst thing for a company to do is to make a decision and then not allow there to be enough time to collect the outcome of that decision, and understand the consequences of that decision, and then they make another decision.

The question of how do you make decisions and run multiple experiments, I don’t think it’s that complicated — as long as you’re paying attention to: What are the outputs from those decisions that you should be looking for? And you should be looking at what’s changing across the business. We’re living in an era today of running companies where it’s much easier to collect and analyze data than it ever has been. You have data lakes where you can bring in product data, your CRM can tie into that product data. We’ve never had… we have BI tools now…

Sonal: Business intelligence tools.

David: That’s right. We have open-source business intelligence tools. We can actually run complex analytics and say, “Wait a minute, my West Coast territory is just doing so much better than my East Coast territory; what is the difference that’s pushing there? Is it because we actually are running more demand-gen campaigns on the West Coast, and the marketing team on the West Coast is separated, or is it just that the West Coast sales reps are better?”

Sonal: You need to be able to tease apart those.

David: You need to be able to tease those things apart. But it’s easier to get access to the data and analyze it quickly and avoid that analysis paralysis than it ever has been in the past.

Narrowing focus and growing the team

Sonal: So you know, a big part of this — so the big theme I’m hearing from you, is a lot of these things have intentionality, even if you don’t know the outcome. And that you can actually control that intentionality by being introspective, understanding your decision-making, understanding what works; that sounds great.

Now, as the leader of the company, how do you, the CEO, figure out what to work on? And depending on what stage you’re at, this whole journey — from technical, to product, to sales, to go-to-market — that’s not necessarily perfectly linear, how do you figure this out?

David: It’s not linear at all. I mean, sometimes in retrospect, we like to look and think that it was linear. <Sonal: Of course, right> I think that there’s different ways to figure out how do you prioritize your time, where do you spend your mental calories?

Sonal: “Mental calories” — I love that phrase.

David: Yeah, I mean that’s how I think about my day: “Where do I want to…” You only have so many mental calories.

Sonal: That’s how I think about my day, too! I think of “nutrition density”. <David: That’s good> I have a phrase that I use for all my editing, which is “ROE (return on energy)”.

David: Ooh, that’s good.

Sonal: So I refuse to spend time on something that the output is gonna be vastly low-proportion outsized win to what the amount of work I put in <David: right> in terms of energy, creative. I have a whole framework for thinking about this stuff because I’m ridiculously productive on this front.

David: Well, hopefully this podcast gets published <chuckles>, because then I’ll know that it had high ROE. <Sonal laughs> I think figuring out how you spend your mental calories is a really important question to ask, and sometimes, the act of asking that question itself is just part of the process of figuring out how to spend your time… and spend it wisely.

And there’s different things that happen along different stages. I always look at what is the problem in the company — is it that we can’t get customers? And then figuring out who that right customer is.

But as a company starts to mature — like, a lot of these companies get to this $2 million, $3 million in annual recurring revenue — that’s a huge milestone; very few companies ever get there. But, yet, it’s tiny when you should be doing $20 million, $30 million.

If you aspire to be there, you can celebrate the milestone, but it’s clear that you have a long way to go to build an enduring, iconic company. At that point, though, you start to have a leadership team.

Sonal: One of the biggest things that we see when we give technical founders advice, is they need to bring on a VP of Engineering, they need to bring on a Head of Sales. They keep resisting this thing because they’re attached to their early startup team. How do they figure out when to really– there’s a lot of religious advice and debates around this.

David: I always go to the question of: What time is it? What is the priority? Are you trying to figure out product-market fit, or are you focusing on going to market? What time is it? Are you hiring salespeople and ramping up? Are you figuring out the customers are churning and you’ve got to go fix your product?

Focus is so important. If you were to ask all your leaders and all the people in your organization — “What is the most important thing for our company right now?” — they should have an answer.

Just one of the more tactical conversations that I have with leaders — especially when they’re a startup and they have this core founding team and then they’re thinking about scaling — and they say, “Well, you know, I have this engineering manager. He or she was with me from the beginning, and I think they’re doing a great job managing.” One of the things I highlight is that bringing in a world-class VP of engineering that could rock the boat, it could cause issues, but it’s not an indictment of your current engineering manager.

You can celebrate the milestone, but you have a long way to go to build an enduring, iconic company. At that point you start to have a leadership team…

Like that’s not what’s happening. Part of bringing on these high-performing leaders and these really well-respected leaders — that have a cult-like following with the people that have worked with them and for them before — is that they are going to help you accelerate your ability to recruit world-class talent. When you deliver that message to that person on your team who’s been there from the beginning and is doing a great job, that should resonate. It’s like, “Oh, wait a minute… We can get way better people, way faster? Yeah, let’s bring that person on.”

Again, you have to be very careful about knowing what are the problems you’re trying to solve in the organization, but oftentimes — and I think VCs have a bad rep for this — they shove in somebody who’s way too senior <Sonal: Yes, exactly!> who comes from way too big of a company. You have to think about what is the right team I need for the right time.

Sonal: I think Ben wrote about this in his book, actually, which is the mistake that people hire for the future instead of hiring for the thing you need now.

David: This often comes up with VP-of-Sales hires, where somebody maybe has run a 10- or 20-person team, but you’re like, can they run a 500-person sales team? Well, you don’t have a 500-person sales-team problem! People often think about the executives they’re hiring and is this person going to do anything for four years, or five years, or six years; I think that’s not always the right question to ask.

In fact, I had a board member once — Dave Strohm, who was a mentor to me, I think of him as the Yoda in my life — and he once said an expression that I’d never heard before, “horses for courses”. Have you ever heard of “horses for courses?”

Sonal: No, I don’t even know what that is.

David: It’s sort of an archaic expression. In horse-track racing, there’s like dirt courses, there’s grass courses.

Sonal: Ohhh I get it!

David: And you want to run the right horse for the right course.

…There’s a bad part of this phrase too, though. <Sonal: Uh-oh> There’s a bad connotation, which is that, sometimes when horses, run their few races and then they’re finished, you don’t have–

Sonal: They’ve “run their course”! That’s where it comes from, that expression!

David: That’s right, they’ve run their course. And do you know what happens to horses that have run their course?

Sonal: No, I don’t want to know. Are they turned into gelatin?

David: Something like that.

I always used to joke (it wasn’t very nice probably) — but I would joke with the VP of Sales I had at OpenDNS — that every quarter was his last quarter because he just constantly outperformed, and we always wondered when we hired him, is this guy going to scale? Now, he scaled wonderfully; he’s an incredible sales leader. He went from a 20-person sales team to, ultimately, a 200- or a 400-person sales team. Then once we got to Cisco, he did wonderfully. But we didn’t know when we hired him how far he’d get past 20 people.

You gotta hire horses for courses.

Sonal: The right team for the right play.

David: And this is a good way of really figuring out, is my leadership team adding capacity for me? Are they helping me understand what’s happening in the business? Because, at a certain point as a CEO, you’re going to start to spend less time on engineering, less time on product. Ideally, you’re going to spend more time in the field with customers, with partners, with customer success. As you start to spend less time with any individual function, you’re going to need to have leaders in place that really are spending all their time really understanding closer to the metal with what is happening.

Importance of product managers

Sonal: I love that you said “close to the metal”, because that’s the exact phrase I use when I think of this: It’s like “bare-metal leadership”.

David: Totally.

Sonal: Because that’s actually the biggest challenge: As a product-oriented person — or a visionary for whatever the product is, in any field — how do you kind of keep that “close-to-the-metal insight”… yet, you can’t actually be close to the metal if you’re scaling.

David: So this comes up a lot in startups, this idea that if you’re the product visionary, you’re the founder of the company, that means you are the product manager for the company. But at the same time, you need to scale an organization. And I think it’s important to differentiate the product manager from the product visionary. <Sonal: Oh, great>

As the founder and CEO, you can always be the product visionary, but there is going to be a time where you’re not going to be able to spend hours of time with the engineers hearing how they’re working on a product, or how it’s technically going to work. You’re not going to spend hours and hours of time looking at all the NPS survey data or the customer support tickets that are coming in.

And so, oftentimes, I’ll meet these startup CEOs who are like, “Oh, I can’t hire a product manager. I am the product manager.”

Sonal: That’s a common thing for technical founders.

David: Totally common! It feels like it’s your baby, you don’t want to let it go. But you’re only going to have five seconds a day to think about different decisions you make. And if your engineering team and the rest of the organization is constantly coming to you, you’re going to end up getting paralyzed. The worst thing for a product visionary is to make some decisions that they know were the wrong decisions — because they lack data, or they lack the time to be thoughtful about it — and then they start to undermine their own thinking about whether or not they even are a product visionary.

When the reality is, just hire a product manager! You’re not offloading the product vision to that person, what you’re offloading is the day-to-day ground war of figuring out: What is customer support telling me? What is sales telling me? What is engineering telling me? What are customers telling me? Synthesizing, analyzing, prioritizing, sorting that data. Obviously, as the founder and visionary, you have the ultimate say, but you’re going to be armed with so much more insight/information, that your intuition — which plays a big role, too — is just going to be further enhanced.

As a visionary, you’re going to have some special secret, some earned power, that you have over the lifetime of your experience, where you’re the domain expert in a problem set and know more about it.

Sonal: Right, because you’ve gone through the idea maze. <David: That’s right> You’ve literally lived and breathed this thing; you’ve built the company, started it. You literally have it seeping out of your pores.

David: That’s right. But how you build a product is not the same thing as having a vision for a product.

Sonal: We had a recent podcast with Safi Bahcall, and he described how Steve Jobs had both the artists and the soldiers, and so not only did he have himself, but he had Tim Cook and Johnny Ive. And when you think about the story of the iPhone, the app store was actually a result of his team coming up with the point that, hey, you can’t just have only Apple apps on this, <David: That’s exactly right> if you want people to use this.

David: Your product managers will come to you when they have conviction on something, and they have the data, and they have the view. You will then be able to make those bets. And nobody would say that he wasn’t a product visionary just because he didn’t come up with the app store.

Sonal: On that note, though, just to probe on one bit — because I’ve always wondered about this — there is a tension between this idea (I hate this idea) of “the head,” and “the hand.” You can’t have one person be the “head” and the other person be the “hand.” How do you reconcile that bit? I guess what I’m asking is, how do you calibrate along this line of visionary to manager?

You can differentiate the product ‘manager’ from the product ‘visionary’… how you build a product is not the same thing as having a vision for a product

David: You want to know what you’re hiring for. Because there are product managers that are much more analytical, and there are product managers much more visionary. You might need different kinds of people at different kinds of times. I think you have to be self-aware and be really intellectually honest. Because, if you actually need someone who is more visionary, then you’re going to have to deal with the fact that you’re going to be going to battle and sitting in a room and duking it out over ideas.

It leads to a secondary insight: Which is that, if you’re a CEO of a company and you do not trust that the information you’re getting from one of your leaders is what’s actually happening on the ground, that’s a tremendous problem.

Sonal: That’s a huge red flag.

David: Massive problem…

Sonal: Fire and move on. Or it could be you, if you’re just not a trusting person.

David: I think you have to work to resolve these things. You don’t just cut and move on immediately, but you have to work to understand, do they understand what’s happening, and are they able to communicate it to you and the rest of your leadership team?

I always like to think of leadership teams, it’s not just, oh, the head of sales reports to the CEO; the head of marketing reports to the CEO — you have these like siloed, pair-wise conversations. The leadership team needs to be working together as a team and communicating with each other because, as a CEO, you don’t want to be interjecting and intervening in every conversation and every decision. You want to start to figure out, are they collaborating? Are they sharing each other’s experiences? Do they understand what’s happening in each other’s businesses? Are they meeting on their own?

I think as a CEO, you actually want your leadership team to meet independent of the CEO.

Sonal: That’s actually really interesting and counterintuitive.

David: Yeah, I think it’s really important, and I think it does happen in a lot of high-performing teams very commonly, maybe not explicitly, but it happens. Then, obviously, in some places, you can do it explicitly — when it’s done in a productive and positive way, not because the CEO is a distraction. <Sonal laughs> Ideally, the CEO is out doing something that’s of high value to the company.

But if you get to this place where you do not have confidence that you are getting the best information from your leaders, if you don’t resolve that, then you have to find someone who’s a better fit. When I talk to a CEO who’s having a tough time in the company, and they’re telling me what’s happening, I’m like, “Just tell me, do you really believe that that is what’s happening?” You either have to go deep (and as a CEO, you do get these occasional bullets where you can cause a little bit of organizational stress to go three levels deep and really figure it out) — and if you find out what’s happening is not what you were being told, you’ve got to make a change of leadership.

You have to think about what is the right team I need for the right time

By the way, I should just say that all my lessons about leaders and management, I have pretty much learned the hard way. So I’m just trying to help save other people from making the same mistakes I made.

The story of OpenDNS

Sonal: Yeah well speaking of that, let’s talk about your story. You’re the founder of a company called OpenDNS. First of all, what is OpenDNS?

David: So OpenDNS is a cybersecurity service that delivers a faster and safer internet. We really innovated on a 25-year-old technology that used to be a cost center, that nobody wanted to innovate on. We proved that you can actually build a business on top of this thing that used to be free, if you make it better. Speed was one part, but then the other part was security.

Let’s say you type in Zamazon.com, you’re meaning to go to Amazon.com, but that could be a phishing site trying to steal your credentials. We would say, wait a minute. We know that from our tens of millions of users, what you really meant to type in was Amazon.com, so we’re going to show you a page that says, “Hey, you typed in zamazon.com, we think it’s a fraudulent site, did you really mean to go to Amazon.com?” And that may help protect you from getting phished.

It was the first third-party DNS provider. In fact, when we started the company, some of the greats of the internet told me: a) what I wanted to do was impossible; and b) even if it was possible, nobody would want it because guys would get it from an ISP (internet service provider).

Sonal: Oh, my God. This reminds me of Marc with Netscape. One of my favorite stories is I saw these old forums that he was on when he was proposing a more of a graphical user interface.

David: The image tag?

Sonal: Exactly. And the thing that I thought was so funny is the people who are the established, kind of old fogies (for lack of a better phrase), they don’t like the change, ironically, even though they were very revolutionary at the time.

So, you mentioned a 25-year-old technology… Why was that almost impossible to them?

David: Think of the DNS like a phonebook. Except that what we wanted to do is not just give the same phonebook to everyone, we wanted to give a custom phonebook to every person. Let’s say you typed in Playboy.com. For some user over here, they may not want content filtering, so they want the answer for Playboy.com, but maybe for someone who has small kids at home, they want a different answer. Doing this at very high speed was thought to be impossible. <Sonal: That’s fantastic> But it turned out that it was possible, and we could do it faster than even if you had no preferences and settings.

Sonal: I have to ask you, how old were you when you had the insight that you wanted to build OpenDNS?

David: So, I had started a DNS company in college that did a different kind of DNS. And through that, I had gotten super interested in cybersecurity. I met an investor when I moved out to California who had asked me, basically, why I wasn’t doing more with my original company. Then, he and I ultimately came up with the idea for OpenDNS.

That original business model that I worked on with him, it was an advertising-supported business model. We pivoted the business at the end of 2009 to having the people that use our service be the people that pay us for our service — it was just a much better alignment of interests, and that journey took a long time. By the time we pivoted the business, it really was a different business than when we started it. When we sold it in 2015 to Cisco, it was really a full-blown cybersecurity company.

Sonal: Why did Cisco want it?

David: If you looked at what happened in between 2007 and 2015. The iPhone came out.

You had more and more people working from coffee shops that all had Wi-Fi; you had workers working from the road, people using mobile devices. So, installing like Norton Antivirus or McAfee Antivirus on your desktop was no longer sufficient security. And so our service, Open DNS, was cybersecurity delivered as a service. It happened intrinsically and as a part of your internet connection, so you didn’t have to have special software, you didn’t have to install an appliance or a piece of hardware. As people we’re working differently and the networks were becoming more ephemeral, Cisco (which is a major cybersecurity company, it’s actually the largest cybersecurity company) wants to evolve to match that shifting IT landscape.

Sonal: You mentioned “The Pivot” a few times. Tell me about that because that’s such an overused — one of those platitude-y words, like, big P, little p, whatever — I know you mean it in actually what happened, but give me a little bit more texture around the pivot. What was that like?

David: The best time I never want to have again. <Sonal chuckles> This might be a podcast unto its own, because there was an 11-month period where I wasn’t even CEO of the company. My original investor had fired me.

Sonal: Oh, my god, I didn’t know that.

David: I was CTO. I’d been demoted.

Sonal: CTO is awesome, though. I think the CTO is the most powerful person in the company.

David: Not when you wanna be CEO.

Sonal: I guess that’s true.

David: We pivoted the business in 2009. What we thought was a consumer business, actually turned out to just be a free business.

One day we got a call from a major oil-and-gas company that had been using us, and we knew they were using us globally on oil rigs of their headquarters and other distributed offices. And then, finally, we got an email: “Look, we need to have a support contract, as a matter of our corporate hygiene. So, figure it out, and give us a quote. We need to have a way to call you if there’s a problem.”

And so we went and got one of these virtual phone numbers on the internet that would route to an engineer’s phone number; and if that person didn’t pick up, it would route to the next engineer’s phone number; and if that person didn’t pick up, it would route to my phone.

Sonal: Oh my, so you were like the support desk?

David: Yeah, it was like a tiered call system. It went to three people, and we sent them a quote for one-hundred grand, and they signed it immediately and returned it. And now it went from us making $2 dollars a year in advertising — which we hated — to paying us $100,000 for something we’re already doing, and we get to turn off the ads.

You don’t need to be a rocket scientist to figure out, wait a minute, maybe there’s something here. We had two or three other companies that had asked previously for something like that, so we went and sent them quotes, and they all signed them and returned them.

Sonal: Why do you think you didn’t know that this would be the business model up front? Why did you have to pivot? Honestly, not to sound judgmental at all, but it seems obvious to me when you say it in hindsight <David: totally> — so I’m confused why you didn’t see that coming up.

David: I think we were sort of enamored with this idea of keeping the whole internet safer, and that meant going after individuals. <Sonal: Idealistic> We had partnerships with Netgear and D-Link, and these people that sold consumer routers, and so we ignored the opportunity that’s right in front of our face.

But as soon as you realize you’re not going to be able to raise money, and you actually have to build a business, you start to open your eyes a little bit. We did that, and then I hired Michelle Law, who actually spent seven years at Greylock, to run BD for us. Ultimately, she became my COO — a wonderful person, and a good friend — she had seen enterprise companies many times, and so she realized as we wanted to go enterprise that a bunch of the team had to change.

First of all, half the team just didn’t care about building an enterprise software company, so they just quit. Then, the other half of the team just could not internalize that we can’t just change the UI overnight. Because it turns out some of our big customers had their own manuals that they had built with screenshots of our product. We got a nasty email once from this major oil-and-gas company that said, “We have all this training material and screenshots and videos we made, and you just totally changed your whole dashboard. You can’t do that.”

You just have to learn how to manage those things, and then you do feature flags, which are things that are common today, but in 2009, feature flagging things and…

Sonal: What’s feature flagging?

David: Feature flagging means some subset or cohort of customers gets the access to a new feature, our new look and feel. A lot of people use it for A/B testing to see if something works, but you can also use it just to keep certain customers on certain packages, or on older features or an older look and feel. You still have one code base, but people have slightly different experiences.

We were starting to do those things. We started implementing feature flags and things of that nature, but it meant that over the course of about 12-18 months, of the 30 people before the pivot, I think only 3 were left at the end.

The role of CEO

Sonal: When did you go from CEO to CTO?

David: So right before all that happened, for most of 2008. The only good thing that ever came out of the total global recession and economic collapse was that my early investor needed cash, and so we found two investors (and that’s actually when I first met Mark and Ben) to come in and buy out my early investor. Those investors came in, started to rebuild the company.

Sonal: But the biggest thing that’s fascinating to me is you came back as the CEO. So, what changed that you didn’t make this — I mean, ‘cause you’re the same person; you didn’t change overnight. <David: Right> Like, how did you… pull that off?

David: Coming back as CEO the second time, after spending almost a year as CTO, one of the things I saw when I wasn’t CEO was all these things that weren’t happening in the company that should have been happening. Of course, I blamed the current CEO, but the reality was, I actually was not doing them either when I was CEO.

Sometimes you just have to have this outside-the-glass-box kind of view, and you’re like: Wait a minute, people don’t know what’s important. Wait a minute, we’re not making it clear what the priorities are. Wait a minute, we’re not firing these low-performing people. But I wasn’t doing any of those things either!

That, to me, was very eye-opening, so when I came back as CEO, I was a much better listener. I think I had this belief the first time I was CEO that I’m expected to have all the answers. It’s just not possible. What is important is CEOs have to make decisions, and I think they have to be able to articulate their decisions, but they don’t have to have all the answers, they don’t have to be the smartest person.

Sonal: That’s a really important point… There’s actually a big difference between an answer and a decision. That’s actually something to reflect on, because I think most people conflate those two things.

CEOs don’t have to have all the answers, but they do have to make decisions, and be able to articulate their decisions

David: Totally. In fact, it turns out, actually, the opposite is true about having the answers. I often tell CEOs — because even before I joined Andreessen Horowitz, people called me for advice — and that’s one of the things I really enjoy.

Sonal: It’s why you’re a VC now.

David: Yeah, it’s why I’m a VC. That’s one of the best parts about the job. I like being the first phone call for a CEO when they’ve had the tough moment, or they need help.

One of the things I often tell CEOs is: When you think about the table of leaders around you, there’s actually room around the table for one person who has no idea what they’re doing. And that’s you, as CEO.

If you have the right leadership team, they’re adding intelligence for you.

I had gone from technical CEO, to product CEO, to sales CEO, but my fault as a sales CEO is that that I loved the dog-and-pony show: I loved the pitch, I hated the close.

Sonal: Why is that? That’s fascinating.

David: You know I thought… you could say that it was ego or ignorance or naïveté. I didn’t like asking for the purchase orders because, first of all, I always thought our pricing was low. So if the customer… customers often like to negotiate.

Sonal: Oh, and you’re ready to fight, like, “Fuck you, I want you to pay more.”

David: Yeah, because the customer goes, “Oh, $100,000? I think it should be $70,000.” I’m like, “Fuck you, it should be $140,000.” <Sonal laughs> Like, “I’m raising the price!”

Sonal: You’re the wrong guy to bring at the close, basically.

David: My sales will be like, “Yeah, so, you can’t negotiate with this customer because you’re going to just blow up this deal.” I’m in a much different place now, obviously, but at that time, and where I was as a CEO, I hated the negotiation. I got uncomfortable.

Everyone, they taught me so much. Because there was only room around the table for me to really not know all the answers.

I will often say that my CMO at OpenDNS, Jeff Samuels — I think of him not just as my CMO but as a mentor to me — he taught me so much. I would say that about my VP of Engineering, my Head of Sales, my VP of Sales. I could take the inputs and use all that to make a decision, and I felt very good about those decisions I made. I think CEOs find it a huge relief when you tell them, you’re allowed to not know. In fact, if you have the best people, you’re going to know the least!

It is not uncommon for a CEO or a leader, a manager… this is good general life advice. You don’t wanna be the whiny person constantly, like, harping on something, but I would say is that you do sometimes need to present an idea more than once. <Sonal: interesting> My old head of finance, who ultimately became my best friend, used to always joke with me that he would just tell me everything he said twice because he knew the first time, I’d ignore him.

Sonal: <laughs> I think I have the same problem.

David: He would tell me some statistic about what’s happening with marketing spend, or with hiring, or sales spends, and I’d be like, “Oh, that’s not really a problem. Like, whatever. I don’t care. You’re just a crazy finance bean-counter.” But then, he’d come back a week later. He’d be like, “Hey, I have more data. I did further analysis. You ignored me, but I know, I’m right here.” And I’d be like, “Oh, you’re right… why didn’t you tell me this last week?”

Sonal: <both laugh> So, in that case, the CEO can get answers from all their management team and then make a decision based on all the answers you’re hearing.

David: That’s right. And I do think sometimes you do have to tell people more than once, and that’s just the function of how human beings operate.

Learning to listen

Sonal: Speaking of this — telling people more than once and learning to listen — that was your big shift between when you came back to be CEO, and you kind of learned your lesson, so to speak.

I honestly feel like that’s kind of a trite thing people say all the time. Like, listen better! I hate the “design-thinking” mindset around “empathy for this scenario” and “this persona”; it’s just so — I can’t diagnose what’s off…

David: Yeah, I think when you’re building a company, being empathetic really means understanding — it doesn’t mean accommodating, right? This happens all the time as a leader, where you may not resolve that thing, but you can still understand it and be empathetic. I can be like, “Yeah, that is terrible. I understand what you’re saying, and I am hearing you.”

Sonal: Honestly, that’s half the battle in relationships. <David: That’s right> You don’t need an answer to 99% of this. You just need someone to say, “Fuck, I feel for you. That sucks.” And you’re already feeling, like, 80% better.

When you’re building a company, being empathetic really means understanding — it doesn’t mean accommodating

David: That’s right, that’s right. When I think about empathy, you really want to be a great listener.

A friend of mine, Wendy MacNaughton, she does this whole New York Times thing every other week where she really goes deep into a topic. She’s written these books. I think she thinks of herself as an artist. I think of her as an artist and entrepreneur. One of the things that she taught me a year or two ago was — when she’s trying to teach people how to be a really, really solid listener — is that when someone’s talking to you be like, “Tell me more about that.”

Sonal: That’s just the phrase: “Tell me more about that.”

David: Five words.

Sonal: So, just make that your first question.

David: When you’re talking to a customer, “Oh, what’s going on?” “Oh we’re doing annual planning?” It’s like, “Tell me more about that. How are you thinking about that? What is happening? What’s the frustrating part about annual planning? Tell me more about that.”

Sonal: So what’s interesting to me about that is, to me, this is the difference between a focus group and an ethnographer. Focus groups and surveys are asking questions for things you already know to ask. <David: That’s right> An ethnographer is embedded in an organization or a setting and is essentially just listening to learn and observe, and letting those patterns reveal things.

David: The deeper insights come out <Sonal: exactly> when you go down the tell-me-more-about-that path. That’s when you get these flashes in your brain of, wait, now I really understand what’s happening. It’s not that annual planning sucks, it’s that you’re having budget issues that aren’t being resolved in the way that you need, or that maybe your tools you’re using to do annual planning, or the way you communicate and collaborate with your team, or the way you work cross-functionally is not working.

Sonal: Totally, totally. I consider myself an “ethnographer-esque editor”. <David: Totally> Because I want the context to know what I’m not hearing, to really understand.

So, it’s interesting because, on the ethnographer side — I don’t think people know this about you, but you started off your career — or, academic career, because you actually started working when you were, like, what?

David: Eighth grade, what is that, like, 14 or 15?

Sonal: That’s when you got your first W-2, right?

David: Yeah, I’ve had a 1099 or a W-2 every year since eighth grade. I worked at a mom and pop ISP in San Diego and learned all about routing and networking. I went to Washington University in St. Louis. I applied to the School of Arts and Sciences. When I went there to interview, they actually then had me interview with somebody in the School of Engineering in the Computer Science department. By the end of first semester of freshman year, I had switched back to the School of Arts and Sciences.

And the reason I switched is I took a class “Introduction to Human Evolution”, and I just found it so fascinating. I’ve always learned in my life I do best in the things I really enjoy working on. I have trouble doing things I don’t want to do.

Sonal: Me, too. I’m the exact same way.

David: Yeah, it sounds obvious, but some people can actually just will their way through the other hard stuff.

Sonal: No, I can’t. I can’t. I have no energy — I have zero. Talk about return on energy, I have no energy to even DO the thing.

David: Yeah, I’m just like, I’ll be okay if I just don’t do this.

Sonal: Yeah, I feel the exact same way.

David: I had trouble in school with things that I really didn’t enjoy, so I learned how to optimize for the things that I like doing. And anthropology, like I just… every book I read I thought was so interesting. I learned about how women enforced power hierarchy in South America in a way that we don’t have elsewhere in the world. I learned about what happens in Southeast Asia around farming. I learned about the Green Revolution in Africa.

And then I find that in my life, I actually think about these things all the time.

Sonal: That was actually my next question, do you think it actually is useful in your career as a technologist?

David: Oh, absolutely. Absolutely. I think about demographic transitions; when I read about what’s happening in Japan, it makes me think about how’s that going to change my investing thesis.

I think it comes up constantly. It comes up both tactically as you think about yourself in leadership and organizational dynamics. It gives you an appreciation that there’s many perspectives in the world. In fact, it gives you an appreciation that more perspectives are better, and you want more.

Sonal: So, that is a perfect way to close this episode. So, David Ulevitch — he’s made a journey from anthropologist, to technical CEO, to product CEO, to sales CEO, to go-to-market CEO — and now, investor. Welcome to the a16z Podcast.

David: Thanks. Glad to be here.

• 

  David Ulevitch is a general partner at a16z where he invests in enterprise and SaaS companies. Prior to joining the firm, he was the founder and CEO of OpenDNS (acquired by Cisco).

  Follow Twitter
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter

Every organization, whether small or big, early or late stage — and every individual, whether for themselves or others — makes countless decisions every day, under conditions of uncertainty. The question is, are we allowing that uncertainty to bubble to the surface, and if so, how much and when? Where does consensus, transparency, forecasting, backcasting, pre-mortems, and heck, even regret, usefully come in?

Going beyond the typical discussion of focusing on process vs. outcomes and probabilistic thinking, this episode of the a16z Podcast features Thinking in Bets author Annie Duke — one of the top poker players in the world (and World Series of Poker champ), former psychology PhD, and founder of national decision education movement How I Decide — in conversation with Marc Andreessen and Sonal Chokshi. The episode covers everything from the role of narrative — hagiography or takedown? — to fighting (or embracing) evolution. How do we go from the bottom of the summit to the top of the summit to the entire landscape… and up, down, and opposite?

The first step to understanding what really slows innovation down is understanding good decision-making — because we have conflicting interests, and are sometimes even competing against future versions of ourselves (or of our organizations). And there’s a set of possible futures that result from not making a decision as well. So why feel both pessimistic AND optimistic about all this??

Show Notes

• Using a football thought experiment to distinguish skill and luck [0:58]
• Balancing outcomes and process [9:49]
• Asking the right questions, especially with a negative outcome [11:17]
• Discussion of timing in forecasting [15:23], and other practical implications [16:59]
• Why not making a decision is also a decision [23:40], and how to evaluate the options you didn’t take [30:15]
• Discussion of how widely this type of decision-making will be adopted by the public [34:10]
• How to communicate probabilistically [37:24] and how to build uncertainty into an organization [40:21]

Transcript

Sonal: Hi, everyone. Welcome to the “a16z Podcast.” I’m Sonal, and today Mark and I are doing another one of our book author episodes. We’re interviewing Annie Duke, who’s a professional poker player and World Series champ, and is the author of “Thinking in Bets,” which is just out in paperback today. The subtitle of the book is, “Making Smarter Decisions When You Don’t Have All the Facts,” which actually applies to startups and companies of all sizes and ages, quite frankly. I mean, basically, any business or new product line operating under conditions of great uncertainty — which I would argue is my definition of a startup and innovation. So that will be the frame for this episode. 

Annie’s also working on her next book right now, and founded howidecide.org, which brings together various stakeholders to create a national education movement around decision education, empowering students to also be better decision makers. So, anyway, Mark and I interview her about all sorts of things in and beyond her book, going from investing, to business, to life. But Annie begins with a thought experiment, even though neither of us really know that much about football.

Skill vs. luck

Annie: So what I’d love to do is, kind of, throw a thought experiment at you guys so that we can have a discussion about this. So I know you guys don’t know a lot about football, but this one’s pretty easy. You’re gonna be able to feel this one. I want you to do this thought experiment. Pete Carroll calls for Marshawn Lynch to actually run the ball.

Sonal: So we’re betting on someone who we know is really good?

Annie: Well, they’re all really good, but we’re betting on the play that everybody’s expecting.

Mark: Yeah, the default.

Annie: This is the default.

Mark: The assumed rational thing to do, right?

Annie: This is the assumed rational thing to do, right. So he has Russell Wilson hand it off to Marshawn Lynch. Marshawn Lynch goes to barrel through the line. He fails. Now they call the timeout — so now they stop the clock. They get another play now, and they hand the ball off to Marshawn Lynch — what everybody expects. Marshawn Lynch, again, attempts to get through that line and he fails. End of game, Patriots win. 

My question to you is, are the headlines the next day, “The Worst Call in Super Bowl History”? Is Cris Collinsworth saying, “I can’t believe the call, I can’t believe the call.” Or is he saying something more like, “That’s why the Patriots are so good. Their line is so great. That’s the Patriots’ line that we’ve come to see this whole season. This will seal Belichick’s place in history.” It would’ve all been about the Patriots.

So let’s, sort of, divide things into, like — we can either say the outcomes are due to skill or luck — and luck in this particular case is gonna be anything that has nothing to do with Pete Carroll. And we can agree that the Patriots’ line doesn’t have anything to do with Pete Carroll — Belichick doesn’t have anything to do with Pete Carroll — Tom Brady doesn’t have anything to do with Pete Carroll — as they’re sealing their fifth Super Bowl victory. 

So what we can see is there’s two different routes to failure here. One route to failure, you get resulting. And basically what resulting is, is that retrospectively, once you have the outcome of a decision — once there’s a result — it’s really, really hard to work backwards from that single outcome to try to figure out what the decision quality is. This is just very hard for us to do. They say, “Oh my gosh, the outcome was so bad. This is clearly — I’m gonna put that right into the skill bucket. This is because of Pete Carroll’s own doing.” But in the other case, they’re like, “Oh, you know, there’s uncertainty. What could you do?” Weird, right?

Sonal: Yeah.

Annie: Okay, so you can kind of take that and you can say, “Aha, now we can, sort of, understand some things.” Like, for example, people have complained for a very long time that in the NFL they have been very, very slow to adopt what the analytics say that you should be adopting, right? And even though now we’ve got some movement on fourth-down calls, and when are you going for two-point conversions, and things like that, they’re still nowhere close to where they’re supposed to be, and why is that?

Mark: So they don’t make the plays corresponding to the statistical probabilities?

Annie: No. In fact, the analytics show that if you’re on your own one-yard line, and it’s fourth down, you should go for it no matter what. The reason for that is if you kick it, you’re only gonna be able to kick to midfield. So the other team is basically almost guaranteed three points anyway, so you’re supposed to just try to get the yards. Like, when have you ever seen a team on their own one-yard line on fourth down be like, “Yeah, let’s go for it.” That does not happen.

Okay, so we know that they’ve been super slow to do what the analytics say is correct, and so you sit here and you go, “Well, why is that?” And that thought experiment really tells you why, because we’re all human beings. We all understand that there are certain times when we don’t allow uncertainty to bubble up to the surface — is the explanation — and there are certain times when we do. And it seems to be that we do, when we have this, kind of, consensus around the decision, there’s other ways we get there. And so, okay, if I’m a human decision-maker, I’m gonna choose the path where I don’t get yelled at.

Sonal: Yeah, exactly.

Annie: So, basically, we can, kind of, walk back, and we can say, “Are we allowing the uncertainty to bubble to the surface?” and this is gonna be the first step to, kind of, understanding what really slows innovation down — what really slows adoption of what we might know is good decision making, because we have conflicting interests, right? Making the best decision for the long run, or making the best decision to keep us out of a room where we’re getting judged.

Mark: Yelled at, or possibly fired. So let me propose the framework that I use to think about this and see if you agree with it. So it’d be a two-by-two grid, and it’s consensus versus non-consensus, and it’s right versus wrong. And the way we think about it, at least in our business, is basically — consensus right is fine. Non-consensus right is fine. In fact, generally, you get called a genius. Consensus wrong is fine, because, you know, it’s just the same mistake everybody else made.

Sonal: You all agreed, right, it was wrong.

Mark: Non-consensus wrong is really bad.

Annie: Horrible.

Mark: It’s radioactively bad. And then as a consequence of that, and maybe this gets to the innovation stuff that you’ll be talking about — but as a consequence of that, there are only two scripts for talking about people operating in the non-consensus directions. One script is, they’re a genius because it went right — and the other is they’re a complete moron because it went wrong. Does that map?

Annie: That’s exactly it. That’s exactly right. And I think that the problem here is that, what does right and wrong mean? In your two-by-two, wrong and right is really just, did it turn out well or not?

Mark: Yeah, outcomes.

Sonal: Not the process.

Annie: And this is where we really get into this problem, because now what people are doing is they’re trying to swat the outcomes away. And they understand, just as you said, that on that consensus wrong, you will have a cloak of invisibility over you — like, you don’t have to deal with it. <Right.> So let’s think about other things besides consensus. So, consensus is one way to do that, especially when you have complicated cost-benefit analyses going into it. I don’t think that people, when they’re getting in a car, are actually doing any, kind of, calculation about what the cost-benefit analysis is to their own productivity, versus the danger of something very bad happening to them. Like, what is this society? Someone’s done this calculation, we’ve all, kind of, done this together — and so, therefore, getting in a car is totally fine. I’m gonna do that.

Mark: And nobody second-guesses anybody. If somebody dies in a car crash you don’t say, “Wow, what a moron for getting in a car.”

Annie: No. Another way that we can get there is through transparency. So if the decision is pretty transparent, another way to get there is status quo. So a good status quo example that I like to give, because everybody can understand it is — you have to get to a plane, and you’re with your significant other in the car, and you go the usual route.

Sonal: This is a common fight for every couple.

Annie: Yeah, so you go your usual route. Literally, this is the route that you’ve always gone and there is some sort of accident, there’s bad traffic, you miss the plane — and you’re mostly probably comforting each other in the car. It’s like, “What could we do?” You know, eh. But then you get in the car and you announce to your significant other, “I’ve got a great shortcut, so let’s take this shortcut to the airport.” And there’s the same accident, whatever — horrible traffic, you miss the flight. That’s like that status quo versus non-status quo decision.

Sonal: Right, you’re going against what’s familiar and comfortable.

Annie: Exactly. If we go back to the car example, when you look at what the reaction is to a pedestrian dying because of an autonomous vehicle, versus because of a human, we’re very, very harsh with the algorithms. For example, if you get in a car accident and you happen to hit a pedestrian, I can say something like, “Well, you know, Mark didn’t intend to do that.” Because I think that I understand — your mind is not such a black box to me. So I feel like I have some insight into what your decision might be, and so more allowing some of the uncertainty to bubble up there. But if this black box algorithm makes the decision, now all of a sudden I’m like, “Get these cars off the road.”

Sonal: Never mind that the human mind is a black box itself ultimately, right?

Annie: Of course, but we have some sort of illusion that I understand, sort of, what’s going on in there, just like I have an illusion that I understand what’s going on in my own brain. And you can actually see this in some of the language around crashes on Wall Street, too. When you have a crash that comes from human beings selling, people say things like, “The market went down today.” When it’s algorithms, they say, “It’s a flash crash.” So now they’re, sort of, pointing out, like — this is clearly in the skilled category. It’s the algorithm’s fault. We should really have a discussion about algorithmic trading and whether this should be allowed, when obviously the mechanism for the market going down is the same either way.

So now if we understand that, so exactly your matrix. Now we can say, “Well, okay, human beings understand what’s gonna get them in the room.” And pretty much anybody who’s, you know, living and breathing in the top levels of business at this point is gonna tell you, “Process, process, process. I don’t care about your outcomes — process, process, process.” But then the only time they ever have, like, an all-hands-on-deck meeting is when something goes wrong. Let’s say that you’re in a real estate investing group, and so you invest in a particular property based on your model, and the appraisal comes in 10% lower than what you expected. Like, everybody’s in a room, right? You’re all having a discussion. You’re all examining the model, you’re trying to figure out. But what happens when the appraisal comes in 10% higher than expected? Is everyone in the room going, “What happened here?”

Outcomes vs. process

Mark: Now there is the obvious reality, which is, like, we don’t get paid in process, we get paid in outcomes. Poker players, you don’t get paid in process, you get paid in outcome, and so there is a…

Sonal: Incentive alignment.

Mark: It’s not completely emotional. It’s also an actual — there’s a real component to it.

Annie: Yeah, so two things. One is, you have to make it very clear to the people who work for you that you understand that outcomes will come from good process. That’s number one. And then number two, what you have to do is try to align the fact that, as human beings, we tend to be outcome driven — to what you want, in terms of getting an individual’s risk to align with the enterprise risk. Because otherwise you’re gonna get this CYA behavior. And the other thing is that we wanna understand if we have the right assessment of risk. So one of the big problems with the appraisal coming in 10% too high, there, could be that your model is correct. It could be that you could have just a tail result, but it certainly is a trigger for you to go look and say, “Was there risk in this decision that we didn’t know was there?” And it’s really important for deploying resources.

Sonal: I have a question about translating this to, say, non-investing context. So in the example of Mark’s matrix, even if it’s a non-consensus wrong — you are staking money that you are responsible for. In most companies, people do not have that kind of skin in the game. <Right.> So how do you drive accountability in a process-driven environment — that the results actually do matter? You want people to be accountable, yet not overly focused on the outcome. Like, how do you calibrate that?

Annie: So let’s think about, how can we create balance across three dimensions that makes it so that the outcome you care about is the quality of the forecast? So first of all, obviously this demands that you have people making forecasts. You have to state in advance, “Here’s what I think. This is my model of the world. Here are where all the places are gonna fall. So this is what I think.” So now you stated that, and whether the outcome is “good or bad” is — how close are you to whatever that forecast is?

So, now it’s not just like, oh, you won to it, or you lost to it. It’s — was your forecast good? So that’s piece number one, is make sure that you’re trying to be as equal across quality as you can, and focus more on forecast quality as opposed to traditionally what we would think of as outcome quality. So now the second piece is directional. So, when we have a bad outcome and everybody gets in the room, when was the last time that someone suggested, “Well, you know, we really should’ve lost more here?” Like, literally nobody’s saying that, but sometimes that’s true. Sometimes if you examine it, you’ll find out that you didn’t have a big enough position. It turned out, okay, well, maybe we should’ve actually lost more. So you wanna ask both up, down, and orthogonal. So, could we have lost less? Should we have lost more? And then the question of, should we have been in this position at all?

Mark: So in venture capital, after a company works and exits — let’s say it sells for a lot of money, you do often say, “God, I wish we had invested more money.” You never, ever, ever, ever — I have never heard anybody say on a loss, “We should’ve invested more money.”

Annie: See, wouldn’t it be great if someone said that? Wouldn’t you love for someone to come up and say that to you? That would make you so happy.

Sonal: I actually still don’t get…

Mark: And what would be the logic of why they should say that?

Sonal: I still don’t get the point. Exactly. Why does that matter? I don’t really understand that.

Annie: Can I just, like — simple, in a poker example?

Sonal: Yeah.

Annie: So let’s say that I get involved in a hand with you, and I have some idea about how you play. And I have decided that you are somebody that, if I bet X, you will continue to play with me. Let’s say this is a spot where I know that I have the best hand, but if I bet X plus C that you will fold. So if I go above X, I’m not gonna be able to keep you coming along with me, but if I bet X or below, then you will — so I bet X. You call, but you call really fast, in a way that makes me realize, “Oh, I could’ve actually bet X plus C.” You hit a very lucky card on the end, and I happen to lose the pot. I should’ve maximized at the point that I was the mathematical favorite.

Mark: Because your model of me was wrong, which is a learning independent of the win or the loss.

Annie: Exactly. So you need to be exploring those questions in a real honest way.

Mark: Right, because it has to do with how you size future bets.

Sonal: This is exactly like a company betting on a product line.

Annie: Correct.

Sonal: And then picking what the next product line is gonna be, and then not having had the information that would then drive a better decision-making process around that.

Annie: Right. So thinking about the learning loss that’s happening because we’re not exploring that — the negative direction — and now you should do this on wins as well. So if you do ever discuss a win, you always think, like, “How could I press? How could I have won more? How could I have made this even better? How could I do this again in the future? Should we have won less?”

Mark: We oversized the bet and then got bailed out by a fluke.

Annie: We should have actually had less in it, and sometimes not at all, because sometimes the reasons that we invested turned out to be orthogonal to the reasons that it actually ended up playing out in the way that it was. And so, had we had that information, we actually wouldn’t have bet on this at all because it was completely orthogonal. We totally had this wrong. It just turned out that we ended up winning. And that can happen. I mean, obviously that happens in poker all the time, but what does that communicate to the people on your team? 

Good, bad, I don’t care. I care about our model. I wanna know that we’re modeling the world well, and that we’re thinking about, “How do we incorporate the things that we learn?” Because we can generally think about stuff in two — stuff we know, and stuff we don’t know. There’s stuff we don’t know we know, obviously — so we don’t worry about that, because we don’t know we don’t know it. But then there’s stuff we could know, and stuff we can’t know. It’s things like the size of the universe, or the thoughts of others, <Exactly.> or what the outcome will actually be. We don’t know that.

Sonal: I have a question about this, though. What is the timeframe for that forecast? So let’s say you have a model of the world — a model of a technology, how it’s gonna adopt, how it’s gonna play out. In some cases, there are companies that can take, like, years to get traction. You wanna get your customers very early to figure that out, right? So you can get that data. But how much time do you give? How do you size that timeframe for the forecast, so you’re not constantly updating with every customer data point, and so you’re also giving it enough time for your model, your plan, your forecast to play out?

Annie: You have to think about —very clearly in advance, “What’s my time horizon? How long do I need for this to play out?” But also, don’t just do this for the big decisions — because there’s things that you can forecast for tomorrow as well, so that you end up bringing it into just the way that people think. And then once you’ve decided, “Okay, this is the time horizon on my forecast,” then you would wanna be thinking about, “What are forecasts we make for a year, two years, five years for this specific decision to play out?” And then just make sure that you talk in advance — at what point you’ll revisit the forecast. So you wanna think in advance, “What are the things that would have to be true for me to be willing to come in and actually revisit this forecast?” Because otherwise, you can start, as you just said, like — it can turn into — super bad.

Sonal: You’re like a leaf in the wind. Right, exactly, because then you’re, like, one bad customer and you suddenly over-rotate on that — when in fact, it could’ve been not even a thing.

Annie: Right, so if you include that in your forecast — here are the circumstances under which we would come in and check on our model — then you’ve already gotten that in advance. So that’s actually creating constraints around the re-activity, which is helpful.

Barriers to logical decision-making

Mark: Two questions on practical implementation of the theory. So what I’m finding is, more and more people understand the logic of what you’re describing, because people are getting exposed to these ideas and, kind of — expanding in importance. And so more and more people intellectually understand this stuff, but there’s two, kind of — I don’t know, so call it emotion-driven warps, or something — that people just really have a hard time with. So one is that you understand this could be true investors, CEO, product-line manager in a company — you know, kind of, anybody in one of these domains — which is you can’t get the non-consensus results unless you’re willing to take the damage, right, the risk on the non-consensus wrong results.

But people cannot cope with the non-consensus wrong outcome. They just emotionally cannot handle it — and they would like to think that they can, and they intellectually understand that they should be able to. But as you say, when they’re in the room it’s such a traumatizing experience that it’s the “touching the hot stove.” They will do anything in the future to avoid that. And so one interpretation would be, that’s just simply flat out human nature — and so, to some extent, the intellectual understanding here — it doesn’t actually matter that much, because there is an emotional override. And so that would be a pessimistic view on our ability as a species to learn these lessons, or do you have a more optimistic view of that?

Annie: I’m gonna be both pessimistic and optimistic at the same time, so let me explain why.

Sonal: Ooh, love it.

Annie: Because I think that if you move this a little bit it’s a huge difference. You, sort of, have two tacks that you wanna take. One is, how much can you move the individual to, sort of, train this kind of thinking for them? And that means, naturally, they’re thinking in forecasts a little bit more — that when they do have those kinds of reactions, which naturally everybody will, they right the ship more quickly, so that they can learn the lessons more quickly, right? I mean, I actually just had this happen. I turned in a draft of my next book — the first part of my next book to my editor — and I just got the worst comments I’ve ever gotten back.

Sonal: Good editor.

Annie: And I had a really bad 24 hours, but after 24 hours, I was like, “You know what? She’s right.” Now, I still had a really bad 24 hours — and I’m the, like, “give me negative feedback” queen. Because I’m a human being. But I got to it fast. I, sort of, got through it pretty quickly after this. I mean, I — you know, on the phone with my agent saying, “I’m standing my ground, this is ridiculous.” And then he got a text the next day being, like, “No, she’s right.” And then I rewrote it, and you know what? It’s so much better for having been re-written, and now I can get to a place of gratitude for having the negative feedback. But I still had the really bad day, so it’s okay.

Sonal: So, it doesn’t go away, right?

Annie: Yeah, and it’s okay. We’re all human, we’re not robots. So number one is, like, how much are you getting the individuals to say, “Okay, I improved 2%, that’s so amazing for my decision making and my learning going forward?” And then the second through-line is, what are you doing to not make it worse? Because obviously for a long time people liked to talk about, “I’m results oriented.” That’s, like, the worst sentence that could come out of somebody’s mouth.

Sonal: Why is that the worst? I’ve heard that a lot, what’s so bad about it?

Annie: Because you’re letting people know that all you care about is, like, “Did you win or lose?” That’s fantastic — be results oriented all you want. You should pay by the piece. You will get much faster work. But the minute that you’re asking people to do intellectual work, results oriented is, like, the worst thing that you could say to somebody. So I think that we need to take responsibility, and the people in our orbit — we can make sure at minimum that we aren’t making it worse. And I think that that — so that’s pessimistic and optimistic. I don’t think anyone is making a full reversal here.

Mark: So the second question then goes to the societal aspect of this. And so we’ll talk about the role of the storytellers — or as they’re sometimes known, the journalists.

Annie: Yeah, and the editors.

Sonal: I love it.

Mark: And the editors, and the publishers. And so the very first reporter I ever met when I was a kid — Jared Sandberg at the Wall Street Journal — you know, the internet was first emerging. There were no stories in the press about the internet, and I used to say, “There’s all this internet stuff happening. Why am I not reading about any of it in any of these newspapers?” And he’s like, “Well, because the story of ‘something is happening’ is not an interesting story.” He said, “There are only two stories that sell newspapers.” He said, “One is, ‘Oh, the glory of it,’ and the other is, ‘Oh, the shame of it.’” And basically he said it’s conflict. So it’s either something wonderful has happened, or something horrible has happened, <Yeah.> those are the two stories. And then you think about business journalism as, kind of, our domain — and you kinda think about it, and it’s like, those are the only two profiles of a CEO or founder you’ll ever read.

It’s just, like, what a super genius for doing something presumably non-consensus and right, or what a moron. Like, what a hopeless idiot for doing something non-consensus and wrong. And so, since I’ve become more aware of this, it’s gotten very hard for me to actually read any of the coverage of the people I know, because — it’s like the people who got non-consensus right, they’re being lavished with too much praise. <Ah.> And the people who got non-consensus wrong, they’re being damned for all kinds of reasons. The traits are actually the same in a lot of cases. And so, I guess, as a consequence — if you read the coverage, it really reinforces this bias of being results-oriented. And it’s like, it’s not our fault that people don’t wanna read a story that says, “Well, you tried something and it didn’t work this time,” right?

Annie: Yes, exactly. But it was mathematically pretty good. If we go back to Pete Carroll, this is a pretty great case. And if we think about options theory, just quickly — the paths preserve the option for two run plays. So if you wanna get three tries at the end zone instead of two, strictly for clock management reasons, you pass first.

Mark: Right, and that’s not gonna kick off ESPN “SportsCenter” that night. And so optimistic or pessimistic that the narrative — the public narrative on these topics will ever move? 

Annie: I’m super, super pessimistic on the societal level, but I’m optimistic on — if we’re educating people better, that we can equip them better for this. So I’m really focused on, how do we make sure that we’re equipping people to be able to parse those narratives in a way that’s more rational? And particularly, you know — now there’s so much information, and it’s all about the framing, and the storytelling — and it’s particularly driven by, what’s the interaction of your own point of view? We could think about it as [a] partisan point of view, for example, versus the point of view of the communicator of the information, and how is that interacting with each other. You know, in terms of, how critically are you viewing the information, for example? I think this is another really big piece of the pie, and somewhat actually related to the question about journalism, which is that third dimension of the space.

So we talked about two-dimension, which is, sort of, outcome quality, and how are you allowing that you’re exploring both downside and upside outcomes in a way that’s really looking at forecast? How are you thinking directionally, so that you’re more directionally neutral? But then the other piece of the puzzle is, how are you treating omissions versus commissions? 

So one of the things that we know with this issue of resulting is, here’s a really great way to make sure that nobody ever results on you — don’t do anything, okay? So if I just don’t ever make a decision, I’m never gonna be in that room with everybody yelling at me for the stupid decision I made, because I had a bad outcome. But we know that not making a decision is making a decision, we just don’t think about it that way. And it doesn’t have to just be bad investing. You can have a shadow book of your own personal decisions.

Sonal: Personal life, I agree.

Not making a decision is a decision

Annie: So, you know, it’s really interesting — I remember I was giving somebody advice, who — I think he was, like, 23. And so, obviously, newly out of college, had been in this position for a year, and was really, really unhappy in the position. And he was asking me, like, “I don’t know what to do. I don’t know if I should change jobs.” And I said, “Well…” So I did all the tricks, you know, time traveling — and so I was like, “Okay, imagine it’s a year from now. Do you think you’re gonna be happy in this job?” “No.” “Okay, well, maybe you should choose this other — go and try to find another position.” And this is what he said to me — and this, I think, shows you how much people don’t realize that the thing that you’re already doing, the status quo thing — choosing to stay in that really is a decision.

So he said to me, “But if I go and find another position, and then I have to spend another year, which I just spent, trying to learn the ins and outs of the company, and it turns out that I’m not happy there, I’ll have wasted my time.” And I said to him, “Okay, well, let’s think about this, though. The job you’re in, which is a choice to stay in, you’ve now told me it’s 100% in a year that you will be sad. Then if you go to the new job, yes, of course it’s more volatile — but at least you’ve opened the range of outcomes up.” But he didn’t wanna do it because it doesn’t feel — like, staying where he was didn’t feel like somehow he was choosing it, so that he felt like if he went to the other place <Yes.> and ended up sad that somehow that would be his fault and a bad decision.

Sonal: That’s so, so profound. In my case — this might be getting a little too personal, but in my case it was a decision I didn’t know I had made, to not have kids. And it’s still an option, but it’s probably not gonna happen. And my therapist, kind of, told me that my not deciding was a choice — and I was so blown away by that that it, then, allowed me to then examine what was going on there in that framework, in order to not do that for other arenas in my life where I might actually want something. Or maybe I don’t, but at least it’s a choice, that there’s intentionality behind it.

Annie: Well, I appreciate you sharing. I mean, I really wanna thank you for that, because I think that people, first of all, should be sharing this kind of stuff so that people feel like they can talk about these kinds of things, number one.

Sonal: I agree.

Annie: And number two, in my book, I’ve got all these examples in there of, like — how are you making choices about raising your kids when it feels so consequential?

Sonal: When you’re doing decisions for other people?

Annie: Right, and you’re trying to decide, like, “Should I have kids, or shouldn’t I have kids?”

Sonal: Or this school, or that school?

Annie: Or, “Who am I supposed to marry, or where am I supposed to live?” And the thing that I try to get across is, you know — we can talk about investing, like, I’m putting money into some kind of financial instrument, but we all have resources that we’re investing. It’s our time.

Sonal: That’s right. Your time, your energy, your heart. It could be whatever, your friendships, your relationships.

Annie: Right, so you’re deploying resources.

Sonal: Yes, I love that.

Annie: And for the kind of decision that you’re talking about, it’s like — if you choose to have children you’re choosing to deploy certain resources with some expected return. Some of it good, some of it bad. And if you’re choosing not to have children, that’s a different deployment of your resources toward other things.

Sonal: And you need to know that there are limits. Everything isn’t a zero-sum game, <No.> but approaching the world, and the fact that evolution has approached the world as a zero-sum game — and our toolkit makes it a zero-sum game — means that we need to still view everything as a zero-sum game when it comes to those tradeoffs and resources. Because you are losing something every time, even in a non-zero game.

Annie: Right. So I don’t feel like the world is a zero-sum game in terms of, like, <Collaborate, coordinate.> most of the activities that you and I would engage around, we can both win, too. But it’s a zero-sum game, to go back to your therapist. It’s a zero-sum game between you and the other versions of yourself that you don’t choose.

Sonal: Exactly. Or an organization, and the other versions of itself it doesn’t choose.

Annie: Exactly. So there’s a set of possible futures that result from not making a decision as well. So on an individual decision, let’s put things into three categories: clear misses, near misses, and hits. There’s some that would just be a clear miss — throw them out — and there’s some that I’m gonna, sort of, really agonize over and I’m gonna, you know, think about it, and I’m gonna do a lot of analysis on it. So the ones which become a yes go into the hit category, and the other one is a near miss. I came close. What happens with those near misses is they just go away. 

So what I realized is that on any given decision — let’s take an investment decision. If I went to you, or you came to me, and said, “Well, tell me what’s happening with the companies that you have under consideration.” On a single decision, when I explain to you why I didn’t invest in a company, it’s gonna sound incredibly reasonable to you.

So you’ll only be able to see in the aggregate, if you look across many of those decisions, that I tend to be having this bias toward missing — towards saying, “You know what? We’re not gonna do it,” so that I don’t wanna stick my neck out. Now this, for you, is incredibly hard to spot because you do have to see it in the aggregate. Because I’m gonna be able to tell you a very good story on any individual decision. So the way to combat that — and again, get people to think about, “What we really care [about] around here is forecast, not really outcomes” — is actually to keep a shadow book. The anti-portfolio should contain basically all of your near misses, but then you have to take a sample of the clear misses as well — which nobody ever looks at. Because the near misses tend to be a little in your periphery if they happen to be big hits.

Mark: So the good news, bad news. So the good news is we have actually done this, and so we call it the shadow portfolio. <Awesome.> And the way that we do it is, we make the investment. We take an equivalent. We take the other equivalent deal of that vintage, of that size — that we almost did but didn’t do — and we put that in the shadow portfolio. And we’re trying to do, kind of, apples-to-apples comparison. In finance theory terms, the shadow portfolio may well outperform the real portfolio, and in finance terms that’s because the shadow portfolio may be higher variance. Higher volatility, higher risk, and therefore, higher return.

Annie: Correct.

Mark: Because, right, the fear is the ones that are hitting are the ones that are less spiky, they’re less volatile, they’re less risky.

Annie: Right. So what’s wonderful about that, when you decide not to invest in a company, you actually model out why. That’s in there.

Mark: It’s often, by the way, a single flaw that we’ve identified. It’s just like, oh, we would do it except for X, <Right.> where X looks like something that’s potentially existentially bad.

Annie: Right, and then that’s just written in there, and so you know that. And then, just make sure those ones that people are just rejecting out of hand, a sample, just a sample.

Mark: Okay, so that’s my question. So we never do that. Let me ask you how to do that, though. So that’s what we don’t do, and as you’re describing, I’m like, “Of course we should be doing that.” I’m trying to think of how we would do that, because the problem is, we reject 99 for every 1 we do.

Annie: Yeah, so you just — literally it’s a sample. You just take a random sample of them.

Mark: A random sample? Okay.

Annie: I mean, as long as it’s just, sort of, being kept in view a little bit. Because what that does is it basically just asks as — pushing against your model. You’re just, sort of, getting people to have the right kind of discussion. So all of that communicates to the people around you, like, “I care about your model.”

Evaluating options you didn’t take

Mark: So let me ask you a different question because you talk about these, sort of, groups of decisions, or portfolios of decisions. So the other question is — so early on in the firm, I happened to have this discussion with a friend of mine, and he basically looked at me and was like, “You’re thinking about this all wrong. You are thinking about this as a decision. You’re thinking about, ‘Invest or not?'” He said, “That’s totally the wrong way to think about this. [The way] you should be thinking about this is, is — is this 1 of the 20 investments of this kind, or of this class size, that you’re gonna put in your portfolio?” When you’re evaluating an opportunity, you are, kind of, definitionally talking about that opportunity. But it’s very hard to abstract that question from the broader concept of a portfolio or a basket.

Annie: Yeah, what I would suggest there is actually just doing some time traveling. That as people are really down in the weeds, to say, “Let’s imagine it’s a year from now, and what does the portfolio look like of these investments of this kind?” So I’m a big proponent of time traveling — of just making sure that you’re always asking that question, “What does this look like in a year? What does this look like in five years? Are we happy? Are we sad? If we imagine that we have this, what percentage of this do we think will have failed? We understand that any one of these individual ones could have failed, so let’s remember that.” And I think that that really allows you to, sort of, get out of what feels like the biggest decision on earth, because that’s the decision you have to be making, and be able to see it in the context of, kind of, all of what’s going on.

Sonal: That’s fantastic. One of the most powerful things my therapist gave me — and it was such a simple construct. It was, sort of, like, doing certain things today is like stealing from my future self. It blew my mind.

Annie: It’s so beautiful.

Sonal: It’s so beautiful. And it seems so, like, you know. Hokey — like, personal, self-helpy — but actually I had never thought of [it]. Because we’re on a continuum. By making discrete individuals — like, Sonal in the past, Sonal today, Sonal, this woman in the future I haven’t met yet. Wow. Like, the idea of stealing from her was, like…

Annie: That’s really a lovely way to put it.

Sonal: Isn’t that so — she’s a fucking awesome therapist, for the record.

Annie: Yeah, she is. I have an amazing therapist.

Sonal: I like talking publicly about therapy because I like lifting the stigma on it.

Annie: No, I’m very, very open about it.

Sonal: Me too.

Annie: Like, let’s not hide it. It’s totally fine.

Sonal: No. There’s no fucking reason to hide it, I totally agree.

Annie: Yeah. Some of the ways that we deal with this is actually prospectively, employing really good decision hygiene — which involves a couple of things. One is some of this good time traveling that we talked about, where you’re really imagining, “What is this gonna look like in the future,” so that that’s metabolized into the decision. Two is making sure that you have pushback once there’s consensus reached. Great, now let’s go disagree with each other. Then the next thing is, in terms of the consensus problem, is to make sure that you’re listening [to] as much input, not in a room with other people. So when somebody has a deal they wanna bring to everybody, that goes to the people individually. They have to, sort of, write their thoughts about it individually, and then it comes into the room after that.

Mark: As opposed to the pile-on effect that tends to happen?

Annie: As opposed to the pile-on effect, and that reduces the sort of effects of consensus anyway. So now this is how you then come up with basically what your forecast of the future is, that then is absolutely memorialized. Because that memorializing of it acts as the prophylactic. First of all, it gives you your forecast, which is what you’re trying to push against anyway. You’re trying to change the attitude to be that the forecast is the outcome that we care about. And it acts as a prophylactic for those emotional issues, right?

Which is now it’s like, okay, well, we all talked about this, and we had our red team over here, and we had a good steel man going on, and we, kind of, really thought about why we were wrong. We questioned — if someone has the outside view, what would this really look like to them? By eliciting the information individually, we were less likely to be in the inside view anyway. We’ve done all of that good hygiene — and then that acts as a way to protect yourself against these kinds of issues in the first place. Again, you’re gonna have a bad 24 hours, I’m just saying. Like, for sure. But you can get out of it more quickly, more often, and get to a place where you can say, “Okay, moving onto the next decision. How do I improve this going forward?”

Sonal: You make better and better decisions.

Mark: Yeah, so building on that, but returning real quick to my optimism, pessimism question. If society is not going to move on these issues, but we can move as individuals — so one form of optimism would be, more of us move as individuals. The other form of optimism could be, there will just always be room in these probabilistic domains for the rare individual who’s actually able to think about this stuff correctly. There will always be an edge. There will always be certain people who are, like, much better at poker than everybody else.

Annie: Oh, I think that’s for sure.

Mark: Okay. Because most people just simply can’t or won’t get there. Like, a few people in every domain might be able to take the time and have the discipline and will power to, kind of, get all the way there, but most people can’t or won’t?

Annie: I think that, in some ways, maybe that’s okay. Like, I mean, I sort of think about it from an evolutionary standpoint. That kind of thinking was selected for for a reason, right? It’s better for survival, likely better for happiness.

Mark: You mean the conventional wisdom of “don’t touch the burning stove twice.”

Annie: Yeah, or run away when you hear rustling in the leaves. Don’t sit around and say, “Well, it’s a probabilistic world. I have to figure out, how often is that a lion that’s gonna come eat me?”

Mark: Most people shouldn’t be playing in the World Series of Poker.

Annie: I have people come up to me all the time and be like, “Oh, you know, I play poker but it’s just a home game,” you know? And I’m like, “Why do you say ‘just a home game?’ There are different purposes to poker. You probably have a great time doing that and it brings you a tremendous amount of enjoyment, and you don’t have an interest in becoming a professional poker player. Just be proud of that, I think that that’s amazing.” Like, I play tennis. I’m not saying, “Oh, but, you know, I’m just playing in USTA 3.5.” I’m really happy with my tennis, I think it’s great.

So I think we need to remember that people have different things that they love. And this kind of thinking, I think that — I would love it if we could spread it more — but of course there are gonna be some people who are going to be ending up in this category more than others, and that’s okay. Not everybody has to think like this. I think it’s all right. So one of the things I get asked all the time is, like, “Well, we can’t really do this because people expect us to be confident in our choices.” <Yes.> Don’t confuse confidence and certainty. So, I can express a lot of uncertainty and still convey confidence. Ready? I’m weighing these three options: A, B, and C. I’ve really done the analysis. Here’s the analysis, and this is what I think. I think that option A is gonna work out 60% of the time. Option B is gonna work out 25% of the time, and option C is gonna work out 15% of the time. So option A is the clear winner. Now I just expressed so much uncertainty in that sentence.

Sonal: But also a lot of confidence.

Annie: But also a lot of confidence. I’ve done my analysis, this is my forecast. And all that I ever ask people to do when they do that is make sure that they ask a question before they bank the decision, which is — is there some piece of information that I could find out that would reverse my decision, that would actually cause — not that would make it go from 60 to 57. I don’t care modulating so much, I care that you’re gonna actually change.

Sonal: Right. And your point is that organizations can then bake that into their process.

Annie: Correct.

Sonal: And not just in the forecasting, but in arriving to that decision. So that then the next time they get to it, right or wrong, they make a better decision.

Annie: Right. And if the answer is yes, go find it. Or sometimes the answer is yes, but the cost is too high. It could be time, it could be actual…

Sonal: Opportunity costs, etc., right.

Annie: Whatever, exactly. So then you just don’t, and then you would say, “Well, then you all recognize as a group, we knew that if we found this out it would change our decision. But we’ve agreed that the cost was too high and so we didn’t.” So then if it reveals itself afterwards, you’re not sad.

Communicating probability to others

Sonal: Yeah, right. Well, you’ve talked a lot about how people should use confidence intervals in communicating — which I love, because we’re both ex-Ph.D psychology people.

Annie: Yes, exactly.

Sonal: Neither finished. So I love that idea. One thing that I struggle with, though, is — again, in the organizational context. If you’re trying to translate this to a big group of people, not just one on one or small group decisions. How do you communicate a confidence interval, and all the variables in it, in an efficient, kind of, compressed way? Like, honestly, part of communication in organizations is emails, and quick decisions — and yes, you can have all the process behind the outcome, but how do you then convey that, even though the people were not part of that room, of that discussion?

Annie: I think that there’s a simpler way to express uncertainty, which is using percentages. Now, obviously, sometimes you can only come up with a range. But for example, if I’m talking to my editor — and this is very quick in an email, I’ll say, “You’ll have the draft by Friday 83% of the time — by Monday, you’ll have it 97% of the time.” Those are inclusive, right?

Sonal: It’s another way of doing a confidence interval, but without making it so wonky.

Annie: Without making it so wonky. So I’m just letting her know — most of the time you’re gonna get it on Friday but I’m building, like, if my kid gets sick, or I have trouble with a particular section of the draft — or whatever it is — and I set the expectations for it that way.

Sonal: That’s fantastic. I mean, we’ve been trying to do forecasting — even for, like, timelines for podcast editing in episodes. And I feel frustrated, because I have a set of frameworks — like, if there’s accents, if there’s more than two voices. If there’s a complex thing, room tone, interaction, feedback, sound effects. I know all the factors that can go into my model, but I don’t know how to put a confidence interval in our pipeline spreadsheet for all the content that’s coming out and predicting it.

Annie: Yeah, so one way to do it is think about — what’s the range? What’s the earliest that I could get it? And you put a percentage on that. And then you think about the latest day they’re gonna get it, and you put a percentage on that.

Sonal: I love that idea.

Annie: And so now, what’s wonderful about that is that — it’s a few things. One is, I’ve set the expectations properly now, so that I’m not getting, you know, yelled at on Friday, like, “Where the hell is the draft?”

Sonal: Exactly, which is half the battle, I’ve learned that.

Annie: And a lot of what happens is that because we think that we have to give a certain answer, it ends up “boy who cried wolf,” right? So that if I’m telling her I’m gonna get it on Friday, and, you know, 25% of the time…

Sonal: Honestly, against your own best judgment sometimes even.

Annie: Right, 25% of the time I’m late, she just starts to not put much stock in what I’ve said anyway. So that’s number one. Number two is — what happens is that you really, kind of, infect other people with this in a good way, where you get them — it just moves them off of that black and white thinking.

Sonal: I love that.

Annie: So, like, one of the things that I love thinking about — and this is the difference between a deadline or, kind of, giving this range — is that I think that we ask ourselves, “Am I sure?” and other people, “Are you sure?” way too often. It’s a terrible question to ask somebody because the only answer is yes or no.

Sonal: So what should we be asking?

Annie: How sure are you?

Uncertainty in an organization

Sonal: How sure are you? I have a quick question for you on this, because earlier you mentioned uncertainty. How do you as an organization build that uncertainty in by default?

Annie: So first of all, we obviously talked a little bit about time traveling and the usefulness of time traveling. So one thing that I like to think about is not [to] overvalue the decision that’s right at hand — the things that are right sitting in front of us, right? So you can kind of think about it, like, how are you gonna figure out the best path? What is it, as you think about what your goals are? And, obviously, the goal that you wanna reach is gonna, sort of, define for you what the best path is. 

If you’re standing at the bottom of a mountain that you wanna summit — let’s call the summit your goal — all you can really see is the base of the mountain. So as you’re doing your planning, you’re really worried about, “How do I get the next little bit,” right? “How do I start?” But if you’re at the top of the mountain, having attained your goal, now you can look at the whole landscape. You get this beautiful view of the whole landscape, and now you can really see what the best path looks like. And so we wanna do this not just physically — like, standing up on a mountain, but we wanna figure out a cognitive way to get there, and that’s to do this really good time traveling. 

And you do this through backcasting and premortem. And now let’s look backwards, instead of forwards, to try to figure out — this is now the headline. Let me think about why that happened. So you could think about this as a simple weight-loss goal. I wanna lose a certain amount of weight within the next six months. It’s the end of the six months, I’ve lost that weight. What happened? I went to the gym, I avoided bread, I didn’t eat any sweets. I made sure that, you know, whatever. So you now have this list. Then in pairing with that, you wanna do a premortem, which is — I didn’t get to the top of the mountain. I failed to lose the weight. I failed to do whatever it is.

Sonal: And then all the things you can do to counter-program against that?

Annie: Exactly, because that’s gonna reveal really different things. It’s gonna reveal some things that are just, sort of, luck, right? Let me think — can I do something to reduce the influence of luck there? Then there’s gonna be some things that have to do with your decisions. Like, I went into the break room every day and there were doughnuts there, so I couldn’t resist them. So now you can think about, how do I counter that, right? How can I bring other people into the process, and that kind of thing? 

And then there’s stuff that’s just — you can figure out it’s just out of your control. It turned out I had a slow metabolism. And now what happens is that you’re just much less reactive, and you’re much more nimble, because you’ve gotten a whole view of the landscape. And you’ve gotten a view of the good part of the landscape and the bad part of the landscape. But I’m sure, as he’s told you, people are very loath to do these premortems, because they think that the imagining of failure feels so much like failure that people are like, “No, and you should, you know — positive visualization, and we should…”

Sonal: I mean, the fact that in brainstorming meetings everyone’s like, “Don’t dump on an idea.” But the exact point is you don’t have to dump on an idea and kill the winnowing of options.

Annie: No.

Sonal: As part of the process you should be, then, premorteming it.

Annie: Exactly. There’s wonderful research by Gabriele Oettingen that I really recommend that people see. The references are in my book. And across domains, what she’s found is that when people do this, sort of, positive fantasizing, the chances that they actually complete the goal are just lower <Interesting!> than if people do this negative fantasizing. And then there’s research that shows that when people do this time travel and this backwards thinking — that increases identifying reasons for success or failure by about 30%. You’re just more likely to see what’s in your way.

So, for example, she did — one of the simple studies was she asked people who were in college, “Who do you have a crush on that you haven’t talked to yet?” She had one group who, you know, it was all positive fantasy. So, “I’m gonna meet them, and I’m gonna ask them out on a date, and it’s gonna be great. And then we’re gonna live happily ever after,” and whatever. And then she had another group that engaged in negative fantasizing. “What if I ask them out and they said no? Like, they said no and I was really embarrassed,” and so on, so forth. And then she revisited them, like, four months later to see which group had actually gone out on a date with the person that they had a crush on. And the ones that did the negative fantasizing were much more likely to have gone out on the date.

Sonal: That’s fantastic.

Annie: Yeah. So one of the things that I say is, like, look — when we’re in teams, to your point, we tend to, sort of, view people as naysayers, right? But we don’t want to think of them as downers. So, I suggest — divide those up into two processes. Have the group individually do a backcast. Have the group individually write a narrative about a premortem. And what that does is, when you’re now doing a premortem, it changes the rules of the game, where being a good team player is now actually identifying the ways that you fail.

Sonal: I love what you said because it’s like having two modes as a way of getting into these two mindsets.

Annie: Right, where you’re not stopping people from feeling like they’re a team player. And I think that that’s the issue, as you said. It’s like, don’t sit there and crap on my goal. Because what are they really saying? You’re not being a team player, so change the rules of the game.

Sonal: You have this line in your book about how regret isn’t unproductive. The issue is that it comes after the fact, not before.

Annie: So the one thing that I don’t want people to do is think about how they feel right after the outcome, because I think that then you’re gonna overweight regret. So you wanna think about regret before you make the decision. You have to get it within the right timeframe. What we wanna do instead is, right in the moment of the outcome, when you’re feeling really sad, you can stop and say, “Am I gonna care about this in a year?” 

Think about yourself as a happiness stock. And so if we can, sort of, get that more 10,000-foot-view on our own happiness, and we think about ourselves as — we’re investing in our own happiness stock — we can get to that regret question a lot better. You don’t need to improve that much to get really big dividends. You make thousands of decisions a day. If you can get a little better at this stuff — if you can just, you know, de-bias a little bit, think more probabilistically — really, sort of, wrap your arms around uncertainty, to free yourself up from, sort of, the emotional impact of outcomes — a little bit is gonna have such a huge effect on your future decision making.

Sonal: Well, that’s amazing, Annie. Thank you so much for joining the “a16z Podcast.”

Mark: Thank you very much.

Annie: Yes, thank you.

• 

  Annie Duke
• 

  Marc Andreessen is a cofounder and general partner at a16z. Marc co-created the highly influential Mosaic internet browser and cofounded Netscape.

  Follow Twitter
• 

  Sonal Chokshi is the editor in chief as well as podcast network showrunner. Prior to joining a16z 2014 to build the editorial operation, Sonal was a senior editor at WIRED, and before that in content at Xerox PARC.

  Follow Twitter