John List is chief economist at Walmart (a title he previously held at both Lyft and Uber) and a professor of economics at the University of Chicago. He recently published a book titled The Voltage Effect: How to Make Good Ideas Great and Great Ideas Scale.
In this interview, List discusses some of the book’s themes, such as the importance of knowing when to quit or pivot, and how practicing the science of scaling can help ensure an idea’s success. He also shares his thoughts on the relationship between economics and technology, the state of behavioral economics and data science, and the prospect of using AI to reanimate promising, but previously unsuccessful ideas.
FUTURE: Can you briefly explain “the voltage effect” – the concept, as well as the book – and how it’s different than just analyzing whether something succeeded or failed?
JOHN LIST: The Voltage Effect is a book on ideas. And it begins with the premise that scaling is a science, not an art. And within that science, there are scientific principles that we should understand about ideas that scale and ideas that don’t scale.
The first half of the book talks about the five signatures, or the five vital signs, of ideas that have features that are scalable. That ranges from making sure your idea has voltage to begin with – that people want it – and it extends all the way to the supply side of scaling. Which is: “What are the marginal cost features of your idea as you expand? Do you have economies of scale or diseconomies of scale?” So, it really spans the demand side and the supply side, in a scientific way, that gives the scholar or the policy maker or the VC or the business planner a checklist to understand what are the key features of ideas that have made it, historically, and ideas that haven’t.
What it’s not is a book about execution. It’s a book that says, “If you execute, these are ideas that will scale.” Of course, if you don’t execute, it doesn’t much matter. You can have a great idea that has wonderful vital signs but it might not scale if you don’t execute.
After you’ve launched your idea, the back half of the book talks about simple economic tools that you can use to maintain high voltage at scale. So, questions like “What are the best incentives to use?” and “What is the best way to think?” (And that, of course, is to think on the margin rather than use averages, and it goes through several examples of that.) It talks about the “optimal quitting rule,” by which I mean understanding or having some principles you can follow in determining when and how you pivot.
And then the final chapter is about building a culture. Culture, from the very beginning, can be built in a way that is sustainable at scale, or you can build a culture that’s very difficult to grow with the company.
We should be constantly spanning the opportunity set and asking, ‘Are there better opportunities out there for me?’
How would an organization go about setting itself up to ensure that it’s able to maximize voltage as much as possible for its ideas or initiatives, or to predict that something’s not going to scale or doesn’t have voltage?
It’s a multi-dimensional problem, in the sense that the world changes and what used to be a great idea might no longer be an idea that is viable and that can scale.
I think the key point there is you have to look at your opportunity set. A lot of times people, when they quit or pivot, they just look at their current lot in life and say, “Does it get soiled?” If it gets soiled, they quit. I think you want to pivot or move just as often when your opportunity set gets better and when your opportunity set dictates that you should be pivoting.
It isn’t natural for people to think this way, because we tend to neglect our opportunity cost of time. Here’s an example: I used to be the chief economist at Lyft and, most of the time, people think, “OK, chief economist at Lyft – that’s really a lot of fun.” But I understood that every day I was a chief economist at Lyft, that meant I couldn’t be a chief economist at a different firm. I’ve recently accepted a position at Walmart to be its chief economist, and I left Lyft. Not because it was soiled or it was a bad firm – it’s not, the people at Lyft are wonderful – but because my opportunity set got better. We should be constantly spanning the opportunity set and asking, “Are there better opportunities out there for me?”
You constantly evolve and build on your data-generating process. The best firms don’t even do this all the time yet, but it’s what everyone needs to do.
How does the job change as you switch sectors between academia, ridesharing, and now retail?
I was a chief economist of Uber for two years and then I moved to Lyft for four. That’s kind of the same – it’s rideshare plus a combination of driver-side incentives, rider incentives, pricing, and strategic mission. It’s thinking about what markets to expand to, what technologies to expand to, and what technologies to contract.
At a company like Walmart, it’s more or less business as usual. Economics is life and life is economics, so economics is going to be everywhere. But there might be different emphases. For example, I think Walmart is going to be very proactive in the tech space and that last-mile delivery, for example, will be relatively new there, whereas we sort of knew how to do that at Uber and Lyft. But you have similar economic incentives driving both sides of that market.
The interesting thing about Walmart is that roughly 90% of Americans live within 10 miles of a Walmart, so you have this comparative advantage to leverage by applying economic thinking and big data by leveraging spatial considerations. Walmart just has a much bigger sandbox to play in. It has international markets, it has new tech, it has 4,000-plus stores. If it was a country, Walmart would have one of the largest GDPs in the world – it would be like Belgium. It’s the largest private employer in the United States. So, you have a question around every corner that economics can help you answer.
I think the big question entering Walmart is: Where do you start? That wasn’t really the case at Uber or Lyft; we sort of knew what we wanted to do there.
Wherever I am, the tool of choice for me is big data: examining big data that the firm has on hand, and also generating big data to provide causal insights where the naturally occurring data fail. What I mean by that is using field experiments to generate mounds and mounds of data that are actionable. A lot of times I see people doing A/B testing and explorations, but they really don’t have a plan for how they’re going to use their data. You need to actually use a bit of game theory – backward induction – and say, “Look, if my data say this, here’s what I’m going to do. If instead the data have this particular causal pattern, then I am going to veer in this way. And, after I make those decisions, this is the next experiment that I’m going to run.” And you constantly evolve and build on your data-generating process. The best firms don’t even do this all the time yet, but it’s what everyone needs to do as an ingrained element of their fabric, their culture of experimentation.
Never undervalue or underrate luck, but luck is more important when you’re using art and luck is less important if you use science.
When big data first came into vogue, Walmart was often cited as a company that had a lot of data and knew what it was doing. But that was a different generation of data analysis, compared with how a company formed more recently thinks about data …
You’re right. When you walk into Uber, you’ll see a sign that says, “Data is our DNA.” It’s not just a slogan, they live that. Walmart in many ways is that firm, but wants to fully embrace that as a cultural identity, and I think it will, but the starting point is important. Where firms or organizations begin culturally has a lasting influence. How you set that up is going to forever influence your genetic makeup.
You can evolve, of course, but you can see the difference between companies where data is in the DNA and those where it’s not. Walmart has the best talent around, and the exec team is world-class, so my job will be partly to ingrain a culture of experimentation, big data, and action.
Back to the voltage idea … How many ideas or ventures do you think are successful because companies planned well and looked at the data, rather than that they got lucky?
When you walk into a billiards hall, the best billiards players look like they have all the physics figured out. But, of course, they don’t. Some people naturally have some of the physics figured out, and some people learn by doing – they make mistakes, experiment, and adjust. Is there a fair amount of luck paired with the billiards player who’s naturally skilled? Absolutely.
And in the course of business, of course, you have some people who got it all right – they have all five vital signs. Did they know it from the beginning? I don’t know, but I don’t expect they did. They either learned it through trial and error, or they stumbled upon it via luck.
In this manner, I don’t think luck should be underrated. Luck is extremely important in every venture. What I hope my book helps to do is add more skill and science so you don’t have to rely on luck. Remember, historically, when we talk about scale, it’s “Move fast and break things.” “Throw spaghetti against the wall.” “Fake it til you make it.” And that’s art, that’s flat-out art.
So, never undervalue or underrate luck, but luck is more important when you’re using art and luck is less important if you use science. When it comes to the problem of scaling, however, the science of using science has never really been top of mind for people. Too often, people and organizations want to turn their great ideas into reality without first getting their hands dirty figuring out whether they’ll scale and how to best implement them. I want to change that.
I don’t think technology is causing people to make new kinds of mistakes . . . but technology gives you new ways to make those errors.
Today, if you’re savvy enough or clever enough or inventive enough, you can generate a lot of data – especially digital data – to feed whatever experiments you want to run. How has this changed how you think about behavioral economics?
A criminal is more dangerous if you allow them to purchase weapons or you give away weapons for free. We know that on the supply side. In the same way, a data analyst is really dangerous now because, at the drop of a hat or the push of a button, they can have mounds and mounds of data and they can get anything to be statistically significant. Because as the sample size goes to infinity, you can get statistical significance every time unless the estimate truly is zero.
So, on the one hand, it has made life easier for people who want to analyze data. We have statistical packages that are easy to use and data that is easy to download. But it has also made it easier to be thoughtless and create nonsensical results and correlations that people want to argue are causal. Remember the old adage: Once you have confidence and ignorance, success is ensured.
And we really need to take great care, because this is still econometrics and I still want to determine what causes what and I want a causal parameter to make a decision. If I have a correlation, it’s not necessarily going to change when I tweak something, because it could’ve been a third variable that’s causing them both. Just tweaking one variable doesn’t necessarily lead to any other changes when you do it yourself versus when nature did it. When nature does things, you have a lot of other variables at play; those correlations will not always be manifest when you tweak the variable itself. A correlation in nature doesn’t equate to a causal relationship.
I see this mistake made all the time, and it’s made much, much easier by technology and by big data being available at the drop of a hat.
So more data isn’t always better?
That’s right. Most of the time, people pride themselves on collecting big data. But the real value is in the refining of data. Oil is valuable, but it’s a lot more valuable because of the oil refinery. That’s what gives it its value. Data is no different. Data, in and of itself, is useless unless you have a good refinery.
If the refinery is top notch then, of course, I would like more data because that allows me to make more precise causal statements. But more data alone can be quite dangerous, as the reckless data refiner then has sharper messages. Think of it this way: a misguided missile is more dangerous the faster it travels.
You can evolve, of course, but you can see the difference between companies where data is in the DNA and those where it’s not.
How has the advent of data science as a discipline changed things?
Well, data science used to be called econometrics. Back in the last 40 or 50 years of the 20th century, economists were econometricians and they focused on trying to generate causal relationships. The new buzzword is trying to do some of the same things, but you call yourself a data scientist. Yet, data scientists begin with correlations often end there. Correlations are interesting, but they’re not as important as most people say they are. To me, they are the beginning, rather than the end of using science to explore data.
You’ve suggested that artificial intelligence and/or automation, to some degree, could help resurrect good ideas that didn’t or couldn’t scale the first time around. What types of scenarios are you thinking about?
I think, in the past, there might have been a series of ideas that a smart person, or a smart set of people, tried to grow into something big. But because of a constraint, whether it was human capital or the consumer base or some other infrastructural constraint, they threw their hands up and said, “We have to pivot. Let’s leave that idea along the wayside and let’s go for something else.” And I wonder, now that we have this fantastic explosion of technology – in particular, technology that is intelligent and can substitute for humans (in some ways) and other inputs that are very scarce – whether you begin to open up the box of ideas that we have discarded over the past several decades. Now that the world has changed and we have better and cheaper alternatives for certain tasks, are those old ideas we scrapped in the past now viable to scale?
I can see a fund doing something like that and I would bet there’s an expected value that’s positive – even an expected value that is greater than the opportunity cost of those funds.
A data analyst is really dangerous now because, at the drop of a hat or the push of a button, they can have mounds and mounds of data and they can get anything to be statistically significant.
Given the emphasis on economics today and all the data we have, are we actually closer to understanding consumer behavior? Or do our irrationalities continue to evolve as our cultures and technologies evolve?
Yes, I think we are dramatically improving our understanding of consumers, and in one dimension especially. In the past, economists have said, “Well, if you buy a tee shirt for $10, that’s your contribution to the market demand curve.” They were then done. Now, we are beginning to use the experimental method and field experiments, for example, to understand the underpinnings for why that particular consumer purchased that tee shirt. Is it for yourself? Is it a gift? Is it for your son or your daughter?
This becomes very important because now you not only know the contribution to market demand, but you also begin to understand the whys behind the purchase. And after you uncover the whys, you begin to bring forward a much better suite of incentives or policies that can help change the world for the positive or help change your firm.
In terms of irrational behaviors, while there are still irrationalities, a lot of them are predictable. What I mean by that is if a consumer has a certain non-standard preference – say, loss aversion – then it’s likely that the consumer across the street will have similar preferences. The same reasoning follows with many mistakes that are predictable. With field experiments, we can begin to understand who’s making mistakes, how, and then why. Why are they making the mistake about choosing the wrong insurance program or the wrong 401k program?
I don’t think technology is causing people to make new kinds of mistakes (sure, on the margins a little bit if it makes things more confusing), but the bigger mistakes tend to be cognitive mistakes that humans have made for decades. The primitives are still kind of the same, but technology gives you new ways to make those errors. However, it also gives us new ways to discover them and treat them in markets.
Views expressed in “posts” (including articles, podcasts, videos, and social media) are those of the individuals quoted therein and are not necessarily the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation.
This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets) is available at https://a16z.com/investments/.
Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.