Listen to the Stepchange Show on Apple Podcasts, Spotify, YouTube, or wherever you listen to podcasts.
Welcome to the third episode of the Stepchange Show—where we explore the technologies and systems that have transformed human civilization.
Every time you stream a movie, send a text message, scroll a feed, or chat with your favorite AI, you’re touching an invisible, physical empire. We call it the cloud, but it isn’t in the sky. It is astonishingly physical—alive in over 12,000 buildings around the world, consuming almost five percent of U.S. electricity, and running through cables laid across the ocean floor.
This is the story of data centers. From the humming punch-card rooms of the 1930s to the Cold War projects that accidentally birthed the internet, and onward to the gigawatt-scale AI factories of today, data centers have quietly become the industrial engine of our era. Six companies—NVIDIA, Microsoft, Apple, Alphabet, Amazon, and Meta—now dominate global markets in part because they command this infrastructure, just as railroads, steel, and oil once defined the fortunes of the last century.
Data centers are the machines behind the modern world. They shape commerce, media, communication, and now artificial intelligence, and raise profound questions about energy, water, and climate.
Join us as we uncover the physical reality of our digital world.
Four main parts:
Part I — Early Mainframes and the Birth of the Internet - 00:00:00
Part II — The Utility of the Cloud and the Hyperscalers - 01:16:13
Part III — The COVID Acceleration + AI Frenzy - 02:32:48
Part IV — Themes & Reflections - 03:40:10
Chapter times:
00:00:00 - The Invisible Empire of Data Centers
00:03:11 - IBM Punch Cards: The First Nerve Center
00:17:12 - SAGE: Connecting the Mainframes
00:27:27 - The APARNET: The Cold War Births the Internet
00:49:15 - The .Com Boom: The World Gets Wired
01:16:13 - AWS: The Utility of the Cloud
01:36:25 - Google: Warehouse Scale Computers
01:58:28 - Microsoft: From Software to Services
02:18:19 - Facebook: Sharing the Blueprints
02:32:48 - COVID: Five Years of Growth in 18 Months
02:45:53 - AI: The Dawn of the AI Age, Powered by NVIDIA
02:53:46 - Power & Water: The Gigawatt Problem
03:10:48 - Chips & Data: The New Cold War
03:19:18 - Climate: The Emissions of Data Centers
03:33:43 - Present Day Scale: Compute, Storage, and Communication
03:40:10 - Themes: Our Takeaways and Reflections
Thank you for joining us for the third episode of Stepchange. Don’t forget to subscribe.
Hosted by Ben Shwab Eidelson and Anay Shah. We would love to hear from you at hi@stepchange.show.
Transcript
This was autogenerated and edited with LLMs. Please be aware that there may be mistakes or typos.
Ben Shwab Eidelson (00:00:00): All right, Anay. Well, kids are in school.
Anay Shah (00:00:04): Kids are in school. We survived the summer. I love how back to school kind of feels like the beginning of a new year.
Ben Shwab Eidelson (00:00:11): It does. I always thought this time of year was the actual new year.
Anay Shah (00:00:16): I like that.
Ben Shwab Eidelson (00:00:28): Welcome to the third episode of the Step Change Show. We're here to cover the stories of human progress. We want to understand the technologies, systems, and infrastructure that shape our world. I'm Ben Eidelson, a co-founder of Step Change Ventures, a fund that invests in the companies that are accelerating today's biggest step changes. I'm based up in Seattle, Washington.
Anay Shah (00:00:46): And I'm Anay Shah, fellow co-founder of Step Change Ventures, based in Los Angeles, California. Today we're going to be telling the story of the hyperscale-up of a recent infrastructure: the story of data centers. Every time you stream a movie, send a text message, call a car to pick you up, or talk to what feels like a fully formed computational consciousness, you are touching an invisible physical empire. We call it the cloud, but it isn't in the sky. It lives somewhere very, very real.
Ben Shwab Eidelson (00:01:20): It lives in nearly 12,000 buildings worldwide, consuming almost 5% of electricity in the U.S. It also lives inside these garden-hose-sized cables that are laid across the darkest parts of the ocean floor. Today we are going to tell the story of this invisible infrastructure—a story that begins in the humming and clattering punch card rooms of the early 1900s, winds through the Cold War projects that accidentally birthed the internet, and leads to the gigawatt-scale AI factories the size of Lower Manhattan that are being built today.
Anay Shah (00:01:50): The U.S. stock market is worth around $60 trillion, made up of over 4,000 companies. But there are just six companies at the top—Nvidia, Microsoft, Apple, Alphabet, Amazon, and Meta—that make up 30% of that market. They are today's railroads, steel, and oil companies, building the modern industrial engine of our time.
Ben Shwab Eidelson (00:02:12): This is the story of data centers. But before we do, just one quick note: We have all of the research links and notes for this episode up at StepChangeShow. And if you are listening to this and you think of a friend or colleague who might enjoy it, please send it over their way. All of this is a new endeavor for us, and we appreciate it getting into the hands—or rather ears—of folks who may dig it. All right, so data centers.
Anay Shah (00:02:36): So Ben, I was recently reading an article saying these new technologies are saving work for everyone nowadays. Pretty soon we'll have nothing to do at all.
Ben Shwab Eidelson (00:02:44): That's right. I think they were saying that we have some new electronic brain that's going to make most workers obsolete.
Anay Shah (00:02:49): Right. But the CEO behind one of these technologies quickly countered, saying that it was a way to save time, not replace jobs. He said it was a small tool to help great minds benefit mankind.
Ben Shwab Eidelson (00:03:00): Was this the CEO of Anthropic, OpenAI, or maybe Google?
Anay Shah (00:03:04): You might think so, but no. This was Thomas J. Watson Sr., the original leader of IBM.
Ben Shwab Eidelson (00:03:11): So, IBM, the computational powerhouse of the last century. Long before we had the Googles, Amazons, Metas, and Microsofts, there was Big Blue IBM. And long before IBM ever built an electronic computer, they were doing computation of a different sort.
Anay Shah (00:03:32): And if you walked into a large company back in the 1930s, you might find yourself in one of their windowless rooms, humming and clattering like a small factory. There'd be long rows of metal cabinets, whirring gears, and clerks feeding stacks of stiff paper cards into these behemoth machines. Each card was a sliver of information: an employee's hours, an invoice, maybe a customer's address. But together, they formed the first centralized nerve center for corporate data.
Ben Shwab Eidelson (00:04:05): And I think that's why we argue that those rooms, with those cards of information being processed on these devices that were doing computing, are the first real data centers.
Anay Shah (00:04:17): That machine has quite an interesting history, born to solve a government problem: the problem of the U.S. census.
Ben Shwab Eidelson (00:04:26): It's wild. The 1880 census data was collected in 1880, and it took seven years before it was tabulated. They realized it just was not working at human computation scale anymore.
Anay Shah (00:04:37): And so one man, Herman Hollerith, had an invention to solve this.
Ben Shwab Eidelson (00:04:42): He observed the way railroad conductors checked tickets, noting that these were papers with punched-out holes that ultimately stored and represented information. If you could build a machine that could count the number of punch holes, you could then do computation at scale without needing humans to count markdowns on a form.
Anay Shah (00:05:05): That's right. These cards stored information, and then you could have machines compute off of them. The 1890 census was the machine's first big break. Even with a population 25% larger than the decade before, it was completed in two years instead of more than seven years, and it came $5 million under budget. That's not something a new technology could typically achieve for a government. So, fast forward to Watson. He sees the punch card machine and understands what it can do. By the mid-1920s, he's convinced.
Ben Shwab Eidelson (00:05:37): So by 1924, he rebranded the company International Business Machines.
Anay Shah (00:05:42): A much more fitting name than CTR.
Ben Shwab Eidelson (00:05:47): Computing-Tabulating-Recording Company.
Anay Shah (00:05:50): "There is no limit for this tabulating business," he told his executives in 1927. He doubled down, selling off all the other less promising lines of business and pouring the company's resources into the tabulating division. They designed a proprietary 80-column IBM card that only worked on IBM machines, which became the IBM card—a sort of early-days lock-in where customers who wanted to use the machines had to buy the cards.
Ben Shwab Eidelson (00:06:22): Everyone obsesses over the Gillette razor blade business model, but the IBM card was not only how you did the computation, but it also stored information and became your record-keeping method inside businesses running on these IBM machines.
Anay Shah (00:06:35): Once you put invoices, time cards, and all of that information onto the IBM card, you're not migrating it off.
Ben Shwab Eidelson (00:06:41): It's on a piece of paper.
Anay Shah (00:06:42): Yeah, that's right. So now you've produced a very steady stream of revenue alongside selling the tabulator machine. The timing for this was perfect. This is the late '20s, early '30s. The New Deal brought in a massive government focus around record-keeping. In 1935, the Social Security Administration signed a contract with IBM, requiring millions of their cards and machines to process benefits and print checks. One of their New York plants was soon printing 10 million cards per day.
Ben Shwab Eidelson (00:07:14): It's amazing.
Anay Shah (00:07:15): And within a couple decades, most large companies had punch card rooms. They had various machines they used to sort, tabulate, and store financial information, payroll information, employee information, customer information—everything you could think of. In order to manage that, they'd hire clerks and technical specialists to operate and maintain it.
Ben Shwab Eidelson (00:07:36): It's easy to take for granted, but this was the first time a company, its accounting, and its invoicing could be calculated at this scale. Think about what was going on in this era: companies were starting to scale in new ways. More global trade was also starting to happen. Thomas Watson Sr. ended up very interested in diplomacy, traveling around the world trying to preach that commerce across borders would create peace. Unfortunately, this backfired. He did a lot of commerce with Germany in the lead-up to the Holocaust, and there are many stories around IBM's computational power being used to help Germans with record-keeping. Ultimately, this product was foundational to so much of the scale of this era, including the scale of war. Every time there was a war or a big government project, regardless of which country was fighting, they needed IBM machines to do calculations.
Anay Shah (00:08:34): This was the way to process and store information.
Ben Shwab Eidelson (00:08:38): But it was all still cards and mechanical switches, ultimately.
Anay Shah (00:08:42): And rooms full of paper.
Ben Shwab Eidelson (00:08:44): Rooms of paper. So everyone was running on punch cards. What was this thing? What was this room? What should we call it?
Anay Shah (00:08:50): This is where we're going to put our stake in the ground and say the earliest data centers came out of the IBM punch card data center. The easiest way to understand it is: it's a physical space or a collection of facilities designed to house and operate an organization's data and computing infrastructure. You can think of it as primarily storing information, computing information, and over time, connecting and allowing for the communication of information. At its basic core, a data center in the evolution we're going to talk about from the mid-1900s to the present day is the physical space around storage, computing, and connectivity.
Ben Shwab Eidelson (00:09:33): At this moment, IBM was already a growing company of import, right? Doing global international business. By 1945, they had around 25,000 employees and annual revenue of approximately $140 million. But something was about to change, and that was also coming out of the war. World War II was this big moment of government investment in R&D. One of the things they often found themselves doing was calculating artillery firing tables, which took into account the wind speed, weather, and all these different factors to figure out where they should shoot artillery. There were teams of human computers, clerks who were doing mechanical calculation to figure things out. But that was a bottleneck. So the army funded a team at the University of Pennsylvania to build the first real electronic computer, a machine that could crank through those calculations an order of magnitude faster. That became the Electronic Numerical Integrator and Computer, also known as ENIAC.
Anay Shah (00:10:33): This was all about math. It was focused on calculating numbers and producing mathematical results that would help them, in this case, be more efficient in war planning.
Ben Shwab Eidelson (00:10:44): Yeah, I mean, they didn't have the...
Anay Shah (00:10:45): TI-83 or the games that go on it.
Ben Shwab Eidelson (00:10:48): It, or the games that go on it. They didn't have Tetris. So they needed a calculator that could go faster than the people who were a bottleneck at this point. The ENIAC had no mechanical parts that slowed its operation. It took up nearly a 2,000-square-foot room, but it did operate over 1,000 times faster than any previous computational device. It could execute 5,000 additions per second. At this time, IBM, with its commercial product—the quickest punch card machine—could complete just four additions a second. So it wasn't an order of magnitude faster; it was three orders of magnitude faster.
Anay Shah (00:11:25): Four additions per second to 5,000 additions per second. This seems like something mathematicians and CEOs privy to this would think, "Oh, I can see the future." But no. A distinguished Harvard mathematician dismissed the idea as foolishness, saying there would be no big market for computers. He believed that the country would need maybe half a dozen, mainly for military and scientific research. Our friend Thomas J. Watson Sr. said general-purpose computers had nothing whatsoever to do with IBM or IBM's mainline of equipment and profitability.
Ben Shwab Eidelson (00:11:59): But Watson's stubbornness to stay wedded to the past was only matched and maybe outdone by his son's stubbornness to push the company forward.
Anay Shah (00:12:09): Runs in the family, but takes a different shape.
Ben Shwab Eidelson (00:12:12): This is a father-and-son story for the ages. Thomas Watson Jr. eventually fought in the war, came back, and was ready to take his seat as a leader inside IBM. He viewed electronics as the future of the company, and this became the intergenerational battle for the control of IBM. Thomas Watson Jr. saw that there was a future for the company in electronics and wanted to push it that way.
Anay Shah (00:12:38): There's some interesting parent-child psychology where you can see him completely frustrated and angry that his father's not going to jump on what he thinks is very clearly the future.
Ben Shwab Eidelson (00:12:50): IBM did do a wartime project with Harvard mathematician Howard Aiken that was a kind of hybrid electromechanical vacuum-tube machine. They followed that with their first machine that they really wanted to demonstrate to the public after the war. This was called the Selective Sequence Electronic Calculator (SSEC). This was in 1948. They showed that in a Madison Avenue showroom, right in the middle of Manhattan, so that people could walk by and see this thing computing.
Anay Shah (00:13:20): And to see this thing, these machines were still the size of rooms.
---
Ben Shwab Eidelson (00:13:25): And it was fast, not by today's standards, but by those days, it was very fast. It didn't have any memory in the sense that we were used to computers today having memory, but it had this punched paper tape as a form of storage. Despite Thomas Watson Sr. not liking this as the new business, he didn't mind the company getting some PR points for it. They'd send reporters down to watch the machine do computation. This is the moment when the press said, "Pretty soon no one's going to have a job to do as a result of this electronic brain."
Anay Shah (00:13:54): The electronic brain that would displace workers. And so, in 1951, IBM's longtime customer, the U.S. Census Bureau, went with Univac instead of IBM tabulators for its next census. Thomas Watson Jr. recalls this moment, saying, "My God, Univac is smart enough to start taking all the civilian business away!" This is really what shakes IBM to once again reinvent itself.
Ben Shwab Eidelson (00:14:22): You go back to 1880, 1890. This whole invention was for the census problem. And that problem gets harder: more people, more things you want to know about people, and more factors you want to track. And so, that is actually a perfect test bed for computation at this time. Here comes the ENIAC team, now commercialized with Univac, and that wakes IBM right up.
Anay Shah (00:14:41): And this company's been growing, right? IBM, in 1940, was doing about $45 million in revenue with 12,000 employees. And a decade later, they're doing $250 million in revenue with 30,000 employees. And if you're betting the company on a new direction, things could go south for a lot of people. It's a big responsibility for Watson Jr.
Ben Shwab Eidelson (00:15:02): And they pushed forward. And so, out of these early computer experiments, they finally built their first commercial product, which was the IBM 701 Electronic Data Processing Machine. This is the first real machine where they thought, "Hey, we're going to build more than one of these, and we're going to sell some of them. We're actually going to try and make this into a business, not just a research project."
Anay Shah (00:15:20): And as we'll see with some of the companies closer to the present day, they had an installed base of customers; they had a sales machine. And so, they started this business line, and within five years, they had 85% of the computer market.
Ben Shwab Eidelson (00:15:34): That's right. So, that team that invented the first vacuum tube computer in the ENIAC and then the Univac, they got outrun by the better commercial go-to-market sales machine that, if you remember, IBM was founded with. So, who was buying these things, and how many did they sell?
Well, the 701 was still a pretty bespoke product. They sold 19. These were going to National Labs, the Weather Bureau, and a lot of aerospace firms that were doing a lot of calculations. And the whole business model was actually not around buying these machines; it was an extension of the tabulating business model that IBM had always had, where they wanted you to lease the machine, rent it, and buy punch cards.
Anay Shah (00:16:08): We got hardware as a service.
Ben Shwab Eidelson (00:16:10): Exactly. It was a beautiful business model. The era of the '50s for IBM was going from this research-centered product to what I'd consider their Model T. So, in this era, they built the first disk drive. They started using tape in addition to punch cards. They wrote Fortran, which is the first real kind of modern programming language where you could write words, and it would get translated into computer instructions by the end of the decade.
In 1959, they launched the IBM 1401, which was available to thousands of companies. And they ended up selling around 12,000 of those machines. They exited the decade with a real...
Anay Shah (00:16:45): computing business, and they're dominating the market.
Ben Shwab Eidelson (00:16:48): So, that first machine shipped 19 systems. They had another machine that shipped 123. After that, they had the first transistorized Model T. The 1401 crossed over 10,000 units. From 1950 to 1962, IBM's revenue rose tenfold, from $260 million to $2.6 billion. And their headcount went from 30,000 employees to almost 130,000 employees. Woo!
Anay Shah (00:17:12): And so, what else is happening right now? We're in the 1950s; World War II has ended. The Cold War is getting colder. And we find that the Soviet Union's long-range nuclear-armed bombers are able to cross the Arctic and reach American cities in a matter of hours. So, this makes the decision window to detect and intercept down to two minutes.
Ben Shwab Eidelson (00:17:36): It's easy to forget, but the Cold War was primarily a technological arms race. Who's going to get there faster if something's going down? If the Soviet Union launches these bombers, how quickly can we respond, know about it, and then respond? And our Air Force and our defense systems were not designed for that speed.
Anay Shah (00:17:53): Yeah, every threat forced a technology upgrade, forced our government to respond, to invest. We had a laboratory project at MIT that, in 1951, could process live radar data in real time, proving that automation could close this gap that we were seeing in national security.
The challenge was: how do you take this radar research prototype and turn it into a 24/7 machine that can operate at government scale? And this is a moment that became the largest computing project to date.
Ben Shwab Eidelson (00:18:23): Thomas Watson Jr. saw this, and he knew that they needed to win this contract. IBM actually had a policy of essentially making only 1% profit on any defense work, which they kept throughout this. But they said this would push them to the future.
And so, the contract awarded in 1954 became one of the largest contracts ever, worth more than $500 million in 1950s dollars. Around $5.5 billion in today's dollars. And so, how did it work? What did it do at full deployment when they got there in the 1960s?
It was a network that spanned 27 different centers, where each center had a pair of special IBM computers designed for this. In case one of them was being serviced or down, the other one would become primary. So they built redundancy into the network. And the scale was unprecedented.
In total, it was 56 computers. These were acre-sized floors, with multi-megawatt power draw to these systems. And they were connected—connected over early modems. This was the moment that really drove the production of the modem, where you could actually have data sent over telephone lines. And so, they would lease special lines between these centers.
Anay Shah (00:19:32): And these centers have characteristics that we're going to talk about more. You have redundancy built into this; you have building-level scale; you've got multiple facilities networked together. These end up being a lot of the same characteristics that drive data center growth and evolution decades later.
Ben Shwab Eidelson (00:19:52): That's right. And I feel like there's the scale of it, but I think the connectivity of this system is the difference. You have storage; you have compute, which was happening; but now you had connection and communication, because you want this center over here to respond to information that was computed over there. And so, this moment in history—the SAGE system, I think—is the first real time that machines were communicating with each other. Pretty big moment.
Anay Shah (00:20:17): So, once we're able to do this in government, it's going to catch the eyes of other industries.
Ben Shwab Eidelson (00:20:23): The airlines had seen the SAGE project take off, and they were sitting there with a new problem on their hands. This was the '60s. Air flight was booming, and the way that reservations got processed was absolutely insane. You would call your travel agent; your travel agent would call the airline; and the airline would have to run clerks around, pulling cards out to book a seat. Supposedly it took up to 90 minutes.
Anay Shah (00:20:49): Well, you had to figure out who had the middle seat in Row 17.
Ben Shwab Eidelson (00:20:52): Who had the middle seat, exactly.
Anay Shah (00:20:54): How were you going to figure that out without looking up the card?
Ben Shwab Eidelson (00:20:56): As American Airlines scaled to a real operational scale, it started to break down. So, they reached out to IBM, and they kicked off this project that would be known as Sabre. It was essentially a commercial version of the SAGE system. Two IBM mainframes were purpose-built, and would be connected over phone lines. Those systems would house the source of truth around the reservations.
And there would be terminals. These weren't terminals with screens; these were terminals with paper where you'd still have a travel agent, but they would be querying over the terminal what seats were available. And things would be booked automatically. So, it would go from 90 minutes to seconds at a terminal. And it removed the clerk in the middle. This set the stage for e-commerce. This was the first time, I think, you were buying something over a computer.
Anay Shah (00:21:44): You had 90 minutes to do a reservation, now down to seconds. And this scaled American Airlines operations by the mid-'60s to be able to do a full 40,000 reservations per day.
Ben Shwab Eidelson (00:21:55): Insane. This system continues to this day to be part of the backbone of airline booking. It's quite an amazing foundational moment where commerce is happening between data centers.
And so, they were running airlines; they were doing these major government projects. Businesses were now buying the 1401 machine off the shelves. But there was a different problem. Entering the mid-'60s, all of these mainframes had their own accessories. There was no compatibility. So, in 1964, they launched the System/360. And that turned this chaotic, messy, ad hoc world of different mainframe models into a platform. It was essentially one architecture; you could choose how powerful you wanted...
Anay Shah (00:22:38): it to be, buy your base package, and then feature it up based on what you need.
Ben Shwab Eidelson (00:22:42): That's right. And so, for facility planners, they could now plan a room and a facility, and then be able to scale up the machine as needed. I believe roughly $5 billion of investment in R&D was spent in this era.
Anay Shah (00:22:55): Another "bet the company" moment.
Ben Shwab Eidelson (00:22:57): And it worked. They were shipping thousands per month of these devices in the '60s. And so, if you zoom out and look across the '50s and '60s, as we talked about earlier. From the '50s to '62, they scaled to $2.6 billion. By 1970, they were doing $7.5 billion in annual revenue and had a team of 270,000 employees.
Anay Shah (00:23:19): In today's dollars, that's $62 billion in revenue.
Ben Shwab Eidelson (00:23:23): And the market noticed. IBM was king at that moment. In 1970, IBM accounted for 6.8% of the total U.S. stock market. It almost feels like an unprecedented thing to say that today Nvidia is currently 7% of the U.S. stock market.
Anay Shah (00:23:44): These mainframes became part of the zeitgeist. They became part of a cultural milieu because they were often nicknamed "the glass houses." You'd have the mainframes and their operators in an enclosed room. And as we mentioned before, you'd have windows and glass walls around it because you wanted to show off how automated your systems were, how advanced your company was.
There was a recognition that these "glass houses"—these mainframes—were powering the billions of dollars of revenue from their enterprise customers. And so, the security, the operational control, the climate control, and cleanliness... It was all part of this cultural moment where mainframes and these early data centers became a meaningful part of how our society and economy ran.
Ben Shwab Eidelson (00:24:34): If any company was using it effectively, they had more demand for use than they had supply for the computer. And so, oftentimes, people would be sitting there waiting for their information to come back, for their computation to run. Sometimes it would be a day or longer, so the utilization was pretty high. But there was this long interactive latency problem.
And an invention that echoes through in multiple ways to this day changed the fundamental way that computing was thought of: from this single-threaded "I'm only working on one problem at once" to "No." The job is to make sure my hardware is as utilized as possible. As a machine, I don't care whether or not it's the same problem from the same person. I just need to be calculating.
Anay Shah (00:25:20): I need to be operating all the time to utilize my capacity. It can be for this problem; it can be for that problem. You can slot them in and out; it doesn't matter.
Ben Shwab Eidelson (00:25:29): That's right. And so, a team at MIT in the early '60s developed what was called CTSS, the Compatible Time-Sharing System, and turned it onto the campus in 1963. And so, students could sit there with electric typewriters and interact with this machine, multiple users at the same time, and feel like they had control over the computer.
Anay Shah (00:25:52): It felt like it was your own.
Ben Shwab Eidelson (00:25:53): And this was a huge moment. This was the first time there were logins, users, files, and instant feedback. And so, it was revelatory to be there in the mid-'60s, experiencing that after the idea of a computer was just this operated thing by someone else.
Anay Shah (00:26:09): That's right. You had to queue up, get in line, hand over your data to the operators—the specialists—who would then go and do their thing, and you would sit and wait. Now you're interacting. It's you and the computer.
Ben Shwab Eidelson (00:26:22): And the way this worked was that you didn't obviously have monopoly control over the computer. The computer was switching what problem it was working on, depending on what free cycles it had. The human perception is, "Oh, how can a computer do that?" Well, the computer is operating at a much faster frequency than we're able to realize. Even if it's something as simple as responding to a keystroke, there are plenty of milliseconds in between my keystroke and your keystroke for the computer to respond and then switch back to the other problem.
And as we'll see later in this story, this notion of intelligently slicing up the hardware and utilizing it becomes the backbone of everything in how the cloud and data centers are architected today. And it started with this timesharing innovation.
Well, we had these mainframes that had taken over the business world, and timesharing. So, multiple people could be on a campus connected to that mainframe, feeling like they had their own computer. They were still fully disconnected from anyone else in the world. They were still islands onto themselves. So yes, now on this island, multiple people could be there, but it's not like you could go visit the next-door island. What are we missing?
Anay Shah (00:27:27): It's something we take quite for granted. Many people in different places, but working on the same system at the same time. Now remember, we are in the late 1950s, in the Cold War, and Eisenhower wanted to ensure that the U.S. would not be blindsided by a technological surprise.
Ben Shwab Eidelson (00:27:47): There's this particular moment right when everyone looked up in the sky and for the first time, a man-made thing was floating in space, and that's Sputnik. The fact that the Russians beat us to the skies with Sputnik kicked off, I think, a level of panic from a geopolitical standpoint that's hard to connect to today.
Anay Shah (00:28:07): And so, after the panic of Sputnik, Eisenhower created the Advanced Research Projects Agency, or ARPA. And in those years, ARPA was pouring money into space, into missiles, into computing. And we wanted to ensure, in this nuclear-armed world, that weapons could not destroy a centralized command system. And so, the network itself needed to be decentralized.
This need to ensure reliability combined with a slightly more mundane problem that Bob Taylor faced when he was at his office in the Pentagon: he was working at three terminals. He had one machine connected to MIT, another connected to UC Berkeley, and a third for a different research system altogether. And he could talk to any of them, but never at the same time. So he was rolling his chair back and forth, and he's quoted saying, "We ought to find a way to connect all of these different machines." He told his boss. After a 20-minute meeting, Taylor walked out with a million-dollar budget and a simple mandate: to make it happen.
Ben Shwab Eidelson (00:29:10): And we won't go into every moment and step because the ARPANET itself is quite a story. But I think a couple of innovations set the stage for the Internet in a really important way. One is the concept of packet switching.
If you think about a phone line, AT&T built its whole business on this idea: you're going to connect a phone call between two people, and there's going to be a circuit that connects those two phones together. This is called circuit switching. But there's this crazy idea: if you want that resilient system so that if one node goes out, the next thing doesn't go out, you need to route things around in different ways and not be dependent on the one route you have.
So, you need to have more of a spiderweb-like network to say, "Well, from here to here, there are actually five, or ideally 50 or 5,000, ways to get between those two points." And to do that, you need a flexible communication system. And so, the idea is to break up information into packets—little pieces of mail that would get sent from one node to another. Different packets could even take different routes, but on the other side, someone would reassemble them into the message.
Anay Shah (00:30:15): It's the same singular message, broken up into many, many different pieces and routed through many, many different pathways.
Ben Shwab Eidelson (00:30:24): It's funny, I hadn't thought about this until we connected to the timesharing story, but it's actually about better utilization of the network. In a circuit-switched mode, you have this direct line; you and I have a direct line together, and you have a direct line to someone else. That line is usually empty. It's pretty hard to fully utilize and build out the circuit-switched thing because the phone call ends, and you think, "Okay, now it's just sitting there unused."
In packet switching, packets are finding their way through this wild world. So, a lot of skepticism obviously came from AT&T, which said, "If you want to have two computers talk to each other, we'll build the phone lines, and you can run them." But a crack team that had kind of spun out of MIT—this company, Bolt, Baranek and Newman—latched onto this packet switching model and ran forward.
And the core thing that they needed to build was what was called the Interface Message Processor, or IMP. This was ultimately the router that would sit in front of the host machine. Universities like MIT and Berkeley would have these mainframes. But to connect those machines together, there would need to be these nodes that would sit in front of them and be able to build this network. And so, BBN used a Honeywell minicomputer that was fridge-sized, cost about $80,000, and built the logic around packet switching and rolled this out. There were a lot of engineering heroics that went into getting the first machine ready. But then it was time.
Anay Shah (00:31:47): Almost 55 years ago, a small group of grad students gathered at UCLA to wait for the machine to be rolled off the truck, champagne in hand, and were able to celebrate the arrival of the first IMP. The second one was delivered to Stanford weeks later. And the prophetic first message, as the story goes, was meant to be "login," right? You now have users that can log in, but after inputting the first two letters, it crashed. And so, the first message was simply "lo," "lo."
Ben Shwab Eidelson (00:32:19): So close to "lol." And, of course, what is a network with two connections? Pretty limited. But then they rolled out UC Santa Barbara and the University of Utah, and then they were just rolling out new IMPs to the large universities, month over month.
Anay Shah (00:32:34): And these became nodes—almost a new node every month. And then every new node that came on the network amplified the value. Because now it's not just bidirectional communication; it's multilateral communication.
Ben Shwab Eidelson (00:32:47): So, they were building all this logic to do the resilient routing, to do the discovery of devices, and everything that you needed to do that. And of course, the original pitch was, "Hey, we've funded as a government all these expensive computers at all these research universities. Let's drive utilization." Maybe one team has a special program for doing something; another team has a special program for doing something. Let's let the departments connect.
Anay Shah (00:33:07): One unexpected application really bubbled to the top that the researchers discovered. Quick, asynchronous messages were often more valuable than logging into someone else's system to run the code. And in 1972, ARPA's Bob Kahn said, "Everyone really uses this thing for electronic mail." And the network's purpose had quietly shifted from sharing machines to connecting people in the form of email.
---
Ben Shwab Eidelson (00:33:36): It's wild how as soon as you have enough connectivity, it's always the killer app. The killer app is always people communicating at a distance.
Anay Shah (00:33:45): That's right. People wanting to do what we are biologically programmed to do, which is connect with other humans. And as sophisticated as these machines were, it was the simple act of asynchronous communication that became the killer app.
Ben Shwab Eidelson (00:34:02): And more people wanted it. There were new ways to connect within your local network over these terminals, so then you could connect into the broader network. It was no longer just about wanting access to a terminal to access the machine on your campus; it was about wanting a terminal so you could email your friends at the other campus.
Sounds like an early social network that we'll get to later. And so the expansion continued. By 1972, there were 29 nodes. By 1975, over 50.
Anay Shah (00:34:27): And some of these connections weren't just from university to university. We were now leaping oceans. It was clumsy, but we were able to go over the Atlantic Ocean from a node in Virginia to the one in Cornwall, England.
Ben Shwab Eidelson (00:34:41): There was some story where they came back from some conference in England and someone had left their electric shaver behind.
Anay Shah (00:34:47): It was the middle of the night, so in England it was 3 o'clock in the morning. But he knew that his colleague was a workaholic, so he sent off a message at 3 o'clock in the morning, England time, to see if this person was awake. And he was. He saw that he was logged on and he's like, "Hey, did I leave my electric razor there?"
Ben Shwab Eidelson (00:35:01): "Can you get my razor, please?" And it all worked. It was phenomenal.
So, by the early 1980s, other research networks started to pop up in the US and elsewhere. People were like, "Oh, this ARPANET thing is pretty cool. We're going to build our own." There was DECnet, there was an IBM Net, there were various nets that companies and other research groups started to stand up.
They had a new challenge: "We want to talk to everyone on any of these networks, and how are we going to do that?" And these IMPs were not designed for that. They had presumed there was only one network they were trying to build out.
And so Vint Cerf and Bob Kahn, who were involved in ARPANET, figured out what they needed to build was a common language to figure out addressing: where to send a packet of mail. And then also, the control protocol for sending information, to make sure duplicate packets weren't sent.
And so this became TCP/IP: TCP for the Transmission Control Protocol, and IP for the Internet Protocol. You can think of IP as solving the addressing problem. You might be familiar with your IP address, which then communicates through routers what you're looking for, and who you're trying to get the packet to. And then TCP is the reliable way that a packet gets acknowledged. "Hey, this little chunk of information has arrived." It's like certified mail.
Anay Shah (00:36:16): Like signing for my package. I have an address, so the package knows where to go. And then I signed for it.
Ben Shwab Eidelson (00:36:21): "And I don't need you to send me the package again, because I got it." And so this combination became the backbone for connecting all of these networks together, and ultimately the backbone for the next stage of not the ARPANET, but the Internet – a network of networks.
In 1983, they took every ARPANET host and said they needed to move to TCP/IP. That switch, on January 1, 1983, became the framework for the modern-day internet: many networks, all speaking the same language, able to connect.
Anay Shah (00:36:54): And that's where we have the advent of the .com, .edu, and .gov, because we now had standard protocols for multiple nets to communicate together.
Ben Shwab Eidelson (00:37:06): 20.3.1.72 doesn't have the same ring as pets.com.
Anay Shah (00:37:11): Pets.com, an important one, but we're not quite there yet. No, but by the mid-1980s, as Ben mentioned, there were multiple nets now connected. And ARPANET itself was showing its age. By 1990, it became apparent that the ARPANET needed to be decommissioned. But the work it had done, the ideas of decentralized networking, packet switching, and open protocols had firmly taken root. Research institutions and large organizations had firmly established how to exchange messages.
Ben Shwab Eidelson (00:37:41): So they were able to send files back and forth, maybe send each other's research papers and data sets. But this was not a thing with a web browser yet. We needed a structure around documents. And so Tim Berners-Lee, a researcher at CERN in Switzerland, proposed the document structure that would become the application-layer infrastructure of the World Wide Web.
Anay Shah (00:38:01): And so the network that started off as a Cold War research project in response to looking up at the sky and seeing the Russians launch Sputnik, had now become the public and commercial Internet – an open highway for data, and the foundation upon which we would build the modern data center economy.
Ben Shwab Eidelson (00:38:18): So we now have this expanding network, but the computing on this network was still this terminal-based interaction between a terminal and a mainframe that then connected to the network, or a terminal and a minicomputer which connected to the network.
And I think it's worth saying a bit more about minicomputers, which were both what became that IMP device and also a primary source of connection. These were not something that you'd buy for your house. They still were the cost of maybe a car when they first came out, but they were much more accessible than a big IBM mainframe, a room-sized computer.
And so the company, Digital Equipment Corporation, or DEC, launched the first minicomputers in the mid-60s. A lot of what innovated on the minicomputer was software like Unix, C programming, and sockets, and all these foundational innovations that would later echo through to today. But there was room for a computer that a normal person could buy.
Anay Shah (00:39:11): On one fateful day, January 1975, the cover of Popular Electronics magazine showcased the Altair 8800, a machine you could buy at home for $439, or $1,500 souped up, to run your own code. And there were a few hobbyists that went to the newsstand and picked up this magazine. Two in particular saw the cover and decided to do something about it.
Ben Shwab Eidelson (00:39:42): And so Bill Gates and Paul Allen, famously, when they were at Lakeside High School in Seattle, they had access to a computer and a terminal, and they would spend hours and hours programming. But that's different than having one in your own house that you can play with.
And so when they saw this come out, it was immediately clear that this was a moment when computing had gotten cheap enough that a normal person could afford their own personal computer, or PC.
Anay Shah (00:40:06): And you could see that if the first one was released at $450, it was only a matter of time before that would become accessible to more and more people.
Ben Shwab Eidelson (00:40:16): Microsoft formed immediately to sell a BASIC compiler for this device. The Apple II launched in 1977, which kicked off the PC wave for many folks, and it paired with VisiCalc in 1979.
So, all of a sudden, you had a killer app – a spreadsheet – that wasn't just for home use, but now had a corporate function, driving the purchase of these PCs. And so someone could model a budget without needing to connect to the mainframe down the hall.
Anay Shah (00:40:43): I can stay at my desk and run my finance operations.
Ben Shwab Eidelson (00:40:47): That's right. And IBM, while not the first here, was actually pretty quick to realize there was a problem for them. Right. If everyone is doing all the computation on their desk, they're not going to need the mainframe down the hall anymore. The glass house had been shattered.
So they kicked off a skunkworks project. It was actually pretty amazing. They isolated it outside of the New York region, down in Boca Raton. They said this team would use off-the-shelf parts to design an IBM PC.
In 1981, they built and launched the IBM PC, and they licensed MS-DOS from Microsoft, which had been founded six years earlier, as the core operating system. They thought hardware was the business, but time would show them to be incorrect.
A bunch of IBM-compatible devices, including Compaq and others, flooded the market through the 80s. So prices fell. Hardware became a commodity, and Microsoft, with DOS and then Windows, became the Wintel duopoly that ultimately took the mindshare and market from IBM.
Anay Shah (00:42:00): That's amazing. You got Lotus 1-2-3 dominating spreadsheets. You've got WordPerfect. I remember using that for word processing. And then Microsoft comes along and says, "I'm going to bundle all this into Office" and really took the cake.
So productivity was through the roof, and we were able to do more personally than we could ever do before. And the way we collaborated? It was that magic floppy disk. We were able to hand that back and forth and keep on rolling.
Ben Shwab Eidelson (00:42:21): And that's great, right? Until it's not.
Anay Shah (00:42:24): Until it's not.
Ben Shwab Eidelson (00:42:25): You want to work with someone across a big building? You're going to run them a floppy disk? And what if they have a different version? It's a mess. I think the "personal" in personal computing started to become a hindrance here. So we needed to connect these devices as well.
Enter the invention of Ethernet and IBM's Token Ring, which wired the floors and connected these PCs into a corporate network. And a new set of companies emerged. Novell, building NetWare, actually turned a server into a hub for shared disks and printers.
You had a PC on your desk, and if there was a printer room and you wanted to print, you'd have a different computer. That was a specialized PC that we'd call a server, connected to that printer or to shared storage. Now we were able to access files and print over a network. A whole company, Novell – which at some point was actually the second-largest software manufacturer after Microsoft –
Anay Shah (00:43:23): Wow.
Ben Shwab Eidelson (00:43:23): was booming in this era. Microsoft, doing what they do very well, responded by building a network-connected operating system – eventually Windows NT and other services – so that you could do it all within Microsoft's ecosystem.
Anay Shah (00:43:37): So throughout our conversations, a lot of the OGs in this industry pointed to this moment as the introduction of the client-server era. What does that mean?
Ben Shwab Eidelson (00:43:46): It means an application was really split in two. You had your computer sitting there on your desk running as the client and accessing a central database or a server in the back room. A bunch of technologies came up to support this, from Unix servers to ERP systems to Windows NT.
But now, all of a sudden, you would think about an application as networked from design, where you'd have local client software that could use the best capabilities of the local PC, connected to the server. Not across the world, but the server in the server room, on site.
Anay Shah (00:44:20): What was that server down the hall or in the other room or across the world?
Ben Shwab Eidelson (00:44:25): Most notably, this was the era when the architecture of the personal computer was adopted. This included the x86 (Intel-based) architecture or the Sun workstation architecture. Increasingly, those would become the servers that could run these applications. This was a notable shift, again, from the IBM mainframe era.
It got to the point where someone selling your company an application would sell you an appliance, which was really a package of the software and the hardware together. So you ended up with a sprawl of different appliances that were being managed, all serving different functions, all written with their own bespoke software and operating systems. So, while the functionality was amazing, people became very inefficient.
Anay Shah (00:45:10): Up until this point, there was a culture and an ethos among data processing professionals that put conservation as the highest ethical principle. To waste a CPU cycle or a byte of memory was embarrassing.
Ben Shwab Eidelson (00:45:23): I think part of it was that they had gotten cheap enough; these servers, which would have been $10,000 a month to rent just 10 or 15 years prior, had now gotten cheap enough. And it wasn't unreasonable to have them sitting there idle at the same time.
In a macro sense, this inefficiency later created the space and value for the evolution of time-sharing into virtualization, where we asked, "Why can't we run these applications on one box? Why do they all need their own box sitting there waiting for a command?"
Anay Shah (00:45:49): That's right. If a box was not being maximally utilized by a single client, share the box.
Ben Shwab Eidelson (00:45:56): We will do a whole episode on how that came to be. But this was still an era of deep innovation, particularly concerning redundancy. You had RAID arrays, which provided redundant storage, and backup tape libraries that were off-site, because you still had these massive points of failure.
Your company server, which might be your system of record for your customers, was sitting there in the closet. I heard from someone who ran some of it at REI at one point that they had a flood near their headquarters, and they were like, "Well, the website's going to go down, we're going to lose all this business." So this model was not designed for scale and redundancy in a major way.
So, to make this moment concrete – of just how computing power and connectivity was starting to shift – let's talk about a deeply innovative retail company that, no, is not Amazon. We're talking about the OGs of major retail and scale. It's Walmart.
Anay Shah (00:46:50): That's right. So, you're at the register at your neighborhood Walmart location: you scan the toothpaste, the barcode beeps. Within seconds, a satellite dish behind the store sends that transaction to the sky and over to Bentonville, Arkansas, where they're hosting their mainframe computer to record the sale.
A few minutes later, a massive data warehouse would update how many tubes of toothpaste you had just bought. Then, perhaps before the end of the day, Procter & Gamble's factory would receive an update that they needed to make more toothpaste.
This was the cutting edge of retail in the late 80s, and Walmart made a massive decision to take this a step further, truly driving innovation across the retail industry. They invested $24 million to build their own private satellite network linking all Walmart stores to headquarters. This was fairly unprecedented at the time, right?
Ben Shwab Eidelson (00:47:43): It was the largest private satellite network.
Anay Shah (00:47:46): That had been built, and it created a unified, real-time machine. So any single event that happened within the Walmart ecosystem rode on this private network, enabling them to mine their data in a way that was unheard of before.
By mining their sales data – which could now be collected in real time across all Walmart stores over their private satellite network – they discovered that when hurricanes approached, the sale of Pop-Tarts increased 7x over their normal rate.
Having spent a summer working in Bentonville, Arkansas, I can tell you this is deep in their DNA. They are constantly looking at signals like this. Specifically, they discovered that it was the Strawberry Pop-Tart that was most in demand before a storm. This led to the legendary insight of meteorologists predicting a severe weather event, and Walmart stocking their affected stores with pallets of Strawberry Pop-Tarts.
Ben Shwab Eidelson (00:48:43): It makes me wonder if the reverse is possible. It's like, instead of checking the weather, you go to Walmart to see if the Strawberry Pop-Tarts are there, because it's such a reliable system. Like, I know that they're watching the weather, but I mean, there's some crazy innovation here.
Both the satellite link, and then Walmart, supposedly, was the owner of the first commercial 1-terabyte enterprise data warehouse they built with a company called Teradata. They were just maniacal about making sure all the sales and customer data flowed into one place. By 2001, just nine years later, that one-terabyte warehouse had grown to 70 terabytes.
Anay Shah (00:49:14): Wow.
Ben Shwab Eidelson (00:49:15): Right. So this was just an explosion of connectivity inside the enterprise, even while there was the personal computer. Some people had fun with their computers at home. I remember playing on an IBM XT as a kid, and some early programming and early games. It was not like most people had a real personal use case. We're still talking about a corporate-centered world.
So we talked about the rise of PCs and the growing client-server LAN inside a company. We talked about these data centers; we talked about Walmart's satellite network. But what was happening now, going into the early 90s, with this whole ARPANET/NSFNET Internet thing? What was happening outside the office?
Anay Shah (00:49:55): Yeah, and perhaps what was happening under the ground.
Ben Shwab Eidelson (00:49:58): So the first network that was available to all these researchers was the evolution of this into the NSFNET. It became the de facto US internet backbone.
Anay Shah (00:50:07): This network was connecting 2,000 computers in 1986 and expanded to over 2 million by 1993.
Ben Shwab Eidelson (00:50:15): It was no longer just researchers who wanted to do stuff with it. So, the design and topology was a high-speed national backbone that had, at this point, gone to a T3 line (at 45 Mbps) that connected a small number of regional networks. So, kind of like a central hub-and-spoke model. These regional networks would then connect to universities, labs, and nonprofits. But this was not designed for commercial scale.
Anay Shah (00:50:39): In fact, it was prohibited, according to their terms of use, to have commercial traffic running on that backbone. It was specifically designed for research, education, and government data.
Ben Shwab Eidelson (00:50:50): So, I think some private commercial network could happen inside just a regional hub, but not across that whole big backbone. So you did not have the beginnings of what could be a commercial internet. If two regional networks – say, New York and Philadelphia – wanted to connect, they had to flow back up to that NSFNET backbone. There was no neutral point where they could connect and exchange.
So this was the moment when commercial ISPs and telcos started to see, "Okay, this internet thing is interesting, this packet-switched model." People wanted to do new things with this. Maybe it was trying to get out of the lab into commercial use. How were we going to connect? What were we going to do?
Anay Shah (00:51:29): In 1992, a group of network providers were sitting in Virginia, drinking a beer, and decided to connect their networks outside of Tysons Corner. This specific group of engineers was from Metropolitan Fiber Systems, the local telco. They chose Tysons Corner outside of Washington D.C. because there was a dense network of defense contractors and early providers, which were heavy users of the current internet.
They famously set up in a repurposed parking garage to become the de facto on-ramp for new ISPs. When you think about an important hub, you wouldn't typically think of a parking garage, but it was the right place at the right time. It turned into what was called Metropolitan Area Exchange East – East Coast. So MAE-East is what formed. And if you connected into MAE-East, that meant you had the internet at your doorstep.
Ben Shwab Eidelson (00:52:27): This became the hub for the internet. If someone sent an email from London to Paris, it most likely went through MAE-East.
Anay Shah (00:52:35): Within a couple of years, roughly half the world's internet packets were flowing across the MAE-East parking garage.
---
Ben Shwab Eidelson (00:52:43): And I think this is something we'll see again and again, which is that the Internet forms around hubs. It's not always obvious why that became the hub specifically, other than it did first. And there's just this gravitational pull of connectivity. This wasn't new to the Internet; this was something that we saw with telcos. There's this concept called carrier hotels. You're in a city like New York, and you had two different carriers that were trying to connect with each other, right? Think Sprint and AT&T. Instead of having to connect all throughout the city in multiple spots, they would all show up in a neutral zone called a carrier hotel and build their connectivity infrastructure there.
Anay Shah (00:53:21): You'd be able to tap into each other's long-haul routes, their local fiber routes, without having to build their own intercity footprints individually.
Ben Shwab Eidelson (00:53:31): And so May east was the first sort of flavor of this, where they would all come in and connect. Ultimately, a device — think a switch — that is connecting. Okay, you're coming in, plugging in your ISP traffic; here I'm plugging in mine. Now users across our ISPs can connect, and it's just one big switch room. The problem is, the Internet is scaling, and you don't necessarily want to all be bottlenecked on one switch. It's not the most secure thing. And if you're, I don't know, eventually building a video streaming service or you make a deal between two ISPs, you don't necessarily want everyone else to be in on that deal. And so you saw the evolution of this model to a different model that became known as MeetMe rooms. These are neutral physical rooms where an ISP or someone trying to hook into an ISP can provide their boxes and their connectivity, and then those two can connect directly.
Anay Shah (00:54:20): So, a bring-your-own-box method. Instead of everyone connecting through the existing box, you BYO your box for the deal you want to do in the private MeetMe room.
Ben Shwab Eidelson (00:54:30): That's right. Something sounds funny about BYO to the private MeetMe room, but we're talking about ISPs connecting for data.
Anay Shah (00:54:38): The farthest thing from a non-platonic conversation as you could be.
Ben Shwab Eidelson (00:54:44): And so, sure enough, this all worked. The NSF kind of officially sanctioned this method and designated these NAP points — Network Access Points. The first one, May east. They designated Sprint to run a NAP in New Jersey near the transatlantic cable landing points, one in Chicago, and one in San Francisco. Then they eventually added Mae west in San Jose.
Anay Shah (00:55:06): And these MeetMe room models started to take off. The carrier hotels, you had One Wilshire, a large building on the west side of Los Angeles, which were law offices, and gave way to a single floor that could host hundreds of carrier routers and thousands of cross-connects, eventually making it one of the most valuable space per square foot on the entire West Coast.
Ben Shwab Eidelson (00:55:29): It's so funny. It's like this ugly building or just relatively nondescript architecture, and you're like, "What's going on in there?" Law offices make sense. And now this is where the West Coast Internet is coming through.
Anay Shah (00:55:40): That's right. And we'll talk a little bit later about undersea cables, but they come in and want to find their shortest path to One Wilshire.
Ben Shwab Eidelson (00:55:48): And the business model was pretty genius for this. These telco hotels, these MeetMe rooms, ultimately provided power, cooling, and cross-connect, and they would charge rent. The dot-com boom to come would boost this model to new heights.
Anay Shah (00:56:03): And it provided an elastic infrastructure that Internet companies in the dot-com boom and after could leverage, including, as we'll see soon, hyperscalers, where you could flexibly increase and decrease your capacity because they were specialized in providing all the necessary infrastructure to host the connectivity.
Ben Shwab Eidelson (00:56:25): So all of this infrastructure being set up to commercialize the Internet, to provide a scalable backbone, to enable the private market of ISPs and telcos and others to invest in making the Internet faster. And we now have the World Wide Web. The NSF has actually funded a little project called Mosaic, which is the first user-friendly web browser. We have personal computing. Enter the mid-90s and the dot-com boom.
Anay Shah (00:56:53): Let's boom!
Ben Shwab Eidelson (00:56:54): The ISPs are ready to build the network. All we need are the users.
Anay Shah (00:57:00): And boy, are they coming! In May of 1995, Bill Gates writes the famous Internet tidal wave memo: "The Internet is the single most important development to come along since the IBM PC." And it was that same year that Netscape went public. Why was Netscape so significant, Ben?
Ben Shwab Eidelson (00:57:21): I think it's significant for two reasons. One is it kicked off the accessibility of the Internet. It turned the Internet from this network for researchers to share files to this browser that you could download, install, and access the web, opening all of that up. And then it was that from a product perspective. But that also captured the economic perspective and interest: "Is there a thing here? Is there a new industry, a boom that you can make your millions or billions off of?" Its journey from founding to IPO in such a short cycle kicked off a mania.
Anay Shah (00:57:56): It kicked off another mania in the business world, which was: having barely any revenue or profits, you could hit a multi-billion dollar valuation, which is what they did. And the web exploded. You had 23,000 websites in 1995 to over 10 million by the year 2000. Global users climbed to over 350 million. Nasdaq tripled in two years. If you put a dot-com at the end of your name, just kind of like you put 'AI' at the end of your name today, you could raise millions on an idea and a slide deck. And in 1999 alone, you had more than 400 Internet companies going public, pulling in $40 billion. What a liquid IPO market that we could only dream of!
Ben Shwab Eidelson (00:58:40): Today Microsoft hits all-time highs in the stock market. The energy was manic, right? Founders in their 20s would become paper millionaires overnight. Engineers were hopping jobs for stock options. This is the rise of the Aeron chair, the foosball tables, and the new economy where the rules of things like revenue no longer apply to business.
Anay Shah (00:58:59): But it wasn't just applications; it was infrastructure as well. Carriers spent half a trillion dollars on fiber and wireless. You had these colocation companies expanding at breakneck speed. This company called Exodus Communications was the world's largest web hosting provider at the time, providing server colocation. Its revenue went from 12 million in 1997 to 250 million two years later. It peaked at a $32 billion market cap three years after that. It was laying the infrastructure in the ground and building the applications above it in a period of unprecedented growth.
Ben Shwab Eidelson (00:59:37): And so to launch a startup, you had to build a site, a service, and a database. You needed money way ahead of time to buy the servers to stick in the colo box.
Anay Shah (00:59:48): Wait, so you're saying, in order for me to launch a web business, I had to buy hardware?
Ben Shwab Eidelson (00:59:53): Exactly. You had to take most of your venture capital dollars and spend it on servers even before you knew if anyone wanted to go to your website to begin with.
Anay Shah (01:00:02): So you couldn't test your idea out, you couldn't A/B test, you couldn't do a landing page that drew in a...
Ben Shwab Eidelson (01:00:07): waitlist, couldn't do any of that.
Anay Shah (01:00:09): Wow, what a different world!
Ben Shwab Eidelson (01:00:11): So all these new servers are trying to run on the new backbones that are being laid to power this boom. The traffic still at this point met at only a handful of public exchange points. Networks are plugged into these shared boxes at places like May east that we just talked about. That can only scale for so long.
Anay Shah (01:00:30): You'd start to hit choke points, and May east, one of the earliest network access points, became one of those major choke points.
Ben Shwab Eidelson (01:00:38): We needed different models for companies, ISPs, non-ISPs, to connect. You had DEC kicking off Palo Alto with the Palo Alto Internet Exchange, a non-telco neutral spot. Then you had the founding of Equinix where they took that model and scaled it with their first site in Ashburn, Virginia.
Anay Shah (01:00:59): Ashburn is the Wall Street for data centers. In 1999, Equinix launches their first data center under this new model in Ashburn, Virginia, right next to May east, the choke point. But the original network access point isn't.
Ben Shwab Eidelson (01:01:13): This, near D.C.? I've never been there. Have you been to Loudoun?
Anay Shah (01:01:16): I grew up right outside of D.C. on the other side of the river. Loudoun county is on the Virginia side. It's rural farmland. It's past Dulles Airport. There's really nothing there. But it's proximal to a large East Coast population, the undersea cables and May East. And you actually had AOL choose Loudoun county in the mid-90s to set up a huge dial-up campus. So they laid fresh fiber and drew even more carriers into the region. And May east had grown so large it outgrew the parking garage it was in. So the Exchange relocated to Ashburn. So you've got this unassuming farmland outside of Dulles Airport. And it was really catalyzed by a couple things. Outside of AOL pioneering the new site, it was policy-led. So Loudoun county ruled that data centers could be treated like ordinary office parks. They eliminated all these special use hearings and provided incredible tax breaks over time to attract data centers into this network. Coupled with that, Dominion Energy, seeing what was coming ahead, offered some of the lowest industrial rates to string high-voltage lines to this empty land to bring power and fiber together to create the new data center model which Equinix pioneered in Ashburn and became the largest Internet hub on planet Earth.
Ben Shwab Eidelson (01:02:42): So it's the combination of perhaps for a moment, cheap land (not so much anymore), fiber and connectivity, cheap and accessible power, and favorable policy. One interesting policy story that I heard on this: Apple was looking for where to put a new site in the late 2000s, 2009. Apple's obviously building more services, needing more storage, and building more data centers. So they run a process, and it turns out that North Carolina gives them a better deal. So Virginia fights back and in reaction passes major tax breaks to say that if you're building a data center, basically if you're building anything more than $150 million of investment, you're going to employ more than 50 people — no tax.
Anay Shah (01:03:22): You, of course, are. If you're building a data center, of course.
Ben Shwab Eidelson (01:03:24): Of course. Yeah, you can't build a data center for $150 million.
Anay Shah (01:03:27): Local policy, as you mentioned, is so important. Federal policy actually played a big role here. In 1996, we passed the Telecommunications Act. And one of the main things it did was force the incumbent telcos — which were regional monopolies from the AT&T breakup — to lease their physical network (their copper pairs, their fiber) to their competitors. So prior to this, a data center was a private enterprise tied to a single carrier because the carriers only used their own fiber. Now it enabled carrier-neutral sites. So you could become a tenant of a data center and choose from multiple fibers provided into that building. And it opened up an explosion of choice for tenants and for this carrier-neutral model.
Ben Shwab Eidelson (01:04:11): And so this is a flywheel of a deregulation environment to build this because once they have the tax incentives in, it becomes a major source of economic prosperity for the region. Now, we'll fast forward later on in the story to today, and maybe hitting some of the first real pushback.
Anay Shah (01:04:27): And similar to Frankfurt, Amsterdam, London, Tokyo, the biggest hubs are where you can find cables and carriers and connect the most networks with the least amount of friction possible. And it's this flywheel that continues to make Ashburn the largest home of data centers in the U.S. And what a time!
Ben Shwab Eidelson (01:04:47): In the first six months of 1995, Internet traffic was doubling every 100 days. The telcos were convinced that you couldn't overbuild. You just needed all the fiber you could put down.
Anay Shah (01:04:56): WorldCom and these other telcos, they just poured billions into this. And the idea of overbuilding — not possible, particularly in this Internet boom, right? But by 2000, how much of that installed fiber was actually being used?
Ben Shwab Eidelson (01:05:11): It was only about 3% — 3% was lit up. So they just laid down fiber, the shards of glass that were just sitting there empty.
Anay Shah (01:05:19): So to give you a sense of the amount of fiber miles laid during this period, you could go around the circumference of the Earth 5,000 times.
Ben Shwab Eidelson (01:05:27): Wow. So this was all over land, right? But how would you connect to Europe?
Anay Shah (01:05:31): Ah, and that overbuild, you're right, was not just limited to land. Nearly all intercontinental Internet traffic rides on undersea cables, which I'd kind of heard of, but I didn't really have a full appreciation for the fact that bundles of glass threads are wound into a garden hose structure and laid down on the ocean floor all over the world. So imagine a planet stitched together with hair-thin strands of glass tucked into an armored hose and laid across the darkest parts of the ocean. That's the undersea cable system, and it's the real physical Internet that connects the continents. So 99% of international data still rides on these cables, not satellites. That's racing pulses of light through fibers thinner than the human hair.
Ben Shwab Eidelson (01:06:23): The story arguably starts way back in the 1850s, when we had the first telegraph cables crossing the Atlantic. They brought those across steamships and landed the first link in 1866, able to send news for the first time across the ocean.
Anay Shah (01:06:38): So instead of weeks, you could get the news in minutes.
Ben Shwab Eidelson (01:06:41): Yeah, you're not sending the news via ship, you're sending it via electrons.
Anay Shah (01:06:44): Amazing!
Ben Shwab Eidelson (01:06:45): Fast forward to 1988, and we land the first transatlantic fiber-optic cable. It runs between the U.S., the U.K., and France, and it kicks off this new era of cross-continental capacity. By the late 90s, as this boom was happening, you can imagine the funding routes going into wiring all of this up.
Anay Shah (01:07:07): You lay all this cable down with specialized cable ships that survey the seabed, unspool the cable, and then near shore they bury it underneath to protect it from anchors and storms, and then raise it up through some nondescript concrete box. This garden hose has optical repeaters that boost the light every 50-100 km, so you can sprint thousands of miles without fading. Then you get to land, and these concrete bunkers have the cable hop up into terrestrial fiber and then run straight to your One Wilshire or any of your nearby hubs that then connect you onward.
---
Ben Shwab Eidelson (01:07:44): What a wild thing. You have a garden hose-shaped thing moving all this data. What do we actually mean? Well, modern cable, as an example, one laid in 2018, can move 250 terabits per second. To conceptualize that, you can send 6,000 HD movies in one second, or about 20% of global internet traffic can go in one garden hose.
How is this working? Well, at the beginning of using fiber optics, you would shine a laser down and blink your zeros and ones. But we've moved from doing that with one wavelength to doing what's called wavelength division multiplexing. That's a fancy way of saying we use the rainbow; we're using multiple. Usually, it's around 80 to 120 different colors that can go down the same cable at the same time. The other thing is we've added more fibers. So instead of there being a pair of fibers, we now have up to 12 to 16 pairs of fibers. And then the last thing is called coherent optics. This is a way of modulating the amplitude of the light. So instead of it just being on and off, you can actually have multiple steps. All of this adds up to today, probably 250-300 terabits maximum capacity in one hose.
There are hundreds of hoses around the world. And I think the other thing that's really cool about this is the thing that was laid was the glass. We keep increasing the capacity of the glass because the glass is the glass—resilient infrastructure. It's kind of like railroads. It's like we're still using the tracks.
Anay Shah (01:09:07): From long ago, even if the engine gets upgraded. And just like we've been talking about with this network effect, when you bring more networks in, it increases the value and the speed. And so that's a big reason why Ashburn, as we talked about, hardened. Once hundreds of carriers and thousands of cross-connects land into a single place, moving it is impossible. And so if you need to reach Europe fast from the East Coast, you're going to colocate where the undersea cable is already coming. You've got New York, New Jersey, Virginia, tying to Cornwall, England, and to Marseille. You've got the Red Sea and the Mediterranean corridor connecting into Djibouti as a critical touch point. You've got Miami as a key touch point into Latin America. Japan, Singapore, Hong Kong, Taiwan are key corridors into Asia Pacific. Mombasa and Lagos now light up Africa's East Coast. Hundreds of cables have converged that have connected the entire world. And this then forms the network of data centers that we have built and will continue to build through this story.
Ben Shwab Eidelson (01:10:16): And so the boom continues, right?
Anay Shah (01:10:19): No, there was something like the bust to that boom.
Ben Shwab Eidelson (01:10:23): That's right. And by 2001, everything collapsed. Advertising folded, startups folded. Exodus, that we talked about earlier, filed for bankruptcy, had nearly $6 billion in debt. PSINet, one of the largest ISPs, had already collapsed. So, was this the death of the internet and the death of the value of all of this fiber that was laid? Clearly not. It was the death of a particular moment and an overbuild and a bubble. But in fact, the actual infrastructure that was built out would prove immensely valuable as services that mattered matured and business models matured. In other words, I'd say the application layer of this era died, but the infrastructure lived on and would eventually thrive.
Anay Shah (01:11:03): So the dot-com tide went out. But the overbuilt assets were exactly what we needed for the next chapter. And it included infrastructure that we can't live without. The carrier hotels, the fiber, the data centers, the glass in the ocean didn't disappear. It just changed owners, and there was a fire sale, right? Assets were being sold at a fraction of the cost. A few key actors survived, and a few new ones stepped in.
Equinix survived the crash. They doubled down on the real asset that they had, which was interconnection. That neutral meet-me room turned into a marketplace where competitors paid you to be neighbors because the value was in the speed, the reliability, and the flexibility that these carrier hotels provided. Meanwhile, private equity swoops in, as they will in every bust cycle, and reframes the category. So, one in particular buys a couple dozen distressed facilities around the world, turns it into a vehicle called Digital Realty Trust, takes it public as the first pure-play data center REIT, and treats compute space like real estate. These weren't high-tech moonshot assets they were buying. They're bringing patient capital in to standardize the shell, finance it cheaply, and get long leases, which becomes a blueprint for the next two decades of data center buildouts.
Ben Shwab Eidelson (01:12:30): In addition to the sobering economic environment of 2001 from the bust, there was also September 11. These financial institutions were still operating in this moment, having their key servers trading information connectivity with banks in their offices or right near their offices. So when 9/11 happened, the Verizon 140 West Street Central Office, one of the largest telecom hubs in the city, was blasted with debris, and dust flooded the equipment rooms. Tens of thousands of voice and data circuits were knocked offline immediately. Most of those circuits were powering exactly that: brokerages, market data providers, the trading floors, low-latency connectivity to the stock exchange and clearinghouses. So imagine the market is just disconnected now in a flash. And this cascaded. Engineers worked night and day to bring things back online.
I was looking into Morgan Stanley's experience. They saw their whole trading system go down. They had a disaster recovery site in New Jersey, but they didn't have the same level of connectivity and data feeds. So it took them a few days. They ran new fiber through building basements, patched hubs into another telco hub that was still operating to restore capacity so they could trade when the stock exchange opened on September 17th. And I think coming out of this, there was a whole reshaping of the data center world to think about resiliency in a new way. I think it showed the physicality in the city of this connectivity right in this moment, where you could take for granted that you could take action on data over there, whether a trade or market data.
Anay Shah (01:14:07): And it wasn't enough to have redundancy on another floor in the same neighborhood.
Ben Shwab Eidelson (01:14:12): Right.
Anay Shah (01:14:12): We had to start thinking about an infrastructure build-out in different locations with different networks, facilities, and routes to really build true resilience and switch over to the point where now, if a data center goes down, there's automatic rerouting, and we don't see those same blips, although it happens from time to time.
Ben Shwab Eidelson (01:14:31): This early 2000s phase is a real maturing and growing up of the entire industry to realize that these servers and data centers are holding important financial data and need to be treated as such. But meanwhile, in consumer land, there's a glimmer of light, and it's a big one. This is the era that we move from that squeaky, squealy phone modem to broadband.
Anay Shah (01:14:55): I remember touring for colleges, and some had Ethernet across the campus, and others didn't.
Ben Shwab Eidelson (01:15:01): I feel like getting broadband to our house was a radically different experience. It was like a different internet.
Anay Shah (01:15:08): The image didn't load from top to bottom.
Ben Shwab Eidelson (01:15:10): Bottom, it would just fly through. And so this is when BitTorrent starts soaring. This is when Skype launches in 2003, and you can actually make VoIP calls.
Anay Shah (01:15:18): Oh, Napster was possible.
Ben Shwab Eidelson (01:15:20): World of Warcraft launches in 2004. I remember that, taking the college campus by storm. Early web video products started to come out.
Anay Shah (01:15:28): By 2005, you had a billion people online, about 16% of the planet.
Ben Shwab Eidelson (01:15:35): So all those folks prognosticating with excitement in the late '90s were not wrong. They were just off by five or six years. And so there's also the advent of CDNs. So Akamai's footprint exploded to provide more and more storage and replication and caching at the edge.
Anay Shah (01:15:52): And so what exactly does this mean?
Ben Shwab Eidelson (01:15:55): It means at the places where your ISP is connecting, if someone's downloading an image—let's say you load the New York Times on your computer, and someone else does on their computer down the street—you don't both need to go all the way back to the New York Times home server for access to that photo. It's now been cached on a nearby CDN that's directly hooked up to your ISP.
Anay Shah (01:16:13): So in the early to mid-2000s, you have the consumer coming back. Applications are flourishing. The internet is becoming a part of the fabric of society. And there are a handful of companies that survived the bust and captured this moment unlike any other. They not only built incredibly large consumer and enterprise businesses, but they actually became critical infrastructure companies that helped build the modern data center world. That is what we're seeing booming today.
The place we're going to start with is the best place to buy books: Amazon.com. So whatever book you wanted to...
Ben Shwab Eidelson (01:16:50): Find, from A to Z. In July 1995, Amazon launches. By '97, they IPO, and by '98, they are no longer just a bookstore. They're on their path to becoming the Everything Store. Now, it was not initially Bezos's and Amazon's intention to become the infrastructure provider of the world. But this high-growth moment of the late '90s set the stage for what they would need to build, not just for themselves, but for everybody. In those early days, they were running expensive, quote, unquote, reliable servers from the likes of DEC. Extremely expensive products—high-margin servers. Now, the problem is Amazon was not a high-margin business. They're trying to go for scale. They're selling things at whatever the cost was to pass through. They are a retailer trying to be the lowest-margin retailer out there. The cheapest way to get your book delivered to your doorstep.
Anay Shah (01:17:45): So, running a retail business, they're always...
Ben Shwab Eidelson (01:17:48): Tight on cash, so spending it on servers stopped making sense. So by 2000, they were spending so much on infrastructure they were worried this was going to bankrupt them. And so they kicked off a big project to rewrite all of Amazon.com onto Linux and to run it on much cheaper HP servers.
Anay Shah (01:18:05): And this is when Amazon was famously a huge monolithic codebase. Every new category they launched, they had to work across their entire codebase. And it became this hairball.
Ben Shwab Eidelson (01:18:17): By, I think, around 2002, Bezos had had enough of that, and he issued the famous API mandate that internally every team had to expose functionality through hardened, documented service interfaces designed not just to be used by internal teams, but eventually potentially externalizable.
Anay Shah (01:18:33): Classic Bezos. There are no exceptions. Every team must communicate through these interfaces. There were no backdoors, no direct threads, no direct linking. It didn't matter what that technology did. It would, without exception, be designed from the ground up to communicate externally to other teams.
Ben Shwab Eidelson (01:18:52): And if you didn't do this, what would happen?
Anay Shah (01:18:54): If you didn't do this, you were canned. So, in this moment of reboot, it seems like the question that the company's leaders are asking themselves is: How are we able to scale our business like a software business and not like a furniture business? What if compute could scale with demand? And if we can do this for ourselves, why not rent it to the rest of the world? They were on the precipice of not just a technical breakthrough, but a business model breakthrough. Because for decades, running an online business meant these multi-year leases, these expensive servers, and over-provisioning to handle peak demand. And Amazon would go on to flip this on its head and fundamentally change internet businesses by saying you can rent a server by the hour and pay for only what you use.
Ben Shwab Eidelson (01:19:47): It is both deeply innovative at the time and also funny because we've had decades—a century—of doing this with our electricity bills in our own houses or water bills, right? This is utilities, utilities. You've always just paid for what you use. But what it continually enables is the driving down of cost and better utilization of centralized infrastructure. And Amazon had just lived through this painful period of having to rewrite their software and change their server architecture. And I think it was two things. One is: we never want to go through this again. A. B. No one should have to go through this again. And C, if we start to build the infrastructure for the world, that's going to accrue to our costs and our benefit. And there's going to be a flywheel here, just like any scale economies provider ever experiences. The bigger we get, the better.
Anay Shah (01:20:39): That's right. And that utilization is such a key point. So, what happens in March of 2006?
Ben Shwab Eidelson (01:20:44): Amazon launches the first real AWS service: S3, or Simple Storage Service.
Anay Shah (01:20:51): What does this Simple Storage Service do for me as a small internet business?
Ben Shwab Eidelson (01:20:56): It lets anyone put a blob of data on this non-physical disk and access it anywhere in the world. And that sounds simple in the name, but it was shockingly hard to put a blob—whether that was a megabyte, a gigabyte, or a terabyte of data—out there and have everyone around the world be able to access it quickly. Amazon abstracted everything away so that you could do that and just pay a monthly fee. You didn't have to build a server and plug in a hard disk and build another one that copied the data over and all these other things. It just gave you what you need as a developer.
Anay Shah (01:21:32): So this enabled me to store information.
Ben Shwab Eidelson (01:21:35): Correct.
Anay Shah (01:21:35): A few months later, what did they do?
Ben Shwab Eidelson (01:21:37): They launched the Elastic Compute Cloud, also known as EC2.
Anay Shah (01:21:41): So I can store, and now you're telling me I can compute. How does this work?
Ben Shwab Eidelson (01:21:45): You can compute. And so what EC2 essentially was the ability to spin up computers—servers, as you saw fit. Now, what they were actually letting you spin up is what's called a virtual machine, where you can say, "I want to run Linux," or "I want to run Windows," or "I want to run some SQL Server OS," and I would have this virtual machine where I could deploy that, run it, and run whatever code I need to. And if I need a second machine, I push a button and get a second machine. If I need a third machine, a third machine. And I pay by the hour, by the machine, only for what I'm using.
Anay Shah (01:22:16): This is outstanding. So I'm building a business, and I think I'm going to grow fast, but I don't know what traffic I'm going to get next month or in six months. And so I just raised a bunch of VC money. Now you're telling me I don't have to buy these expensive HP and Sun servers, and as I grow, I can just rent more compute and rent more space?
Ben Shwab Eidelson (01:22:37): Yeah, not just that, but I think the activation energy here was brought way down. Before there was the challenge of getting a server and getting space and all that. But in this model, now you just put down your corporate card, and you're off and running with the foundational building blocks that you need.
Anay Shah (01:22:51): This is the Cambrian explosion for startups.
Ben Shwab Eidelson (01:22:53): Key to this is the utilization point. So many startups had bought servers that then never hit full utilization, or we talked about all these appliances sitting in the backroom closet not hitting full utilization. What enabled Amazon to drive utilization? It was the fact that, yes, they gave you a quote-unquote "server" to run your operating system on. They did not give you a server; in reality, they gave you a virtual machine. Not a machine, a virtual machine.
Well, what is a virtual machine? A virtual machine is the flavor of the concept of a machine. And this goes back to a company, VMware, that was founded in 1998 by Diane Green and Mendel Rosenblum. Seid and I got to take operating systems in college from Mendel. And then Diane, as we get to later in the story, was running Google Cloud around the time I was leaving Google. So, legends in this field and in this industry. What VMware did is they made it possible to take a normal computer—whether it was a server or a PC, for that matter—and run virtual machines on that computer.
And why that's typically hard is a computer is usually made to run one operating system at a time, and that operating system is managing applications and making sure that the computer doesn't crash. Well, if all of a sudden you have multiple machines running on a computer at a time and one does something that you can consider kind of unsafe—right?—that would stall out the machine or do something that they weren't supposed to do, that could break the whole model. But VMware's first product enabled an unchanged Windows and Linux to run side by side on the same x86 box. This accelerated to even more interesting use cases. So you can actually hot-swap VMs on machines at the same time. Let me give you a concrete example here. Let's say you're playing a game on a PC. It was as though all of a sudden, in the snap of a half-second, that game moved to another machine mid-frame. So this was actually designed to be able to hot-swap a virtual machine from one server to the next.
Anay Shah (01:24:55): So it didn't matter that you didn't actually have your own server, because your virtual machine could float around as needed. So let's say that you have a hard drive crash. Well, you could have a snapshot running in the background and you could flip over to that one in real time. And so this notion of a virtual machine becomes the backbone for both the utilization point, because one physical server can be used to actually host multiple virtual machines, and a lot of the redundancy and fallback designs.
Ben Shwab Eidelson (01:25:24): And VMware wasn't the only one to do this. Eventually there was the open-source Xen project. That is what Amazon and AWS first used. This kind of thread becomes better and better over time as all the hyperscalers have figured out how to maximally virtualize everything that they do.
Anay Shah (01:25:38): And so now, rather than me trying to run my pets.com and ensure that my product is getting to my customer and my website is doing everything... And then when it crashes, me having to stop everything, file the ticket, and pause business to fix the crash—that's just abstracted out to the specialists, to a business that is designed to solve this problem for me. And all I get then is continuous production, continuous service.
Ben Shwab Eidelson (01:26:08): And it gets cheaper for you every year.
Anay Shah (01:26:10): It gets cheaper.
---
Ben Shwab Eidelson (01:26:11): The cost of S3 and EC2 has just gotten cheaper and cheaper and cheaper. It's an amazing business. And for many others, as we'll get into the story, building these data centers and the utility business is not their high-margin business. But for Amazon, this business has margins, which makes it a high-margin business compared to their retail business.
Anay Shah (01:26:30): It's almost like if I were an airline spending all this time trying to figure out how to book a reservation, and then you gave me a program that could do it for me. I can now focus on serving the customer and serving more customers faster, and it explodes.
Ben Shwab Eidelson (01:26:45): Right. If you look at S3 in 2007, there were 10 billion items stored in S3. By 2009, that was about 64 billion. And by last year, 400 trillion items were stored in S3.
Anay Shah (01:26:58): That's a really big number.
Ben Shwab Eidelson (01:27:00): It's a very big number. It's a lot of items. This powers the startup ecosystem and industry. Let's go back in time. So, what was driving the growth of all these numbers?
Anay Shah (01:27:11): We're in Hackathon City. We're having happy hours with engineers and folks with ideas coming together. Now, if you have an idea, you can drop a credit card down, and you don't need to negotiate and buy hardware. No invoices, no contract, no sales calls. You're up and running within hours. AWS very brilliantly saw this as a pathway for short-term and long-term growth. In the short term, they can get a bunch of early-stage startups using their compute, and that's not going to amount to a lot of money. But some of them are going to grow, and they're going to be built on AWS. So they actually had a business model of giving out free credits at these happy hours, at these hackathons, to make AWS the default infrastructure platform to build a new company.
Ben Shwab Eidelson (01:28:03): I think it feeds through from that go-to-market to their product design. Their product design was deeply unopinionated about what you were going to do. It was to make it easy for you to get a server to go do what you want to do with it. Here's storage, as simple as it can be. These are the simplest Lego blocks you can build on. And so simple, in fact, that people then built what feels like the same business on top of them. Ever heard of Dropbox? Dropbox is just an S3 application for this whole early period.
Anay Shah (01:28:31): It's storage.
Ben Shwab Eidelson (01:28:32): It's storage. Let's make it easy for syncing files from your computer to this new cloud thing, backing them up, syncing them, and sharing them with other people. Dropbox and S3 are intimately linked, and Dropbox is built to do exactly that. It wasn't until 2015 that they were like, "Okay, we should probably look at the cost of this." They eventually moved off of AWS to their own servers because all they are is the storage layer. So that's a rare case where it made sense.
Anay Shah (01:28:59): That's all they are. They're still getting $11.99 a month for me.
Ben Shwab Eidelson (01:29:01): That's right. Good lock-in. The Dropbox AWS story is a classic one, but there's probably no better partnership to exemplify this time than the one that Netflix had with AWS.
Anay Shah (01:29:14): Their ability to scale with these companies is really something to behold. In 2008, Netflix was still primarily a DVD-by-mail company. For those of us that remember, it had launched a streaming service as a side feature. The leadership knew that there was something here. But in August of 2008, Netflix suffered a major database corruption in its primary data center. For three days, it disrupted their DVD shipping and their streaming ability. It stopped their business. This was a huge wake-up call. The recovery was very painful for Netflix. They realized that their on-prem, vertically scaled systems were too fragile, and they couldn't recover quickly enough from major failures. The leadership decided they needed architecture that was designed to be more fault-tolerant, elastic, and globally available, because they had aspirations of being able to stream their future business all over the world. They concluded that they needed to focus on their core business. Building this out in-house was slow and costly. So Netflix actually became AWS's first marquee, all-in, public reference of a customer that scales. This proved to be vital. By 2015, Netflix was delivering billions of hours of content annually, almost entirely over AWS and their own CDN.
Ben Shwab Eidelson (01:30:33): They were running thousands of EC2 instances and had all of their videos, the canonical system of record, in S3 — like the actual videos that we are watching now. They did at some point realize that it was so important for them to own the latency and cost of that last mile of delivery—that edge. This is a perfect example of a use case for CDNs. They launched in 2012 what they call OpenConnect and the Open Connect appliances. This means that they would go into those "meet-me rooms" that we had talked about before and drop a Netflix peering box that would directly connect to your local ISP, and they would do this for free. Right? The ISP just says, "Hey, we have a lot of people trying to access Netflix. Let's make it better, faster, and cheaper."
Anay Shah (01:31:17): It's a win-win.
Ben Shwab Eidelson (01:31:18): It's a win-win. They cut out the CDN that they were using at the time. Netflix saves money. Customers are happier. Everyone gets their videos faster. This way, when someone on the ISP accesses the newest popular movie, it's already close to them. Amazon doesn't actually even — they barely get hit in that moment. It's obviously running what's called the control plane for Netflix.
Anay Shah (01:31:38): Yeah, because if I have to wait three or four seconds for that preview to load, I might not watch that show.
Ben Shwab Eidelson (01:31:43): You might churn. This is what powers streaming to this day. The same model is what enables others to enable streaming at scale. All of this — this whole Netflix case study — is the perfect flywheel and customer to show that AWS can scale with you, scale in a really challenging environment, be resilient, and power global reach. Netflix was not just a U.S. company at this point.
Anay Shah (01:32:05): Yeah. For anyone that saw AWS and the cloud as a concept — as risky, unable to scale, or not enterprise-ready — Netflix helped debunk that for chief information officers around the world.
Ben Shwab Eidelson (01:32:22): They would just do whatever it took. I can't remember the exact story — I think this is from the Acquired episode — where Amazon would allow you to ask them to roll in a big truck, slurp up all your data into hard drives, then they'd bring it to their data center to plug in and dump all your data into your AWS instance. So, they figured out what was needed to close the enterprise customers. It's just amazing that someone known for selling books was able to so quickly build the brand around how to do this new, private, secure utility thing at scale.
Anay Shah (01:32:54): So AWS continued to rethink what data centers are used for and how to build them out to serve the customer. Rather than having one mega-facility per market, they introduced the concept of Availability Zones, which became clusters of independent data centers within a region — each on separate power lines, each on separate fiber paths, linked with millisecond private connections. This Availability Zone concept and clustering was yet another piece of the data center evolution that AWS contributed to the ecosystem.
Ben Shwab Eidelson (01:33:29): I feel like this notion of U.S. East and U.S. West — this is how people think about their servers now — is this abstracted notion of an Availability Zone.
Anay Shah (01:33:39): AWS continued to do this globally, and this actually became a blueprint for hyperscale.
Ben Shwab Eidelson (01:33:44): To end the Amazon story here for now, I think it would not be wrong to credit them with really kicking off the utility-scale cloud story, right? And getting so many things right: giving developers the simple building blocks they could use; going after cost, reliability, and redundancy; making sure to build the right guardrails for virtualization so that they could actually do this efficiently and build for the long term, using the same APIs internally so that it created a flywheel for them to move faster and bring those cost benefits back to the core business. It propelled them so far ahead in this business; they are still the leader to this day in the core cloud infrastructure product.
Anay Shah (01:34:28): As a result, as we've been talking about, the data center is simple at its core, right? It's a facility with storage, compute, and connectivity. What AWS does is take the data center and enable it to build an economy on top of it.
Ben Shwab Eidelson (01:34:46): To frame it another way, it used to be that if you wanted lights in your house — if you wanted electricity in your house to power those lights — you would put a dynamo to burn coal underneath your house, right? And if you wanted a website on the internet, you would buy a server and plug it into your home ISP. That is insane to us today. "What do you mean? You're going to power my house with electricity from underneath my house, right?" Instead, I'm going to hook up this shared infrastructure with all of my neighbors, and we're going to centralize our demand and build the cheapest, biggest infrastructure we can to generate the power. That's the same thing happening here for the first time. This is the introduction of the concept of the cloud, right? People have had the internet where they're interacting with publicly visible websites. Private enterprises have servers, but the idea that you have your own storage over there in a Dropbox folder, or you as a developer could build whatever you wanted and access this floating blob of data — that's to you — invisible where specifically it is, but it's out there. That was, I think, the conceptual notion of the cloud. I've come to somewhat dislike the phrase because it feeds into the invisibility of the infrastructure, right? It says, "It is nowhere. Your photos are nowhere, your storage is nowhere."
Anay Shah (01:36:07): That's so far from the truth.
Ben Shwab Eidelson (01:36:09): It is somewhere.
Anay Shah (01:36:10): It is in multiple locations.
Ben Shwab Eidelson (01:36:12): In fact, the cloud is almost the design antithesis of the glass room. The glass room where you want to show off, "This is where it is. This is where the compute is happening."
Anay Shah (01:36:22): It is only happening here; it is happening right now.
Ben Shwab Eidelson (01:36:25): But it served a good purpose in explaining this notion of migrating to this unknown location, and has remained pretty sticky, I think. So, while Amazon was out there selling books, a little startup out of Stanford was helping you organize and access the information on the growing World Wide Web.
Anay Shah (01:36:44): Google was founded in 1998, and shortly after, it was handling about 10,000 search queries per day. By the end of 2006, it was processing the same number of searches every second. Google acquired YouTube in 2006, a year after its founding, and at the time, it was already one of the fastest-growing websites in the world with 100 million video views per day.
Ben Shwab Eidelson (01:37:07): I remember folks having their first webmail-based accounts, like a Hotmail account. They'd have 2 to 4 megabytes. Well, Google notoriously on April 1, 2004, announced and launched Gmail as an invite-only service that had a full gigabyte of storage.
Anay Shah (01:37:22): I remember you could get access if you referred in.
Ben Shwab Eidelson (01:37:25): If you referred in, or they started going on eBay for like $150, it was a hot ticket. By 2010, Gmail had over 150 million users. And of course, with more gigabytes of.
Anay Shah (01:37:36): Storage available, it was amazing. It was from the very beginning the front door to the internet. But it was also going to need to be an infrastructure company.
Ben Shwab Eidelson (01:37:46): So, you go back to 1999, a year into Google's life, and an engineer named Urs is being shown around as part of his recruitment by Larry Page, the CEO and founder. Urs says you couldn't really set foot in the first Google cage because it was so tiny. The cage was 7ft by 4ft with 30 PCs arranged on the shelves, providing the world with more Google than it could handle. Our direct neighbor was eBay. A bit further away was a giant cage housing Deck machines and AltaVista. All of this was hosted at Exodus in Santa Clara, one of those colo locations that we talked about earlier.
Anay Shah (01:38:21): So, Google was competing with AltaVista and all those others at the time. It was powering part of Stanford Search.
Ben Shwab Eidelson (01:38:28): That's right.
Anay Shah (01:38:29): And running off of 30 PCs.
Ben Shwab Eidelson (01:38:32): Running off of 30 PCs in this colo center, it cost Google about $1,400 per month per Mbps of data. So they had to purchase 2 Mbps. At the time, 1 Mbps was about a million queries per day. From the beginning, Google looked at its servers and infrastructure differently. It was never interested in taking the tried-and-true path of buying the Sun or even HP boxes.
Anay Shah (01:38:56): One of the most memorable stories of this time that explains this is the infamous corkboard. So, in the early days, Google engineers literally mounted motherboards on corkboard. They had $15 box fans pushing air across. They had zip ties holding it together. In a traditional IT philosophy, this is heresy. But the idea was that if a part doesn't add reliability at fleet scale, strip it away.
Ben Shwab Eidelson (01:39:24): This all set in motion an ethos of questioning the assumptions and, ultimately, deep, deep vertical integration of their infrastructure. Some might argue that that was Google's superpower, and still remains so to this day. Apple's piling all of that thinking into making the perfect iPhone; Google's doing it to their data centers.
Anay Shah (01:39:43): So Google had a number of innovations that came from this early time. One was around how to do power backup. So, the standard was to spend for a facility-wide big battery. Well, by 2000, Google was questioning this model and seeing the waste in the large battery. So they actually put little batteries on each server. Again, they were accepting that one might go down, but that was okay as long as the whole fleet was reliable. The key to all of this working was moving reliability up the stack into the software layer. Google was not using off-the-shelf file systems and off-the-shelf software.
Ben Shwab Eidelson (01:40:22): They built everything here themselves. They assumed that hard drives would break, so they built the Google File System. This was around 2001-2002. So, this is a distributed file system that would take files, chunk them up, and spread them across three different servers no matter what. So you could always have resiliency. Also, this would make accessing a search index much faster. In 2003, they built Borg, a cluster manager, which ultimately was all about managing where jobs were running on which servers and machines. And it really was an extension of the virtualization and predates Kubernetes, which we'll talk about a little bit later. So, these were all the software components, but they were also questioning many of the physical constraints that people had looked at with servers. Most notably, they did some of the first experiments in "hot aisle, cold aisle" airflow containment. Traditionally, data centers were just a bunch of hot computers in a room, and they would blast cold air into the room. So you'd have to keep the room as cold as possible, often uncomfortably cold. There was no real thought about getting the hot air out of the room. Over time, it became accepted wisdom that you should point servers in one direction, get the hot air out of the room, and encourage that to happen. Google realized that the more you took that to an extreme, the better. So they would heat up and isolate the hot aisle from the cold aisle because then you could actually extract the hot air more efficiently and not mix it back into the cold. So, what we mean by cold aisle and hot aisle is that cold air needs to blow across those exposed chips and motherboards that would extract the heat off of the server into the hot aisle, and then the hot air from the hot aisle would need to be extracted out of the room. Google became a leader in designing the best airflow there.
Anay Shah (01:42:05): This is a concept that continued for years. You also think about the traditional data center and this idea of assuming something's going to fail and building around that. Whereas in a colo facility, or when you're renting out these cages, every server had to work because it was a different company utilizing it. Not only that, 20 years ago you had specialty cleaning crews that were moving around the data center, and they would sweep up the room and then analyze the contaminants they had just swept up to understand how to continue to optimize the cleanliness and space around it. The idea of laying in a corkboard and a zip tie around it, and assuming something would fail, really takes the data center concept that existed then and completely flips it on its head and, as we'll see, drives better performance.
Ben Shwab Eidelson (01:42:59): All of this comes together when they build their own first scaled, homegrown data center.
Anay Shah (01:43:05): I love this idea: Don't think of a data center the way we've been defining it, as a room packed with servers, power, and connectivity. Think of it as a warehouse-scale computer. Urs wanted us to think of it not as many machines, but as one machine.
Ben Shwab Eidelson (01:43:22): So, you think about the components of a computer, right? The storage, the memory, the compute. Urs saw what they were building out over time in their data centers and realized that they were just building a large computer that happened to be the shape of the warehouse.
Anay Shah (01:43:38): Yeah. If you accept this premise, then you stop trying to make each server perfect, but you start thinking about making the fleet reliable, and you design entirely differently. So, this is about 2004. You have a wild man named Chris Sacca. So, before he was a famed investor, he was apparently a young, sloppily dressed individual walking around rural Oregon looking for "shovel-ready" enterprise zones where he could find some tax breaks. He was walking around asking for such astronomical quantities of power that, allegedly, a nearby town suspected him as a terrorist and called the Department of Homeland Security. But this was just what he was looking for. The Dalles, Oregon, had a site for him: 30 acres next to a decommissioned aluminum smelter that once drew enough power to power the needs of a small city. Sacca was ecstatic. He said it was visionary. This little town with no tax revenues had figured out that if you want to transform an economy from manufacturing to information, you've got to pull fiber. So Google went on to build out The Dalles, Oregon, and bring all of their innovation to bear to increase the performance of this data center. So, what that meant was low-cost, steady hydropower from the Columbia River; the Bonneville Corridor, providing high-voltage transmission; a cool, dry climate that lets you run a cooling process most of the year that's very economical; long-haul fiber that traces the river's edge; and a town that was hungry for a new tenant. This all came together to launch Google's first warehouse-scale computer to drive down unit economics, improve the performance of a data center, and therefore the unit economics of the entire fleet.
---
Ben Shwab Eidelson (01:45:35): And I would argue this is potentially the first built-from-the-ground-up, hyperscale data center. They're building to solve their problems, but their problems are at this point scaling so rapidly that they're thinking about how to optimize all these things, tuned with software, tuned with virtualization, and so this is the new blueprint for the future. That is still how we're generally designing data centers today.
Anay Shah (01:46:01): They had a head start in thinking about infrastructure from the get-go, making it part of their DNA. It's not as if the other companies were standing still; they were building concurrently. This was a race, and they had to do it quietly. Google was the first to get operational in 2006. They quickly replicated this elsewhere in North Carolina and then in Finland, testing different climates to optimize. This included using seawater, hot aisle containment, and other ways to optimize cooling and performance.
Ben Shwab Eidelson (01:46:32): Yeah, there were other areas they kept pushing on, like the power supply system. Normally, each server you think about plugging into the wall converts AC to DC and then steps down DC for all the various components. Google instead realized that if they could just have a higher voltage DC current coming into the rack, they could then convert once cleanly and do it as late as possible. With that higher voltage, you'd have less loss, and you'd get efficiency gains across the whole path. So, at the scale Google was building servers, they could optimize every single part of the stack.
Anay Shah (01:47:07): They later announced this 48-volt DC in 2016, publicly revealing double-digit efficiency gains that the world could then incorporate into data center design. Now, we've talked quite a bit about performance and efficiency, and there's good reason for that. Many data center companies were hemorrhaging money on power. The impact on the bottom line was significant enough that it needed focus. We had the distinct pleasure of talking to one of the OGs in data centers, Christian Belady. He told us this fantastic story of bringing the metric for power efficiency to the industry.
Ben Shwab Eidelson (01:47:55): It's called PUE, or Power Usage Effectiveness. This is a simple ratio of the total energy consumed by a data center, divided by the actual IT equipment you're trying to power. Right? So, it very easily spits out the overhead in running your data center. If it's two, that means twice as much energy is used for overhead than for the actual servers. And how did this metric come to be? Such a simple, beautiful metric.
Anay Shah (01:48:21): Christian Belady was working at HP in the late 90s, and HP had a customer in Japan called NTT Docomo. Christian had been in the industry for so long that he had formulated enough standards through HP and the industry to come up with 10 best practices. He took these 10 best practices to Docomo, and they said, "Yes, this is great, we're going to implement all of this!" He replied, "Fantastic, come back in three months and we'll do a review." So he goes back to Japan, and they had printed out everything, with big stacks of paper on the desk. Everyone was in a suit and tie in a hot room when they told him, "Mr. Belady, we've done everything you said. Here are all of our reports, and nothing's changed. The servers are all still the same."
Ben Shwab Eidelson (01:49:07): "It's even hotter in the hot aisle, and no one wants to go back there. So it doesn't seem like it's any better."
Anay Shah (01:49:12): "That's right. We're still hot when we go in there. We don't think this is working. We're going to go back to the way we were doing it."
Ben Shwab Eidelson (01:49:18): And it drives Christian nuts. The whole point was that you actually want the hot aisle to be hotter and the cool aisle to be cooler. That separation is what drives the efficiency. You don't want the air to mix. He knew in his bones, "No, you're running in a more efficient way. You have to see that."
Anay Shah (01:49:35): NTT couldn't see the change or measure the efficiency gains of the hot aisle/cool aisle containment or the other recommendations that Christian was making. So he invented this incredibly elegant metric and kept it internal to HP. They continued to implement it in HP for six years until his good friend Chris Malone suggested he publish a paper and present it at Ken Brill's Uptime Institute in 2006.
Ben Shwab Eidelson (01:50:02): At the Uptime Institute, they presented the paper and kicked off a new organization called The Green Grid to publish PUE and other metrics. The team included Christian Belady, Paul Perez, Bruce Shaw, and Larry Vertel. These folks later became CTOs of Dell and led big teams at AMD. They thought it was just going to be an internal "here's the thing we tried" kind of paper. It hit the industry like a storm because the PUE race was now on, and Google wanted to win. Many enterprises had PUE numbers around 2, meaning twice as much power went to lights, cooling, and everything else compared to the amount of power that actually ran the IT hardware. Google pushed it down to 1.1, meaning only 10% of the power was not directly used to power the computers.
Anay Shah (01:50:50): Their internal teams took hold of this and ran with it. They used this rethinking of the data center from the ground up as one computer system, with software being the reliability layer, not the hardware. This, along with innovative geographic placement, drove performance and PUE. It's come to a point where it essentially can't be optimized past one.
Ben Shwab Eidelson (01:51:16): Before we get too far ahead of ourselves, let's talk about what it is to go to one of these data center campuses.
Anay Shah (01:51:23): You're driving up into vast open space where you suddenly see massive buildings growing out of the ground. From the sky, it maybe looks like a distribution center. You can see some steam flowing out from the building. But as you drive into this campus...
Ben Shwab Eidelson (01:51:42): You start noticing: "Why would there be so much power infrastructure if this is just a warehouse?" Because you're seeing big battery packs, extra turbines, or generators.
Anay Shah (01:51:52): The other thing you'll notice is that security is everywhere. You'll have to pass through a series of checkpoints as you get to the front desk. Maybe they're powering Netflix, but they're also powering Pentagon operations. They have an enterprise obligation, so security is paramount. So, once you're approved, you manage to get through multiple layers of gates.
Ben Shwab Eidelson (01:52:14): And then there's a moment—the moment of big reveal—when you walk in, and it finally hits you: the rows and rows and rows of machines that just go on further than...
Anay Shah (01:52:24): The eye can see. These facilities are bigger than a football field, and they're lined with racks of blinking lights and machines. You're hearing this constant whir, and you're noticing the temperature. When you walk in, it might be fairly comfortable.
Ben Shwab Eidelson (01:52:42): I think 80 degrees is average in a Google data center today. What did it used to be like?
Anay Shah (01:52:47): You had to keep these machines cool, and the best way to do that was to chill the room. So it was frigid; the entire room was blasting with AC.
Ben Shwab Eidelson (01:52:57): Back in the 80s, you'd just wear a thick sweater all the time. Now they can comfortably hover around 80. That's because of the really good containment they did of hot and cold aisles. Now you've taken in the machines, adjusted to the noise, and they've given you earplugs to get comfortable. You look up and start to see the infrastructure coming in, powering these servers and connecting them. Google famously had bright-colored wires and marked where the plumbing was. But it wasn't always like this. If you go back to the 80s, I think it was quite different.
Anay Shah (01:53:27): You're now building right on the floor. But one of the ways to cool the data center decades ago was to raise the platform. So you'd actually look down and see that you're walking on panels above a hidden space where cold air was being pushed through.
Ben Shwab Eidelson (01:53:41): Per the trend of everything we've talked about, with Google driving the evolution, the question was: "What is the cheapest way to most efficiently do the thing we're doing?" And it stopped being those raised floors over time. Hopefully, that gives you a little taste of what it's like to walk into one of these. There is a nice Google podcast called 'Where the Internet Lives' that really gives a nice tour of exactly this.
Anay Shah (01:54:00): It's a Google and Latitude Media production where you can actually hear the sounds as they record inside the data center. So the whir is very palpable.
Ben Shwab Eidelson (01:54:12): Google's investments continue to grow. They continue to vertically integrate, deeper and deeper. They go to their own sites, start buying up their own fiber. They took advantage of some of that dark fiber that we mentioned earlier, snapping a lot of it up. All of this is a capacity strategy. Ultimately, they want to have the capacity redundancy to drive their own cost down. While, mind you, they're building the highest-margin, best business in the world in the form of search and ads. So that whole flywheel allowed them to continue to invest in this infrastructure. Google had explosive demand, as they were the web company here. They had explosive internal use driving the value of the vertical integration. But the whole industry saw what was going on with AWS after its launch. And Google was one of many companies that realized they had some assets to put toward that, too. So they put together a team that launched App Engine in 2008. It let you run apps on Google's infrastructure, but was very opinionated about the app. It only supported Python with a particular framework. So it was quite different than Amazon's basic building block EC2 approach. It wasn't until 2013 that Google launched a general-purpose virtual machine capability. As a result of this approach, as well as Google's lack of enterprise sales and marketing motion, it took quite a while for their cloud program to get running. Things started to shift in 2014-2015. Google went public about its containerization strategy. If you recall our discussion of virtualization: you don't actually need to package up the whole operating system. You don't really care that it's running Windows or Linux. You actually generally just want to run the application. Containers are really a way to do that at the application level. This continued even further more recently with the idea of serverless functions.
Anay Shah (01:55:59): So Google in 2014-2015 took the world by storm by open-sourcing Kubernetes and the Google Kubernetes Engine. It very quickly became the default way for programmers to use containers and migrate away from virtual machines. Now, they still didn't have an enterprise sales and partnerships muscle. But Ben, you were there at this time, right?
Ben Shwab Eidelson (01:56:18): That's right. That's when they brought in Diane Greene, who we previously talked about as the founder of VMware, to help build that muscle. By 2018, their cloud share was 7%, AWS was 34%, and Azure was 15%. So they were a distant third. Following through to 2019, a new leader came in who has been the leader since, and he has made big pushes, growing massively. Their cloud business is now a $13 billion business, growing 32% net-net. You look at Google's story, and it's interesting. They've always been technically ahead, especially internally. They've always had amazing dogfooders—meaning people inside using their infrastructure. The problem was they did not know how to sell to this customer. They were not really a developer platform, especially to the enterprise. This is a market Amazon had cracked, so they really had to learn new skills despite having the best tech in the entire ecosystem. The other story is that Google's vertical integration sometimes went too far. They figured out the best way to do this for themselves, but sometimes that led to blind spots in what someone not in the Google ecosystem might need. So, the idea that every developer was going to build a new Python framework that Google released was a little self-serving when it turned out engineers might just want maximum flexibility, which Amazon deeply understood.
Anay Shah (01:57:33): And it all starts with giving the industry an entirely new way of thinking about it: Treat the data center as a single, evolving machine. Build the hardware to suit the software that will survive its own failures. Put the buildings in thermodynamically optimal and grid-optimal locations, and publish your math. Showcase your performance and your PUE so that everyone can have FOMO and try to play catch-up. So, Google was spending billions on land, buildings, and hardware, designing its own infrastructure and overturning the data center world by building software to bind it all. Which forced the entire industry to meet hyperscalers like Google on its own terms. Competition was heating up. It wasn't just Amazon and Google. There were even more household names quietly transforming themselves.
Ben Shwab Eidelson (01:58:28): By the mid-2000s, Microsoft was unquestionably the software king. Windows was powering 90% of desktops. Office was licensed to print money. But things were changing. Broadband penetration was climbing. We started to see the emergence of connected phones—BBM and BlackBerrys and things like that at the time. And internet-native companies. They saw Google and Amazon building, climbing, and proving that you didn't need to buy software on a CD to run interesting computing applications. Remember, the cloud, as we've been talking about, isn't just about backend developers. Applications themselves were starting to move—not just their backend, but the front end to the browser as well.
Anay Shah (01:59:10): Right, on the browser. You remember, Salesforce was leading the charge here. They launched with this idea of "no software, all in the browser." By 2005, they were doing nearly $200...
Ben Shwab Eidelson (01:59:20): Million in revenue and capturing the mindshare, proving that this could be the future—not just of consumer applications, but also a business product like a CRM.
Anay Shah (01:59:31): CRM enterprise B2B software.
Ben Shwab Eidelson (01:59:34): So you're there, sitting at Microsoft. That is your greatest fear: that you're not going to be part of the application and operating system of the future. That is your whole business. At this point, it's not like Microsoft didn't run any web services. Ten years prior, Gates had sent his memo of the "Internet Tidal Wave." They had invested; they bought and scaled Hotmail; they had MSN; they had Xbox Live. But these were all self-contained services. They hadn't yet thought about how to expose an infrastructure to others to build on. They loved building operating systems that would ultimately run on others' infrastructure. They had built one of the best businesses in history doing this. They would write this code once, print as many CDs as they could, and it was this amazing, high-margin business. They didn't have to go buy other people's hardware.
Anay Shah (02:00:16): They'd print the CDs, they'd print the money. This idea of infrastructure as a service was radical. Inside Redmond, Washington, the pivot for Microsoft was framed as nothing less than existential.
Ben Shwab Eidelson (02:00:28): As is often the case in a company motion like this, it's about people and talent. So they bought a company, Groove Networks, that brought in the famous Ray Ozzie, who had previously led the development of Lotus Notes. They brought him into Microsoft and made him co-CTO of the company. This was so existentially important. Think about what Meta's doing today to staff up talent. It was *that* important at the time. The cloud was the existential thing. "We need the leader of the cloud, Ray Ozzie, to come in and lead the way through this charge."
Anay Shah (02:00:58): "If we're going to bet everything we've built on this pivot, we're going to bring in the best talent."
Ben Shwab Eidelson (02:01:03): So they bought the company, put them in charge, and this was a 5,000-word manifesto about what the future of Microsoft needed to be. It's worth reading the whole thing, and we'll link to it in the show notes. But Anay, would you read that key section that really stood out?
Anay Shah (02:01:17): In Ray's memo, he says: "'Computing and communications technologies have dramatically and progressively improved to enable the viability of a services-based model. The ubiquity of broadband and wireless networking has changed the nature of how people interact, and they're increasingly drawn towards the simplicity of services and service-enabled software that just works.'"
Ben Shwab Eidelson (02:01:40): He ends that section by saying: "'Businesses are increasingly considering what services-based economics of scale might do to help them reduce infrastructure costs, or deploy solutions as needed and on a subscription basis.'"
Anay Shah (02:01:54): "'—just give people the end result that they want and demystify everything that's happening.'"
Ben Shwab Eidelson (02:02:00): That's right. And he ends with: "We must respond quickly and decisively." For a company of Microsoft's scale and history, that was really trying to be a...
Anay Shah (02:02:09): Wake-up call. And didn't Gates and Ballmer kind of give Ray Ozzie a blank slate, like, "You're the leader here. Help us understand where the future is"?
Ben Shwab Eidelson (02:02:17): Not just that, they let him carve out and run this separate from the Microsoft Server and Tools business, which had built Windows Server and SQL Server—which would have been the obvious place to try these things. But they knew that this was bigger than an evolution from what they were doing in their previous software business. This was a transformation: a fundamentally different product, a different business model, and a different go-to-market motion. So when everything had to change like that, it was very hard for the leader of the incumbent thing to keep going. Let's make this concrete. What does this mean for a server or a data center? Microsoft had Windows Server and SQL Server. When we talked about those server rooms that a company would run, they would very often be running Microsoft software. This was the IT sale they had perfected by this point. You would sell the Windows Server license, it would be Active Directory, it would be running Exchange and Outlook for your email, and it would be running your calendar and all these things inside your company's network. It'd be running Windows on their PCs.
Anay Shah (02:03:15): So it's this beautifully connected thing. And this is saying, "No, no, no, we've got to move that server from your bottom line CapEx as a company into OpEx in our data centers." This is a transformational move from what you're providing at the end of the day—from just software that's going to run on someone else's computer—to actual services.
Ben Shwab Eidelson (02:03:32): Ray Ozzie goes for it. He staffs up a team they call Project Red Dog. They bring in Dave Cutler, who had previously led Windows NT and is a legendary programmer, to help lead the architecture of this new project.
Anay Shah (02:03:46): So they're putting together this cloud strategy that is revamping Microsoft internally, and at the same time, ramping up direct competition to the front door of the Internet: Google. They evolved an old MSN Search product into what then became Live Search and eventually Bing, as we know it, in 2009. This was a moment that kind of rippled across the industry, especially for Google's dominance.
Ben Shwab Eidelson (02:04:12): I was at Microsoft at this time.
Anay Shah (02:04:13): Oh, yeah.
Ben Shwab Eidelson (02:04:14): I joined Microsoft out of college in 2008 and was there in this moment: they had Windows Live Search. They briefly called it Live Search, then came this big rebrand to Bing. Microsoft was willing to spend a lot on this repositioning, on this marketing—so much so that they gave Yahoo a very sweet deal to power Yahoo's search and ads and gain some infrastructure market share.
Anay Shah (02:04:35): This catapulted them into getting up to 20% share, which is no small feat in an exploding market that Google has been dominating.
---
Ben Shwab Eidelson (02:04:45): It's important to remember that doing search well and search ads well is a really hard scale problem that essentially only one other company had cracked, and that was Google. To respond quickly with an active, updated index of the internet, and to do this auction for the right ad unit—all of these things that you have to build—is what propelled Google to build their vertically integrated servers. And Microsoft was now putting themselves in the position of having to solve those same really hard technical problems, in addition.
Anay Shah (02:05:12): To those really hard technical problems, we saw with Google that in order to effectively run this search business, you have to have a lot of compute power behind you. You now have to get into the data center business and build out infrastructure that is your own to drive down the cost and drive up performance. We mentioned Christian Bellady previously. He was at HP in 2007, gets a recruiting call from Microsoft, and his initial reaction is, "What the hell's a mechanical engineer going to do at Microsoft?" Because Microsoft is known as the software company, and so he turns them down a couple times.
After the third call, he decides to go up for an interview, enjoys the process, gets an offer, but doesn't think much of it. And I believe he's at his parents' house when he suddenly gets an email from Bill G. (Bill@microsoft.com) explaining why he should accept the job offer. And Bill essentially says, "Everything's going to the cloud. We're investing for the cloud business, and we need to build out this new infrastructure as core to the future of Microsoft."
Ben Shwab Eidelson (02:06:24): This seems like a wacky idea to go there, but Bill G emailing me? Like, you're there with your parents, and you've got to give this a go. Let's pack our things, move to the Seattle area, and join Microsoft. And so at this point, he starts there, and Microsoft had just finished building their first real, more integrated data center build-out. This is in Quincy, Washington, in Eastern Washington, a 13-megawatt build. And he walks in there, and everyone's looking around, and the first thought they have is, "We're going to get fired because there's no way we're going to fill this thing with servers." There's just no way that we have the demand.
Anay Shah (02:07:00): And this was their first massive-scale, purpose-built cloud data center campus. This was Microsoft entering the infrastructure business.
Ben Shwab Eidelson (02:07:11): And similar to what we saw with Google at The Dalles, a lot of the same things that attracted them to Quincy were power from the same Columbia River. They're getting cheap 1.9-cent per kilowatt-hour power at the time to power this. And the climate was that dry, cool climate that allowed them to be really economical with using outside air cooling for much of the year. They had multiple fiber routes. It's in Washington, so they could connect their Redmond headquarters directly to Quincy.
Anay Shah (02:07:34): The environment, the fiber, the power—it all comes together. And they, like Google, knew they had to work with the local community and the local government. And so they actually worked with the City of Quincy to build a Quincy water reuse system to treat and recirculate the cooling water to reduce the dependence on the local community.
Ben Shwab Eidelson (02:07:53): They built Quincy. They have it up and running. He's worried they're not going to fill it, but they kept going. Azure ends up launching in 2008, which we'll talk about in a moment. But I think it's very easy for a big company that's bet a lot of resources building something internally to think that they have an idea of what demand is going to look like. But they overbuilt in Chicago. Then they had it mothballed, they tried to sell it, and then nine months later, they were so happy. They didn't sell it because they needed the space.
Anay Shah (02:08:16): It was hilarious. They tried to sell it to the government.
Ben Shwab Eidelson (02:08:18): So, what's going on there, and what's happening, is they're launching this project, 'Red Dog,' which would become known as Microsoft Azure. And it's hard to predict the scale of the utilization of infrastructure as you launch it. So let's go back to October 2008—the big reveal. This is Microsoft's big annual Professional Developers Conference, and Microsoft unveils not Microsoft Azure at the time, but Windows Azure.
Anay Shah (02:08:46): And this wasn't just Windows running somewhere else. It's a platform for developers to build and host applications on Microsoft infrastructure, paying only for what they used. Rent some storage and compute, pay for what you use.
Ben Shwab Eidelson (02:09:00): And the key thing that you mentioned is, it was designed for those Windows developers. So it used all the familiar tools: the .NET Framework and Visual Studio and all these things. And it made it easy for you to take your work that you'd done over there that would run on Windows Server right inside of your office, and take that same code and shift it to run on Microsoft servers. What a powerful integration and move! It kind of split the thread between AWS, which gave these really basic building blocks—S3 and EC2—of this unopinionated 'run your own operating system, bring your own full thing, and we'll just make it work' approach.
Anay Shah (02:09:35): Right. They were agnostic to what you were doing on top of it and how you were doing it.
Ben Shwab Eidelson (02:09:37): Exactly. To Google App Engine, which was overly specific and cute about it, in a way Microsoft was almost more Google's approach, but it was the same code and the same thing that a developer had already used. And so they had this massive developer ecosystem and integrators, and this whole kind of motion around it.
Anay Shah (02:09:54): It wasn't just easy for the developer community; this is Microsoft that has an enterprise sales engine. They've got a go-to-market motion. They have the finance department that knows how the sales team is going to operate and how to budget for it. And so they're bringing this world-class distribution and, most importantly, the trust from this massive customer base to bear with Azure. And the adoption is extremely rapid because people are already embedded in the ecosystem, and here you are coming with an improved product, and the enterprise base adopts it very seamlessly. Yeah.
Ben Shwab Eidelson (02:10:28): And it's not the story that Amazon has to make for the first time, which is, "Hey, come build your thing with us in the cloud," and introduce this. This is four or five years later, and it's sitting there like, "How do we think about the cloud?" We don't have an agreement with Amazon; that's not an existing vendor relationship. Whereas with Microsoft, you're like, "This is the agreement we have, this is the IT relationship we have, this is the full trust." If you're telling us this is how we can move to the cloud in a seamless, secure, safe way that's going to integrate with Active Directory and Outlook, and I just don't have to run the machines in my closet anymore? Sign me up.
Ironically, I think some of the people they had the hardest time signing up were the engineers and product managers in Redmond, Washington. We were actually talking with Ben Gilbert of Acquired about this, and he at the time (this was 2012, 2013) was working on the web version of Microsoft Word. So Microsoft is finally going to compete head-to-head with Google Docs on product and ship a version of Word that you can use via your browser. And of course, the desire was for everyone to use Azure. Did they? Absolutely not. They said, "No. This is the custom server we need. We need this. Many of them can scale this big, and we had to do some custom JavaScript rendering, all this stuff." We want our own servers, probably still in Quincy, but we're not using the Azure APIs and layers.
Anay Shah (02:11:43): So the own employees are demanding all this heterogeneity, resulting in a wide variety of server SKUs.
Ben Shwab Eidelson (02:11:52): Amazon, for context, building out AWS, I think had something like 20 different types of machines that were running AWS. Google, in their whole build-out, standardized on this vertically integrated, commodity-built server design. And so they would just keep it down to a handful of machine types. So supposedly at Microsoft, there were dozens and dozens of different SKUs that Microsoft was maintaining. So instead of calling this a server farm, someone called it a server zoo. You just think about where they're coming from in the evolution of all these different web services and different teams building their own thing. It's a whole different world. And around this time, I think energy starts to become an interesting layer as well.
Anay Shah (02:12:31): So similar to Christian getting recruited in 2007, thinking, 'What the hell am I going to do at Microsoft?' he then goes out to recruit an energy expert. Brian Janes. In 2011, Microsoft saw the need for an energy person to be in-house.
Ben Shwab Eidelson (02:12:47): I'd say Microsoft saw the need, but I think Brian was confused.
Anay Shah (02:12:51): Brian was like, 'I do energy, you don't need an energy person.' And his thought was, 'This is a dead-end job.' Being an energy person at a software company doesn't make sense because what they're trying to do is all about land, it's all about fiber. Energy's a distant third or lower down the list in the priority of how you think about scaling this type of infrastructure.
Ben Shwab Eidelson (02:13:10): Yeah, I think at this time, for context, utilities were happy you'd show up with your 10 Quincy, 13-megawatt data center, and utilities are like, 'Sure, great, sign up!'
Anay Shah (02:13:20): Yeah, generally power demand is relatively flat. There's excess power available, and they're happy to sign you up and increase their profits because it's not going to drive any new infrastructure needs on their end yet.
Ben Shwab Eidelson (02:13:34): And how things change! This is 2011, 2012. Fast forward a little more than a decade to where we are now, and we'll get there.
Anay Shah (02:13:43): But Brian took the job and ended up building a phenomenal team. And within a few years, his team was actually the decision-maker of where they were going to build out Microsoft infrastructure. And so it's a lead-in to us actually starting to care about where our energy is coming from and the whole complexity around where you're sourcing and doing these clean energy build-outs that Brian's team was leading.
Ben Shwab Eidelson (02:14:03): The team thought it was a real kind of planning challenge, right? And so they're bringing this up to leadership around, 'Hey, we need a lot of money to invest now to build this out.'
Anay Shah (02:14:12): They're doing their annual budgets, they're projecting years in advance, they're getting into their mid-year review.
Ben Shwab Eidelson (02:14:19): Microsoft's in this weird transitional phase. They have Bing that they're still growing; they have other services. Xbox is booming, Azure is still early innings. So it's not the dominant use of their internal server build-out, but it's a lot of different teams to juggle and predict. And so this forecasting challenge is a real one. You bring this to leadership, and what are you there to say?
Anay Shah (02:14:38): And you've talked to all the business groups to figure out what they're thinking, and you go to Ballmer with your prepared notes, and within a few sentences you get interrupted, and Steve Ballmer's like, 'Ah, the business groups don't know what they need. Let me tell you how to do this.'"
Ben Shwab Eidelson (02:14:52): And he says, "Give me the Excel sheet. You just draw a straight line from how growth has been going the last few years, and you just project it out, and you just build the data centers that that line shows you to do."
Anay Shah (02:15:04): Just a straight line from here to 2020. The business groups, they don't know what the plan is.
Ben Shwab Eidelson (02:15:08): There's actually a lot of wisdom to what Ballmer's saying because you realize that every team is trying to be excited about what they're building and Azure is going to have all this demand, but no one really knows—especially more than 12, maybe 18 months out. So how can you plan if you're building long-term, multi-year build-outs for what anyone's going to need?
Anay Shah (02:15:28): This moment of growth? It was quite difficult. And Janes' team found that he was consistently under-forecasting. But he did go back many, many years later, and he was kind of curious: how accurate were his forecasts compared to the straight line in 2020?
Ben Shwab Eidelson (02:15:43): He looked back, and Ballmer was off by how much?
Anay Shah (02:15:46): 5%.
Ben Shwab Eidelson (02:15:47): 5%. So they should have just followed the line. So sure enough, Microsoft keeps at it, and the power team at Microsoft—the data center team at Microsoft—becomes one of the biggest builders in this industry. They shift from leasing square footages to megawatt-based deals, as they become a major, major hyperscaler.
Anay Shah (02:16:08): There's a big learning journey for them, right, because they were a software company, and they had to grow very quickly into an infrastructure company. It was not an easy pivot to make, and they had to do that globally. The Microsoft team had the blessing of this entrenched enterprise customer base that was very loyal, very profitable, and very global. And so their go-to-market strategy was actually to follow their customers. And that meant that within a few years of launching Azure, they were getting calls from customers in Frankfurt, in Amsterdam, in Singapore. If their German multinational had data sovereignty questions, Microsoft would go and stand up capacity in the region.
And so by 2015, they had announced over 20 regions, which was actually more than Amazon at this time. Each anchored on an interconnection-rich metro and typically supported by entering with colocation facilities—perhaps the Digital Realties of the world—before moving on to dedicated Microsoft campuses once they built up enough critical demand. But this go-to-market strategy forced them to innovate and expand all over the globe, and it drove massive results quite quickly.
Ben Shwab Eidelson (02:17:33): And to that point, you fast forward to today, and Microsoft's business is in significant part the Azure business. Azure is a core growth driver. It's now 75 billion of revenue a year and growing 40% year-over-year. And so this company that was 'the software company' ('we write software, we sell software, those are our core platforms, those are our products, that's what we do') is no longer just a software company; they are, in large part, a data center company. So we've gone through AWS, Google and GCP, Microsoft and Azure. But there was another company that many thought for many, many years was just a toy. Let's talk about Facebook.
Anay Shah (02:18:16): The Facebook, the Facebook.
Ben Shwab Eidelson (02:18:19): In February 2004, Facebook's running out of a single server in Zuckerberg's dorm room at Harvard University. Early days caused crashes. In fact, many of his earlier products, including FaceMash, the 'hot or not' for Harvard campus, melted down the servers in the Kirkland House, and the IT department at Harvard was not pleased. But then it becomes a real company, right?
Anay Shah (02:18:40): And they start expanding to other campuses, and you've got different schools putting on different servers in different data centers.
Ben Shwab Eidelson (02:18:49): In a way that I think creates automatic resiliency because if the server crashed for, whatever, Penn, it's not going to bring down Harvard.
Anay Shah (02:18:56): And this isn't their concern, right? They're building this viral, game-changing platform that is exploding, probably beyond their expectations. And new campuses are signing up. So they're not really concerned about the infrastructure, but it's just growing organically. By the time they launch News Feed, you've got hundreds of millions of people refreshing this site, uploading photos by the billions, opening up Messenger just as a reflex of the thumb.
Ben Shwab Eidelson (02:19:23): And the company's spending all this money; they raised VC dollars and they're not happy spending all this money on fancy servers and colo cages.
Anay Shah (02:19:29): And so they're realizing, 'Why is everyone making so much money on us?' There's got to be a better way to do this. And there are two broad ways that they go about this. One is similar to what we've seen with Amazon, Google, and Microsoft, which is the thought that, 'Hey, we can do this better, cheaper if we build our own custom data center.' And so their first purpose-built facility was announced on January 10th and began operations in 2011 in Prineville, Oregon. And they soon followed with North Carolina, Sweden, and Iowa, and expanded rapidly. But Prineville was, just like all the others, quite intentional and quite strategic.
Ben Shwab Eidelson (02:20:09): Yeah. And so similarly, in this kind of Northwest region, you had cheap power, you had a dry, cool climate, and you had Oregon's incentives starting to come into play where they offered long property tax abatements on any improvements they made. All of this combined to make a great site for their first 'Lighthouse' data center. And if you look at the numbers of server growth, if you go back just to 2008, they had 10,000 servers; in 2009, 30,000 servers. Supposedly by 2010, when they are really kicking off and building the Prineville project, they have around 60,000 servers. They're trying to figure out how to scale up.
Anay Shah (02:20:42): And so they decide to engineer the building and the servers together. This very purpose-built facility. And just like the others, they realized that they can beat industry standards. And so they very famously and publicly came out with this first purpose-built facility with a very aggressive target of a 1.15 PUE. Industry average is 1.5. Historically, they were north of 2, and they were able to report 38% less energy, 25% lower cost against prior facilities. And do you think they ended up beating their PUE?
Ben Shwab Eidelson (02:21:15): No way.
Anay Shah (02:21:16): Smoked it: 1.07.
Ben Shwab Eidelson (02:21:20): So that means that only 7% of the energy going to run the data center went to anything other than powering the IT equipment. That is phenomenal.
Anay Shah (02:21:30): Amazing.
---
Ben Shwab Eidelson (02:21:30): Did they keep this to themselves, how they did this? I think this is what sets Facebook's approach apart.
Anay Shah (02:21:38): This first part they needed to do, and they executed extremely well. But the second thing they did was far more revolutionary. In a very secretive world of data center development where each hyperscaler kept their builds to themselves, in April of 2011, Facebook open sourced their blueprints. They announced the Open Compute Project.
Ben Shwab Eidelson (02:22:01): It's hard to overstate how radically different an approach this was. This is an industry that kept the design of the servers in the data centers extremely secretive. They viewed that as core IP, differentiation, and notions of security. Google would publish papers, especially on things like PUE, and be visible about metrics that they wanted to highlight. But they famously didn't let anyone into their data centers until this really changed the game.
Anay Shah (02:22:29): Their motivation is pretty interesting. They look around at all the other big data center companies and they're seeing all the margins that are being made. They're like, 'Well, wait a minute, if we publish our blueprints and we get everyone else to buy in and publish theirs, that means we can drive down the costs by standardizing what we're building.'
Ben Shwab Eidelson (02:22:50): Ultimately, what they want is for their suppliers to be in increased competition. The best way to do that is not to make one deal with one supplier, but to say, 'Hey, suppliers, this is what we need, this is what we like. You make this, we'll buy it as long as it's the cheapest one out there.'
Anay Shah (02:23:04): Yeah, it flips the power dynamic. Now, instead of the vendors and suppliers dictating what the specs are and having multiple different specs for multiple different customers, they're able to standardize it and say, 'You're going to respond to the Open Compute Project standards.'
Ben Shwab Eidelson (02:23:17): And it was directionally aligned with all the innovation that we saw at Google. There's actually a big parallel echo here, right? Google built all of these in-house software orchestration tools that we talked about that made the data center reliable. Facebook grew up in that open-source world and benefited from that, and was able to build on top of that.
So it's not surprising that they're the ones to look at the hardware design and say, 'Why is this any different? We don't want to be the sole developers of this part of our infrastructure.' It actually benefits us to have other developers and the ecosystem in part because — and this really gets to a business model motivation — we're not trying to sell that innovation. We're not trying to sell anything to another business other than ads on our feed. The better and the cheaper this infrastructure is, the better our business margins are going to be.
Anay Shah (02:24:05): Yeah, we want to take this cost item and reduce it, so we can put more of our budget, more of our focus, onto the core business. They're looking at it and they're like, 'Well, the industry is using a 19-inch rack because that's what the telcos did.' 'Well, that's not suiting what we needed. There's no reason for the 19-inch rack. It was just a legacy element.' So Facebook decides to do it differently. They widen it to 21 inches and publish that to make it more efficient and fit their needs better.
Ben Shwab Eidelson (02:24:33): It's this funny thing how companies are open strategically, so it's important to be clear-eyed about what they're open about and what they're not. Would Facebook publish publicly the exact way that their News Feed ranking works? No. But that's very different than this infrastructure layer that benefits them when the ecosystem adopts it.
I think one thing that's amazing about it is how much the industry rallied around it and started to join it. OCP ripples across the industry. Microsoft starts bringing their designs into it. Google contributes big improvements, like we talked about, their 48-volt design. Telcos and everyone joins this, and it has the intended desired flywheel effect.
Anay Shah (02:25:12): It did normalize the idea of transparency, and this was right around the time where companies like Google at first were publishing their PUE. So, this notion that we can be transparent with our results as a way to drive the industry forward, bring down costs, and make this whole part of the business easier for everyone, spurred more competition.
Ben Shwab Eidelson (02:25:33): This kicked off the sustainability race among these four hyperscalers. I would say this started all the way back in 2007 when Google announced that they would achieve operational carbon neutrality.
Anay Shah (02:25:43): Each of the companies followed, and they were serious. Right? Microsoft committed to carbon-neutral operations in 2012. It drove site selection, their power purchase agreements, and how they negotiated and determined who they were going to work with and their relationship with their utilities.
Ben Shwab Eidelson (02:26:01): And it just accelerates. Amazon kicks off the Climate Pledge to get net-zero by 2040. You have zero-waste goals. Microsoft announced water-positive by 2030 in 2020. Google catches up with a similar announcement a year later. So you just have this era of accelerating commitments and goals.
I think it's good to ask, why is this? Does it just feel good? Having worked at some of these companies, I do genuinely think the leadership wants the company to have a positive environmental impact, and it is the economically right thing to do as well. These goals of efficiency lead to your data centers, which are a growing item of operational scale, costing you less and less.
The other thing this does is catalyze different renewable buying and procurement habits, right? These teams, like the one led by Brian Janis at Microsoft, become energy-buying procurement machines. They're looking to help accelerate that next solar project, that next wind project, to then be able to buy — via what's called a virtual power purchase agreement — the accounting for the clean electrons. Google pushes this even one step further to say, 'Hey, we want to buy electrons that are generated at the same time that our data center is using electrons.' So we want this notion of 24/7 carbon-free electricity. This really pushes the clean energy ecosystem forward, I think, in a material way.
If we look at these three commercial cloud businesses of AWS, GCP, and Azure, they are behemoths, as we said. Azure hit $75 billion last year, GCP surpassed $50 billion, and AWS was on a run rate of $111 billion for that business. You total that up, that's $236 billion a year spent only on the direct cloud infrastructure businesses.
Anay Shah (02:27:51): Let's put this in perspective. That's $236 billion across these three companies spent on the infrastructure. US consumers spend $500 billion annually on electricity. So we're talking about half the spend of all residential electricity across this country.
Ben Shwab Eidelson (02:28:10): So I think it is safe to say that we have now entered the utility era of computing. It just so happens that the utilities are Amazon, Microsoft, and Google. Meanwhile, Meta, first of all, doesn't want to buy power from those three utilities. They also use so much themselves that they want to build their own and have no interest in selling access to it. They just need their own data centers to run everything that they're doing in-house at such an immense scale.
So we end the 2010s with a very mature cloud, right? I think it's almost in the background. People don't even need to talk about and explain the cloud anymore. It's like, 'Of course, this is how you're going to start and run your company.' The functionality is all grown up. All of the companies are now running containerized Kubernetes things. Your services are portable. Yes, they have their functional differences and their sales differences, but they're all kind of mature and stable, multi-region. All of this stuff going on.
Anay Shah (02:29:04): Nobody asks, 'Do you have a cloud strategy anymore?' It just is the way business is done.
Ben Shwab Eidelson (02:29:08): You also have the new startups that have built out the missing pieces. You have the Snowflakes of the world that are helping build more specialized databases. You have Datadog helping you with orchestration — real significant public-scale businesses that have built the missing components of the cloud moment.
Anay Shah (02:29:23): And the feeling right now is one of maturity of the cloud infrastructure, maturity of a new large-scale data center build-out, and at the same time, a huge pressure to continue to build. There is a global race; it is fierce, it is competitive, and it has many, many more players than just these hyperscalers. To give you a sense, Microsoft scaled from 35 regions to 75 regions over the period of a couple of years. So they were adding an entire region a month. This is data center region build-outs. Deal size was ballooning at the same time. A decade ago, a large lease might have been 5 megawatts. You fast forward to the end of the 2010s, and every one of the hyperscalers is reserving 100-megawatt campuses. So, a 20x growth in the size of the data center over this decade.
Ben Shwab Eidelson (02:30:21): These two big points fit. The whole idea that the cloud now can be taken for granted and invisible is because of these scaled, global build-outs. You don't get to do that if the thing isn't just working as usage is exploding across the entire industry.
Anay Shah (02:30:35): The industry has matured around private equity money coming in. You've got developers treating this as a real estate asset class, and they're stockpiling land, pre-building substations ahead of demand because they know what the hyperscaler or the next large company is going to need, and they're going to snap up that capacity. So this build-out is fierce on land.
Ben Shwab Eidelson (02:30:56): Not only is it fierce on land, it's fierce in the seas. It's not enough just to have your own servers and your own buildings and your own power infrastructure. You need confidence that you can stay connected. The best way to have confidence is to put your own cables underwater. So instead of just renting capacity, which of course they continue to do, Microsoft, Meta, Google, and others start financing and building their own undersea network. This way, they can control end-to-end.
You think about Google's vertical integration: it starts with the box, it goes to the rack, then it goes to the building, then it goes to the campus, then it goes to the wires that connect the campus. That's how you get confidence in what you're building. This leads to improved economics and improved performance.
Anay Shah (02:31:38): And performance continues to drive forward. But by the end of the 2010s, PUE had plateaued around an incredible 1.1. Only 10% of the power being used in this facility is not directly for the server and IT equipment. So, as we talked about in the sustainability race, the bragging rights had shifted. The marketing teams had moved off of PUE, and now they're talking about carbon intensity (CUE). Now we're talking about water intensity (WUE), and the holy grail metric here is who is launching 24/7 real-time renewable-powered data centers. Everyone's putting out their climate pledges, building into their plans, and driving the entire renewable energy and PPA world forward through this genuine focus on carbon intensity.
Ben Shwab Eidelson (02:32:28): Intensity. The cloud by 2020 was doing what it needed to do. So we now had this scaled, mature global technology for storing things, computing things, connecting all of us, streaming content, and powering video calls. Why might that be helpful as we go into 2020?
Anay Shah (02:32:48): Well, Ben, we're now upon everyone's favorite moment in recent history: the dark days of the COVID pandemic — a time where we all had to stay at home and a time where society, systems, public health, everything was stretched to the max. The same actually goes for our data center infrastructure. While the 2010s saw fast, consistent growth of cloud computing, COVID compressed five years of adoption into about 18 months with everyone at home. Video calls, gaming, online collaboration, telehealth, e-commerce, SaaS — everything surged all at once. We all remember the endless video calls, but let's put that into perspective a little bit.
Ben Shwab Eidelson (02:33:35): Going into this era, Zoom had something like 10 million folks as daily meeting participants. Fast forward to March 2020: they hit 200 million in four months. One month later, 300 million. So, just explosive growth. Google Meet similarly saw 3 million new users per day in April. Peak usage was up 30x since January, and gaming was exploding. They're at home, looking for entertainment, so Steam is shattering records.
Anay Shah (02:34:03): Netflix and YouTube famously had to throttle down the network speed to lower resolutions to avoid essentially breaking the Internet.
Ben Shwab Eidelson (02:34:11): I was at Stripe at the time, and to the e-commerce point, it was just this explosive moment where, sure, you saw some businesses, perhaps in travel in particular, struggle and go under. But the majority of Stripe's users were exploding. Everyone was ordering stuff online. Instacart and DoorDash were booming. It's easy to say five years of growth in 18 months. What does it actually feel like when you're working on the infrastructure? It feels like everything is breaking, and you're trying to make sure no one in the outside world feels it. We managed to accomplish that, and I'm very proud of that. The API stayed up, and payments were processed even as they scaled rapidly.
Anay Shah (02:34:45): Another company in the crosshairs that seized the day was obviously Zoom. They had to scale infrastructure from 10 million to 300 million users in months. They added servers in colocation metros, they used AWS, they used Azure, and they expanded aggressively on Oracle Cloud Infrastructure. They were just doing everything they could to ensure that those video calls didn't have delays.
Ben Shwab Eidelson (02:35:09): Under the hood, the way Zoom works, you can think of it split into two: the Zoom Control Plane, which was making sure that it would respond to your logins and understand what you are as a participant in a meeting. But then you actually had the video feed, and that would have these meeting zones where Zoom would make sure to connect you to the closest meeting zone in a carrier-dense building. So it's kind of very similar to the CDN networks that we talked about earlier with Netflix, but it had this dual, bi-directional nature to it. They were in a race to add more of these peering machines in those telco hotels that we talked about earlier.
Anay Shah (02:35:41): And Zoom became proof that the cloud's elastic promise was real. In the greatest stress test of its time, capacity could materialize as fast as humanity demanded it. This was the moment. We were in an era of exploding demand, and there are a few key themes that affect the data center world in this moment. One of them is famously known as ZIRP. On March 15, 2020, the U.S. Federal Reserve cut its target rate to the range of 0 to 25 basis points. What does that mean in the infrastructure world? That basically lowers the hurdle rate for your risk for speculative builds. So now campuses and shells can be built out way ahead of leases being signed because the carrying cost of that infrastructure was so low, because your cost of capital is so low.
Ben Shwab Eidelson (02:36:33): You're buying a house. This was the time when your interest rate on the mortgage was as low as possible, so you could buy more house. Well, if you are a company trying to build data centers, it's the same story. You can buy more data centers. Your same amount of principal is going to go much further in building out more.
Anay Shah (02:36:50): As a company in this infrastructure environment, you can take more risk.
Ben Shwab Eidelson (02:36:54): The hyperscalers took advantage of this and started pouring money into building new data centers. But so did Blackstone. In 2021, Blackstone purchased QTS Realty Trust, one of the largest data center builders in the industry, for $10 billion.
Anay Shah (02:37:07): There's so much capital floating around, and the beauty of the data center asset class was that it had started to look very predictable. It was looking like utility-style returns. Everyone's investing in the REITs and the infrastructure debt. This is fantastic for the counties that were building this out, namely Loudoun County, which we've talked about before. Right? Their budgets were suffering in the COVID era, but because they're Data Center Alley, they saw their server equipment tax go past $400 million annually, and that prevented them from needing to raise property taxes to fill their budget gaps.
Ben Shwab Eidelson (02:37:46): I think at some point their vacancy was below 1% in all of this. So there's just so much demand for space, colocation, interconnection. Everything is just flying off the shelves.
Anay Shah (02:37:56): Another major theme here is what is defining the build-out. What you find now is that it starts to grow the panic around the urgency for electrical supply. It was both the amount of supply of power that you want, as well as the time to power, because there is this capital looking to go to work. The question was actually, 'How soon can you get me the power?'
Ben Shwab Eidelson (02:38:23): This is new, right? The scale of the data centers earlier — these 5, 10, 15-megawatt data centers — didn't necessitate a big conversation with the utility. We're now talking into the 50 to 100-megawatt builds, and this changes the dynamic with suppliers. We had folks in Ireland saying for the first time, 'We're going to pass regulation that says the utility can block a data center from coming in because it's too much load.'
Anay Shah (02:38:47): You can't just walk up to a utility and plug in for 100 megawatts. For the first time, grid capacity and planning was a throttling factor in how fast data infrastructure could grow.
Ben Shwab Eidelson (02:39:01): The era from 2010 to 2018-2019, which was this big shift to the cloud that we talked about, was one of increased efficiency in these systems that ended up balancing out the increase in demand and capacity. So there's actually relatively flat power-use growth despite there being so much growth. While there was more demand, it was a steady increase, and the efficiency jumps could go in concert with the growth jumps.
Anay Shah (02:39:30): So you buy a new phone, you buy a new computer — most of it's all the same, except you just get more processing power because of Moore's Law. That brought about these efficiency gains. We were building a lot of new data centers and compute, but total energy consumption of data centers actually remained relatively flat throughout this period. It was largely due to a few factors. One is that we moved into the...
Ben Shwab Eidelson (02:39:54): ...cloud, and everything we talked about earlier with utilization can then come into play, right?
Anay Shah (02:39:59): Exactly. One study found that the cloud is 93% more efficient than running your compute power on-premises because you have less wasted assets combined with everything we talked about with PUE and driving down that efficiency.
Ben Shwab Eidelson (02:40:14): Well, now all of a sudden you had COVID plus ZIRP, where you're not waiting to get more efficient; you're just trying to build more and more.
Anay Shah (02:40:22): We'd also kind of run out of that low-hanging fruit, right? PUE is at 1.1.
Ben Shwab Eidelson (02:40:28): So if you look at the big four that we talked about before — Amazon, Microsoft, Google, and Meta — between 2017 and 2021, in those four years, they doubled their energy use to 72 terawatt-hours in 2021.
Anay Shah (02:40:40): So much greater demand, combined with a loss of the PUE benefits and the cloud efficiency, means we're in a new paradigm of power usage, and the industry is having to reckon with that.
Ben Shwab Eidelson (02:40:52): It's pretty amazing that we had such a flat period for how much growth that we just talked through. I mean, the whole scale-up of the cloud led to growing usage, while the efficiency gains were so significant. We've run out of those tricks. Right?
Anay Shah (02:41:05): Right.
Ben Shwab Eidelson (02:41:06): The hot and cold aisles are contained. We've virtualized everything; we've containerized everything. We need to come up with some new tricks. I hold out some optimism that necessity is the mother of invention, and we'll figure out some new tricks because we'll have to.
Anay Shah (02:41:18): We have to.
---
Ben Shwab Eidelson (02:41:20): That being said, there is an extreme way that a new power user is showing up in this era. Let's talk about the crypto miners for a bit.
Anay Shah (02:41:29): Let's talk a little bit about crypto because it's not just a hobby, but actually ends up being a competitive buyer in the data center infrastructure build-out.
Ben Shwab Eidelson (02:41:40): By 2022, 100 to 150 terawatt hours. So that's almost 50% of what the rest of the data centers are consuming now going to crypto mining. Go back to 2009, Bitcoin launches, and it's pretty much just a dark corners of the internet thing, right? Someone mining on a PC in their bedroom.
Then, finally, folks figure if you buy GPUs off the shelf, you can go faster, and people start even designing custom ASICs that are designed to do nothing but mine Bitcoin and other coins. By the mid-2010s, mines had become significant power loads, and folks had started to build these out wherever they could find cheap electricity.
Anay Shah (02:42:14): And while the technology behind crypto was interesting from a business perspective, it was less of an IT business and more of a power arbitrage business, because it was all based on how efficiently you could mine and where you're doing it.
Ben Shwab Eidelson (02:42:29): Hard math computation as many times as you can. This is purely solving problems using power. And this is very, very different than everything we've said about data centers before. This is just compute as cheaply as possible.
Anay Shah (02:42:43): And this really comes to a collision point in the real world where you had markets like Iceland and Eastern Washington where Bitcoin miners were outbidding these cloud providers for the cheap hydro and the power that was available.
Ben Shwab Eidelson (02:42:57): Then the crypto industry starts to realize that there's power that's not useful to others. Let's look for ways and places to put crypto mining that you couldn't even put a data center or a normal industrial load.
You're in an oil field and you're flaring excess gas. What if you put a turbine there, burn that, and power a crypto mine? This is how Crusoe, which later became one of the premier AI data center build-outs, got started finding that cheap power source. And so, it's interesting how many of these power-related innovations came from the crypto moment.
Anay Shah (02:43:27): Then, crypto loads were different. They were large, but they were fairly mobile. So that was an advantage in terms of where and how quickly they could place. But it also meant that policy could step in and shape the industry fairly quickly.
Ben Shwab Eidelson (02:43:39): By 2021, most crypto mining was happening in China. There was the cheapest power. It was close to where the hardware was being manufactured. But in 2021, China cracked down. There was no more crypto mining. So all the miners fled to Texas, to Kazakhstan, to Canada. And so, by early 2022, the US share of crypto miners was back up to almost 40%. This is up from 3 to 4% in 2020.
Anay Shah (02:44:03): What started out as a hobby and took over the fintech world and financial infrastructure forced real estate, power, and the compute industry to wrestle with how to connect very large, very flexible loads to stressed grids. And what innovations can be born out of trying to wrestle with that.
On that point, crypto as an industry didn't care much about its footprint. But the rest of the industry, and the data center industry in particular, continued to be very focused on the carbon intensity of their operations. Now, these hyperscalers were tracking their carbon footprints with extreme precision.
Ben Shwab Eidelson (02:44:40): By 2020, Amazon became the world's largest buyer of renewable energy. It's a scale thing. Amazon signed the climate pledge. And for Amazon to hit net zero, they didn't just have to decarbonize their data centers and offices; they had fulfillment centers and travel of unprecedented scale, delivering all the packages.
Anay Shah (02:44:59): And this is beyond just the quote-unquote "right thing to do." As we've talked about, oftentimes the cheapest, most economical way to get new load onto the grid and scale is through renewable power. And in a zero-interest-rate environment, it actually makes these PPAs more and more attractive, and it lowers the hurdle rate. So, a wind project that might otherwise have been too expensive in this environment is profitable and able to be signed on to a Microsoft new build-out.
Ben Shwab Eidelson (02:45:30): It's hard to appreciate the flywheel of more demand for the data center services driving data center build-out, which can be funded by a zero-interest-rate environment, which then all drives all the renewable energy build-out, which is similarly funded by the zero-interest-rate environment. You just had an infrastructure build flywheel running that set us all up for what might come next.
Anay Shah (02:45:53): It was an extremely timely moment in the evolution of data centers and actually served as a dress rehearsal for the unprecedented scale that we are going to see in the AI boom. This moment during COVID, with cheap capital flooding the system, was driving a massive new build-out. Power was starting to become a much more limiting reagent in the development of our infrastructure, and there was a breaking down of our global supply chains which was feeding this build-out. All this happened at this moment, right before a massive boom.
So, now we've spoken about the companies building the big infrastructure, the big servers powering our life, and we have a $4 trillion elephant in the room that I think it's time to introduce.
Ben Shwab Eidelson (02:46:40): It's now time to talk about Nvidia. Nvidia started as a company that was trying to figure out how to make gaming run faster on computers. So, they made gaming chips. They did this for years and years and years. And I remember them recruiting many a student out of the Stanford EE department to go work on gaming chips.
Anay Shah (02:46:59): They were very much tied to the gaming industry. Their growth was correlated to it.
Ben Shwab Eidelson (02:47:05): But it turns out the same thing that you're doing in a game — which is doing a bunch of math to figure out how to render a pixel on a screen — is what you sometimes need to do complex science: a bunch of tough math problems at once. And if you're going to build a cryptocurrency, what are you doing? As we said earlier: a bunch of tough math problems at once.
Anay Shah (02:47:21): A lot of compute, matrix multiplication at...
Ben Shwab Eidelson (02:47:24): Scale, to such an extent that their stock price starts to get tied to what's going on in crypto mining.
Anay Shah (02:47:29): First, they had been tied to the game industry; now they became even more correlated to crypto. And then, in 2018, you have the crypto winter. That fall in the crypto industry dragged Nvidia's stock down 17% in one day.
Ben Shwab Eidelson (02:47:45): Over that period, they got cut in half.
Anay Shah (02:47:47): And so, enter COVID, and you hit a crypto boom-bust cycle. But machine learning algorithms are spitting out more and more recommendations to us as we're sitting on YouTube and Netflix and Instagram. And you know what powers a recommendation model? An Nvidia GPU. And so, growth of these training models takes off.
Ben Shwab Eidelson (02:48:08): And you can see it as a line item, right? In their earnings, they have this carve-out for what they call their data center business, which is really the growth of this business. You see 60% growth.
Anay Shah (02:48:18): This hit nearly $4 billion in just a quarter. And this consisted mostly of their A100 chips. They were selling thousands of these.
Ben Shwab Eidelson (02:48:26): Right. And so, these A100 chips became the backbone of the early era of training models. So, in 2020, Microsoft announced the next stage of their partnership with OpenAI. They committed to buying 10,000 or more of these A100 chips to build a special data center, a special supercomputer for OpenAI to continue developing their models. And remember, at this time, OpenAI was not really a household name; it was...
Anay Shah (02:48:47): ...a research project, right?
Ben Shwab Eidelson (02:48:48): Yeah, I mean, that's right. It was a leading, well-funded AI lab with this deep collaboration with Microsoft, building models that were available via API. And in the same year, they published a paper that really demonstrated something that they had discovered and what they call their scaling laws, showing that the bigger the compute and the larger the data set you threw at the problem, the better the model. And Google was discovering very much the same thing: the more chips, the better the model.
Anay Shah (02:49:15): How convenient. The more chips, the better the model.
Ben Shwab Eidelson (02:49:18): Music to Jensen's ears, I think.
Anay Shah (02:49:21): Indeed. And so, in 2022, right as the crypto crash was coming, OpenAI released a new model, GPT-3.5. It was a huge upgrade from GPT-3, and they had built it in their new Microsoft data center. And they were thinking about new ways to show to the world the value of what they've been able to build.
Ben Shwab Eidelson (02:49:41): Are you saying not everyone wants to use an API sandbox to understand how good a model is?
Anay Shah (02:49:45): Right now it's a little bit hard to access, but maybe if you put a simple box as a UI and create a little chat feature on the GPT, it'd be a...
Ben Shwab Eidelson (02:49:59): Nice demo at least, right?
Anay Shah (02:50:00): Right. Few people might be interested, so they decided to put out this chat function. On November 30, 2022, ChatGPT launched. In five days, they had a million users. This was far and away beyond the team's expectations, as well as their capacity planning. According to Sam Altman, they expected an order of magnitude less interest by January 2023. ChatGPT has become the fastest-growing consumer software application in history. So, after getting a million users in five days, they crossed 100 million users in just two months.
Ben Shwab Eidelson (02:50:41): By this last summer, ChatGPT's website was among the top five most visited sites in the world, right after Instagram. So there was real consumer and enterprise pull. The world had been waiting for AI to be a thing for a very long time, but this was the moment where everyone said, "Okay, this is it. This is the time."
Anay Shah (02:50:59): It ended up with over a billion prompts a day. People were loving this thing.
Ben Shwab Eidelson (02:51:02): And everyone knew, to make it better, we just needed more chips. Why is Jensen so delighted? Where are we now?
Anay Shah (02:51:09): Three years later, more chips, better models, right? Three years ago, in 2022, Nvidia's data center revenue was growing rapidly: $4 billion in revenue that quarter. Last quarter, their data center revenue was a staggering $39 billion. That's it growing nearly 10x from what it was in three years.
Ben Shwab Eidelson (02:51:30): To add $36 billion in quarterly revenue in three years, it's unprecedented.
Anay Shah (02:51:37): This is powered by this discovery of the scaling laws.
Ben Shwab Eidelson (02:51:41): And the timing of their product launch could not have been better. They had the A100. That was essentially for training — scaling your training. But they launched the H100 in 2022. That was designed for this moment. It was the rocket boost of a chip. It added new special math modes tailored for transformers. It added more high-speed memory. As a result, it slashed training time and made inference faster.
Anay Shah (02:52:07): And these H100s are flying off the shelf, from selling a million and a half in 2023 to over 2 million in 2024. And the big players, having learned from past years of the cloud boom and the COVID demand boom, are ordering these in the hundreds of thousands.
Ben Shwab Eidelson (02:52:23): These things are not cheap. Each chip is like a car, right? $40,000 apiece, supposedly. You couldn't buy one chip at a time.
Anay Shah (02:52:32): No. Why would you sell one car at a time when you can put them in boxes?
Ben Shwab Eidelson (02:52:35): They really sold in two models, right? There are training boxes and inference boxes. The training box is an 8-GPU box with a CPU in the middle. They're hardwired with this NVLink connection. What's really interesting about this is it's all about making it effectively one big GPU. It's all about creating this really high-bandwidth connection between these GPUs. Because when you're training, that's what you want. In NVLink, if anyone's ever built a computer, you would have what's called a PCI connection in a motherboard. The way your Nvidia graphics card would plug into your computer was using PCI. It felt like a really fast connection. But it's not fast enough for this. So Nvidia invented NVLink. It's up to 15 times faster than that connection. So these eight GPUs can communicate extremely quickly. One training box is about $500K to buy. Then they also launched these inference boxes. And so, this is two GPUs coupled with a bunch of memory. To be clear, inference is when you're asking ChatGPT a question and it's responding. That process of responding is inference. You want a lot of memory to be able to understand all the content in that response. These boxes were designed with a lot of fast memory, connected to be able to share context across the model.
Anay Shah (02:53:46): Now, selling these GPUs in a box actually has a physical space implication, right? So you've heard us talk about the racks and racks in the data center as far as the eye can see. These racks have slots for servers, and those servers have a traditional power consumption. But when you put in a specialized server, it starts to change the power consumption of that rack and therefore of the square footage and of the data center. So let's unpack, just for a moment, what's happening at the rack level when AI arrives.
Ben Shwab Eidelson (02:54:16): Historically, one rack would be maybe 400 watts. So you'd build a rack with 20 to 30 of these servers. You'd have some networking equipment. So, somewhere around 5 to 10 kilowatts of power going to a rack — that's like 10 hair dryers running at the same time, to give you a sense. Now, these GPU boxes arrived — those eight-GPU training boxes. Each of those is up to 10 kilowatts by itself. So it's the same power consumption and heat as the whole rack was. But you could now pack a rack physically with four to eight of those boxes. So, all of a sudden, you're an order of magnitude up in power consumption, up to 90 kilowatts in a rack. This has a bunch of implications, not just for power, but also for cooling. An inference box is somewhere in between; so a whole rack, maybe, is up to 40 kilowatts.
Anay Shah (02:55:01): One of the things driving this power conversation we've been having is, again, not only the size of these data centers, but now the density of power required within these data centers. This is what makes an AI-driven data center so much different from a traditional data center.
Ben Shwab Eidelson (02:55:21): Just 20 of these training racks in a pod is one megawatt of power. That's on the order of a small, 800-person town powering electricity. So, if you get 200 of these racks, you get to an 8,000-person neighborhood very quickly. You get to a gigawatt of Seattle-scale power.
Anay Shah (02:55:41): As you pack these GPUs that have higher power density, they run hotter, and you can't let them run hotter. So now, how you cool your GPUs and your racks needs to evolve.
---
Ben Shwab Eidelson (02:55:55): The problem is, air can only move heat so quickly. And if you were trying to do that, even with the inference racks, you would need leaf blower-level fans to move the air out. And so you need to start bringing more cooling down to the rack level.
So the first thing that you can look at is retrofitting the racks with what's called rear door heat exchangers. This puts some water cooling directly alongside the rack so that when air is moving over those hot chips, some of that air, instead of just going straight into the hot aisle, can get extracted via some tubes of water running inside the rack. That is a nice convenient half-step because you don't have to change how you're actually cooling the overall system.
The next jump is the jump to direct chip liquid cooling. This is where you mount cold plates directly on your chips and you run water right near the chip to actually extract the heat away, instead of expecting air and heat sinks to do it. And so, increasingly, especially with the most powerful GPUs—and to be clear, each generation of GPUs gets more powerful and therefore has more heat to remove—we are moving to a water-cooled world.
Anay Shah (02:57:03): We've crossed the threshold of physics where air can cool this amount of heat being emitted at this level of density from the racks. And so now it is a liquid-cooling world. Now, at the end of the day, you've got to move the heat from inside to outside. But it's very environment-dependent. In the right climates, you can expel it outside. And that's why we saw some of these data centers looking at Finland and Oregon and certain geographies that enable that to happen. And this consumes very little water. But in other climates, you have large evaporated water cooling towers. This is where the problem of the water impact on the local community starts to come into play.
Ben Shwab Eidelson (02:57:45): Fundamentally, what we're trying to do is we're removing heat through that phase change, right? You're evaporating water; that's a massive transfer of heat. These towers increase the surface area of the water. Water evaporates out. It looks like steam going up from these cooling towers. And it's a very effective and efficient way to cool things down. But it can use up your water.
Anay Shah (02:58:03): And so you end up with this energy-water trade-off in certain environments. And that becomes a big planning focus not only for data centers that are working with their local community, but also those that are power-constrained and built by companies that care about carbon intensity. One study Google did in 2022 found that water-cooled data centers use about 10% less energy, which means they emit about 10% less carbon emissions than many of the air-cooled data centers. Now, the problem is, you can't do this as easily in water-stressed areas.
Ben Shwab Eidelson (02:58:38): So there's essentially this knob you can dial up and down. To be clear, it's possible to build data centers that don't use any water, but they're going to use more electricity to run an air conditioner in your house. You're not using a bunch of water, but you're taking a ton of energy. And so I would argue if we care about water, the best thing we can do is have clean electricity to make this an easier trade-off to make.
Anay Shah (02:59:00): So if you zoom out and look at water, globally, data centers consume over 550 billion liters of water annually, which is a big number. To put that into context, a single 100-megawatt data center can consume 2 million liters a day, or the amount equivalent to 6,500 U.S. households per day, both direct and indirect water usage. So this is a very big, real, and current issue for data centers in terms of where they site their data center, how they get permitted, and the ongoing operations.
Ben Shwab Eidelson (02:59:35): The local question is hyper-important. Some reporting that The New York Times did about the situation in Georgia where, supposedly, when Meta broke ground on their billion-dollar data center build-out, water taps in some residents nearby went dry. And there's some back and forth around whether or not you can prove direct causality. Still working out whether or not that is actually the case, caused by that data center. But it is known to be true that that data center is using about 10% of the county's total water and it's driving water prices in the region to go up.
Anay Shah (03:00:06): So now let's dive into what is the largest constraint to data center build-outs today, and that is power. We've talked about the evolution of power moving up the decision stack. From the early 2010s through the COVID era, and now in the AI era, power is really the limiting reagent.
Ben Shwab Eidelson (03:00:30): The image in my mind is: we have the largest, most complex machine that humankind has built in the power grid that has brought us such evolution in how we live our life. And it is colliding with the new, complex, most interesting machine of the data center. And this collision and the force of it that we're living through right now is defining this period.
Anay Shah (03:00:53): And it is no small collision. It is a function of two things, right? It's a function of speed and a function of scale. So, as we've seen, there is an absolute arms race on compute, because the faster you can get compute online, the better your models are, the more revenue you're going to make. And timing matters a lot here because a model trained in 2025 can become obsolete by 2027. So if you have a two-year delay in getting your data center online because of the power grid, it becomes a dealbreaker. It has to happen now, coupled with the scale of these data centers. And so, as we talked about, a typical data center started as 5 to 10 megawatts. Then, just a few years ago, we were in the 50 to 100 megawatts. And now this year we're starting to see gigawatt-scale announcements. Now, to be clear, these are campuses that make up a gigawatt. There are multiple facilities, but they are still being built out cohesively and will put a strain on the grid and the local community at a gigawatt scale. Now, to put a gigawatt in perspective.
Ben Shwab Eidelson (03:02:00): This is the scale of a city's power consumption, right? Maybe a Pittsburgh or Cleveland. And Google's data center electricity use doubled over four years and was up to 30 million megawatt-hours in 2024 from 14 and a half million megawatt-hours in 2020. To put that all into perspective, that's about 3 million U.S. homes, or around three-quarters of a percent of all U.S. electricity consumption.
Anay Shah (03:02:22): Which checks out, right? If data centers are currently about 4% of U.S. electricity use, I guess it makes sense that Google's about a fifth of that.
Ben Shwab Eidelson (03:02:31): And it's worth pausing there for a second to talk about that 4% electricity use. When I hear that number, it actually seems shockingly small. For all of the discussion of data centers and electricity, it's like, okay, it's 4%. Let's say it goes to 8%. Compared to industry, compared to cooling and heating buildings. It all feels small. So why might this be such an issue of conflict?
Anay Shah (03:02:54): It's a great question. It might have to do with the fact that it's misleading to think about it in terms of total national electricity use, because a data center has a localized and concentrated impact, right? You're not spreading this load across multiple utilities. And not only is it localized and concentrated, it's localized and concentrated in similar areas. Because, as we've been talking about, the network effect of the value of a data center being positioned near the undersea cables and near other network points is where you get a lot of performance gains. And so you end up concentrating yourself in Virginia, in California, in Texas. There's just basically 10 states where you're seeing new data centers come online. And so there's a tremendous impact at a local level—at the electricity prices for that community, that county, and in that state—but not necessarily at a national level.
Ben Shwab Eidelson (03:03:52): We talk gigawatt scale. That is all about these large training clusters. When we're talking about inference—which is, hey, ChatGPT, processing your response—you actually probably want that spread out, right? You want that closer to the edge, and that's where you need it close to the interconnect, because you want it to not have latency when interacting with the user and their data.
Anay Shah (03:04:10): So let's talk a little bit about why you need a gigawatt data center or a 5-gigawatt data center to do the best training possible.
Ben Shwab Eidelson (03:04:17): You don't really want to think about it as one GPU or a rack of GPUs or a building of GPUs. You want, as much as possible, the whole data center to act together. Because the way that a transformer actually trains is that it makes guesses about what should come next, and then it compares that to reality, and then it tunes the weights across it. And you're doing that collectively across the whole model. And to do that, you need all of the computers working on the problem to be able to communicate quickly. Otherwise, the whole thing is training slowly. The faster those exchanges can happen, the better. And so you want the connectivity between chips to be as fast as possible. You want the connectivity between boxes of chips to be as fast as possible. And so it does not work if all of a sudden half of your computers are on the East Coast and half are on the West Coast, because the speed of light across the country is going to slow you down by multiple orders of magnitude than within one campus, within one center. And that's why Meta wants to build a 5-gigawatt campus because that's going to get them the biggest training model possible.
Anay Shah (03:05:21): There's also an interesting nuance here where training models hit higher load factors than an inference model.
Ben Shwab Eidelson (03:05:30): And in a typical data center, you're trying to run this all at once and you're utilizing ideally all your expensive chips. And so all of that leads to a higher utilization and more power and more compute.
Anay Shah (03:05:43): So we talked about the speed. More time to power is time to revenue. We've talked about the scale, now going from 100 megawatts to gigawatts. So this creates an objective function when we're thinking about the power constraint on data center growth. And that ends up being: how do we secure large 24/7 power where you need it, when you need it, quickly and ideally as clean as possible? Should be easy, right?
Ben Shwab Eidelson (03:06:13): Oh, wait. Not to mention there are supply chain bottlenecks in transformers, bottlenecks in gas turbines, and bottlenecks—
Anay Shah (03:06:20): In labor and in specialty skills.
Ben Shwab Eidelson (03:06:22): We don't have the most up-to-date, freshest grid with transmission lines. So this gets into a very tense moment and an area that arguably breeds innovation.
Anay Shah (03:06:33): So the reality is, getting affordable power that is 24/7, quickly and cleanly, is going to be very rare. That Goldilocks situation doesn't really exist today. And so we're going to have to compromise on at least one of those variables. Now you've got a few different options to get these data centers online, right? You can either hook it up to the grid and use power that already exists. You can build new generation of power that's off the grid and power that data center, do some sort of hybrid in between, and over the long term, we can build out large new generation like nuclear, geothermal, and others.
Ben Shwab Eidelson (03:07:14): Watching this closely, there are both a lot of moments of concern, right, because we're at times doing things like keeping a coal plant running longer than we might have otherwise. But there are also a lot of interesting moments of innovation. A lot of growth in storage to help supplement what's going on and smooth out power generation, and co-locating these in new hybrid ways.
Anay Shah (03:07:34): And there's a tension that's largely dictated by timing, right? There's a near-term pain that local communities are facing when you bring a data center online and potentially increase electricity prices. But there's also cause for optimism because you've got well-funded companies who have an interest in long-term sustainable power. And the fact of the matter is, the most affordable, long-term sustainable power we have is going to be clean and renewable. And so this moment actually offers an opportunity for us to accelerate a lot of new clean build on the grid and reshape our electricity infrastructure.
Ben Shwab Eidelson (03:08:10): I think, at the end of the day, there's also a big issue of incentives. These hyperscalers, they want to build these models. Now, many of them are used to mostly still building their businesses on software-based timescales. And the reality is, despite all the discussion of the utilities and electricity, for them, it's a bottleneck. The actual cost of energy for them to build the biggest models is like 2 to 6%. It is all in the people and the hardware. It is all in the chips and the humans. And so for them, it's all, as you said earlier, this speed to getting out there. And that's not how our system is set up. That's not how our incentive structures are set up, right?
Anay Shah (03:08:51): That's right, because the utility supplying that power operates on an entirely different paradigm. In a typical electric utility, power delivered in 2027 is valued at the same amount as power delivered in 2030. Utilities don't differentiate those products, but for the customer—a Google or Meta—there's a massive difference for their business. They need power today.
Ben Shwab Eidelson (03:09:17): How much more would they pay if they could get power now versus three years from now?
Anay Shah (03:09:21): And how much could we unlock in terms of investment into our utilities if they could take advantage of the fact that companies are willing to pay a lot more today than in two years?
Ben Shwab Eidelson (03:09:31): Brian Janis calls this the Watt-bit spread. It's the economic arbitrage between an available Watt and the ability to turn that into a bit. Reality is, the speed of regulatory change in utility tariff pricing—which is a fancy way of saying how quickly the utility can change anything about their economics—is slow. And so despite there being an obvious market demand structure, it's not going to change in the next 12 months the way that utilities price their product.
Anay Shah (03:10:03): Utilities are public goods, right? They power our lives and our schools and our hospitals. And so for good reason, they are regulated. But because they are regulated, they take time to change. And moreover, we have a federated system. We have some 3,000 different utilities across the United States. And so making that change across this country, or even in the 10 states where data centers are getting built out, will take a lot of time.
Ben Shwab Eidelson (03:10:27): Power has become the dominant question of: where can you build your data center? Because you can't go do it without that. But it didn't remove the fact that the other questions still remain. For example, you need people to go build a data center; you need a lot of electricians; you need all the components and the supply chain and the things to build the transformers to step down the power.
Anay Shah (03:10:48): Based on how we've been talking about this, you might think that data centers are almost entirely a U.S. story. And while the U.S. is the dominant builder of data centers, there is actually a global story here and global infrastructure being built out, and a strong need for sovereignty for other nations. So in order to help understand what's going on here, let's go back in time a little bit to understand the drama that's setting in globally.
Ben Shwab Eidelson (03:11:19): I think it's worth looking at just fairly recently. In 2018, Europe passed GDPR, and that set this world baseline for privacy. Thank you. All the cookie pop-ups, but also really tight controls on where EU data sits and what happens if you have EU resident data outside of the EU. And so this is really, I think, a reaction to where people are storing data about our citizens. And so there are controls about what happens if you're holding or transferring data outside of the EU.
Anay Shah (03:11:51): In 2018, the U.S. passed the CLOUD Act, which essentially lets U.S. authorities lawfully demand data from U.S. providers, even if that data sits on servers outside of—
Ben Shwab Eidelson (03:12:03): the U.S. And so the net effect is if you're working with EU data at scale, you start to need to have servers and hard drives in the EU. And this becomes, I think, almost another lock-in that the big cloud providers start to have because they go and build the infrastructure to do that.
Anay Shah (03:12:19): And we saw this in the Microsoft story, right? A big part of their early scaling was following their enterprise customers globally and needing to build out data centers in Germany as one of the primary cases.
Ben Shwab Eidelson (03:12:31): And so if the EU-U.S. looks like a little tiff drama, I think it's time to move to the major geopolitical data center drama, which some have—
Anay Shah (03:12:41): deemed the new Cold War. And there's a book called *The Chips War*, which centers around this notion of the U.S. and now China in a fight over everything.
Ben Shwab Eidelson (03:12:57): But centered on this technology, right? So, I mean, let's go back to 2012, which I think really kicked off this last decade. Fifteen years of animosity here. The House Intelligence Committee report warned U.S. carriers away from using Huawei and ZTE equipment, saying that we shouldn't use this in any sensitive government system. Now that, of course, starts to spiral outside of just government use.
Anay Shah (03:13:21): That was first with the carriers. Beijing responded on a different front: the data front. So in 2017, China's Cybersecurity Law, followed by other laws, pushed data localization and tighter state control. And so the U.S. companies had to respond by localizing operations, which essentially means AWS, Apple, Azure: they either had to sell off to Chinese partners or find a Chinese government-linked company to be a partner.
Ben Shwab Eidelson (03:13:49): That's right. So it's not just that those companies' hard disks and servers had to be in China, but they actually had to be owned and operated by a Chinese company. And so then the U.S. reacted even further. They barred all federal agencies from buying or using equipment from a long list of covered Chinese vendors. And they barred the FCC from approving any new authorizations from Huawei, ZTE, and others. So this just ramped up further and further.
Anay Shah (03:14:14): This brewing war was not just limited to land, but also involves our friends, the undersea cables.
Ben Shwab Eidelson (03:14:21): Right.
Anay Shah (03:14:21): And so in 2016, Google and Facebook partnered with a Hong Kong-based company to build the Pacific Light Cable Network, a massive 12,000-kilometer undersea cable linking Los Angeles to Hong Kong, Taiwan, and the Philippines. Part of that global network of undersea cables we were talking about earlier. Seems like a great idea. What could go wrong?
Ben Shwab Eidelson (03:14:44): And fast forward four years later, to 2020. U.S. national security officials say, "We don't know about this Hong Kong landing point anymore." We think that's going to be a vehicle for Beijing to have surveillance across this important backbone of the internet. And so the FCC ultimately blocked that route. And so now the cable hits Taiwan and the Philippines, and those are lit up. But the Hong Kong branch that's been laid, I believe, is just lying dark.
Anay Shah (03:15:10): So I'm sitting here in Los Angeles without that cable to Hong Kong. Now, moving back to land and to chips. The battle continues. And so in 2022, the Commerce Department rolls out these sweeping controls on advanced AI chips and fab tools. And so, remember, this is the moment where we've gone through a COVID boom. We're seeing an uptake in usage from crypto and streaming. And now AI, and the Commerce Department tightens these controls again in 2023 and again in 2024.
Ben Shwab Eidelson (03:15:43): And this feels very active, right? I mean, literally every quarter of Nvidia's earnings, there are discussions of this. A few weeks ago, the U.S. opened a channel that said Nvidia could ship certain chips that are dialed back, like the H20. But they have to give the U.S. government a 15% cut of the revenue. It's a pretty unprecedented pay-to-export arrangement, like a lot of things that are unprecedented that are going on. But if you zoom out, it raises some good geopolitical strategic questions on how we want to engage, right?
Anay Shah (03:16:11): Yeah. It's foundational to the new economy that's being built on this advanced compute power. And it remains to be seen how this plays out. It may backfire. Right. China is not terribly keen on being dependent on U.S. production, so they are infusing massive amounts of capital in their chip sector. It's still a step behind Nvidia in terms of raw performance and software ecosystem dominance. But given China's resources, you have to assume that that gap is going to narrow and narrow quickly.
---
Ben Shwab Eidelson (03:16:46): There's a reason China wanted domestic energy. They want to have their own domestic production of silicon here, and so they've invested heavily and they're catching up. Some estimates are that they're 60 to 70% of Nvidia performance; it depends on what they're doing. But they're investing out of a $50 billion fund to improve chip development.
Even more so, they're passing policy that says data centers need to source at least 50% of their chips domestically. So they're making sure to really stand up not just the supply side of the market, but also simultaneously the demand side of the market.
Anay Shah (03:17:17): The story is less about whether Chinese chips can equal Nvidia's top-end designs and more about how quickly that's going to happen given the demand and the capital that they're putting in. So in short, at the moment, the US is still ahead, but the China chip ecosystem is accelerating rapidly. Their backlog of demand for Nvidia chips, which is being withheld, is going to matter less and less over time.
Ben Shwab Eidelson (03:17:43): The other dynamic that you have geopolitically is governments are seeing the AI data center boom as an important future part of their economy. A deal that exemplifies this is the Emirates deal with OpenAI around this UAE Stargate buildout, a one-gigawatt AI cluster in Abu Dhabi, and all the ecosystem players from Oracle to Nvidia to Cisco to SoftBank coming in to finance this and collaborating on the buildout. I think it just raises a lot of interesting questions: What about the location of a data center matters to a government?
Anay Shah (03:18:16): And the map is stark. If you look at where AI data centers are located today, only 32 nations have them, and most of them are in the Northern Hemisphere. You have large swaths of Latin America and Africa as fully dark. Governments are deeply, deeply concerned because they are wondering that if you continue to rent compute power from faraway data centers, you remain at the whims and you remain vulnerable to foreign entities and foreign companies, and you aren't able to support domestic enterprise or domestic scientific research or academia with the same level of control.
So this idea of compute sovereignty is top of mind for a lot of people, and a lot of emerging markets are worried that the AI era runs the risk of leaving them even further behind economically.
Ben Shwab Eidelson (03:19:09): There's almost a rough analogy to energy independence, right? Do you have it regardless of your relationship with another country? That feels like the important thread here.
Anay Shah (03:19:18): The other tricky dimension here, which if you talk a little bit about, is the actual impact on the climate, which gets a lot of attention these days, rightly so. There are a few questions that continue to come up, and one that Ben, maybe your mom has asked you, I know mine has, is if a ChatGPT query is bad for the climate. If I care about the climate, should I really be using AI? And we have some data on that.
Ben Shwab Eidelson (03:19:44): There's been some reporting on this, and I think it's an area that's become a pretty hot-button issue for folks. There's been, I think, some people even shaming others for using these tools because of the climate impact. So it's worth looking at the latest numbers. Google actually just put out a publicly readable paper on Gemini where they go down to the details on the energy consumption and the carbon intensity of that energy.
The median Gemini text prompt, which for all intents and purposes I think could be viewed as fairly equivalent to a Claude or a ChatGPT or a Copilot prompt, uses 0.24 watt-hours of energy and emits 0.03 grams of carbon dioxide equivalent and consumes 0.26 milliliters or about five drops of water. Those are figures that are substantially lower than what the public estimates have been. For some kind of sense of scale, that per-prompt energy impact is about the same as watching your TV for less than nine seconds. Why do we think the public reporting might be so far off?
Anay Shah (03:20:40): There have been a lot of improvements over time, and so it's important to kind of ground ourselves in this moment. We know, through that same report, that the AI systems that we're using are becoming more efficient. There's constant innovation in the software and the hardware to drive more efficiency. So over 12 months, the energy of a standard Gemini text prompt dropped by 33-fold.
Ben Shwab Eidelson (03:21:04): Wow.
Anay Shah (03:21:05): A 33x improvement in the energy consumption. Even more so, from a total carbon footprint per se, we have 44x, all while delivering higher quality responses. So we are moving in a world where the efficiency gains are continuing, and this question of "is your query bad for the climate?" is fairly insignificant. Hannah Ritchie from Our World in Data has independently corroborated this fact. It's an interesting cocktail conversation, but largely misses the forest from the trees.
Ben Shwab Eidelson (03:21:37): I think the pressure is good positive pressure because one individual query you should not feel bad about. But the pressure for all of these companies to monitor this and drive these numbers down and get these efficiency improvements is a fantastic thing. That's right. And to drive the deployment of clean energy so that the carbon intensity of the energy drives down is a good flywheel to keep pressure on.
Anay Shah (03:21:56): Indeed. So, Mom, keep asking a question because Google will then continue to improve.
Ben Shwab Eidelson (03:22:00): I think there's another area that's been under-discussed.
Anay Shah (03:22:03): You don't hear a lot of discussion on the actual carbon intensity of building these very large pieces of infrastructure, pieces of real estate. That's called embodied carbon. So what is the embodied carbon impact of the steel, the cement, the energy used to build a shell and stand up the facility in and of itself?
Ben Shwab Eidelson (03:22:27): Thankfully, Google did another report on this and shared that in their analysis. Running an AI data center, the operational emissions, which really means the energy going into running the data center ongoing, is going to be about 70 to 90% of the total emissions. Manufacturing emissions, which is really the manufacturer of the server components, the memory, the flash storage, the GPUs, is going to be around 25%. And then the data center construction, so this is the steel and cement and the logistics around that, are around 5%.
So this is all to say, I think when you look at this, we should probably be paying a bit more attention to the manufacturing of the components and make sure that we're looking at the LCAs for solid-state disk drives and GPUs and making sure we're taking that into account. But the attention's probably correctly centered on the electricity.
Anay Shah (03:23:14): And on this point of focusing in on the manufacturing, the procurement muscle of one of these companies that are building this out is actually a very big leverage. So when Microsoft requires its suppliers to use 100% carbon-free energy by 2030, it will pull their suppliers, all the fab building and the motherboard building and the assemblers. It's moving everyone down the supply chain — it's called Scope 3 emissions — into a cleaner world, which has tremendous power.
Ben Shwab Eidelson (03:23:43): I think the other thing that's interesting is some of the circularity conversations are starting to happen. So I was at a talk where the Microsoft CSO was talking about how they recently launched, I think it was just in April of this year, a circularity program where they would take the rare earth material in the hard disk drives that they're using in the data centers, and they have a new way to recycle those while disposing of the data. That yields a 90% recovery of those rare earth materials, which also helps stand up a US supply chain around rare earth materials, which we don't really have today, which is pretty phenomenal.
Anay Shah (03:24:13): Yeah. Tremendous. And it's going to be a growing discussion over the coming years as we mine for more and more of these critical minerals and the geopolitical control over who has those minerals.
Ben Shwab Eidelson (03:24:25): All right, so given all of that operational load, is the main contributor to greenhouse gases something like 70% or more of the emissions? How should we think about the scale of that from a meta-climate perspective?
Anay Shah (03:24:38): So when we talk about operations, remember we're talking about the energy consumption and not the materials that go into building the data center. Recent studies show that data centers account for about 4% of total U.S. electricity consumption. And with more than half of that electricity derived from fossil fuels, that means that data centers generate more than 105 million tons of CO2 every year.
Ben Shwab Eidelson (03:25:04): Until reading the study, I didn't realize that data centers' carbon intensity is actually higher than average, exceeding the U.S. average by 48%.
Anay Shah (03:25:12): The big point here is how grid-dependent that impact is. If these data centers are located on a dirtier grid, let's say a coal-heavy grid in the Mid-Atlantic region of Virginia, then it's going to have a very different climate profile than a data center that's in Eastern Washington that's entirely powered by hydro. So when we talk about 105 million tons of CO2, how should we think about that?
Ben Shwab Eidelson (03:25:36): A few comparisons to help think about the scale of 105 million tons of CO2 emissions. One is you look at US aviation emissions: this is about half of that. Another would be looking at enteric methane, which is a fancy way of saying cow burps, is around 178 million tons. So that means data centers are about 60% of the equivalent of that. Or all US passenger vehicles are 1,000 million tons, or a gigaton. And so data centers are about 10% of that today.
Now, if you look globally, this goes up by approximately a factor of three, as do the rest of these things. So overall, data centers are not insignificant. They're worth tracking on the map here, but quite fractional compared to passenger vehicles, big emitters like livestock, and a third to a half of something like global aviation. The interesting thing, I think, what makes this a hot topic, is not just the current scale, but the projections, right? If all of a sudden we double on our current grid in the US, then now we're caught up to the emissions of aviation in the U.S. And if you keep going from that and you keep the trend line up, you could imagine data centers becoming a fairly dominant story in emissions.
Anay Shah (03:26:50): The other leg of the stool in this conversation is how it's affecting the local communities that these data centers are being placed in. One of the themes that we've found come out through this is the impact of these data centers are concentrated and they are disproportionately felt by the environment, the ecology, and the people that immediately surround it. We metaphorically went to Memphis, where XAI is building a very large data center on the banks of the Mississippi, 15 minutes from downtown. And there, Elon is commissioning a 200,000 GPU unit data center. There's been a significant backlash from the Memphis community for good reason.
Ben Shwab Eidelson (03:27:37): I think a lot of it centers on how are we going to provide power to that? Because the utility said that they can provide XAI power for about 50 megawatts of load, but XAI wants triple that amount. That's a lot of GPUs, and 50 megawatts isn't going to cut it. So they brought in gas turbines — 35 turbines — which could theoretically power 420 megawatts.
The problem with that — I mean, it's good they're solving their power problem — but the other problem with that is these are highly polluting gas turbines. So they have the potential to emit a couple thousand tons of smog-forming nitrous oxides each year. For context, that's more than the smog caused by the Memphis airport. So it's like you're adding another airport of smog to the region.
Anay Shah (03:28:18): And this impact is not continuous, right, because these data centers ebb and flow in terms of how much power they are consuming. So it's important to look at these peak moments. We found public satellite data from NASA and the European Space Agency that shows that, on average, nitrogen dioxide concentration increased by 3% when comparing for a year before in this area. But in the times of peak consumption, we're talking about a 79% increase in nitrogen dioxide concentration from pre-XAI levels in that area. So you can imagine it's like sitting in a traffic jam, and those moments have outsized impact on the community at that moment in time.
Ben Shwab Eidelson (03:29:00): Yeah. And so there's been a lot of pushback from this pollution and health risk standpoint. It's hitting South Memphis neighborhoods that already have elevated asthma and cancer rates from past industrial waste. This temporary electricity generation infrastructure isn't going to sustain in the long term. So all of this is kind of a case study in both the local impacts and also the time-to-power issue.
Anay Shah (03:29:24): And this notion of time to power, essentially what it means is there is enough demand to get these data centers online quickly. The companies are willing to pay more and more. So when you have that kind of economic pressure, the community resistance sometimes can only go so far. As we're seeing in this case, XAI, despite the pressures, is currently building out a second location a few miles away, which will be double the size.
Ben Shwab Eidelson (03:29:50): Of the first: half a million of these GPUs.
Anay Shah (03:29:53): I mean, just insane. That's the current moment we're living in.
Ben Shwab Eidelson (03:29:59): So far we've been trying to keep it focused on what has been built more than what is proposed. But I think it's worth taking an extreme view on some of what's coming. Meta has announced their big long-term project, their big training center, the Hyperion project, located in Richland Parish, Louisiana. It's a $10 billion buildout, online by 2030. They'll have a few steps in between. The goal is for this to be a five-gigawatt canvas covering over 2,000 acres.
Anay Shah (03:30:28): What is 2,000 acres?
Ben Shwab Eidelson (03:30:29): Yeah. So a different way to visualize this is this data center is about the size of Lower Manhattan. It is at a scale that is kind of unfathomable, right? That we're going to build buildings of that scale in a few years here to power such a new technology.
Anay Shah (03:30:48): That Louisiana project is a great example of what we are living, the step-change function that we are living through right now. This is the biggest tech infrastructure project since either the 1960s, the dawn of the computer age, or even the 1880s, the heyday of the railroad period.
Ben Shwab Eidelson (03:31:09): I think Nvidia is on pace to capture the highest share of market-wide capital spending since IBM peaked in their percentage of that in 1969. And so a lot of comparisons are being made to other past booms like the Gilded Age or the telco buildout.
Anay Shah (03:31:25): And when you mentioned booms, obviously you end up thinking about busts, right? So we talked about the telecom boom of the 1990s, which contributed to the dot-com crash and a bust on fiber. You also look back at the 1870s and the huge railroad boom that led to a crash, and both of these pose the question of, like, did we overbuild? Are we outrunning our demand? In both of those cases, it's not that the CapEx spenders were wrong; they were just early. We saw this vividly in the fiber overbuild.
Ben Shwab Eidelson (03:32:00): Right.
Anay Shah (03:32:01): By the year 2000, we were only using 3% of the fiber we had laid. But it was absolutely necessary to power the next couple decades. And that's what it looks like for AI is that there may be an overbuild in this moment, but the foundational nature of this technology suggests that it's going to get utilized.
Ben Shwab Eidelson (03:32:20): Yeah, I think there's such a rush here because the potential AI market creates this economic imperative to go and plug in as many GPUs as quickly as possible, and that will build the largest and smartest AI model, which will create a moat. I think there's an interesting question here, though, of what is the infrastructure? Because yes, we still use the railroad tracks and we still use the fiber, but remember that all the companies that were building the fiber and the ones that were on top of that, it was too early for them to capture that value.
But what's in part different about this boom is how it's being funded now. Yes, there are VC dollars and other dollars going into it, but at least historically, and if you look at the major capital expense, it is these big companies we've spent the episode talking about — it is Amazon, Microsoft, Google buying from Nvidia. And they're using the fact that their business models have thrown off billions and billions of dollars every quarter to then go buy the chips and build this out. So there's a real kind of concentration in some of the buildout of the infrastructure.
Anay Shah (03:33:20): Those previous boom-busts financed by the banks end up drawing in the rest of the economy. And while there are some private credit actors and venture capitalist actors, because it's largely financed by free cash flow of these private companies, a bust hurts them, which will drag down the stock market, but is hopefully insulating the rest of the capital providers that are not involved.
Ben Shwab Eidelson (03:33:43): There is supposedly now more private credit starting to come in to fund this, and that could create linkages to other parts of the system. We will see how that plays out. We've talked about the story of data centers up until present day. Let's zoom out for a second and talk about just where we are today from a sense of scale.
Anay Shah (03:34:04): Let's start with the most basic question: How many data centers exist today? There are globally approximately 11,800 data centers worldwide. Now, of course, the counting of this gets nuanced when you think about the closets that still exist with servers as they did back in the day. But by and large, we're talking about independent construction. And the US is by far the largest with over 5,000 data centers, followed by Germany, the UK, China, and Canada.
Ben Shwab Eidelson (03:34:33): So the US has almost half the data centers. Wow.
Anay Shah (03:34:36): Yeah, heavy concentration in the U.S. Obviously other countries are catching up. China is building very quickly. So this begs the question of who owns all of these 11,000 data centers and who's spending these hundreds of billions of dollars. And you can kind of think of it in four broad categories. That's representative of the US but also it fits globally.
And so the first category are our friends, the colocation centers, the ones we've talked about: Equinix, Digital Realty, and others. There are about a dozen of these that are building 10 to 100 megawatt blocks in the US and around the world. The second category are the hyperscalers: the Facebooks, Amazons, Microsofts; you can add in Apple, Oracle in there. They account for almost half of global data center capacity, and they're the lion's share of new growth. But importantly, you still have thousands of private server rooms in banks and retail facilities and public institutions that are out there. So while large in count, they're relatively smaller in capacity, but today still make up about 35%. And finally, you've got legacy telcos that are still owning and operating data centers.
Now the interesting thing about this is the trend, right? You're seeing enterprise and public sector data centers trending down, and new build increasingly going to hyperscalers and these purpose-built colocation facilities.
Ben Shwab Eidelson (03:35:54): So how much actual physical real estate space are all of these data centers taking up? This is like anything that gets into this level of detail. It can be hard to pull exact numbers, but one reasonable estimate is that maybe there's one and a half billion square feet globally in data center buildout. For context, if you took an average median US home of around 2,200 square feet, that's around 730,000 of those homes of square footage, or 28,000 football fields.
Anay Shah (03:36:20): Importantly, including end zones.
Ben Shwab Eidelson (03:36:24): That feels like a big number. But for some comparison to some other infrastructure in our life: If you just took the U.S. interstate system, so this is, you know, I-5, I-90, the big interstates, just the asphalt of those interstates is 10 times the square footage. So let's talk about power.
Anay Shah (03:36:40): Remember the impact of data centers on power: it's quite localized and concentrated to its physical location. But let's talk about it in a sense of scale. We mentioned that in the US it's about 4.5% of total electricity. That's roughly 17 million households' annual use. How can I conceive of that power?
Ben Shwab Eidelson (03:36:59): Yeah, I think if you look at the electricity consumption of the city of New York, it's about three New Yorks to power all the data centers in the US. Or if you look at all data centers globally, it's about equivalent to the power consumption of the United Kingdom. If you look at other comparable industries, it's pretty close to the power consumption of the chemical industry or the primary metals industry. This is extracting iron ore and smelting aluminum.
Anay Shah (03:37:26): Similarly, on the water question, there's a tremendous amount of nuance in how to aggregate total water consumption. But let's just do the average of the average. Let's take direct water for cooling, indirect water from the power generated for the plant, and get a sense of how much water this is using US data.
---
Ben Shwab Eidelson (03:37:45): Centers together consume about 250 million gallons of water per day. That's about a quarter of what the city of New York uses.
Anay Shah (03:37:53): So we've talked about the space, the data centers, how many of them, who owns them, how is it being powered and cooled? What's actually inside of them?
Let's think about the scale of the compute power. It's actually a little bit difficult to figure out what humanity's overall compute power is, but let's just say there are about 50 to 100 million servers globally.
Ben Shwab Eidelson (03:38:13): Yeah. And so, I think the best way to get at this was to go back to the electricity usage numbers and then look at how much compute power we get per unit of electricity. What's surprising here is that if you run the numbers, we've grown humanity's compute power around 40x over the last decade, but have only grown power consumption about two and a half times because we've gotten more efficient.
And if you look at that new compute power that's come online, only around 10% of data centers are AI-focused today. Of course, that's a growing percentage.
Anay Shah (03:38:45): So compute power is one part of what the data center is doing. Let's give a sense of the scale on storage.
Ben Shwab Eidelson (03:38:51): Yeah. So in the last decade, we've 3x to 4x total installed storage capacity to around 15 zettabytes.
Anay Shah (03:38:57): And so we're speaking to a friend of the show, Byron, who made me understand the modern marvel of hard drives. So imagine the head of the hard drive is scaled up to the size of a Boeing 747.
It's flying at 560 miles an hour, and it is about the thickness of one piece of paper above a football field. That hard drive head is reading and writing to every single blade of grass moving at 560 miles an hour.
Ben Shwab Eidelson (03:39:27): That is absolutely insane. And so we have all these hard drives and all this compute. Now they're connected via these submarine cables. How many of those do we have?
Anay Shah (03:39:34): We have approximately 600 active submarine cables that are powering 99% of international usage on the internet.
Ben Shwab Eidelson (03:39:45): So in total, we're about 100 times the global bandwidth that we were a decade ago. And we've seen internet bandwidth continue to grow 25% year over year.
Anay Shah (03:39:53): And so these numbers are today's snapshot.
Now, there are a lot of projections out there of where growth is going to go over the next five years. It can get fairly outlandish, but suffice to say, growth is continuing to accelerate now and in the years to come.
Ben Shwab Eidelson (03:40:10): All right, well, we talked about Byron's plane reading and writing blades of grass. Now it's time to land our plane.
Anay, let's do the themes. What did you realize bubbled up for you over the course of this research and conversation?
Anay Shah (03:40:24): It's been a journey. I did not know a lot coming in and have learned way more about the ins and outs of data centers than I ever imagined. A few big things will continue to stick with me.
One of the first books we read is called *Tubes*, and it's this nonfiction, nerdy detective story to discover where this thing called the internet lives. It turns out the cloud is quite physical, but even more interesting, it's strangely concentrated. The internet has almost infinite edges, but it's got a shockingly small number of centers and has been built in this hub-and-spoke structure.
And so for me, that starts with May East, right? Ashburn, Virginia, home to 13% of global data capacity and has once carried north of 30% of the world's internet traffic. That is the undisputed capital of the internet.
Ben Shwab Eidelson (03:41:25): And it's so wild that that became the hub just by being the hub. There's just this center of gravity that spiraled on itself, a black hole of data center investment, almost by accident, right? Yeah.
Anay Shah (03:41:38): The internet's structure is based on this mesh connectivity of global cities on coastal shores. It's Virginia, New York, D.C., London, Paris, Amsterdam, Tokyo, Seoul, Lagos, all connected by these 600 undersea cables that are literally powering the world's economy.
And so I think about data centers that started off occupying closets, growing to whole floors, then buildings, then warehouses. And I start to think of the cloud as a building, as a factory, where a bit comes in, gets massaged, gets put together in the right way, and then it gets sent out to its destination, and it really brings the internet home.
Ben Shwab Eidelson (03:42:21): For me, this connects to one of my reflections, which is threaded through the invisibility of this infrastructure and what makes data centers and the connecting internet almost the perfect abstracted infrastructure. And what I mean by that is you can access them, you can use them, you can get all of the power of them without ever seeing them or touching them.
Now, with us living in a mostly wirelessly blanketed internet ourselves, you have access to all of these warehouses and all these buildings via all these undersea cables. And if you're an engineer developing software, you can deploy to all these regions around the world and never actually look at a CPU or a hard drive in your life.
It is, I think, the best-abstracted physical infrastructure that humanity has built. It is physical, it is silicon, it is electricity, it is fiber. There is no real magic. It is all physics.
But the only infrastructure I can think of in humanity that is as well abstracted is maybe money, but money is actually not physical anymore. In the same way, this actually is still doing physical work. And that struck me.
Anay Shah (03:43:34): It does. It feels very rare to have global infrastructure that is defining our life and how the world operates, to be so.
Ben Shwab Eidelson (03:43:44): Invisible, and so frequently in our life. Even though it is invisible, our entire day is mediated by this infrastructure. But if your entire day is spent on the train, you're very aware of where the tracks are, and you can see them, and you're on the train. That's the engine, right?
Anay Shah (03:44:01): Even fire, we probably used it selectively throughout the day. The railroads, we got on and off. The car, we got in and out of. Electricity, we turned the lights on and off. This is a modern marvel that we're only not using when we sleep. And even then, it's doing stuff for us.
Ben Shwab Eidelson (03:44:20): It's tracking my sleep. I'm wearing a ring that's tracking my sleep, sending bits to Oura's servers.
Yeah, as you were saying that, it's like you're getting on and off a train. You're kind of turning electricity on and off in a transactional moment. This is almost more like you're living in it, right? You're living in this infrastructure, but we never see it.
Anay Shah (03:44:44): I just had a flash of *Ready Player One*.
Ben Shwab Eidelson (03:44:46): Yes.
Anay Shah (03:44:47): Putting on the full suit and immersing myself in another reality. We're close, when you think about it this way.
Ben Shwab Eidelson (03:44:54): I think that's right. I think this gets to one of my other realizations when comparing this to other physical infrastructure, which is, despite it being truly physical—buildings and silicon and electricity—we are upgrading it continuously and rapidly. Part of that is because of Moore's Law. Part of that is because of the improvement of the hardware and our techniques for cooling and all of these things.
But I feel that with most other infrastructure, you get to, "Okay, this is a reasonably good way to build a train track. Now we need to go build more of it," or, "This is a reasonably good way to transport electricity. So now we're going to build more transmission."
And certainly you get a little bit better at transmission, but we don't just get a little bit better here. We're getting 40x better. In the last decade, which was off the previous decade, it's gotten much better.
So it's this weird thing where, yes, it's infrastructure, but it is changing what it is. We're getting at such an order of magnitude scale so quickly. And I can't think of other analogies to that. Maybe the first 20, 30 years of electricity felt like this, but this feels... something feels even more drastic here because what is being delivered is not commoditized exactly the same way as electricity.
Anay Shah (03:46:02): That sense of movement that you're describing reminds me of another one of my takeaways, which is where I'm feeling stuck. And that's on the question of power. To me, it's easy to think about the complexities of why the power problem is so hard to solve, and yet the answers are right in front of us.
The problem is huge, right? Data centers are pulling more and more electricity from the grid.
The problem is hard because the grid is strained, the turbines and transformers are backlogged, and we don't really have utilities that are built to capture the incentive structures to value what hyperscalers are willing to pay today.
And it's going to take years to reform the grid to actually meet the needs. And at the same time, the fastest, cheapest energy is available today. It's solar, storage, putting renewable energy online quickly and cheaply. That is already accounting for 90% of new capacity going on the grid.
Despite every effort from our current administration to execute an ideological war on clean power and claim that we're in some national energy emergency, and yet deny the grid affordable electrons being placed on it.
So if we can just get out of our own way, we can put a lot of power on the grid fairly quickly. The other side of that story is the way we're building data centers: accounting for 24/7 peak load.
And there's a famous study that circulated earlier this year from Duke University that suggested that if we can limit grid-facing power of data centers by just 1% a year—90 hours out of the entire year—we can unlock 100 gigawatts of load. We can unlock the equivalent of two nuclear fleets on the US energy grid.
Ben Shwab Eidelson (03:47:54): Right. You saw Google do some recent announcements where they're collaborating with some of the utility providers to do exactly that. There are new startups that we see all the time that are building businesses around that concept.
This connects really closely to my major takeaway on the power story, which is that limitations drive innovation, especially when they come up against a legacy incumbent system—something that generally works, and reasonably well. Which for all of our general disgruntledness about utilities in our life, it is phenomenal that the electricity works most of the time as well as it does.
And it has not gotten a kick in the behind in a long time in terms of needing to try and evolve. There have been decades of people talking about a smarter grid, but nothing's forced the need like the AI data center competition and the geopolitical competition with China, which drives innovation across the stack.
It's exactly what you're saying in terms of driving the next cheap, clean electron on the grid in the forms of renewable energy. But it's also pushing us on transmission and permitting, and it's also pushing us inside the data center to think even further about AI models.
And we've invested in companies that improve the efficiency of actually running and training the model.
Anay Shah (03:49:12): That's right.
Ben Shwab Eidelson (03:49:12): And so up and down this whole stack, a point we found again from talking to people and from research, these types of crunches drive efficiency innovation and the evolution of the grid.
Anay Shah (03:49:24): And arguably, according to folks who have been in this for decades, we've been fairly lazy on innovation.
It's been a lot of low-hanging fruit. And so this type of forcing function can drive step-change improvements in something as complicated as our national grid.
Ben Shwab Eidelson (03:49:41): So to me, all of that points to long-term optimism: first, in solving the problems, but second, in solving them in a way that leads to more clean energy deployment. However, it also means needing to recognize and not dismiss the near-term local impacts for both power and water in these environments.
Anay Shah (03:50:01): I love that point because as we talked through these stories, that theme kept recurring of how it is different when you think about the problem locally than when you think about the problem nationally. And I think that's really important for people to understand as this is more and more headline news.
Ben Shwab Eidelson (03:50:16): My personal takeaway is less and less anxiety about that next incremental Gemini Query, ChatGPT, Anthropic use case or storing a video, and a greater desire to have more awareness locally within a given county that's struggling with air quality or water shortages.
Because that's where activism needs to center within communities in the moment, rather than trying to slow the valuable uses and progress that we're all building toward.
Anay Shah (03:50:45): Absolutely. This point on communities and geography and where things are located brings up another takeaway that's close to my heart as a former international politics major and someone who spent 20 years living and working across emerging markets. And that's the geopolitical implications of the moment we're living through right now.
And so we talked a little bit about this, but to me, it really crystallizes into a twofold idea: One is the notion that we're living through this chip war, right? That microchips have replaced oil as one of the world's most critical resources and it's a key determinant of economic and military power.
And we're in a new Cold War. This time, it's largely with China. As Americans, we don't want a country like China accessing our data or having superior performance over us.
And so we're actively preventing US companies from selling chips to China. But it's not so straightforward, right? Because you think of, "Okay, Nvidia as an American company, we can prevent that."
But Nvidia is the architect of the chips, right? The manufacturer of over 90% of the world's advanced chips is a Taiwanese company, TSMC, and they're the ones actually constructing it.
So this geopolitical tension has various layers to it, but the notion that we're living through a new Cold War and the chip is the center of that story is quite striking.
Ben Shwab Eidelson (03:52:07): I think in the long term, or mid to long term, this type of technology is a genie that can't be put back into the bottle. China, in particular, is hyper-aware of what they want to domestically build for themselves here and actually is in a great position from a supply chain perspective to pull that off.
Even if they're a couple years behind. You've seen them spin up AI labs, spin up their own domestic chip manufacturing, and do impressive jobs.
And you can say, "Well, 18 months ahead," but what is that in the scale of this foundational technology?
Anay Shah (03:52:40): Yeah, 18 months is a blink of an eye.
Ben Shwab Eidelson (03:52:42): I lean again toward long-term optimism that this competition creates good incentives for us to rethink our fundamental constraints, which are mostly on the clean energy deployment side.
One big thing China has going for it that we don't have is its ability to grow generation of clean energy much, much faster than we do in the US. Again, because of their solar panel supply chain, their battery supply chain.
Anay Shah (03:53:07): Their government structure, their ability to build, is unmatched.
Ben Shwab Eidelson (03:53:11): And so this is giving a lot of our structures a kick to rethink what's slowing us down. If this is so existentially important to stay at least on par, if not ahead on.
Anay Shah (03:53:22): I agree strongly. In this notion of the chip war, I think it's interesting. In this moment, I think over the medium to long term, it's less.
So here's where I don't share as much optimism: the AI boom and data center build-out that's potentially creating a new digital divide, exacerbating global inequality on yet another dimension.
So we've talked about how US hyperscalers run almost 50% of data center compute. The U.S., China, and the European Union host more than half of all powerful data centers.
What does that mean for the billions of other people living around the world?
And what is the implication for a country that is renting compute power from a faraway data center? It's potentially higher costs, slower connection speeds, and compliance to different laws.
But more importantly, as we unpack this—as we're trying to wrestle with this idea—it's this question of having a critical national security and economic infrastructure input owned, operated, and housed outside of your control.
Ben Shwab Eidelson (03:54:28): The techno-optimist in me is just screaming to refute this, and I'll tell you why. And I think we can study the internet as an example. This feeds into, I think, another meta theme for us, which is that these companies are the modern-day utilities, but they're actually the opposite in a way of the governing structure of an electricity utility, which is hyper-localized and granted local monopoly.
These are global utilities. They're providing much more non-commodity functionality, everything down to a bespoke AI model that they're training, down to more commodity storage.
And the business model of a utility is to, as we've seen again and again, provide more as cheaply as possible to as many customers as possible.
And you've seen the biggest internet companies grow by incentivizing and subsidizing free services to more and more people. And that has led to a lot of the world getting connected that was not connected just 10 years ago.
There's reason to be cynical about what Google and Meta are doing, trying to get more of the world connected. Don't they just want more customers?
Obviously, yes, they want more customers. But in doing so, a lot more of the world has access to the internet than ever before, and information and knowledge.
And I think that's where I fall on the side of optimism: that there's nothing more that OpenAI wants than more people having access to OpenAI.
Now, that to me goes to the individual human level, which is very different than how a nation-state or the government in control of a nation-state might feel about it.
But I lean on the individual human in a global context, and what powers I want them and their children to have in the decades to come.
And that's where I get a source of optimism.
Anay Shah (03:56:07): I mean, it's true. Sixty-six-plus percent of the world's population is now connected to the internet, and that number is increasing.
I like the optimism. I wonder if the nature of this technology, particularly AI compute power, functions differently than access to the internet, access to electricity, and access to other similar infrastructures—having that controlled by private equity and public companies that have very specific incentive structures.
Ben Shwab Eidelson (03:56:39): I think there's a big thing that you just hit on for me, which is my optimism doesn't really account for the hosting government of hyperscalers. And so it's not just that these are companies, but they're American companies.
If they're American companies, they exist a little bit at the pleasure of the American government.
And that means that another country's access to them exists at the pleasure of the American government.
And that, I think, gets to a more complex nuance, and to me, it just drives value to the idea of more global models and more global competition, even if it's not hosted within a given country, so that it's not just one or two governments that have control over these technologies.
Anay Shah (03:57:22): More models trained on different languages, of course, and different speech patterns. And so, how this unregulated utility—as we're talking about, this new digital utility, this new critical infrastructure that society will increasingly depend on—will continue to be seen as a public good and delivered to more and more people as that...
You know, they say hindsight is 20/20, but there's something even more interesting about creating a historical narrative to something like data centers.
And it's the marvel of the narrative process, this story, that starts to make so much sense of how one thing came after another.
So you start off with IBM and Watson's journey and the introduction of the punch card that is storing data, and then the machines that start to compute off that storage, and that grows and evolves.
And you've got the Cold War introducing the need for connecting networks, and suddenly you've got compute and storage power. Network connections bring that together, and that continues to evolve.
And then you get the explosion of the '90s. And in this time, first slightly accidental and then more purposeful infrastructure starts to get built in very important nodes, and the fiber boom happens.
I think I'll now always associate the late '90s with the overbuild of fiber and this notion that by the time the dot-com bust happened, we only used 3% of the fiber that we built. It's wild.
And yet that was exactly what we needed to go through the run-up of the 2000s and the growth of consumer and enterprise internet, mobile, and streaming, and then eventually the cloud.
Ben Shwab Eidelson (03:59:10): And then you had crypto creating this market for the AI chips before we had the models. Cycles of building ahead of the curve, there being some bust, but then there being this new utilization of those fundamental building blocks.
Anay Shah (03:59:22): Absolutely. And these brilliant people located within very important companies that took the lead to build what is now foundational infrastructure for the world.
Ben Shwab Eidelson (03:59:34): And then there's another dance that's happening—in where storage and compute lives for the average consumer or average business—where you go from it being all on the mainframe.
Then it moves eventually to the PC. And you feel so powerful with your PC, but you have no collaboration, no backup, no redundancy.
Then it moves back to the server room.
Then in the mobile era, your connection is really slow. Early days, you think about the first iPhone: it was a really slow edge connection.
So your phone actually had to do most of the stuff on the phone.
But now we live in this world where the connection's back, fat, and happy.
And so we do everything now on the server except for rendering the UI on the phone. And so this dance—of where both compute and storage happens, mediated by the communication bandwidth—is a really interesting dance that I see played out in the story.
Anay Shah (04:00:26): That dance then drove the cloud boom of the 2010s and the maturing of enterprise cloud and infrastructure, where now you're able to do everything you want at higher and higher utilization in the cloud.
And all of that build-out, and that fine-tuning and optimization, allowed us to enter March of 2020—the dark days of the beginning of the pandemic—and switch over from in-person to online from the perspective of the internet, largely without much of a hiccup.
Ben Shwab Eidelson (04:01:04): Yeah, going to slightly lower resolution Netflix videos for a month. I think we handled it pretty well from a technology infrastructure perspective.
Anay Shah (04:01:13): Correct. And how that just so happened to juice the infrastructure even more, to lay the foundation for GPUs to finally be utilized for what was growing in the background: machine learning and large language model training.
And then November of 2022, putting a chatbot on top of a model, and the AI boom and the infrastructure that was laid to capture that moment, and the cash coffers of the hyperscalers that were ready to meet that moment, are now propelling the greatest technological infrastructure build-out in human history. What a story.
Ben Shwab Eidelson (04:01:59): Yeah. Do you think it's an overbuild?
Anay Shah (04:02:02): There are things that I worry about in this world, as we noted from the geopolitical situation, but once I strap in, the optimist in me takes over.
So I am fully convinced that we will utilize every photon, electron, and square footage. I hope that this is the opportunity where we collectively can seize the moment to get more clean, firm power online, more affordable power online, and where we can use technology and renewable power to reduce the depletion of water resources in critical areas.
The opportunity to build a newly-skilled workforce to build out this new infrastructure. The opportunity to modernize our aging grid, our transmission lines, the way we do siting and permitting, the way we do environmental review processes, the way utilities are incentivized to meet demand.
The opportunity to spur local economic development in communities all over the country, and in the world. The opportunity to have large enterprises build out infrastructure that will be a net benefit to the consumer where they're locating their assets.
And I think it's really important for us to be mindful of the cost of moving so quickly and not thinking through the implications of water, power, climate, communities, and global equity.
And my real hope is that when you have this much capital and this much demand coming on to reshape a foundational technological infrastructure, we come out the other end with an ability to—as one of our new friends that we interviewed said—build these assets as ecologically invisible as possible.
I think we have the opportunity to do that. And when we come out the other end having done that, I think our children's generation will look more kindly on us.
---
Ben Shwab Eidelson (04:04:04): I think so, yeah. My big takeaway is that, of all the things we might study on this show, this one feels like the major piece of infrastructure that we're living in the middle of the step change. We are in the sharp part of the curve that's going up.
This level of capital investment and build-out is what it looked like if you were to put yourself back into the initial build-out of the electricity grid or the railroad system, but further compressed. What's so interesting about that is you get to experience personally how quickly you adapt to amazing infrastructure and power—how quickly we've adapted, I think, to having the latest large language models at our fingertips.
Now I have Claude Code, and I'm a few weeks into using that. I have what feels like new powers, but I'm kind of getting used to it again. We lived through most of this story and have vivid memories. Like, I remember Blockbuster, and I remember when Netflix was a DVD rental service. That was because it was too expensive to move streaming video at that scale, and now it's blindingly cheap.
The business model changes; the products completely change. We are living through this cycle of infrastructure iteration and evolution at a pace that's rapidly changing what we can do with it. I don't know when we'll again get to the upper step of that curve, but we are living in the step change, and I'm just trying to relish that experience.
Anay Shah (04:05:35): The rocket ship is in takeoff, feeling some G-forces. That's right.
Ben Shwab Eidelson (04:05:39): But you get used to it, don't you?
Anay Shah (04:05:41): Very quickly. Shockingly fast. I mean, even since we launched the fund two years ago to now, our workflows have changed dramatically.
Ben Shwab Eidelson (04:05:53): That's right.
Anay Shah (04:05:54): Despite the fact that you and I continue to look at each other 12 hours a day in the same screen. Yeah, yeah. The normalization of humans is astounding. And then you start thinking about our kids and how they're growing up in...
Ben Shwab Eidelson (04:06:06): This moment: speaking to ChatGPT, Santa Claus if they want, or making a game. My four-year-old programmed a game that runs in JavaScript by telling Claude what he wanted.
Anay Shah (04:06:18): Amazing.
Ben Shwab Eidelson (04:06:20): It is wild, wild times.
Anay Shah (04:06:22): We are living through a step change moment. Thanks for listening. Hopefully you're that much more informed about the invisible infrastructure that's not so invisible in powering the modern world.
Ben Shwab Eidelson (04:06:39): You know how you got this podcast into your ears? It's amazing. That's right.
Anay Shah (04:06:43): If you enjoyed this and have friends or colleagues that you think may find the story of data centers valuable, please send it their way.
Ben Shwab Eidelson (04:06:51): And please make sure you're subscribed to Step Change in your podcast player of choice so you find out when our next episode comes out, which at this current pace will be at least once more this year, I hope. And sign up for emails from us at StepChange Show. Podcast ratings make a huge difference to help people discover this, so we always appreciate them and love hearing from listeners. Shoot us an email anytime. It'll bounce from your data centers to ours over at hi StepChange Show.
Anay Shah (04:07:20): At Step Change, we invest in early-stage companies accelerating energy abundance and building critical infrastructure. So if you're a founder working on software to help make all of this work, from data center management to efficiency to power generation, reach out to us. We'd love to check, chat, and learn more.
And last but not least, we talked to a number of folks in the data center and cloud worlds who have helped us tremendously with this research. Deep appreciation and gratitude to Christian Baledi, Peter Gross, Brian Janis, Sean James, Byron...
Ben Shwab Eidelson (04:07:52): Rakitsas, Brandon Middaugh, Nat Bullard, John Kumi. And a thank you to Ben Gilbert and David Rosenthal for all of their Acquired episodes on AWS, Microsoft, Google, and Nvidia. And a big thank you to Nick Petrie, our editor, and to our wives, Shiva and Ana, for hearing us talk about data centers for the last six months. And, all right, until next time, thank you, Sam.
