ON GOOGLE SPENDING HUNDREDS OF MILLIONS ON "THERMONUCLEAR" NETWORK INFRASTRUCTURE

RE-BUILDING THE FOUNDATION

(UPDATED BELOW)

Apologies for the long and unwieldy title to this post, but I wanted to let you know upfront what this longish post is about.  This topic is important if you have an interest in how Google and others need to scale their infrastructure for the next generation of consumer internet services. 

Tech columnist Robert X. Cringely has another entertaining, "what-if" post titled "Google-Mart" where he explains why Google might need all the "dark fiber" they've been reportedly buying:

"The probable answer lies in one of Google's underground parking garages in Mountain View. There, in a secret area off-limits even to regular GoogleFolk, is a shipping container. But it isn't just any shipping container. This shipping container is a prototype data center.

Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support ind power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid.

While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.

Two years ago Google had one data center. Today they are reported to have 64. Two years from now, they will have 300-plus. The advantage to having so many data centers goes beyond simple redundancy and fault tolerance. They get Google closer to users, reducing latency. They offer inter-datacenter communication and load-balancing using that no-longer-dark fiber Google owns. But most especially, they offer super-high bandwidth connections at all peering ISPs at little or no incremental cost to Google.

Where some other outfit might put a router, Google is putting an entire data center, and the results are profound. Take Internet TV as an example. Replicating that Victoria's Secret lingerie show that took down Broadcast.com years ago would be a non-event for Google. The video feed would be multicast over the private fiber network to 300+ data centers, where it would be injected at gigabit speeds into each peering ISP. Viewers watching later would be reading from a locally cached copy. Yeah, but would it be Windows Media, Real, or QuickTime? It doesn't matter. To Google's local data center, bits are bits and the system is immune to protocols or codecs. For the first time, Internet TV will scale to the same level as broadcast and cable TV, yet still offer soemthing different for every viewer if they want it.

As for the coming AJAX Office and other productivity apps, they'll sit locally, too. Two or three hops away from every user, they'll also be completely backed-up by two to three data centers down the line. Your data never goes away unless you erase it. Your latency and system response are as low as they can possibly be made for a network app.

And remember the Google Web Accelerator that came and disappeared? It's back! Only this time the Web Accelerator will have the proper hardware and network infrastructure to make it worth using.

This is more than another Akamai or even an Akamai on steroids. This is a dynamically-driven, intelligent, thermonuclear Akamai with a dedicated back-channel and application-specific hardware.

There will be the Internet, and then there will be the Google Internet, superimposed on top. We'll use it without even knowing. The Google Internet will be faster, safer, and cheaper."

Quite a scenario.

Just the bit about a "datacenter in a container" got me jazzed. 

As someone who worked closely with a host of data center companies in the last go around like Exodus (lead research analyst on it's IPO)  and others, the thought of city blocks of data center equipment being condensed into the size of a container is something to contemplate.

It's like that bit in Apollo 13, when Tom Hanks early on is giving a group tour of the NASA facilities, and says:

"What did the man say?       
"Give me a lever long enough
and I'll move the world"?   

Well, that's exactly  
what we're doing here.

This is divine     
inspiration, folks.
 
It's the best part of     
each one of us: the belief
that anything is possible.
 
Things like a computer that can
fit into a single room...      
 
and, and hold millions   
of pieces of information."

(Subtext alert:  Internet Datacenters today are roughly where computers were in the sixties, EVEN WITH Google's alleged and if true, very cool,  "container data centers").

But coming back to earth,  it's consistent with my thesis that in this "Web 2.0" cycle, the consumer internet companies are going to have to shoulder their fair share and possibly more, of building out the next generation of  network infrastructure, and possibly take the hit on their financials.

Cringely offers some back of the envelope calculations:

"Say the containers cost $500,000 each in volume and $500,000 per year to run. That's $300 million to essentially co-opt the Internet."

As I explained in a post on internet infrastructure needed for Web 2.0, back in October,

"What's truly different since 1999 is that we don't have a whole horizontal layer of infrastructure companies this time around, funded by the same VC and investor exuberance ready and waiting to provide the storage and ISP (Internet Service Provider) infrastructure to the consumer online services companies. "

As a result,

"(It)...means that the consumer Internet companies are going to have to get their hands dirty this time.  The effort, R&D, and the expense of the infrastructure build this time around will have to be borne on the financial models and statements of the consumer companies themselves, with or without the concurrent understanding, support and enthusiasm of their public shareholders. 

And these companies will need to do this globally given the much greater maturity of international markets in Web 2.0 vs. Web 1.0."

And it'll result in an arms race.  If Cringely's scenario is true, each of the GYMA consumer companies (Google, Yahoo!, Microsoft, and AOL) will have to step up their infrastructure spend if any of them accerates more than the norm. 

And they'll be joined by the cable and telco companies since all of this involves the fusion of video, audio, text, and every form of multimedia imaginable.

This'll obviously be good for infrastructure companies, who'll wind up getting billions in additional orders.  (UPDATED)  All the more reason for Cisco to spend over $ 5 billion for cable box vendor Scientific Atlanta.  Combined with the home wireless technologies from it's Linksys unit, it offers much easier off and on-ramps for content and services from Google and others into the home.  Om Malik has a good post on the dynamics here titled "Now it's Cisco vs. Motorola, Microsoft".

It's likely that in response to Google's potential infrastructure plans and the general heavy lifting to be done ahead, that we see the emergence of a new set of infrastructure companies over the next year or so.  And that'll be helpful for a number of the GYMAAAE companies who'd rather not do the infrastructure thing themselves and partner with outside vendors.

My guess is that Google would do it's own infrastructure even if outside options were to become available, since it's always the way they've operated, from the very beginning.  Their own-home built, world-class computing infrastructure. 

And consider this...if the end-game is truly about shifting not all, but SOME computing resources off individual PCs into the network, then you need NOT just storage near the access peer points on the broadband networks, but also data computing as well. 

It'd be the logical way to get processing queries back individually to millions of Google users who would presumably not just be using Google for search (data base searches of stored indices), but for every day computing tasks (live processing of everyday applications off the network).  This would then be served near-instantaneously to any kind of device, PC, cell phone, PDA, tablet or some other thingamajig.

It's about "Edge Competences" indeed, as Bubblegeneration would say.

Heady stuff...and a technical sumbofabitch to get done.

And Google is the only one amongst the GYMAAAE companies who's got the internal chops to get it done.  No, not Microsoft...you'd think yes, but they've outsourced almost every infrastructure aspect of MSN, their online service to third-party providers.  They may decide to grow that capability, but it'll get time.  Yahoo!'s re-getting this, if the recent pace, quantity and quality of their technical hiring is any indication.  AOL is still trying to find it's next mommy.

It's not that  these companies can't get the job done from a technical point of view.  It's about who get it done with the most technical AND financial efficiencies, both from a cost and revenue/profit perspective, that's going to matter to shareholders.

But that's what the next set of consumer internet services are potentially all about.  So that's what'll have to be done.  And somehow paid for...hopefully handsomely to the one most efficient.

For consumers, it'll be all the more  goodness from all the providers large and small, and will likely see some very cool services from their favorite portals.  As the song goes, "It's only just begun..."

Comments

[Brian]

AT&T once owned more backbone and infrastructure than Google could dream of owning.

Now they're gone because they depended upon competitors for the "last mile" of connection to customers.

With all the infrastructure in the world, unless Google supplants Windows somehow (Google Internet Appliance? GoogleOS?) aren't they in the same situation AT&T was in?

[Michael Parekh]

Good point Brian...the distinction here is that I'm not talking about the pipes, but the computing and storage at the critical connection points in the pipe network.

AT&T had backbone pipes, and for a while a dial-up network infrastructure on the consumer side, and a "broadband" private data network for corporate customers. There was very little "computing power" in the network, which is the core focus of what Google may be trying to do.

Infrastructure is too broad a word, but I've used it here as short hand. What I'm really talking about is the "computing, storage and networking" infrastructure in terms of what Google and the other portals need to do vs. the "pipes" backbone and network which is what the carriers are focused on.

The latter is increasingly a commodity...the value is putting "intelligence" in the network, preferably at the edges (i.e., as closer to major customer nodes as possible.

Hope that helps clarify my unclear language in the post.

Thanks again.

[Harish Pillay]

Michael -

I think your analysis is partially correct in that there has to be infrastructure investments by GYMAAAE, but of them all, only G is best suited to do this - because, apart from Y, the rest *sell* products/services, which is dramatically different in Gs and Ys case. Y could be a crucial player, but my sense is that Y has peaked a while back - after they went IPO - and that G is therefore in the best position to do the Right Thing.

Also, with the Google Internet overlaying on the Internet, we will see the value of what the Web brought to the text-internet 10 years ago.

I would also suggest that the in the true design genious of the Internet, the end points will be where the intelligence will reside and bootable USB drives (possibly in a handphone) will be the way people will access everything. The phone will give the means to authenticate with multi-factor means.

Harish

[Brock]

I wonder if Google will sell their Data Center Appliance at the Google Store, and if so, for how much?