NEVER SAY NEVER
"Any start-up trying to displace Google, Yahoo, or even MSN or Ask (or for that matter any VC trying to fund them) should just get in their car (or hop on a plane) and go look at Google’s new server farms at The Dales in Oregon. And if that doesn’t convince them they should head up the Columbia river a bit and check out Microsoft and Yahoo’s digs. The costs to compete in core search , are now simply to high."
He goes on to argue that:
"One can imagine a world in the not to distant future in which an application designer can easily leverage the billions of dollars being spent by Google, Yahoo et. al., by having programmatic access to what is essentially a custom crawl list and a highly filtered index.
In this way search engines, in some respects, may become an infrastructure layer not too dissimilar from the telecommunications networks and internet standards that they themselves are built upon."
In a foll0w-up comment to the post, he goes on to outline why even with the on-going improvement in the price-performance in hardware/bandwidth infrastructure costs, the price of entry into search is high, and will likely remain high. I summarize his five points here...read the full comment for exposition.
"What has changed since 1999 that has made entry into core search so much more expensive? I would say five main things but others may have more. These include:
1. Size of the web.
2. Frequency of crawl.
3. Size of index.
4. Complexity of indexing algorithms.
5. Query response."
He makes a very persuasive case that's worth reading, along with the various insightful comments by readers.
However, it does remind me a bit of the time Francis Fukuyama proclaimed "The End of History" in his controversial book at the end of the 20th century, predicting a world of global, liberal democracies in the 21st century following the end of the cold war.
As insurmountable the obstacles seem to a new company being able to do search with the kind of pervasive adoption and economic success of a Google, I can't help but feel that someone can find a way.
After all, it wasn't long when the tech world proclaimed that startups focused on productivity applications on the PC were dead.
I mean who would invest the hundreds of millions or billions it would take to go up against Microsoft?
And for a long time, no one did.
But technology continued to evolve, changing the current realities and future possibilities.
And now, many believe the formerly impossible may be possible after all.
Bill may very well be right in his thesis for quite some time.
But "never' is a long time.
To paraphrase fictional mathematician Ian Malcolm from Crichton's Jurassic Park, "Technology will find a way", with a little help from technologists, entrepreneurs, and investors.