Part 11: WebCrawler, Yahoo, and AltaVista

As I alluded to in last week’s post, this week we will be discussing three more early web-based search engines: WebCrawler, Yahoo, and AltaVista. Why these three, you may be asking? Well, it’s because they have three highly compelling storylines that perfectly illustrate one of the most interesting times in United States’ history – the Dotcom Bubble. It was a great era for some, and catastrophic for others. As it relates to search engines, it was a trying time, as we shall see whilst examining these three entities.

First, we will start with WebCrawler. WebCrawler was the brainchild of Brian Pinkerton, a computer science student at the University of Washington. He initially created WebCrawler as a desktop application in 1994, then quickly realized its potential as a web-based application. Three months later, he launched WebCrawler with an initial database of 4,000 websites. The application implemented the first true text-based query system, aka the Google Search homepage, which made it quite popular. In November, WebCrawler had its one-millionth query. The query was about Nuclear Weapon Design and Research, which is awfully suspicious, but regardless the achievement was highly significant. It would not be the only search engine launched in 1994 though.

Yahoo was created by two Stanford Electrical Engineering students, Jerry Yang and David Filo. It started out as a simple website titled “Jerry and David’s Guide to the World Wide Web”, and all it contained were links to websites the two college kids found interesting. As it would happen, a lot of other people found these links interesting, and by the end of 1994 Yahoo had been accessed by over 100,000 visitors with a total of 1 million hits. Realizing the potential this application had, the two of them quickly began seeking investors to grow their business. Six months later, in April 1995, they raised over $2 million in capital. 12 months after that, April 1996, Yahoo IPO’d. This venture into the business world is unique to our story of search engines thus far. The typical story has been the search engine’s creator would tinker it into oblivion, but not Yahoo. They went straight for the money, for better or worse.

Our third search engine is AltaVista, whose story is a unique combination of all the search engines we’ve discussed in this series. It was created by a team of researchers at Digital Equipment Corporation (DEC), a large computer manufacturing company, as a way for the company to showcase its hardware’s capability. AltaVista was unique in its ability to search for websites across multiple different devices using a multi-threaded crawler. Single-threaded crawlers, as the name suggests, could only search one device at a time. The multi-threaded crawler combined with the company’s excellent hardware proved to be quite the combination. DEC initially tested the tool internally with 10,000 employees before making it publicly available in December 1995. The launch was a resounding success. 300,000 people used it on the first day, and by the end of 1996 the website had 19 million hits. By the end of 1997, it had 80 million hits per day. One would think with numbers like this we would be searching AltaVista today and not Google. Well, that’s not what happened.

In the midst of the Dotcom Bubble, early 1998 to be exact, DEC was acquired by Compaq for $9.6 billion. Compaq elected to spin-off AltaVista a year later to CMGI, an internet investment company. CMGI intended to take AltaVista public in 2000, but the Dotcom Bubble burst, and the IPO was cancelled, thus spinning AltaVista into irrelevancy. WebCrawler was also a victim of the mad business culture during this time. It too was sold and spun-off several times to firms which, as a result of the Bubble busting, went under and took all their acquisitions with them. Among these firms were AOL, who initially acquired WebCrawler in 1995 before selling it to Excite, who eventually went bankrupt in 2001. Before Excite died off, it sold WebCrawler to InfoSpace.

There’s a colloquialism that money ruins everything which I typically do not agree with, but in the case of these two search engines it’s hard to argue that statement. Yahoo did feel the effects of the Dotcom Bubble, but unlike the other two it made strategic partnerships with other up-and-coming players such as Inktomi and Google which kept it alive just long enough to ride the subsequent economic wave in 2002.

We will leave off there, as I’ve led you all on long enough. The next blog, or maybe two, will be about Google. But before we leave this post, it’s important to acknowledge the timing factor in all of these early search engines. Had WebCrawler and AltaVista come around slightly earlier or later, there’s a high probability they would’ve had more staying power than Google or Yahoo. Similarly, had Google or Yahoo come around at different times, there’s a high probability they would not have experienced the same success. Paul Otlet and Ted Nelson, similarly, were too far ahead of their times to be taken seriously. I’m generalizing, but hopefully the point has been illustrated it’s not enough to just have a good idea. You need to have the right resources and be in the correct time period for an idea to take hold.

Appreciate that fact for a second, not just in the context of search engines but in every object you see. Chances are, the objects and the companies who make them were not the first to develop the idea, and they stood on the broad shoulders of the giants before them.

Now, on to Google...