DuckDuckGo founder on trying to fly with Google, Microsoft

AltaVista, one of the leading search engines of the 1990s, has died. It was 18 years old. It had languished for years before its owner, Yahoo, finally pulled the plug.

Silicon Valley is littered with the graves of firms that tried and failed to gain traction in the search market. Today, the U.S. market is dominated by two companies: Microsoft (whose technology also powers Yahoo’s search engine) and Google (whose technology also powers the AOL and Ask search engines). Internationally, Yandex and Baidu have significant market share in Russia and China, respectively.

Why does Google have so few competitors? Gabriel Weinberg is one of the few entrepreneurs who have managed to gain some traction in the search-engine market. His site, DuckDuckGo, touts industry-leading privacy features and “zero click info” boxes, which show information pulled from sites like Wikipedia and Wolfram Alpha.

Q: Why is it so hard to compete with Google? Are there large barriers to entry?

A: If you look at the search engines that died, especially the ones that raised a bunch of money in the mid-2000s, almost all of them tried to copy the indexing Google does. They tried to copy the whole Internet to their servers and then data-mine it. That is a very expensive proposition, both in human capital and physical capital. That is a large barrier to entry that only Microsoft, Yandex and Baidu have been able to do. All of those are public companies, and they all spend a lot of money per year on it. Way more than any startup has ever raised.

Mahalo started as human-powered search and eventually died. Cuil innovated on the design a bit, which was interesting, but couldn’t get traction with it. Another one that innovated on design was SearchMe, which had a cover-flow look.

The problem with spending a lot of money is that companies only get a few years before they run out of money. We’ve been around for five years now. We’re just now getting more understanding and mainstream adoption. It takes a while. If you have such high burn rate, you don’t have much runway.

Q: There are a few other competitors still on the market, right?

A: Blekko is still around. They developed the concept of “slashtags,” where you can search phrases like “gun control/liberal” [to get a liberal perspective on the topic] or “gun control/conservative” [for conservative views]. That’s been interesting. I think they also attempted a big crawl but did it in an intelligent way where they backfill [supplement their own crawl results with data from larger firms].

Other than them, there are not many new ones that are out there. Ask and AOL are still around, with 1 percent or 2 percent of the market. They are both white label of Google [e.g. Google’s search results repackaged]. AOL did really interesting things early in the 2000s, some of the stuff we’re doing right now. They showed that they were getting relevancy metrics that were higher than Google at the time. But it was expensive and wasn’t creating enough revenue.

Ask is generally white label. They’ve generally focused on the question-and-answer portion of it. Neither are attempting to be a general search engine, as far as I can tell.

Yahoo is trying to reboot itself now. And obviously, Microsoft has been the major competitor.

Q: There used to be a lot of search startups in the 1990s. Why is it so much harder to enter the market today? Has the Web gotten that much bigger?

A: It’s orders of magnitude bigger. It’s insanely bigger. It’s a power law. Just like you click on the top result way more than you click on the second result, most of those top results are covered by a small portion of the Internet: sites like Facebook or Twitter.

Even Google and Bing are structured this way. They have a primary index and a secondary index. Some of the companies are focused on getting the primary index better. It’s still expensive even to do the primary index. I think you’d have to ask them, but it seems like they’re spending on the order of $10 million per year on it.

There were a ton of search engines in the ‘90s. It was a much cheaper problem. Also, before Google came along, [the incumbents] weren’t very good. AltaVista was the best one before Google.

Q: How have you managed to gain traction, given these challenges?

A: I make an analogy to the Web-browser market. Just a few years earlier, Internet Explorer had been dominant. All the sudden, you had Firefox come on the scene, then Safari and Chrome came on the scene as well. That’s because the underlying tech had reached diminishing returns. More and more browsers have the same underlying technology. They’re differentiating by adding features [on top of shared rendering software].

Our approach was to treat the “copy the Internet” part as the commodity. You could get it from multiple places. When I started, Google, Yahoo, Yandex and Microsoft were all building indexes. We focused on doing things the other guys couldn’t do.

One was instant answers. People were going directly to Wikipedia, Yelp and other large communities with user-generated content. Those answers were not appearing on conventional search engines. You were still getting links. We thought, why can’t you aggregate that information, give people those results?

The reason the big guys can’t do that easily is scale. They only like to work across billions of things. They have a culture of doing everything in-house. Instead of working with any third party, Google or Microsoft buys the company they want to work with. That is a limited strategy. They can’t do that for smaller things.

We have plug-ins for Lego parts, for example. Very few people search for that, but people who do search like it a lot.

The big guys also have legal issues. Google in particular could get in trouble for favoring their own stuff over other people, favoring one thing over another.

Q: What else do you do to set DuckDuckGo apart?

A: We remove spam from content farms and parked domains. Google kind of responded to that with one of their updates, but not fully. A lot of those pages are still in Google’s search results. They have a couple of reasons they can’t respond to that very easily. I think they think those pages are more valuable than I think they are. I think a lot of their human raters say those pages are higher-quality than some people think they are.

Another advantage of DuckDuckGo is privacy. We don’t track our users. We don’t have to do that because we focus on Web search, where we can make money without tracking people. All the rest have other products that require tracking.

When you add all that stuff up, you have a search experience that is different than the other guys. It was never my argument that you would overtake them or kill them but that a significant percentage would prefer this experience over here, just like the Web-browser makers.