Better search for the Twitter age

Search engines are cropping up to sift through constantly updated sites like Twitter, Flickr, and blogs, but they still have a long way to go.

EMAIL  |   PRINT  |   SHARE  |   RSS
google my aol my msn my yahoo! netvibes
Paste this link into your favorite RSS desktop reader
See all RSS FEEDS (close)
By Jessi Hempel, writer

301 Moved Permanently

301 Moved Permanently

How will the economy fare in the second half of 2009?
  • It will get worse
  • It will get better
  • It will stay about the same

NEW YORK (Fortune) -- When Iran cracked down on journalists following its recent election, international focus turned to Twitter as citizen journalists posted 140-character reports and links to photos and videos to the site. Trouble was, it was hard to sift the useful and reliable nuggets of information from scores of tweets that included plenty of spam, useless remarks, and stray sentiments.

Few events more clearly define the newest problem on the Web: how to make sense of all that real-time information bubbling up from Twitter, Facebook, Flickr, blogs, and really every single other self-publishing platform in cyberspace. It simply overwhelms.

Google (GOOG, Fortune 500) -- the tool consumers have relied on for a decade to organize the Internet for us -- isn't cutting it. At Google's Zeitgeist conference in May, co-founder Larry Page acknowledged the company has fallen behind Twitter, saying they've "done a relatively poor job of doing things that work on a per second basis."

A fast-growing group of startups aspires to pick up the slack by offering real-time search results. With names like Collecta, OneRiot and Scoopler, these companies attempt to provide answers to the question: what is happening on the web right now? Nearly every week, a new one is added to the mix. IDC analyst Hadley Reynolds explains, "Yahoo, Microsoft and Google will take a while to figure out how to cope with this. There is definitely a window of business opportunity for startups."

But providing these results is not straightforward. Inevitably, there is tension between information that is most recent, stuff that is most popular, and a subjective concept of content that is most important. There is no perfect solution.

Information filtered only by time is nearly as unwieldy as the data stream itself. But once you begin to add filters, weighting the results according to the authority of the publisher, for example, or the rate at which they're spreading across the web, you risk missing important trends and information tidbits because they are not popular.

The bulk of this stream of constantly updating information comes through Twitter. The site's search engine, a startup it purchased in 2008 called Summize, turns up results filtered only by time. Collecta, a search firm started this month, also filters by time, but draws from other blogs and social media sites on the web.

Time might not be the magic filter for search results -- or the best way to make sense of the information stream. Many of these startups, like Scoopler, have developed algorithms that attempt to unearth not just the latest tweet, but also the most popular. Among the more well known so far, OneRiot relies on a system called PulseRank that takes into account how fresh information is, how much authority the information's author has, and how quickly it's spreading to determine its rank.

So far, however, none of these search engines is particularly good. The interfaces are largely gaudy and none of the results are reliably useful. The day after Bernie Madoff was sentenced to 150 years in prison, a quick search on a smattering of these sites revealed: a "goodbye Bernie" YouTube video, a tweet linking to a Daily Show video, and another tweet asking who should play Bernie Madoff in a potential movie.

Maybe it's inevitably true that even a short lag lets the web sort out the kind of information that's ultimately useful. Google's search the day after Madoff's sentencing yielded a Los Angeles Times news story on the hearing followed by the fraudster's Wikipedia entry. To top of page

Company Price Change % Change
Apple Inc 99.76 2.09 2.14%
Bank of America Corp... 16.26 0.05 0.31%
Pfizer Inc 27.93 0.10 0.36%
Facebook Inc 76.95 1.00 1.32%
Microsoft Corp 44.08 0.45 1.03%
Data as of 4:04pm ET
Index Last Change % Change
Dow 16,399.67 19.26 0.12%
Nasdaq 4,316.07 57.63 1.35%
S&P 500 1,904.01 17.25 0.91%
Treasuries 2.18 -0.02 -0.82%
Data as of 9:27pm ET
More Galleries
Some Converse copycats cost big bucks A few bargain brands got swept up in Chuck Taylor's net, but others cost a pretty penny. More
Urban infrastructure gets a second life Railroad beds become parks, power plants become aquariums and slaughterhouses are now art centers as an industrial past turns people-centric. More
Boomtown moms From working mothers raising their kids in RVs to stay-at-home moms who spend their days organizing events for the Oil Wives club, meet the moms of North Dakota's oil boom. More
Worry about the hackers you don't know 
Crime syndicates and government organizations pose a much greater cyber threat than renegade hacker groups like Anonymous. Play
GE CEO: Bringing jobs back to the U.S. 
Jeff Immelt says the U.S. is a cost competitive market for advanced manufacturing and that GE is bringing jobs back from Mexico. Play
Hamster wheel and wedgie-powered transit 
Red Bull Creation challenges hackers and engineers to invent new modes of transportation. Play

Market indexes are shown in real time, except for the DJIA, which is delayed by two minutes. All times are ET. Disclaimer Morningstar: © 2014 Morningstar, Inc. All Rights Reserved. Disclaimer The Dow Jones IndexesSM are proprietary to and distributed by Dow Jones & Company, Inc. and have been licensed for use. All content of the Dow Jones IndexesSM © 2014 is proprietary to Dow Jones & Company, Inc. Chicago Mercantile Association. The market data is the property of Chicago Mercantile Exchange Inc. and its licensors. All rights reserved. FactSet Research Systems Inc. 2014. All rights reserved. Most stock quote data provided by BATS.