Better search for the Twitter age

Search engines are cropping up to sift through constantly updated sites like Twitter, Flickr, and blogs, but they still have a long way to go.

EMAIL  |   PRINT  |   SHARE  |   RSS
google my aol my msn my yahoo! netvibes
Paste this link into your favorite RSS desktop reader
See all RSS FEEDS (close)
By Jessi Hempel, writer

How will the economy fare in the second half of 2009?
  • It will get worse
  • It will get better
  • It will stay about the same

NEW YORK (Fortune) -- When Iran cracked down on journalists following its recent election, international focus turned to Twitter as citizen journalists posted 140-character reports and links to photos and videos to the site. Trouble was, it was hard to sift the useful and reliable nuggets of information from scores of tweets that included plenty of spam, useless remarks, and stray sentiments.

Few events more clearly define the newest problem on the Web: how to make sense of all that real-time information bubbling up from Twitter, Facebook, Flickr, blogs, and really every single other self-publishing platform in cyberspace. It simply overwhelms.

Google (GOOG, Fortune 500) -- the tool consumers have relied on for a decade to organize the Internet for us -- isn't cutting it. At Google's Zeitgeist conference in May, co-founder Larry Page acknowledged the company has fallen behind Twitter, saying they've "done a relatively poor job of doing things that work on a per second basis."

A fast-growing group of startups aspires to pick up the slack by offering real-time search results. With names like Collecta, OneRiot and Scoopler, these companies attempt to provide answers to the question: what is happening on the web right now? Nearly every week, a new one is added to the mix. IDC analyst Hadley Reynolds explains, "Yahoo, Microsoft and Google will take a while to figure out how to cope with this. There is definitely a window of business opportunity for startups."

But providing these results is not straightforward. Inevitably, there is tension between information that is most recent, stuff that is most popular, and a subjective concept of content that is most important. There is no perfect solution.

Information filtered only by time is nearly as unwieldy as the data stream itself. But once you begin to add filters, weighting the results according to the authority of the publisher, for example, or the rate at which they're spreading across the web, you risk missing important trends and information tidbits because they are not popular.

The bulk of this stream of constantly updating information comes through Twitter. The site's search engine, a startup it purchased in 2008 called Summize, turns up results filtered only by time. Collecta, a search firm started this month, also filters by time, but draws from other blogs and social media sites on the web.

Time might not be the magic filter for search results -- or the best way to make sense of the information stream. Many of these startups, like Scoopler, have developed algorithms that attempt to unearth not just the latest tweet, but also the most popular. Among the more well known so far, OneRiot relies on a system called PulseRank that takes into account how fresh information is, how much authority the information's author has, and how quickly it's spreading to determine its rank.

So far, however, none of these search engines is particularly good. The interfaces are largely gaudy and none of the results are reliably useful. The day after Bernie Madoff was sentenced to 150 years in prison, a quick search on a smattering of these sites revealed: a "goodbye Bernie" YouTube video, a tweet linking to a Daily Show video, and another tweet asking who should play Bernie Madoff in a potential movie.

Maybe it's inevitably true that even a short lag lets the web sort out the kind of information that's ultimately useful. Google's search the day after Madoff's sentencing yielded a Los Angeles Times news story on the hearing followed by the fraudster's Wikipedia entry. To top of page

Company Price Change % Change
Ford Motor Co 8.29 0.05 0.61%
Advanced Micro Devic... 54.59 0.70 1.30%
Cisco Systems Inc 47.49 -2.44 -4.89%
General Electric Co 13.00 -0.16 -1.22%
Kraft Heinz Co 27.84 -2.20 -7.32%
Data as of 2:44pm ET
Index Last Change % Change
Dow 32,627.97 -234.33 -0.71%
Nasdaq 13,215.24 99.07 0.76%
S&P 500 3,913.10 -2.36 -0.06%
Treasuries 1.73 0.00 0.12%
Data as of 6:29am ET
More Galleries
10 of the most luxurious airline amenity kits When it comes to in-flight pampering, the amenity kits offered by these 10 airlines are the ultimate in luxury More
7 startups that want to improve your mental health From a text therapy platform to apps that push you reminders to breathe, these self-care startups offer help on a daily basis or in times of need. More
5 radical technologies that will change how you get to work From Uber's flying cars to the Hyperloop, these are some of the neatest transportation concepts in the works today. More
Worry about the hackers you don't know 
Crime syndicates and government organizations pose a much greater cyber threat than renegade hacker groups like Anonymous. Play
GE CEO: Bringing jobs back to the U.S. 
Jeff Immelt says the U.S. is a cost competitive market for advanced manufacturing and that GE is bringing jobs back from Mexico. Play
Hamster wheel and wedgie-powered transit 
Red Bull Creation challenges hackers and engineers to invent new modes of transportation. Play

Most stock quote data provided by BATS. Market indices are shown in real time, except for the DJIA, which is delayed by two minutes. All times are ET. Disclaimer. Morningstar: © 2018 Morningstar, Inc. All Rights Reserved. Factset: FactSet Research Systems Inc. 2018. All rights reserved. Chicago Mercantile Association: Certain market data is the property of Chicago Mercantile Exchange Inc. and its licensors. All rights reserved. Dow Jones: The Dow Jones branded indices are proprietary to and are calculated, distributed and marketed by DJI Opco, a subsidiary of S&P Dow Jones Indices LLC and have been licensed for use to S&P Opco, LLC and CNN. Standard & Poor's and S&P are registered trademarks of Standard & Poor's Financial Services LLC and Dow Jones is a registered trademark of Dow Jones Trademark Holdings LLC. All content of the Dow Jones branded indices © S&P Dow Jones Indices LLC 2018 and/or its affiliates.