Many search engines fall short of capacity

Joe Kauzlarich and Aaron Klemm

Trying to harness information from the Internet becomes more difficult as more Web pages go online.

One of the ways to make sense of the information overload is to use one of the many Web search engines available, and even their effectiveness has been called in question.

A study from the July 10 issue of New Scientist reported that many of the most popular search engines, such as Yahoo, Lycos or AltaVista, index only about 16 percent of the 800 million publicly available pages on the World Wide Web.

The 11 most popular search engines’ combined index reached only 43 percent of all the Web pages, according to the study.

The study also concluded that when a new Web page is made publicly available, it can take up to an average of six months before it shows up on search engines.

Some Iowa State professors understand the dilemma with Web search engines.

Anthony Hendrickson, associate professor of management information systems, said search engines are constantly being improved.

“A lot is being done to bring the most critical and most important information to the decision-maker’s desktop,” he said.

Search engine designers are concentrating on ways to bring the best search results to the user, not necessarily the most results, he said.

This area of development is called “knowledge management.”

“Knowledge management is probably one of the hottest areas of information technology right now,” he said.

Hendrickson said many of the search engines are developing “portals,” which allow the user to find appropriate pages through a series of menus, rather than by typing keywords.

Search engines differ in many ways, he said. One way is by how they gather information for their searches, Hendrickson said.

While some search engines receive information about new pages directly from their authors, others scour the Web themselves to locate new pages, he said.

Unfortunately, the search engines currently are unable to keep up with the growth of the Web, he said.

“We are almost at a point of information overload,” Hendrickson said. “We’re putting more things on the Web much faster than we’re developing engines to pull the right information off there.”

With the search engines finding only 16 percent of the information that is available on the Web, making the most effective searches is becoming more and more important.

Ed Jones, university professor of electrical and computer engineering, said the answer is simply to be reasonably specific with a choice of keywords.

“The more you can narrow down your search to begin with, the better off you’ll be,” he said.