How are we searching the World Wide Web? A comparison of nine search engine transaction logs

Bernard J. Jansen a,*, Amanda Spink b

a School of Information Sciences and Technology, The Pennsylvania State University, 329F IST Building,

University Park, PA 16802, USA

b School of Information Sciences, University of Pittsburgh, 610 IS Building, 135 N. Bellefield Avenue, Pittsburgh, PA 15260, USA

Received 26 May 2004; accepted 21 October 2004

Available online 7 January 2005

Abstract

The Web and especially major Web search engines are essential tools in the quest to locate online information for many people. This paper reports results from research that examines characteristics and changes in Web searching from nine studies of five Web search engines based in the US and Europe. We compare interactions occurring between users and Web search engines from the perspectives of session length, query length, query complexity, and content viewed among the Web search engines. The results of our research shows (1) users are viewing fewer result pages, (2) searchers on US-based Web search engines use more query operators than searchers on European-based search engines, (3) there are statistically significant di erences in the use of Boolean operators and result pages viewed, and (4) one cannot nec- essary apply results from studies of one particular Web search engine to another Web search engine. The wide spread use of Web search engines, employment of simple queries, and decreased viewing of result pages may have resulted from algorithmic enhancements by Web search engine companies. We discuss the implications of the findings for the devel- opment of Web search engines and design of online content.

3 2004 Published by Elsevier Ltd.

Keywords: Web search engines; Web searching; Transaction log analysis

1. Introduction

The Web is now the primary source of information for many people (Cole, Suman, Schramm, Lunn, & Aquino, 2003; Fox, 2002). Over 80% of Web searchers use Web search engines to locate online information

* Corresponding author. Tel.: +1 814 865 6459; fax: +1 814 865 6426.

E-mail addresses: [email protected] (B.J. Jansen), [email protected] (A. Spink).

0306-4573/$ - see front matter 3 2004 Published by Elsevier Ltd. doi:10.1016/j.ipm.2004.10.007

B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 249

or services (Nielsen Media, 1997). There is a critical need to understand how people use Web search en- gines. Amichai-Hamburger (2002) presents a review of the e ect of the Web and the lack of awareness of the user in the design of Web systems and site content. The research reported in this article attempts to contribute to such a dialogue. Most research of Web searching provides little longitudinal, regional, or across system analysis. We need a clearer understanding of emerging Web searching trends across dif- ferent global regions and between di erent Web search engines in order to design better searching systems.

This important research area directly impacts pay-per-click marketing, Web-site-optimization strategies, and Web and Intranet search engine design. It complements research such as that conducted by Liawa and Huangb (2003), who showed that individual experience, individual motivation, search engine quality, and user perceptions of technology acceptance are all factors a ecting individual desire to use Web search engines.

In this paper, we present a comparison of nine major Web studies, four European and five US-based Web search engines, over a seven-year period. We provide a temporal comparison of di erences in Web searching among and between US and European-based Web searches as one might expect some divergence due to linguistics and interface factors (Spink, Ozmutlu, Ozmutlu, & Jansen, 2002b). We specifically inves- tigate the interactivity between searchers and Web search engines, identifying changes in the complexity of Web search interactions. In addition, we present a longitudinal analysis of the types of information people are searching for on the Web.

We center our research analysis on the interactions between the user and the search engine. Interaction has several meanings in information searching, although the definitions generally encompass query formu- lation, query modification, and inspection of the list of results, among other actions. Belkin, Cool, Stein, and Theil (1995) have extensively explored user interaction within an information session. Efthimiadis and Robertson (1989) present and categorize interaction at various stages in the information retrieval pro- cess from information seeking research. Bates (1990) presents four levels of interaction, which are move, tactic, stratagem, and strategy. Lalmas and Ruthven (1999) present two groups of interaction, that which occurs across sessions and that which occurs within a session.

This within-session category is the type of interaction that we examine in this study. We consider an interaction as any specific exchange between the searcher and the system (i.e., submitting a query, clicking a hyperlink, etc.). We define a searching episode as a series of interactions within a limited duration to ad- dress one or more information needs. This duration is typically short, with Web researchers using between 5 and 120 min to define a session duration (c.f., He, Go¨ker, & Harper, 2002; Montgomery & Faloutsos, 2001; Silverstein, Henzinger, Marais, & Moricz, 1999). The searcher may be multitasking (Spink, 2004) within a searching episode, or the episode may be an instance of the searcher engaged in successive searching (Lin, 2002; Spink, Wilson, Ellis, & Ford, 1998).

We begin with an extensive review of literature concerning the rapidly growing area of Web search engine research. We then present the datasets used in this study. We discuss the analysis, results, and impli- cations of the results for the design of Web searching systems.

2. Related studies

There have been a few review articles on Web searching. Jansen and Pooch (2001) provide a review of Web transaction log research of Web search engines and individual Web sites through 2000. Hsieh-Yee (2001) reviews studies conducted between 1995 and 2000 on Web search behaviors. The researcher reports that many studies investigate the e ects of certain factors on search behavior, including information orga- nization and presentation, type of search task, Web experience, cognitive abilities, and a ective states. Hsieh-Yee (2001) also notes that many studies lack external validity. Bar-Ilan (2004) presents an extension and integrative overview of Web search engines and the use of Web search engines in information science

250 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263

research. Bar-Ilan (2004) provides a variety of perspectives including user studies, social aspects, Web struc- ture, and search-engine evaluation. We extend these review articles in this section, setting the stage for our analysis.

Web searching studies fall into three categories: (1) those that primarily use transaction-log analysis, (2) those that incorporate users in a laboratory survey or other experimental setting, and (3) those that examine issues related to or a ecting Web searching. In this paper, we focus on studies using transaction log ana- lysis. Romano, Donovan, Chen, and Nunamaker (2003) present a methodology for general qualitative analysis of transaction log data. Wang, Berry, and Yang (2003) and Spink and Jansen (2004) also present detailed explanations of approaches to transaction log analysis.

In investigations of single Web sites, Yu and Apps (2000) use transaction log data to examine user behavior in the SuperJournal project. For 23 months (February 1997 to December 1998), the researchers recorded 102,966 logged actions, related these actions to four subject clusters, 49 journals, 838 journal is- sues, 15,786 articles, and three Web search engines. In another study covering the period from 1 January to 18 September 2000, Kea, Kwakkelaarb, Taic, and Chen (2002) examined user behavior in Elsevier s ScienceDirect, which hosts the bibliographic information and full-text articles of more than 1300 journals with an estimated 625,000 users. Loken, Radlinski, Crespi, Millet, and Cushing (2004) examined the trans- action log data of the online self-directed studying of more than 100,000 students using a Web-based system to prepare for US college admissions tests for several months of use. The researchers noted several non- optimal behaviors, including a tendency toward deferring study and a preference for short-answer verbal questions. The researchers discuss the relevance of their findings for online learning.

Wen, Nie, and Zhang (2001) conducted research on a Web-based version of the Encarta encyclopedia. The researchers investigated the use of click-through data to cluster queries for question answering. The researches explored the similarity between two queries using the common user-selected documents between them. The results indicate that a combination of both keywords and user logs is better than using either method alone. Using a Lucent proxy server, Hansen and Shriver (2001) used transaction-log analysis to cluster search sessions and to identify highly relevant Web documents for each query cluster.

Continuing the rich tradition of using transaction logs to investigate the remote use of library systems (Peters, 1993), Chen and Cooper (2001) clustered users of an online library system into groups based on patterns of states using transaction logs data. The researchers defined 47 variables, using them to classify 257,000 sessions. Then they collapsed these 47 variables into higher order groupings, identifying six distinct clusters of users. In a follow-on study, Chen and Cooper (2002) used 126,925 sessions from the same online system, modeling patterns using Markov models. The researchers found that a third-order Markov model explained five of the six clusters.

In what appears currently to be one of the longest temporal studies, Wang et al. (2003) analyzed 541,920 user queries submitted to an academic-Website-search engine during a four-year period (May 1997 to May 2001). Conducting analysis at the query and term levels, the researchers report that 38% of all queries con- tained only one term and that most queries are unique. Eiron and McCurley (2003) used 448,460 distinct queries from an IBM Intranet search engine to analyze the e ectiveness of anchor text.

Rather than focusing on single Web sites, other researchers have investigated information searching on Web-search engines. Ross and Wolfram (2000) analyzed queries submitted to the Excite search engine for subject content based on the co-occurrence of terms. The researchers categorized more than 1000 of the most frequently co-occurring term pairs into one or more 30 developed subject areas. The cluster analyses resulted in several well-defined high-level clusters of broad subject areas. He et al. (2002) examined contex- tual information from Excite and Reuters transaction logs, using a version of the Dempster–Shafer theory (Voorbraak, 1991) to identify search engine sessions. The researchers determined the average Web user ses- sion duration was about 12 min. Ozmutlu and Cavdur (Forthcoming) investigate contextual information using an Excite transaction log. The researchers explore the reasons underlying the inconsistent perfor- mance of automatic topic identification with statistical analysis and experimental design techniques.

B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 251

Xie and O Hallaron (2002) investigated caching to reduce both server load and user-response time in dis- tributed systems by analyzing a transaction log from the Vivisimo search engine, from 14 January to 17 February 2001. The researchers report that queries have significant locality, with query frequency following a Zipf distribution. Lempel and Moran (2003) also investigated clustering to improve caching of search en- gine results using more than seven million queries submitted to AltaVista. The researchers report that pre- fetching of search engine results can increase cache-hit ratios by 50% for large caches and can double the hit ratios of small caches.

Pu (2000) explored the searching behavior of users searching on two Taiwanese Web search engines, Dreamer and Global Area Information Servers (GAIS). The average length of English terms on these two Web search engines is 1.0 term for Dreamer and 1.22 terms for GAIS. Baeza-Yates and Castillo (2001) examined approximately 730,000 queries from TodoCL, a Chilean search system. They found that queries had an average length of 2.43 terms. A lengthier analysis is presented in Baeza-Yates and Castillo (2000). Montgomery and Faloutsos (2001) analyzed more than 20,000 Internet users who accessed the Web from July 1997 through December 1999 using data provided by Jupiter Media Metrix (http://www.jupiter- research.com). The researchers report users revisited 54% of URLs at least once during a searching session. They also report that browsing patterns follow a power law and the patterns remained stable throughout the period of analysis.

Rieh and Xu (2001) analyzed queries from 1,451,033 users of Excite collected on 9 October 2000. The researchers examined how each user reformulated his/her Web query over a 24 h period. Out of the 1,451,033 users logs collected, the researcher used various criteria to select 183 sessions for manual ana- lysis. The results show that while most query reformulation involves content changes, about 15% of the reformulation relate to format modifications.

Huang, Chien, and Oyang (2003) propose an e ective term-suggestion approach for interactive Web search using more than two million queries submitted to Web search engines in Taiwan. The researchers propose a transaction log approach to relevant term extraction and term suggestion using relevant terms that co-occur in similar query sessions.

Jansen and Spink (2003) determined that the typical Web session was about 15 min from an analysis of click through data from AlltheWeb.com. The researchers report that the Web search engine users on aver- age view about eight Web documents, with more than 66% of searchers examining fewer than five docu- ments in a given session. Users on average view about two to three documents per query. Over 55% of Web users view only one result per query. Twenty percent of the Web users view a Web document for less than a minute. These results would seem to indicate that the initial impression of a Web document is extremely important to the user s perception of relevance.

Beitzel, Jensen, Chowdhury, Grossman, and Frieder (2004) examine hundreds of millions of queries sub- mitted by approximately 50 million users to America Online (AOL) over a 7 day period from 26 December 2003 through 1 January 2004. During this period, AOL used results provided by Google. The researchers report that only about 2% of the queries contain query operators. The average query length is 2.2 terms, and 81% of users view only one results page. The researchers report changes in popularity and uniqueness of topically categorized queries across hours of the day.

Park, Bae, and Lee (Forthcoming) analyzed transaction logs of NAVER, a Korean Web search engine and directory service. The data was collected over a one-week period, from 5 January to 11 January 2003 and contained 22,562,531 sessions and 40,746,173 queries. Users of NAVER implement queries with few query terms, seldom use advanced features, and view few results pages. Users of NAVER had an average session length of 1.8 queries.

There is a growing breadth and depth in research concerning Web searching and interest in a variety of issues from interactions, cognitive processes, to algorithm enhancements, with a notable emphasis on clus- tering. There is an increasing common lexicon in the analysis and presentation of results, which permits the contrasting of results among this body of research. However, there has been little comparison of findings

252 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263

across studies. Therefore, we do not know if these finding have external validity across the larger Web pop- ulation and among the various Web search engine user groups. It is this issue that we address in this re- search by comparing results at key levels of analyses across a set of Web searching studies that provided significant data.

3. Research questions

We present the results from a comparative analysis across Web search engines focusing on following re- search questions:

1.What are the trends and di erences in the number of one query sessions?

2.What are the trends and di erences in the number of one-term queries?

3.What are the trends and di erences in the number of result pages viewed?

4.What are the trends and di erences in search topics?

In the next section, we present our research methodology.

4. Research design

4.1. Data collection

We utilize nine studies from currently published or forthcoming articles that provide significant data from searching on Web search engines. The nine studies we compare in this paper are shown chronologi- cally in Table 1.

The nine studies include: (1) a 1997 study of the Excite Web search engine (Jansen, Spink, & Saracevic, 2000), (2) a 1998 study of the Fireball Web search engine (Ho¨lscher & Strube, 2000), (3) a 1998 study of the AltaVista Web search engine (Silverstein et al., 1999), (4) a 1999 study of the Excite Web search engine (Wolfram, Spink, Jansen, & Saracevic, 2001), (5) a 2000 study of the BWIE Web search service (Cacheda & Vin˜a, 2001a, 2001b), (6) a 2001 study of the AlltheWeb.com Web search engine (Spink et al., 2002b), (7) a 2001 study of Excite Web search engine (Spink, Jansen, Wolfram, & Saracevic, 2002a), (8) a 2002 of the AlltheWeb.com (Spink et al., 2002b), and (9) a 2002 study of AltaVista (Jansen & Spink, Forthcoming). Collectively, the nine studies represent 287,212,000 (nearly 300 million) Web searching sessions and 1,015,126,814 (over 1 billion) queries that people submitted to the Web search engines.

If one views the studies from the geographical perspective of the Web search engine, there is a Euro- pean and an US grouping. For the analysis of European Web searching trends, we examined results from four studies over a five year period from three Web search engines. Fireball (http://www.fire- ball.com) is a predominantly German Web search engine. BWIE (http://www.bwie.com/) is a Spanish Web search service, and AlltheWeb.com (http://www.allthewebcom) is a Web search engine based in Norway.

Our analysis of US-based Web search engines covers five studies and data samples over a six period from two Web search engines. Excite (http://www.excite.com) was a major Web search engine at the time of the studies and is now a meta-search service. AltaVista (http://www.altavista.com) was an independent Web search engine from 1998 through 2002 and is now a Web search engine within the Yahoo! Search (http://www.yahoo.com) network. Other published studies did not provide a rich enough dataset for com- parison at the time of the study. We could not obtain data from other Web search engines in either Europe or the US (e.g., Google, MSN) at the time of the study.

Table 1

Aggregate data from Web search engine studies from 1997 through 2002

Study no. 1 2 3 4 5 6 7 8 9
                   
  Excite Fireball AltaVista Excite BWIE AlltheWeb.com Excite AlltheWeb.com AltaVista
                   
Region US European US US European European US European US
Data collection Tuesday 16 1–31 July 1998 2 August–13 Wednesday 1 3–18 May Tuesday 6 Monday 30 Tuesday 28 Sunday 8
  September 1997   September 1998 December 1999 2000 February 2001 April 2001 May 2002 September
                  2002
Sessions 211,063 Not reported 285,474,117 325,711 83,232 153,297 262,025 345,093 369,350
Queries 1,025,908 16,252,902 993,208,159 1,025,910 71,810 451,551 1,025,910 957,303 1,073,388
Terms 1,277,763 Not reported Not reported 1,500,500 116,953 1,350,619 1,538,120 2,225,141 1,073,388
                   

B.J.Jansen,A.Spink/InformationProcessingandManagement42(2006)248–263

253

254 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263

4.2. Data analysis

We compare the changes in session length, query length, operator usage, and number of results pages viewed across these nine studies.

•Session length is the number of queries that a searcher submits in one episode with a Web search engine. We define an episode as the period from the first recorded time stamp to the last recorded time stamp on the search engine server from a particular searcher in a particular day.

•Query length is the number of terms in a query.

•Term is a series of alpha-numeric characters separated by white space or other delimiter.

•Operator usage is the number of Boolean or other operators in a query (i.e., AND, OR, MUST APPEAR, PHRASE).

•A results page is the set of usually 10 ranked uniform resources locators (URL) of Web documents (i.e., organic results) and other information (i.e., sponsored results) that a search engine presents to the user in response to a query.

•A results page viewed is the viewing of a results page by a searcher while trying to locate relevant documents.

The nine studies all use large-scale Web transaction logs that contain records of the interactions between searchers and the particular Web search engine. Web transaction logs allow for the analysis of aggregate Web search characteristics and trends, and are beneficial for understanding aspects of the real search pro- cess (i.e., a real user with a real information need using a working system and content). However, data on individual identities is typically not in a Web transaction log. A Web transaction log also does not record the reasons for the search, the searcher s motivations, or other qualitative aspects of user. In addition, cli- ent-side caching may result in incomplete data logging of the number of identical Web queries from users. However, Web transaction logs have the advantage of unobtrusively recording real interactions by real users in the pursuit of real information needs in the complex Web information environment. This natural interaction in such a realistic environment is di cult to recreate in a laboratory setting (Dumais, 2002, 7–11 May).

Web transaction logs follow a standard format and usually contain at least the following fields: (1) Time of day: measured in hours, minutes, and seconds from some daily time mark, (2) User identification: an anonymous user code assigned by the server representing the Internet Protocol address of the client s com- puter, and (3) Query: terms entered by the user. (4) Results page: a code representing a set of URLs and result abstracts returned by the Web-search engine in response to a query.

5. Results

We present the results of our comparative analysis at the session, query, and results page levels of ana- lysis from 1997 to 2002 across the nine datasets. Since the absolute numbers of sessions, queries, and results pages vary for each study, we use the percentages for comparison.

5.1. Sessions

At the session level, we analyze the percentage of sessions with only one query (i.e., a searcher submits one query and then departs) on each Web search engine. The trend in the percentage of one query sessions will inform us whether or not the number of queries per user is increasing or decreasing. Fig. 1 displays the results of this session analysis.

B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 255
Sessions 90%              
80%   AV          
             
70%       EX   EX ATW ATW
Query 60%        
           
50% EX           AV
One              
40%              
of 30%              
Percentage              
20%              
10%              
0%              
  1997 1998 1999 2000 2001 2002
   
            Year of Study    
      Fig. 1. Percentage of single query sessions.  

All figures in this paper follow a similar layout. The x-axis is the year of the study. The y-axis is the mea- sured percentage for a particular metric. The dark bar columns show the data points for the European stud- ies. The light bar columns show the data points for the US studies. There is a label on the columns identifying each study (i.e., ATW—AlltheWeb.com, AV—AltaVista, BWIE—BWIE, EX—Excite, FB— Fireball).

Fig. 1 shows that for the US Web search engines, it does not appear that the complexity of interactions is increasing as indicated by longer sessions (i.e., users submitting more Web queries). We conducted a Chi-Square goodness of fit procedure to evaluate whether or not the percentage of one query session across Web search engines was significantly di erent (i.e., homogeneity of proportions). A Chi-Square test indicated only marginally significance di erence among the Web search engines in terms of percent- age of one query sessions (Chi-Square (6) = 11.09, p = 0.086). However, if the 1998 AltaVista dataset is removed, there is no significant di erence among the remaining search engine datasets. This would indi- cate that the temporal cut-o used for analysis in the 1998 AltaVista study (Silverstein et al., 1999) was too short.

In 2002, approximately 47% of searchers on AltaVista submitted only one query, down from 77% in 1998. In the 1998 study, however, a session was artificially limited to 5 min. Subsequent research has shown that the typical Web session is about fifteen minutes (He et al., 2002; Jansen & Spink, 2003). Therefore, the 1998 AltaVista study probably over estimates the number of one query sessions. The downward trend also appears with Excite users from 1999 to 2002, dropping from 60% to 55%, although not a significant de- crease. The data analysis methods were similar for all Excite studies and did not impose a session time limit.

The session data for European users is available from two Web search engines, BWIE and Allthe- Web.com. For these European Web search engines, there is also no significant change in one query sessions. So, for session length, the trend appears to be one of stability, with no di erences among search engines.

5.2. Queries

At the query length level, we analyze the percentage of queries with only one term. The percentage of one-term queries will inform us whether or not the length of queries is increasing or decreasing. Fig. 2 dis- plays the results for the analysis of Web query lengths.

A Chi-Square test did indicate a significant di erence among the Web search engines in terms of percent- age of one-term queries (Chi-Square (7) = 26.43, p = 0.01). However, if the 1998 Fireball dataset is re- moved, there is no significant di erence among the remaining search engine data. This would signify

256 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263
  60%   FB        
             
Queries 50%            
40%           ATW
           
of       EX    
        EX  
Percentage 30% EX AV      
      ATW  
           
20%           AV
           
             
  10%            
  0%            
    1997 1998 1999 2000 2001 2002
        Year of Study    

Fig. 2. Percentage of one-term queries.

that something in the Fireball user base, content, or system di erentiates it from users of the other Web search engines.

For the US-based Web-search engines the percentage of one-term queries is holding steady, within a range of 20–29% of all queries. Using data from 1999 onward, the trend with US-based Web-search engines appears to be of one-term queries declining as a percentage of all queries, dropping from 30% to 20%.

For the Europe-based Web-search-engine users, the trend appears to be one of little change, although there is a spike in 2002 with AlltheWeb.com users. Otherwise, we see a percentage of one-term queries on these European-based Web-search engines within a range of about 25–35%, excluding the 1998 Fireball study.

5.3. Query operators

We also analyze the percentage of Web queries containing searching operators. The trend in the percent- age of queries with searching operators will inform us whether or not the complexity of query structure is increasing or decreasing.

Based on the use of advanced operators, the complexity of interaction appears to be at least remaining stable. Fig. 3 shows the results for query operator usage on the various Web search engines.

The usage of query operators appears to be search-engine dependent, and there is a notable regional dif- ference. A Chi-Square test indicated significant di erence among the US Web search engines in terms of percentage of usage of query operators (Chi-Square (4) = 16.383, p = 0.01). A Chi-Square test indicated no significant di erence among the three Excite search-engine datasets in terms of percentage of usage of query. A Chi-Square test indicated no significant di erence among the two AltaVista search-engine data- sets in terms of percentage of usage of query operators. This indicates that there is a search engine depen- dency in terms of the use of query operators with a particular search-engine system.

For the AltaVista Web search engine, the usage of query operators has held steady at approximately 20%. For the Excite Web search engine, the usage increased steadily from 1997 to 2001, although not a sta- tistically significant variation between datasets.

For the European-based Web search engines, the usage also varied among the three Web search engines, but these searchers seldom use advanced operators. A Chi-Square test indicated no significant di erence among the four European search datasets in terms of percentage of usage of query operators, with the usage extremely low on all.

B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 257
  25%          
    AV     EX AV
Queries 20%      
       
         
15%   EX      
of        
EX        
Percentage         ATW
         
10%          
5%     BWIE    
FB     ATW  
       
         
           
  0%          
  1997 1998 1999 2000 2001 2002
      Year of Study    

Fig. 3. Percentage of operator usage.

The most notable feature of operator usage is the rather large gap between usage on the US and Euro- pean-based Web search engines. The usage of query operators on the US-based Web search engines varied from 11% to 20%. The usage on the European-based Web search engines varied from 2% to 10% and held fairly stable at under 5% from 1998 to 2001.

5.4. Results pages

We analyze the percentage of users viewing only one results page. This trend will inform us how persis- tent searchers are when locating information or services on the Web. Overall, it appears that Web searchers are tending to view fewer documents per Web query, which might indicate a move to less complex interac- tions. Fig. 4 presents results-page-viewing findings.

We see that the percentage of searchers viewing only one results page is increasing for users of both US and European-based Web search engines. The percentage of searchers viewing only the first results page has increased from 29% in 1997 to 73% in 2002 for US-based Web search engines users. Again, the 1998

Viewed 90%   AV     ATW  
        ATW
         
80%          
          AV
        BWIE  
Pages 70%          
  FB        
60%          
        EX  
           
Result 50%          
    EX      
           
40%            
of            
  EX          
Percentage 30%          
           
20%            
10%            
  0% 1997 1998 1999 2000 2001 2002
   
        Year of Study    

Fig. 4. Percentage of single result page viewing.

258 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263

AltaVista study limited sessions to five minutes, which probably increased the percentage of sessions with only one page result. For European searchers, the variability ranged from 60% to 83%, although there was a dip to 76% in 2002.

A Chi-Square test indicated significant di erence among the Web search engines in terms of percentage of single result page viewing (Chi-Square (8) = 45.743, p = 0.01). A Chi-Square test indicated a significance di erence among the three Excite Web search-engine datasets in terms of percentage of single result page viewing (Chi-Square (2) = 6.049, p = 0.05). A Chi-Square test indicated no significance di erence among the two AltaVista search-engine datasets in terms of percentage of single result page viewing (Chi-Square (1) = 0.911, p = 0.34). A Chi-Square test indicated no significance di erence among the four European search datasets in terms of percentage of percentage of single result page viewing (Chi-Square (3) = 4.136, p = 0.247). Therefore, there was trend among Excite users to view fewer result pages. Excite users viewed more result pages than users of other Web search engines. However, as time processed, the tendency was to view fewer.

5.5. Topical classification

For the six Web query datasets that we had access to, we qualitatively analyzed a random sample of approximately 2600 queries from each in order to determine trends in the type of information people are searching for on the Web. We classified each query into eleven non-mutually exclusive, general topic categories developed by Spink et al. (2002a). At least two independent evaluators manually classified que- ries from each dataset independently. The evaluators then met and resolved discrepancies.

Tables 2 and 3 display the topical evaluation results for European and US-based Web search engines, respectively.

For searching on AlltheWeb.com, People, Places or Things category remained the top ranked category with a large percentage increase from 2001 to 2002, accounting for over forty percent of queries. Commerce, Travel, Employment or Economy and Computers, Internet or Technology accounted approximately 25% of the queries. Noticeably percentage decreases occurred in Computers or Internet, Entertainment or Recrea- tion, and Sex or Pornography. A Chi-square goodness of fit test indicates a significant di erence between the Web search-engine datasets based on category of People, Place or Things (Chi-Square (3) = 5.554, p = 0.05).

Table 2

Distribution of AlltheWeb.com general topic categories

  Categories 2001 (2503 English queries) (%) 2002 (2525 English queries) (%)
       
1 People, places or things 22.5 41.5
2 Computers or Internet 21.8 16.3
3 Commerce, travel, employment, or economy 12.3 12.7
4 Sex or pornography 10.8 9.5
5 Entertainment or recreation 9.1 4.9
6 Health or sciences 7.8 4.5
7 Society, culture, ethnicity or religion 4.8 2.6
8 Performing or fine arts 4.7 2.5
9 Education or humanities 2.9 2.3
10 Government 2.7 2.1
11 Unknown or Other 0.6 1.1
    100.0 100.0
       

Note: Bolded percentages indicate the highest ranked topic in a given year.

  B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 259
Table 3        
Distribution of Excite and AltaVista general topic categories      
           
  Categories 1997 Excite 1999 Excite 2001 Excite 2002 AltaVista
    (2414 queries) (%) (2539 queries) (%) (2453 queries) (%) (2603 queries) (%)
           
1 People, places, or things 6.7 20.3 19.7 49.3
2 Commerce, travel, employment, or economy 13.3 24.5 24.7 12.5
3 Computers or Internet 12.5 10.9 9.7 12.4
4 Health or sciences 9.5 7.8 7.5 7.5
5 Education or humanities 5.6 5.3 4.6 5.0
6 Entertainment or recreation 19.9 7.5 6.7 4.6
7 Sex and pornography 16.8 7.5 8.6 3.3
8 Society, culture, ethnicity, or religion 5.7 4.2 3.9 3.1
9 Government 3.4 1.6 2.0 1.6
10 Performing or fine arts 5.4 1.1 1.2 0.7
11 Non-English or unknown 4.1 9.3 11.4 0.0
    102.9 100.0 100.0 100.0
           

Note: Bolded percentages indicate the highest ranked topic in a given year.

On the US-based Web search engines. Queries for People, Place or Things account for nearly half of the queries in 2002, with Commerce, Travel, Employment or Economy and Computers, Internet or Technology accounting for another 25% of the queries. There appears to be a steady rise in searching for People, Place or Things and Commerce, Travel, Employment or Economy, with decreased searching for Sex and Porno- graphy and Entertainment or Recreation. A Chi-squared goodness of fit test indicated significant di erences among the Web search-engines datasets based on distribution of queries among categories in the areas of

People, Places, or Things (Chi-Square (3) = 39.317, p = 0.01), Entertainment or Recreation (Chi-Square (3) = 13.80, p = 0.01), and sex and pornography (Chi-Square (3) = 10.892, p = 0.05). There was a marginally significant di erence with the category of Commerce, Travel, Employment, or Economy (Chi-Square (3) = 4.136, p = 0.06). There was no significant di erence among the datasets in the other categories. The percentages of People, Places, or Things and Commerce, Travel, Employment, or Economy are increasing at the expense of Entertainment or Recreation and Sex and Pornography.

6. Discussion

As the Web is becoming a worldwide phenomenon, we need to understand better the emerging trends in Web searching given the tremendous influence Web search engines have on directing tra c to online infor- mation and services. Our findings indicate that the interactions between Web search engines and searchers are not becoming more complex, and in some respects, are becoming less complex. Our comparative ana- lysis also indicates that finding from a study focusing on one Web search engine cannot be applied whole- sale to all Web search engines.

Sessions lengths are not increasing as measured by number of queries. The percentage of one-term ses- sions is remaining stable over time and across Web search engines. There was a di erence with the 1998 AltaVista study, but this appears to be caused by an artificially short session duration that the researchers used. Queries lengths are also not increasing as measured by number of terms. There was a statistical dif- ference in the percentage of one-term queries on the German Fireball Web search engine, which may be due to linguistic di erences with the other Web search engines. The percentage of single-term queries is holding steady, and the use of query operators is also remaining stable. Web search engines in the future may better leverage the implicit feedback from this interaction to provide more personalized results (Callan &

260 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263

Smeaton, 2003). However, the use of query operators between Web search engines varies significantly, so in this area findings from one study cannot necessary be applied to predict behaviors on other Web search engines.

The viewing of only the first page of results is extremely high, and it significantly increased over time on the Excite Web search engine. This may indicate increasing simplicity in interactions. It may also be an indi- cation of the increasing ability of Web search engines to retrieve and rank Web documents more e ectively. There is certainly a need for more studies that focus on the Web document and virtual document (Watters, 1999) level of analysis.

The trend toward viewing fewer result pages with Excite users may be related to a changing user base during the time of the study as the Web population dramatically increased during this time. Excite was the second most popular Web site in 1997 (Munarriz, 1997), and was the fifth most popular in 1999 and 2001 as measured by number of unique visitors (Cyber Atlas, 1999, 2001).

There are both similarities and di erences between usage on US and European-based Web search en- gines. Searchers on both are similar in session length, query length, and number of results pages viewed. Additionally, the use of Web query operators on both is fairly stable. However, the usage of these advanced Web-query operators is much higher on US-based Web search engines than on their European counter- parts. In investigating this di erence, we ruled out size of content collections (they are all immense), user bases (they all number in the millions), or algorithmic sophistication (they are all similar in performance tests). Fireball and BWIE did not prominently display the advanced Web searching options; however, it may be that users of these Web search engines just do not use query operators. This increases the criticality of keyword and phrase selection for Web providers targeting these users.

Fireball is a general purpose Web search engine, but, BWIE is also a search directory. A search directory supplements query matching of the entire content collection with directory-based search (c.f., Yahoo http:// www.yahoo.com or Open Directory http://dmoz.org/). The idea behind directory services is to provide additional organization to the content. However, some research has shown that directory-based searching does not improve searching performance and also takes longer (Dennis, Bruza, & McArthur, 2002). There are variations of the search directory including specialized or niche Web search engines that provide content within a specific Web search engines, including computer science literature (CiteSeer http://www.research- index.com), e-commerce (Froogle http://froogle.google.com/), or personal information (c.f., http:// www.switchboard.com). Some Web search engines provide clustering (Vivisimo http://vivisimo.com/), which one can view as an automated, real time, and virtual directory service.

AlltheWeb.com has extensive advanced Web search features, however. Additionally, the results of the 2002 AlltheWeb.com dataset do not conform to the results from studies of the other European-based Web search engines. One possible reason may be that AlltheWeb.com is attracting searchers outside of its traditional European market. From our analysis of the AlltheWeb.com transaction log, nearly 90% of the query requests are in English, with 6% French, 1% each Spanish, German, Italian, and a variety of other languages making up the rest. Further research will be needed to isolate the e ects of linguistic di erences.

Web searching topics are changing. There was a decrease in sexual searching as a percentage of overall Web searching on both European and US-based Web search engines. The overall trend is towards using the Web as a tool for information or commerce, rather than entertainment. This trend is more pronounced with US as opposed to European searchers. This analysis certainly confirms survey and other data that the Web is now a major source of information for most people (Cole et al., 2003; Fox, 2002). There is increased use of the Web as an economic resource and tool (Lawrence & Giles, 1999; Spink et al., 2002a), and people use the Web for an increasingly variety of information tasks (Fox, 2002; National Telecommunications & Information Administration, 2002).

The decreased level of interaction of Web searches may be unwelcome news for Web-search engine developers and for those providing Web-based information content, products, and services. Web users ap-

B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 261

pear unwilling to invest additional e ort to locate relevant Web content. The trend towards viewing only the first results page is a challenge for those seeking to draw visitors to their Web sites or for Web search engines attempting to generate revenue via ad impressions. Users have a low tolerance of viewing any re- sults past the first page. They prefer to reformulate the Web query rather than wade through result listings. Placement within the first page of Web search engine results of an accurate abstract appears to be a deter- mining factor in drawing tra c to a particular Web site.

We continue to conduct ongoing analysis of Web searching trends to provide a valuable insight into this important and critical area of human computer interaction and electronic commerce.

References

Amichai-Hamburger, Y. (2002). Internet and personality. Computers in Human Behavior, 18, 1–10.

Baeza-Yates, R., & Castillo, C. (October 2000). Relating web characteristics [in Spanish] [Website]. University of Chile. Retrieved 15.07.02 from the World Wide Web: http://www.todocl.cl/stats/rbaeza.pdf.

Baeza-Yates, R., & Castillo, C. (2001). Relating web structure and user search behavior. In Proceedings of the 10th World Wide Web conference (pp. 1–2). Hong Kong, China. 1–5 May.

Bar-Ilan, J. (2004). The use of web search engines in information science research. In B. Cronin (Ed.). Annual review of information science and technology (Vol. 33, pp. 231–288). Medford, NY, USA: Information Today.

Bates, M. J. (1990). Where should the person stop and the information search interface start? Information Processing & Management, 26(5), 575–591.

Beitzel, S. M., Jensen, E. C., Chowdhury, A., Grossman, D., & Frieder, O. (2004). Hourly analysis of a very large topically categorized Web query log. In Proceedings of the 27th annual international conference on Research and development in information retrieval (pp. 321–328). She eld, UK, 25–29 July.

Belkin, N., Cool, C., Stein, A., & Theil, S. (1995). Cases, scripts, and information-seeking strategies: on the design of interactive information retrieval systems. Expert Systems with Applications, 9(3), 379–395.

´

Cacheda, F., & Vin˜a, A. (2001a). Experiences retrieving information in the World Wide Web. In Proceedings of the 6th IEEE Symposium on Computers and Communications (pp. 72–79). Hammamet, Tunisia. July.

´

Cacheda, F., & Vin˜a, A. (2001b). Understanding how people use search engines: a statistical analysis for e-business. In Proceedings of the e-Business and e-Work Conference and Exhibition 2001 (pp. 319–325). Venice, Italy, October.

Callan, J., & Smeaton, A. (2003). Personalisation and recommender systems in digital libraries. Joint NSF_EU_DELOS working group report. Joint NSF-EU DELOS Working Group Report. Retrieved 1.1.02 from the World Wide Web: http://www-2.cs.cmu.edu/ ~callan/papers/personalisation03-wg.pdf.

Chen, H.-M., & Cooper, M. D. (2001). Using clustering techniques to detect usage patterns in a web-based information system. Journal of the American Society for Information Science and Technology, 52(11), 888–904.

Chen, H.-M., & Cooper, M. D. (2002). Stochastic modeling of usage patterns in a web-based information system. Journal of the American Society for Information Science and Technology, 53(7), 536–548.

Cole, J. I., Suman, M., Schramm, P., Lunn, R., & Aquino, J. S. (February 2003) The ucla internet report surveying the digital future year three [Website]. UCLA Center for Communication Policy. Retrieved 1.2.2003 from the World Wide Web: http://www.ccp.ucla.edu/ pdf/ucla-internet-report-year-three.pdf.

Cyber Atlas. (1999). US Top 50 internet properties, December 1999, at home/work combined [Website]. CyberAtlas. Retrieved 1.7.2000 from the World Wide Web: http://cyberatlas.internet.com.

Cyber Atlas. (2001). US Top 50 internet properties, may 2001, at home/work combined [Website]. CyberAtlas. Retrieved 1.7.2000 from the World Wide Web: http://cyberatlas.internet.com.

Dennis, S., Bruza, P., & McArthur, R. (2002). Web searching: a process-oriented experimental study of three interactive search paradigms. Journal of the American Society for Information Science and Technology, 53(2), 120–133.

Dumais, S. T. (2002). Web experiments and test collections [Presentation]. Retrieved 20.4.03 from the World Wide Web: http:// www2002.org/presentations/dumais.pdf.

Efthimiadis, E. N., & Robertson, S. E. (1989). Feedback and interaction in information retrieval. In C. Oppenheim (Ed.), Perspectives in information management (pp. 257–272). London: Butterworths.

Eiron, N., & McCurley, K. (2003). Analysis of anchor text for web search. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (pp. 459–460). Toronto, Canada. 28 July–1 August.

Fox, S., (2002, July 2002). Search engines [website] The Pew Internet & American Life Project. Retrieved 15.10.2002 from the World Wide Web: http://www.pewinternet.org/reports/toc.asp.

Hansen, M. H., & Shriver, E. (2001). Using navigation data to improve ir functions in the context of web search. In Proceedings of the tenth international conference on information and knowledge management (pp. 135–142). Atlanta, Georgia, USA. October.

262 B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263

He, D., Go¨ker, A., & Harper, D. J. (2002). Combining evidence for automatic web session identification. Information Processing & Management, 38(5), 727–742.

Ho¨lscher, C., & Strube, G. (2000). Web search behavior of internet experts and newbies. International Journal of Computer and Telecommunications Networking, 33(1–6), 337–346.

Hsieh-Yee, I. (2001). Research on web search behavior. Library & Information Science Research, 23(1), 168–185.

Huang, C.-K., Chien, L.-F., & Oyang, Y.-J. (2003). Relevant term suggestion in interactive web search based on contextual information in query session logs. Journal of the American Society for Information Science and Technology, 54(7), 638–649.

Jansen, B. J., & Pooch, U. (2001). Web user studies: a review and framework for future work. Journal of the American Society of Information Science and Technology, 52(3), 235–246.

Jansen, B. J., & Spink, A. (2003). An analysis of web information seeking and use: documents retrieved versus documents viewed. In

Proceedings of the 4th international conference on Internet computing (pp. 65–69). Las Vegas, Nevada. 23–26 June.

Jansen, B. J., & Spink, A. (Forthcoming). An analysis of web searching by European alltheweb.Com users. Information Processing & Management.

Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the web.

Information Processing & Management, 36(2), 207–227.

Kea, H.-R., Kwakkelaarb, R., Taic, Y.-M., & Chen, L.-C. (2002). Exploring behavior of e-journal users in science and technology: transaction log analysis of Elsevier s Sciencedirect onsite in Taiwan. Library & Information Science Research, 24(1), 265–291.

Lalmas, M., & Ruthven, I. (1999). A framework for investigating the interaction in information retrieval. In Proceedings of 9th European–Japanese conferences on information modeling and knowledge bases (pp. 222–239). Iwate, Japan. 24–28 May.

Lawrence, S., & Giles, C. L. (1999). Accessibility of information on the web. Nature, 400, 107–109.

Lempel, R., & Moran, S. (2003). Predictive caching and prefetching of query results in search engines. In Proceedings of the twelfth international conference on World Wide Web (pp. 19–28). Budapest, Hungary.

Liawa, S.-S., & Huangb, H.-M. (2003). An investigation of user attitudes toward search engines as an information retrieval tool.

Computers in Human Behavior, 19, 751–765.

Lin, S.-J. (2002). Design space of personalized indexing: enhancing successive web searching for transmuting information problems. In

Proceedings of the eighth Americas conference on information systems (pp. 1092–1100). Dallas, Texas. 9–11 August.

Loken, E., Radlinski, F., Crespi, V. H., Millet, J., & Cushing, L. (2004). Online study behavior of 100,000 students preparing for the SAT, ACT, and GRE. Journal of Educational Computing Research, 30(3), 255–262.

Montgomery, A., & Faloutsos, C. (2001). Identifying web browsing trends and patterns. IEEE Computer, 34(7), 94–95.

Munarriz, R. A. (1997). How did it double? Daily double. Retrieved 10.11.2002 from the World Wide Web: http://www.fool.com/ ddouble/1997/ddouble970812.htm.

National Telecommunications and Information Administration. (2002). A nation online: How Americans are expanding their use of the internet. Washington, DC: US Department of Commerce.

Nielsen Media. (1997). Search engines most popular method of surfing the web [Website]. Commerce Net/Nielsen Media. Retrieved 30.8.2000 from the World Wide Web: http://www.commerce.net/news/press/0416.html.

Ozmutlu, H. C. & Cavdur, F. (Forthcoming). Application of automatic topic identification on excite web search engine data logs.

Information Processing & Management.

Park, M., Bae, J., & Lee, S. (Forthcoming). End user searching: a Web log analysis of NAVAR, a Korean web search engine. Library & Information Science Research, 27(2).

Peters, T. (1993). The history and development of transaction log analysis. Library Hi Tech, 42(11), 41–66.

Pu, H. T. (2000). An exploratory analysis on search terms of network users in Taiwan [in Chinese]. Central Library Bulletin, 89(1), 23–37.

Rieh, S. Y., & Xu, H. (2001). Patterns and sequences of multiple query reformulation in web searching: a preliminary study. In

Proceedings of the 64th annual meeting of the American society for information science and technology, pp. 246–255.

Romano, N. C., Donovan, C., Chen, H., & Nunamaker, J. F. (2003). A methodology for analyzing web-based qualitative data. Journal of Management Information Systems, 19(4), 213–246.

Ross, N., & Wolfram, D. (2000). End user searching on the internet: an analysis of term pair topics submitted to the excite search engine. Journal of the American Society for Information Science, 51(10), 949–958.

Silverstein, C., Henzinger, M., Marais, H., & Moricz, M. (1999). Analysis of a very large web search engine query log. SIGIR Forum, 33(1), 6–12.

Spink, A. (2004). Multitasking information behavior and information task switching: an exploratory study. Journal of Documentation, 60(3), 336–345.

Spink, A., & Jansen, B. J. (2004). Web search: public searching of the web. New York: Kluwer.

Spink, A., Jansen, B. J., Wolfram, D., & Saracevic, T. (2002a). From e-sex to e-commerce: Web search changes. IEEE Computer, 35(3), 107–111.

Spink, A., Ozmutlu, S., Ozmutlu, H. C., & Jansen, B. J. (2002b). US versus European Web searching trends. SIGIR Forum, 32(1), 30–37.

B.J. Jansen, A. Spink / Information Processing and Management 42 (2006) 248–263 263

Spink, A., Wilson, T., Ellis, D., & Ford, F. (1998). Modeling users successive searches in digital environments. D-Lib Magazine. Voorbraak, F. (1991). On the justification of Dempster s rule of combination. Artificial Intelligence, 48(1), 171–197.

Wang, P., Berry, M., & Yang, Y. (2003). Mining longitudinal web queries: trends and patterns. Journal of the American Society for Information Science and Technology, 54(8), 743–758.

Watters, C. (1999). Information retrieval and the virtual document. Journal of the American Society for Information Science, 50(11), 1028–1029.

Wen, J.-R., Nie, J.-Y., & Zhang, H.-J. (2001). Clustering user queries of a search engine. In Proceedings of the 10th international conference on World Wide Web (pp. 162–168). Hong Kong. 1–5 May.

Wolfram, D., Spink, A., Jansen, B. J., & Saracevic, T. (2001). Vox populi: the public searching of the web. Journal of the American Society of Information Science and Technology, 52(12), 1073–1074.

Xie, Y., & O Hallaron, D. (2002). Locality in search engine queries and its implications for caching. In Proceedings of the twenty-first annual joint conference of the IEEE computer and communications societies (pp. 307–317). New York City, New York, USA. 23–27 June.

Yu, L., & Apps, A. (2000). Studying e-journal user behavior using log files: the experience of superjournal. Library & Information Science Research, 22(3), 311–338.