Categorizing

Web

Queries

According

to

Geographical

Locality

Web

pages

-LRB-

and

resources

in

general

-RRB-

can

be

characterized

according

to

their

geographical

locality

For

example

a

web

page

with

general

information

about

wildflowers

could

be

considered

a

global

page

likely

to

be

of

interest

to

a

ge-ographically

broad

audience

In

contrast

a

web

page

with

listings

on

houses

for

sale

in

a

specific

city

could

be

regarded

as

a

local

page

likely

to

be

of

interest

only

to

an

audience

in

a

relatively

narrow

region

Similarly

some

search

engine

queries

-LRB-

implicitly

-RRB-

target

global

pages

while

other

queries

are

after

local

pages

For

example

the

best

results

for

query

-LSB-

wildflowers

-RSB-

are

probably

global

pages

about

wildflowers

such

as

the

one

discussed

above

However

local

pages

that

are

relevant

to

say

San

Francisco

are

likely

to

be

good

matches

for

a

query

-LSB-

houses

for

sale

-RSB-

that

was

issued

by

a

San

Francisco

resident

or

by

somebody

moving

to

that

city

Unfortunately

search

engines

do

not

analyze

the

geographical

locality

of

queries

and

users

and

hence

often

produce

sub-optimal

results

Thus

query

-LSB-

wildflowers

-RSB-

might

return

pages

that

discuss

wildflowers

in

specific

U.S.

states

-LRB-

and

not

general

information

about

wildflowers

-RRB-

while

query

-LSB-

houses

for

sale

-RSB-

might

return

pages

with

real

estate

listings

for

locations

other

than

that

of

interest

to

the

person

who

issued

the

query

Deciding

whether

an

unseen

query

should

produce

mostly

local

or

global

pages

without

placing

this

burden

on

the

search

engine

users

is

an

important

and

challenging

problem

because

queries

are

often

ambiguous

or

underspecify

the

information

they

are

after

In

this

paper

we

address

this

problem

by

first

defining

how

to

categorize

queries

according

to

their

-LRB-

often

implicit

-RRB-

geographical

locality

We

then

introduce

several

alternatives

for

automatically

and

efficiently

categorizing

queries

in

our

scheme

using

a

variety

of

state-of-the-art

machine

learning

tools

We

report

a

thorough

evaluation

of

our

classifiers

using

a

large

sample

of

queries

from

a

real

web

search

engine

and

conclude

by

discussing

how

our

query

categorization

approach

can

help

improve

query

result

quality

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

CIKM

November

New

Orleans

Louisiana

USA

Copyright

ACM

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Information

Search

and

Retrieval

query

formulation

search

process

Algorithms

Experimentation

Human

Factors

Web

pages

-LRB-

and

resources

in

general

-RRB-

can

be

characterized

according

to

their

geographical

locality

For

example

a

web

page

with

general

information

about

wildflowers

could

be

considered

a

global

page

likely

to

be

of

interest

to

a

ge-ographically

broad

audience

In

contrast

a

web

page

with

listings

on

houses

for

sale

in

a

specific

city

could

be

regarded

as

a

local

page

likely

to

be

of

interest

only

to

an

audience

in

a

relatively

narrow

region

Earlier

research

-LSB-

-RSB-

has

addressed

the

problem

of

automatically

computing

the

geographical

scope

of

web

resources

Often

search

engine

queries

-LRB-

implicitly

-RRB-

target

global

web

pages

while

other

queries

are

after

local

pages

For

example

the

best

results

for

query

-LSB-

wildflowers

-RSB-

are

probably

global

pages

about

wildflowers

discussing

what

types

of

climates

wildflowers

grow

in

where

wildflowers

can

be

purchased

or

what

types

of

wildflower

species

exist

In

contrast

local

pages

that

are

relevant

to

say

San

Francisco

are

likely

to

be

good

matches

for

a

query

-LSB-

houses

for

sale

-RSB-

that

was

issued

by

a

San

Francisco

resident

or

by

somebody

moving

to

San

Francisco

even

if

San

Francisco

is

not

mentioned

in

the

query

The

user

's

intent

when

submitting

a

query

may

not

be

always

easy

to

determine

but

if

underspecified

queries

such

as

-LSB-

houses

for

sale

-RSB-

can

be

detected

they

can

be

subsequently

modified

by

adding

the

most

likely

target

geographical

location

or

by

getting

further

user

input

to

cus-tomize

the

results

Unfortunately

search

engines

do

not

analyze

the

geographical

locality

of

queries

and

users

and

hence

often

produce

sub-optimal

results

even

if

these

results

are

on-topic

and

reasonably

popular

or

authoritative

Thus

query

-LSB-

wildflowers

-RSB-

might

return

pages

that

discuss

wildflowers

in

specific

U.S.

states

-LRB-

and

not

general

information

about

wildflowers

-RRB-

In

fact

as

of

the

writing

of

this

paper

the

first

results

that

Google

provides

for

this

query

include

pages

each

of

which

discusses

wildflowers

in

only

one

U.S.

state

-LRB-

e.g.

Texas

Wildflowers

-RRB-

Similarly

the

top

results

that

Google

returns

for

query

-LSB-

houses

for

sale

-RSB-

include

real

estate

pages

for

Tuscany

United

Kingdom

and

New

Zealand

These

pages

are

likely

to

be

irrelevant

to

say

somebody

interested

in

San

Francisco

real

estate

who

types

such

an

underspecified

query

Deciding

whether

a

query

posed

by

a

regular

search

engine

user

should

produce

mostly

local

or

global

pages

is

an

important

and

challenging

problem

because

queries

are

often

ambiguous

or

underspecify

the

information

they

are

after

as

in

the

examples

above

By

identifying

that

say

query

-LSB-

wildflowers

-RSB-

is

likely

after

global

information

a

search

engine

could

rank

the

results

for

this

query

so

that

state-specific

pages

do

not

appear

among

the

top

matches

By

identifying

that

say

query

-LSB-

houses

for

sale

-RSB-

is

likely

after

local

information

a

search

engine

could

filter

out

pages

whose

geographical

locality

is

not

appropriate

for

the

user

who

issued

the

query

Note

that

deciding

which

location

is

of

interest

to

a

user

who

wrote

an

underspecified

query

such

as

-LSB-

houses

for

sale

-RSB-

is

an

orthogonal

important

issue

that

we

do

not

address

in

this

paper

Our

focus

is

on

identifying

that

such

a

query

is

after

local

pages

in

nature

and

should

therefore

be

treated

differently

by

a

search

engine

than

queries

that

are

after

global

pages

By

knowing

that

a

user

query

is

after

local

information

a

search

engine

might

choose

to

privilege

pages

whose

geographical

locality

coincides

with

that

of

the

user

's

or

alternatively

attempt

to

obtain

further

input

from

the

user

on

what

location

is

of

interest

In

this

paper

we

first

define

how

to

categorize

user

queries

according

to

their

-LRB-

often

implicit

-RRB-

geographical

locality

We

then

introduce

several

alternatives

for

automatically

and

efficiently

classifying

queries

according

to

their

locality

using

a

variety

of

state-of-the-art

machine

learning

tools

We

report

a

thorough

evaluation

of

our

classifiers

using

a

large

sample

of

queries

from

a

real

web

search

engine

query

log

Finally

we

discuss

how

our

query

categorization

approach

can

help

improve

query

result

quality

The

specific

contributions

of

this

paper

are

as

follows

A

discussion

on

how

to

categorize

user

queries

according

to

their

geographical

locality

based

on

a

careful

analysis

of

a

large

query

log

from

the

Excite

web

site

-LRB-

Section

-RRB-

A

feature

representation

for

queries

we

derive

the

feature

representation

of

a

query

from

the

results

produced

for

the

query

by

a

web

search

engine

such

as

Google

-LRB-

Section

-RRB-

A

variety

of

automatic

query

classification

strategies

that

use

our

feature

representation

for

queries

-LRB-

Section

-RRB-

A

large-scale

experimental

evaluation

of

our

strategies

over

real

search

engine

queries

-LRB-

Section

-RRB-

Preliminary

query

reformulation

and

page

re-ranking

strategies

that

exploit

our

query

classification

techniques

to

improve

query

result

quality

-LRB-

Section

-RRB-

Traditional

information-retrieval

research

has

studied

how

to

best

answer

keyword-based

queries

over

collections

of

text

documents

-LSB-

-RSB-

These

collections

are

typically

assumed

to

be

relatively

uniform

in

terms

of

say

their

quality

and

scope

With

the

advent

of

the

web

researchers

are

studying

other

dimensions

to

the

data

that

help

separate

useful

resources

from

less-useful

ones

in

an

extremely

heterogeneous

environment

like

the

web

Notably

the

Google

search

engine

-LSB-

-RSB-

and

the

HITS

algorithm

-LSB-

-RSB-

estimate

the

impor-tance

of

web

pages

by

analyzing

the

hyperlinks

that

point

to

them

thus

capturing

an

additional

dimension

to

the

web

data

namely

how

important

or

authoritative

the

pages

are

Ding

et

al.

-LSB-

-RSB-

extract

yet

another

crucial

dimension

of

the

web

data

namely

the

geographical

scope

of

web

pages

For

example

the

Stanford

Daily

newspaper

has

a

geographical

scope

that

consists

of

the

city

of

Palo

Alto

-LRB-

where

Stanford

University

is

located

-RRB-

while

the

New

York

Times

newspaper

has

a

geographical

scope

that

includes

the

entire

U.S.

To

compute

the

geographical

scope

of

a

web

page

Ding

et

al.

propose

two

complementary

strategies

a

technique

based

on

the

geographical

distribution

of

HTML

links

to

the

page

and

a

technique

based

on

the

distribution

of

geographical

references

in

the

text

of

the

page

Ding

et

al.

report

on

a

search-engine

prototype

that

simply

filters

out

from

the

results

for

a

user

query

any

pages

not

in

the

geographical

scope

of

the

user

This

technique

does

not

attempt

to

determine

whether

a

query

is

best

answered

with

global

or

local

pages

which

is

the

focus

of

our

paper

Ding

et

al.

built

on

the

work

by

Buyukkokten

et

al.

-LSB-

-RSB-

who

discussed

how

to

map

a

web

site

-LRB-

e.g.

-RRB-

to

a

geographical

location

-LRB-

e.g.

Palo

Alto

-RRB-

and

presented

a

tool

to

display

the

geographical

origin

of

the

HTML

links

to

a

given

web

page

This

tool

then

helps

visualize

the

geographical

scope

of

web

pages

-LSB-

-RSB-

A

few

commercial

web

sites

manually

classify

web

resources

by

their

location

or

keep

directory

information

that

lists

where

each

company

or

web

site

is

located

The

North-ernLight

search

engine

extracts

addresses

from

web

pages

letting

users

narrow

their

searches

to

specific

geographical

regions

-LRB-

e.g.

to

pages

originated

within

a

five-mile

radius

of

a

given

zip

code

-RRB-

Users

benefit

from

this

information

because

they

can

further

filter

their

query

results

McCurley

-LSB-

-RSB-

presented

a

variety

of

approaches

for

recognizing

geographical

references

on

web

pages

together

with

a

nav-igational

tool

to

browse

pages

by

geographical

proximity

and

their

spatial

context

-LRB-

Please

refer

to

-LSB-

-RSB-

for

additional

references

-RRB-

of

these

techniques

addresses

our

focus

problem

in

this

paper

automatically

determining

the

geographical

locality

associated

with

a

given

unmodified

search

engine

query

As

discussed

above

queries

posed

to

a

web

search

engine

can

be

regarded

as

local

if

their

best

matches

are

likely

to

be

local

pages

or

as

global

if

their

best

matches

are

likely

to

be

global

pages

In

an

attempt

to

make

this

distinction

more

concrete

we

now

discuss

several

examples

of

local

and

global

queries

Global

queries

often

do

not

include

a

location

name

as

is

the

case

for

query

-LSB-

Perl

scripting

-RSB-

A

user

issuing

this

query

is

probably

after

tutorials

about

the

Perl

language

and

hence

pages

on

the

topic

with

a

restricted

cal

scope

are

less

desirable

than

global

pages

Other

global

queries

do

not

mention

a

location

explicitly

either

but

are

topically

associated

with

one

particular

location

An

example

of

such

a

query

is

-LSB-

Elgin

marbles

-RSB-

which

is

topically

associated

with

the

city

of

Athens

We

consider

these

queries

as

global

since

their

best

matches

are

broad

global

pages

not

localized

pages

with

a

limited

geographical

scope

In-terestingly

global

queries

sometimes

do

include

a

location

name

For

example

a

query

might

be

just

a

location

name

-LRB-

e.g.

-LSB-

Galapagos

Islands

-RSB-

-RRB-

or

a

request

for

concrete

information

about

a

location

-LRB-

e.g.

-LSB-

Boston

area

codes

-RSB-

-RRB-

General

resources

about

the

location

-LRB-

e.g.

tourist

guides

-RRB-

are

arguably

to

be

preferred

for

such

queries

which

are

hence

regarded

as

global

Other

global

queries

include

locations

that

are

strongly

associated

topic-wise

with

the

rest

of

the

query

Query

-LSB-

Woody

Allen

NYC

-RSB-

is

an

example

of

such

a

query

The

location

mentioned

in

this

query

-LRB-

i.e.

NYC

for

New

York

City

-RRB-

is

not

used

to

restrict

query

results

to

pages

of

interest

to

New

York

residents

but

rather

expresses

a

topic

specification

Query

-LSB-

Ansel

Adams

Yosemite

-RSB-

is

another

example

photographer

Ansel

Adams

took

a

famous

series

of

photographs

in

Yosemite

Local

queries

often

include

a

location

name

as

is

the

case

for

query

-LSB-

Wisconsin

Christmas

tree

producers

association

-RSB-

The

location

mentioned

in

this

query

-LRB-

i.e.

Wisconsin

-RRB-

is

used

to

localize

the

query

results

Query

-LSB-

houses

for

sale

New

York

City

-RSB-

is

a

related

example

Other

local

queries

do

not

include

a

location

name

but

still

implicitly

request

localized

results

Query

-LSB-

houses

for

sale

-RSB-

is

an

example

of

such

a

query

These

queries

tend

to

be

underspecified

but

are

still

asked

by

-LRB-

presumably

ive

-RRB-

search

engine

users

We

conducted

a

thorough

examination

of

a

large

number

-LRB-

over

-RRB-

of

real

search

engine

queries

Most

queries

that

we

encountered

can

be

cleanly

categorized

as

being

either

global

or

local

However

other

queries

are

inherently

ambiguous

and

their

correct

category

is

impossible

to

determine

without

further

information

on

the

user

intent

behind

them

For

example

query

-LSB-

New

York

pizza

-RSB-

could

be

con-strued

as

a

local

query

if

it

is

say

after

pizza

delivery

web

sites

for

the

New

York

area

In

contrast

the

same

query

could

be

regarded

as

a

global

query

if

the

user

who

issues

it

wants

to

learn

about

the

characteristics

of

New

York-style

pizza

We

earlier

established

that

queries

are

associated

with

local

or

global

status

which

influences

the

kind

of

results

that

are

desirable

Since

current

search

engines

do

not

directly

take

into

account

geographical

information

for

certain

types

of

queries

they

produce

a

large

number

of

on-topic

but

un-wanted

results

as

in

the

-LSB-

houses

for

sale

-RSB-

example

discussed

earlier

In

this

section

we

discuss

automatic

methods

that

can

determine

given

a

query

whether

the

query

is

a

local

or

global

one

To

build

the

two-class

classifier

we

experimented

with

several

state-of-the-art

classification

techniques

using

widely

available

implementations

for

each

We

describe

below

the

features

used

in

the

classification

how

we

extract

them

from

web

pages

and

the

classifiers

with

which

we

experimented

Classification

Features

Web

queries

which

we

treat

in

this

paper

as

ordered

bags

of

words

with

no

other

structure

are

typically

fairly

short

In

the

collection

of

real

queries

that

we

used

in

our

experiments

-LRB-

Section

-RRB-

were

five

words

long

or

shorter

Because

few

words

are

available

per

query

basing

the

classification

directly

on

the

words

in

the

query

may

lead

to

severe

sparse

data

problems

Even

more

importantly

some

of

the

characteristics

that

make

a

query

local

or

global

are

not

directly

observable

in

the

query

itself

but

rather

in

the

results

returned

For

example

a

query

that

returns

results

that

contain

few

references

to

geographical

locations

is

likely

to

be

global

while

a

query

that

returns

results

spread

uniformly

over

many

locations

without

including

a

significant

percentage

of

results

with

no

locations

is

likely

to

be

local

For

these

reasons

we

base

our

classification

on

a

sample

of

results

actually

returned

for

a

given

query

rather

than

the

words

in

the

query

itself

By

observing

distributional

characteristics

in

the

unmodified

results

the

classifier

can

infer

the

type

of

the

query

-LRB-

global

or

local

-RRB-

so

that

the

results

can

be

appropriately

filtered

or

re-ordered

or

the

query

modified

In

a

way

the

approach

is

similar

in

spirit

to

query

expansion

techniques

that

rely

on

pseudo-relevance

feedback

-LSB-

-RSB-

In

our

experiments

we

use

Google

-LRB-

via

the

Google

API

-RRB-

to

obtain

the

top

web

pages

that

match

the

query

For

simplicity

we

limited

our

search

to

HTML

pages

skipping

over

non-HTML

documents

We

chose

Google

because

it

represents

state-of-the-art

web

search

technology

and

offers

a

published

interface

for

submitting

large

numbers

of

queries

We

represent

the

web

pages

returned

by

Google

as

text

documents

This

conversion

is

achieved

by

using

the

lynx

HTML

browser

with

the

dump

option

We

base

our

classification

features

on

measures

of

frequency

and

dispersion

of

location

names

in

these

text

files

For

this

purpose

we

have

constructed

a

database

of

location

names

by

concatenating

lists

of

all

country

names

of

the

capitals

of

these

countries

of

the

fifty

U.S.

states

and

of

all

cities

in

the

United

States

with

more

than

people

We

then

compare

the

words

in

each

text

document

with

the

database

of

location

names

and

output

any

matching

entries

and

their

count

per

document

This

matching

is

case

insensitive

because

we

found

capitalization

of

location

names

in

web

pages

to

be

erratic

Note

that

we

do

not

attempt

to

disambiguate

words

that

match

location

names

but

also

have

other

senses

-LRB-

e.g.

China

-RRB-

as

this

is

a

hard

problem

in

natural

language

analysis

instead

we

count

such

words

as

locations

An

alternative

approach

that

would

detect

and

disambiguate

location

names

would

be

to

use

a

named-entity

tagger

We

experimented

with

a

well-known

third-party

named-entity

tagger

but

we

encountered

a

very

high

error

rate

because

of

the

noise

often

introduced

in

web

pages

Our

classification

features

combine

these

location

counts

in

various

ways

For

each

query

we

measure

the

average

Obtained

from

the

United

Nations

Overview/unmember.html

Obtained

from

the

CIA

World

Factbook

capitals.com

Obtained

from

the

U.S.

Census

Bureau

-LRB-

census

figures

-RRB-

-LRB-

per

returned

web

page

-RRB-

number

of

location

words

in

the

retrieved

results

We

count

the

average

frequency

of

location

words

at

different

levels

of

detail

-LRB-

country

state

city

-RRB-

as

well

as

the

average

of

the

aggregate

total

for

all

locations

We

obtain

these

frequencies

for

both

the

total

count

-LRB-

tokens

-RRB-

and

the

unique

location

words

in

each

page

-LRB-

types

-RRB-

as

it

is

possible

that

a

few

locations

would

be

repeated

many

times

across

the

results

indicating

a

global

query

or

that

many

locations

would

be

repeated

few

times

each

indicating

a

local

query

We

also

consider

the

total

number

of

unique

locations

across

all

the

returned

documents

taken

together

divided

by

the

number

of

retrieved

documents

For

the

average

token

frequencies

of

city

state

and

country

locations

we

also

calculate

the

minimum

and

maximum

across

the

set

of

returned

web

pages

To

account

for

the

hierarchical

nature

of

location

information

we

calculate

an

alternative

frequency

for

states

where

we

include

in

the

count

for

each

state

the

counts

for

all

cities

from

that

state

that

were

found

in

that

text

this

allows

us

to

group

together

location

information

for

cities

in

the

same

state

We

also

include

some

distributional

measures

namely

the

fraction

of

the

pages

that

include

at

least

one

location

of

any

kind

the

percentage

of

words

that

match

locations

across

all

pages

and

the

standard

deviation

of

the

total

per

page

location

count

Finally

we

add

to

our

list

of

features

the

total

number

of

words

in

all

of

the

returned

documents

to

explore

any

effect

the

local/global

distinction

may

have

on

the

size

of

the

returned

documents

These

calculations

provide

for

distinct

features

that

are

passed

on

to

the

classifier

The

core

data

needed

to

produce

these

query

features

-LRB-

i.e.

the

locations

mentioned

in

each

web

page

-RRB-

could

be

efficiently

computed

by

a

search

engine

such

as

Google

at

page-indexing

time

Then

the

final

feature

computation

could

be

quickly

performed

at

query

time

using

this

core

data

Classification

Methods

We

initially

trained

a

classifier

using

Ripper

-LSB-

-RSB-

which

constructs

a

rule-based

classifier

in

an

incremental

manner

The

algorithm

creates

an

initial

set

of

very

specific

rules

based

on

the

data

similar

to

the

way

in

which

decision

trees

are

generated

The

rules

are

then

pruned

iteratively

to

eliminate

the

ones

that

do

not

seem

to

be

valid

for

a

large

enough

subset

of

the

training

data

so

as

to

prevent

overfitting

Although

Ripper

provides

a

robust

classifier

with

high

accuracy

and

transparency

-LRB-

a

human

can

easily

examine

the

produced

rules

-RRB-

it

outputs

binary

local

global

decisions

In

many

cases

it

is

preferable

to

obtain

a

measure

of

confidence

in

the

result

or

an

estimate

of

the

probability

that

the

assigned

class

is

the

correct

one

To

add

this

capability

to

our

classifier

we

experimented

with

logistic

regression

-LSB-

-RSB-

Logistic

or

log-linear

regression

models

a

binary

output

variable

-LRB-

the

class

-RRB-

as

a

function

of

a

weighted

sum

of

several

input

variables

-LRB-

the

classification

features

-RRB-

Con-ceptually

a

linear

predictor

is

first

fitted

over

the

training

data

in

a

manner

identical

to

regular

regression

analysis

i.e.

w

k

i

w

i

F

i

where

F

i

is

the

i-th

feature

and

w

i

is

the

weight

assigned

to

Studying

the

effect

on

classification

accuracy

of

a

richer

feature

set

-LRB-

e.g.

including

as

well

all

words

on

the

result

pages

-RRB-

is

the

subject

of

interesting

future

work

that

feature

during

training

Subsequently

is

transformed

to

the

final

response

C

via

the

logistic

transformation

C

e

e

which

guarantees

that

C

is

between

and

Each

of

the

endpoints

of

the

interval

-LRB-

-RRB-

is

associated

with

one

of

the

classes

and

C

gives

the

probability

that

the

correct

class

is

the

one

associated

with

In

practice

the

calculations

are

not

performed

as

a

separate

regression

and

transformation

but

rather

as

a

series

of

successive

regressions

of

transformed

variables

via

the

iterative

reweighted

least

squares

algorithm

-LSB-

-RSB-

We

used

the

implementation

of

log-linear

regression

provided

in

the

R

statistical

package

Another

desideratum

for

our

classifier

is

its

ability

to

support

different

costs

for

the

two

possible

kinds

of

errors

-LRB-

misclassifying

local

queries

versus

misclassifying

global

queries

-RRB-

Which

kind

of

error

is

the

most

important

may

vary

for

different

settings

for

our

search

modification

application

we

consider

the

misclassification

of

global

queries

as

local

ones

a

more

serious

error

This

is

because

during

our

subsequent

modification

of

the

returned

results

-LRB-

Section

-RRB-

we

reorder

the

results

for

some

of

the

queries

that

we

consider

global

but

we

modify

the

original

queries

for

some

of

the

queries

classified

as

local

returning

potentially

very

different

results

Consequently

the

results

can

change

more

significantly

for

a

query

classified

as

local

and

the

potential

for

error

is

higher

when

a

global

query

is

labeled

local

than

the

other

way

around

Both

Ripper

and

log-linear

regression

can

incorporate

different

costs

for

each

type

of

error

We

experimented

with

a

third

classification

approach

that

also

supports

this

feature

Support

Vector

Machines

-LRB-

SVMs

-RRB-

-LSB-

-RSB-

which

have

been

found

quite

effective

for

text

matching

problems

-LSB-

-RSB-

SVM

classifiers

conceptually

convert

the

original

measurements

of

the

features

in

the

data

to

points

in

a

high-dimensional

space

that

facilitates

the

separation

between

the

two

classes

more

than

the

original

representation

While

the

transformation

between

the

original

and

the

high-dimensional

space

may

be

complex

it

needs

not

to

be

carried

out

explicitly

Instead

it

is

sufficient

to

calculate

a

kernel

function

that

only

involves

dot

products

between

the

transformed

data

points

and

can

be

calculated

directly

in

the

original

feature

space

We

report

experiments

with

two

of

the

most

common

kernel

functions

a

linear

kernel

K

-LRB-

x

y

-RRB-

x

y

and

a

Gaussian

-LRB-

radial

basis

function

-RRB-

kernel

K

-LRB-

x

y

-RRB-

e

x-y

where

is

a

parameter

-LRB-

representing

the

standard

deviation

of

the

underlying

distribution

-RRB-

This

latter

kernel

has

been

recommended

for

text

matching

tasks

-LSB-

-RSB-

Regardless

of

the

choice

of

kernel

determining

the

optimal

classifier

is

equivalent

to

determining

the

hyperplane

that

maximizes

the

total

distance

between

itself

and

representative

transformed

data

points

-LRB-

the

support

vectors

-RRB-

Finding

the

optimal

classifier

therefore

becomes

a

constrained

quadratic

optimization

This

is

because

the

modeled

distribution

is

binomial

rather

than

normal

and

hence

the

variance

depends

on

the

mean-see

-LSB-

-RSB-

for

the

technical

details

Set

Original

number

of

queries

Number

of

appropriate

queries

Global

Local

Training

-LRB-

-RRB-

-LRB-

-RRB-

Development

-LRB-

-RRB-

-LRB-

-RRB-

Test

-LRB-

-RRB-

-LRB-

-RRB-

Table

Distribution

of

global

and

local

queries

in

our

training

development

and

test

sets

problem

In

our

experiments

we

use

the

SVM-Light

implementation

-LSB-

-RSB-

In

many

binary

classification

tasks

one

of

the

two

classes

predominates

and

thus

trained

classifiers

tend

to

favor

that

class

in

the

absence

of

strong

evidence

to

the

contrary

This

certainly

applies

to

our

task

as

we

show

in

Section

of

web

queries

are

global

Weiss

and

Provost

-LSB-

-RSB-

showed

that

this

imbalance

can

lead

to

inferior

classifier

performance

on

the

test

set

and

that

the

problem

can

be

addressed

through

oversampling

of

the

rarer

class

in

the

training

data

Their

method

examines

different

oversampling

rates

by

constructing

artificial

training

sets

where

the

smaller

class

is

randomly

oversampled

to

achieve

a

specific

ratio

between

samples

from

the

two

classes

For

each

such

sampling

ratio

a

classifier

is

trained

which

assigns

a

score

to

each

object

indicating

strength

of

evidence

for

one

of

the

classes

By

fixing

a

specific

strength

threshold

we

divide

the

classifier

output

into

the

two

classes

Further

by

varying

this

threshold

we

can

obtain

an

error-rate

curve

for

each

class

as

a

function

of

the

threshold

The

entire

process

results

in

a

Receiver-Operator

Characteristic

-LRB-

ROC

-RRB-

curve

-LSB-

-RSB-

for

each

sampling

ratio

Specific

points

on

the

curve

that

optimize

the

desired

combination

of

error

rates

can

then

be

selected

and

the

performance

of

the

classification

method

across

the

different

thresholds

can

be

measured

from

the

area

between

the

curve

and

the

x-axis

Weiss

and

Provost

use

the

classifier

-LSB-

-RSB-

a

decision

tree

classifier

with

additional

pruning

of

nodes

to

avoid

overfitting

We

use

a

software

package

provided

by

them

-LRB-

and

consequently

also

the

algorithm

-RRB-

to

explore

the

effect

that

different

ratios

of

global

to

local

queries

during

training

have

on

classifier

performance

We

now

describe

the

data

-LRB-

Section

-RRB-

and

metrics

-LRB-

Section

-RRB-

that

we

use

for

the

experimental

evaluation

of

the

query

classifiers

-LRB-

Section

-RRB-

Data

For

the

experiments

reported

in

this

paper

we

used

a

sample

of

real

queries

submitted

to

the

Excite

search

engine

We

had

access

to

a

portion

of

the

December

query

log

of

Excite

containing

queries

We

randomly

selected

initial

sets

of

queries

for

training

development

-LRB-

tuning

the

parameters

of

the

classifiers

we

train

-RRB-

and

testing

purposes

by

selecting

each

of

these

queries

for

inclusion

in

each

set

with

a

constant

-LRB-

very

small

-RRB-

probability

These

probabilities

were

set

to

and

Available

from

Setting

the

threshold

to

each

extreme

assigns

all

or

of

the

data

points

to

that

category

for

the

three

sets

respectively

Subsequently

we

combined

the

training

and

development

set

and

reassigned

the

queries

in

the

combined

set

so

that

three-fourths

were

placed

in

the

training

set

and

one-fourth

in

the

development

we

kept

the

test

set

separate

This

process

generated

and

queries

in

the

initial

versions

of

the

training

development

and

test

sets

We

further

eliminated

queries

that

passed

any

of

the

following

tests

Upon

examination

they

appeared

likely

to

produce

results

with

explicit

sexual

content

When

supplied

to

Google

and

after

filtering

out

any

non-HTML

results

and

any

broken

links

the

queries

produced

fewer

than

files

This

constraint

is

meant

to

ensure

that

we

are

not

including

in

our

experimental

data

queries

that

contain

misspellings

or

deal

with

extremely

esoteric

subjects

for

which

not

enough

material

for

determining

locality

would

be

available

They

had

already

been

included

in

an

earlier

set

-LRB-

we

constructed

first

the

training

set

then

the

development

set

and

finally

the

test

set

-RRB-

Since

multiple

people

may

issue

the

same

query

duplicates

can

be

found

in

the

log

Although

our

algorithms

take

no

special

advantage

of

duplicates

we

eliminated

them

to

avoid

any

bias

Taking

into

account

variations

of

upper/lower

case

and

spacing

between

queries

-LRB-

but

not

word

order

-RRB-

this

constraint

removed

queries

from

the

test

set

These

filtering

steps

left

us

with

queries

in

the

training

set

queries

in

the

development

set

and

queries

in

the

test

set

We

then

classified

the

queries

using

the

criteria

laid

out

in

Section

Table

shows

the

size

of

the

three

sets

before

and

after

filtering

and

the

distribution

of

local

and

global

queries

in

each

set

We

observe

that

in

general

most

queries

-LRB-

-RRB-

tend

to

be

global

Evaluation

Metrics

We

consider

a

number

of

evaluation

metrics

to

rate

the

performance

of

the

various

classifiers

and

their

configurations

Since

a

large

majority

of

the

queries

are

global

-LRB-

in

the

training

development

and

test

sets

combined

-RRB-

overall

classification

accuracy

-LRB-

i.e.

the

percentage

of

correct

classification

decisions

-RRB-

may

not

be

the

most

appropriate

measure

This

is

because

a

baseline

method

that

always

suggests

the

most

populous

class

-LRB-

global

-RRB-

will

have

an

accuracy

equal

to

the

proportion

of

global

queries

in

the

evaluated

set

Yet

such

a

classifier

will

offer

no

improvement

during

search

since

it

provides

no

new

information

The

situation

is

analogous

to

applications

in

information

retrieval

or

medicine

where

very

few

of

the

samples

should

be

labeled

positive

-LRB-

e.g.

in

a

test

for

a

disease

that

affects

only

of

patients

-RRB-

While

we

do

not

want

overall

accuracy

to

decrease

from

the

baseline

-LRB-

at

least

not

significantly

-RRB-

we

will

utilize

measures

that

capture

the

classifier

's

improved

ability

to

detect

the

rarer

class

relative

to

the

baseline

method

Two

standard

such

metrics

are

precision

and

recall

for

the

local

queries

Precision

is

the

ratio

of

the

number

of

items

correctly

assigned

to

the

class

divided

by

the

total

number

of

items

assigned

to

the

class

Recall

is

the

ratio

of

the

number

of

items

correctly

assigned

to

a

class

as

compared

with

the

total

number

of

items

in

the

class

Note

that

the

baseline

method

achieves

precision

of

but

recall

of

For

a

given

classifier

with

adjustable

parameters

often

precision

can

be

increased

at

the

expense

of

recall

and

vice

versa

therefore

we

also

compute

the

F-measure

-LSB-

-RSB-

-LRB-

with

equal

weights

-RRB-

to

combine

precision

and

recall

into

a

single

number

F-measure

Precision

Recall

Precision

Recall

Finally

we

argued

earlier

that

one

kind

of

misclassification

errors

may

be

assigned

a

higher

cost

We

can

then

calculate

the

average

cost

-LSB-

-RSB-

Average

cost

X

-LCB-

Global

Local

-RCB-

Cost

-LRB-

X

-RRB-

Rate

-LRB-

X

-RRB-

where

Cost

-LRB-

X

-RRB-

is

the

cost

of

wrong

X

classifications

and

Rate

-LRB-

X

-RRB-

is

the

rate

of

wrong

X

classifications

Average

cost

is

the

measure

to

minimize

from

a

decision

theory

perspective

The

rate

of

wrong

classifications

for

a

class

is

equal

to

the

number

of

data

points

that

have

been

misclassified

into

that

class

divided

by

the

total

number

of

classification

decisions

and

the

costs

for

each

misclassification

error

are

predetermined

parameters

If

both

costs

are

set

to

then

the

average

cost

becomes

equal

to

the

total

error

rate

i.e.

one

minus

accuracy

In

our

experiments

we

report

the

average

cost

considering

the

mislabeling

of

global

queries

as

local

twice

as

important

as

the

mislabeling

of

local

queries

for

the

reasons

explained

in

the

previous

section

Results

We

trained

the

classifiers

of

Section

on

the

queries

in

our

training

set

Ripper

and

the

regression

model

were

trained

on

that

training

set

without

modification

For

and

SVMs

we

explored

the

effect

that

different

proportions

of

local

queries

in

the

training

set

have

on

overall

performance

For

that

purpose

we

used

our

development

set

to

evaluate

the

performance

effects

of

different

local

query

proportions

and

select

the

optimal

classifier

within

each

family

For

the

classifier

we

used

the

software

provided

by

Foster

Provost

and

Claudia

Perlich

to

explore

the

effect

of

different

proportions

of

local

and

global

queries

We

created

training

sets

by

randomly

oversampling

or

un-dersampling

the

minority

-LRB-

local

-RRB-

class

as

needed

in

increments

of

For

any

given

proportion

of

local

queries

between

and

we

started

from

our

training

set

modified

it

according

to

the

above

sampling

method

to

have

the

desired

proportion

of

local

queries

trained

the

corresponding

classifier

and

evaluated

its

performance

on

our

development

set

The

natural

proportion

of

the

local

class

in

the

unmodified

training

data

is

also

included

as

one

of

the

proportions

used

to

build

and

evaluate

a

classifier

In

this

manner

we

obtain

curves

for

the

various

metrics

as

the

proportion

of

local

queries

varies

-LRB-

Figure

-RRB-

We

observe

Figure

Evaluation

metrics

for

classifiers

trained

on

different

proportions

of

local

queries

that

the

highest

value

for

precision

and

F-measure

and

the

lowest

value

for

the

average

cost

are

obtained

when

the

classifier

is

trained

with

a

significantly

amplified

proportion

of

local

queries

-LRB-

-RRB-

Further

running

with

local

queries

also

produced

the

largest

area

under

the

ROC

curve

obtained

when

different

precision/recall

tradeoffs

in

the

development

set

are

explored

On

the

basis

of

this

information

we

selected

the

proportion

of

local

queries

as

the

optimal

configuration

for

We

refer

to

that

configuration

as

-LRB-

-RRB-

and

this

is

the

version

of

that

we

evaluated

on

the

test

set

Using

our

own

implementation

for

constructing

extended

training

sets

with

a

given

proportion

of

local

queries

we

performed

similar

experiments

for

Support

Vector

Machines

with

linear

and

Gaussian

kernels

For

these

classifiers

we

also

experimented

with

versions

trained

with

equal

error

costs

for

the

two

kinds

of

classification

errors

and

with

versions

where

during

training

a

false

local

assignment

counts

for

twice

as

much

as

a

false

global

assignment

We

found

that

the

optimal

proportion

of

local

queries

is

closer

to

the

natural

proportion

with

SVMs

compared

to

classifiers

the

proportions

chosen

from

our

development

set

were

for

the

linear

SVM

classifier

with

equal

error

costs

for

the

linear

SVM

classifier

with

unequal

error

costs

for

the

Gaussian

SVM

classifier

with

equal

error

costs

and

for

the

Gaussian

SVM

classifier

with

unequal

error

costs

We

denote

the

optimal

classifiers

from

these

four

families

as

SVM-Linear-E

-LRB-

-RRB-

SVM-Linear-U

-LRB-

-RRB-

SVM-Gaussian-E

-LRB-

-RRB-

and

SVM-Gaussian-U

-LRB-

-RRB-

respectively

Figure

shows

the

curve

obtained

for

the

SVM-Gaussian-U

family

of

classifiers

Having

determined

the

best

value

for

the

proportion

of

local

queries

for

and

SVM-based

classifiers

we

evaluate

these

classifiers

as

well

as

the

classifiers

obtained

from

Ripper

and

log-linear

regression

on

our

test

set

Table

shows

the

values

of

the

evaluation

metrics

obtained

on

the

unseen

test

set

The

classifier

using

a

linear

kernel

SVM

with

unequal

error

costs

achieves

the

highest

F-measure

We

also

experimented

with

variable

error

costs

for

the

Ripper

classifier

using

the

same

error

cost

correspondence

but

the

resulting

classifier

was

identical

to

the

Ripper

classifier

obtained

with

equal

error

costs

Classifier

Recall

Precision

F-Measure

Average

Cost

Accuracy

Ripper

Log-linear

Regression

-LRB-

-RRB-

SVM-Linear-E

-LRB-

-RRB-

SVM-Linear-U

-LRB-

-RRB-

SVM-Gaussian-E

-LRB-

-RRB-

SVM-Gaussian-U

-LRB-

-RRB-

Baseline

-LRB-

always

global

-RRB-

Table

Evaluation

metrics

on

the

test

set

of

selected

classifiers

optimized

over

the

development

set

Figure

Evaluation

metrics

for

Support

Vector

Machines

with

Gaussian

kernel

and

false

local

assignments

weighted

twice

as

much

as

false

global

assignments

trained

on

different

proportions

of

local

queries

while

the

log-linear

classifier

achieves

the

lowest

average

classification

cost

As

expected

the

SVM

classifiers

that

were

trained

with

unequal

error

costs

achieve

the

same

or

lower

average

cost

-LRB-

which

also

utilizes

the

same

unequal

error

costs

-RRB-

compared

to

their

counterparts

trained

with

equal

error

costs

Overall

Ripper

log-linear

regression

and

the

two

SVM

classifiers

with

linear

kernels

achieve

the

highest

performance

with

small

differences

between

them

They

are

followed

by

the

two

SVM

classifiers

with

a

Gaussian

kernel

function

while

trails

significantly

behind

the

other

classifiers

The

features

used

for

classification

vary

considerably

from

classifier

to

classifier

Ripper

achieves

one

of

the

best

classification

performances

using

only

one

simple

rule

based

only

on

the

average

number

of

city

locations

per

returned

web

page

if

that

number

exceeds

a

threshold

the

query

is

classified

as

local

otherwise

as

global

On

the

other

hand

the

and

SVM

classifiers

utilize

all

or

almost

all

the

features

The

log-linear

regression

classifier

falls

in-between

these

two

extremes

and

primarily

utilizes

the

average

numbers

of

unique

city

state

and

country

names

per

retrieved

page

as

well

as

the

total

number

of

unique

locations

per

page

-LRB-

features

-RRB-

For

concreteness

and

to

conclude

our

discussion

Table

Most

classifiers

automatically

ignore

some

of

the

provided

features

to

avoid

overfitting

shows

the

performance

of

our

classifiers

on

a

few

representative

examples

of

local

and

global

queries

The

core

of

this

paper

is

on

classifying

queries

as

either

local

or

global

In

this

section

we

present

preliminary

ideas

on

how

to

exploit

this

classification

to

improve

the

quality

of

the

query

results

Further

exploration

of

these

and

other

directions

is

the

subject

of

interesting

future

work

Consider

a

query

that

has

been

classified

as

local

using

the

techniques

of

Section

By

definition

this

query

is

best

answered

with

localized

pages

We

can

easily

determine

if

the

query

includes

any

location

name

by

using

the

dictionary-based

approach

of

Section

If

no

locations

are

present

in

the

query

-LRB-

e.g.

as

in

query

-LSB-

houses

for

sale

-RSB-

-RRB-

in

the

absence

of

further

information

we

can

attempt

to

localize

the

query

results

to

the

geographical

area

of

the

user

issuing

the

query

for

which

we

can

rely

on

registration

information

provided

by

the

user

for

example

Consequently

we

can

simply

expand

the

query

by

appending

the

user

's

location

to

it

to

turn

say

the

query

-LSB-

houses

for

sale

-RSB-

into

-LSB-

houses

for

sale

San

Francisco

-RSB-

for

a

San

Francisco

resident

Alternatively

a

search

engine

might

attempt

to

obtain

additional

information

from

the

user

to

further

localize

the

query

as

appropriate

For

example

the

query

-LSB-

houses

for

sale

-RSB-

can

then

be

transformed

into

-LSB-

houses

for

sale

New

York

City

-RSB-

for

a

San

Francisco

resident

who

is

moving

to

New

York

City

In

either

case

the

expanded

query

will

tend

to

produce

much

more

focused

and

localized

results

than

the

original

query

does

As

of

the

writing

of

this

paper

all

of

the

results

returned

by

Google

for

query

-LSB-

houses

for

sale

San

Francisco

-RSB-

are

results

of

relevance

to

a

person

interested

in

Bay

Area

real

estate

In

contrast

most

of

the

results

for

the

original

query

-LSB-

houses

for

sale

-RSB-

are

irrelevant

to

such

a

person

as

discussed

in

the

Introduction

An

alternative

more

expensive

strategy

for

handling

these

queries

is

to

compute

and

exploit

the

geographical

scope

of

web

pages

as

defined

in

-LSB-

-RSB-

Then

pages

with

a

geographical

scope

that

includes

the

location

of

the

user

issuing

the

query

would

be

preferred

over

other

pages

In

contrast

a

local

query

in

which

locations

are

mentioned

is

likely

to

return

pages

with

the

right

locality

making

any

further

modification

of

the

query

or

reranking

of

the

results

unnecessary

Consider

now

a

query

that

has

been

classified

as

global

using

the

techniques

of

Section

By

definition

this

query

is

best

answered

with

broad

pages

Rather

than

attempting

to

modify

a

global

query

so

that

it

returns

broad

pages

we

can

follow

a

result

reranking

strategy

to

privilege

these

pages

over

more

localized

ones

One

possible

reranking

strategy

is

to

reorder

the

results

from

say

Google

for

Class

Query

Classifier

Ripper

Regression

-LRB-

-RRB-

SVM-LE

SVM-LU

SVM-GE

SVM-GU

Global

-LSB-

Perl

scripting

-RSB-

Global

Global

-LSB-

world

news

-RSB-

Global

Local

-LSB-

wildflowers

-RSB-

Global

Global

-LSB-

Elgin

marbles

-RSB-

Local

Local

-LSB-

Galapagos

Islands

-RSB-

Global

Global

-LSB-

Boston

zip

code

-RSB-

Local

Local

-LSB-

Woody

Allen

NYC

-RSB-

Global

Global

Local

-LSB-

houses

for

sale

-RSB-

Global

Global

-LSB-

Volkswagen

clubs

-RSB-

Local

Global

-LSB-

Wisconsin

Christmas

tree

producers

association

-RSB-

Global

Local

-LSB-

New

York

style

pizza

delivery

-RSB-

Global

Global

Table

Classification

assignments

made

by

different

classifiers

on

several

example

queries

SVM-LE

SVM-LU

SVM-GE

and

SVM-GU

stand

for

classifiers

SVM-Linear-E

-LRB-

-RRB-

SVM-Linear-U

-LRB-

-RRB-

SVM-Gaussian-E

-LRB-

-RRB-

and

SVM-Gaussian-U

-LRB-

-RRB-

respectively

For

regression

and

SVM

classifiers

positive

numbers

indicate

assignment

to

the

local

class

and

negative

numbers

indicate

assignment

to

the

global

class

the

absolute

magnitude

of

the

numbers

increases

as

the

classifier

's

confidence

in

its

decision

increases

-LRB-

We

linearly

transformed

the

regression

output

from

the

-LRB-

-RRB-

to

the

-LRB-

-RRB-

range

-RRB-

The

scale

of

the

numbers

is

consistent

across

queries

and

between

all

SVM

classifiers

but

not

directly

comparable

between

regression

classifiers

-LRB-

bound

between

and

-RRB-

and

SVM

classifiers

-LRB-

unbounded

-RRB-

the

unmodified

query

based

on

the

geographical

scope

of

the

pages

as

defined

in

-LSB-

-RSB-

Thus

pages

with

a

broad

geographical

scope

-LRB-

e.g.

covering

the

entire

United

States

-RRB-

would

prevail

over

other

pages

with

a

narrower

scope

A

less

expensive

alternative

is

to

classify

the

result

pages

as

local

or

global

following

a

procedure

similar

to

that

of

Section

for

queries

Specifically

we

implemented

this

alternative

by

training

Rules

a

rule-based

version

of

the

decision-tree

classifier

with

a

collection

of

web

pages

categorized

in

the

Yahoo!

directory

Pages

classified

under

individual

states

in

the

Regional

portion

of

the

directory

were

regarded

as

local

while

pages

under

general

categories

were

regarded

as

global

The

feature

representation

for

the

pages

was

analogous

to

that

for

the

queries

in

Section

but

restricted

to

features

that

are

meaningful

over

individual

pages

-LRB-

e.g.

total

number

of

locations

on

a

page

-RRB-

as

opposed

to

over

a

collection

of

pages

-LRB-

e.g.

minimum

number

of

locations

per

page

in

the

result

pages

for

a

query

-RRB-

At

query

time

we

reorder

the

results

so

as

to

privilege

global

pages

over

local

ones

This

is

based

on

the

locality

classification

of

the

pages

which

can

be

precomputed

off-line

since

it

is

query-independent

or

performed

on

the

fly

as

we

do

in

our

prototype

implementation

This

procedure

is

efficient

and

produced

promising

initial

results

for

a

handful

of

global

queries

-LRB-

e.g.

-LSB-

wildflowers

-RSB-

-RRB-

that

we

tried

Our

preliminary

approach

to

query

modification

is

therefore

as

follows

Given

a

query

specified

by

the

user

we

supply

first

the

unmodified

query

to

the

search

engine

and

collect

the

top

results

We

extract

location

names

from

these

results

and

calculate

the

features

of

Section

Using

one

of

the

best

performing

classifiers

of

Section

we

determine

if

the

query

is

global

or

local

If

it

is

local

and

As

noted

earlier

these

names

could

be

cached

along

with

each

web

page

at

the

time

of

indexing

to

increase

efficiency

contains

at

least

one

location

name

nothing

is

done

the

results

returned

from

the

unmodified

query

are

presented

to

the

user

If

the

query

is

local

and

contains

no

location

we

add

the

user

's

location

-LRB-

or

alternatively

request

further

information

from

the

user

as

discussed

-RRB-

reissue

the

query

and

present

the

results

Finally

if

the

query

is

global

we

calculate

the

scope

of

each

retrieved

web

page

using

part

of

the

location

features

computed

earlier

and

the

Rules

classifier

and

rerank

the

results

so

that

more

global

pages

are

higher

in

the

list

shown

to

the

user

We

have

built

a

prototype

implementation

of

this

algorithm

using

the

classifier

obtained

from

Ripper

-LRB-

because

of

the

relative

simplicity

of

its

rules

-RRB-

for

query

classification

and

Google

as

the

search

engine

We

have

described

an

attribute

of

queries

locality

that-to

the

best

of

our

knowledge

has

not

been

explored

before

either

in

theoretical

work

or

in

practical

search

engines

but

can

significantly

affect

the

appropriateness

of

the

results

returned

to

the

user

We

defined

a

categorization

scheme

for

queries

based

on

their

geographical

locality

and

showed

how

queries

can

be

represented

for

purposes

of

determining

locality

by

features

based

on

location

names

found

in

the

results

they

produce

Using

these

features

automatic

classifiers

for

determining

locality

can

be

built

We

explored

several

state-of-the-art

classification

approaches

and

evaluated

their

performance

on

a

large

set

of

actual

queries

The

empirical

results

indicated

that

for

many

queries

locality

can

be

determined

effectively

The

bulk

of

the

paper

discussed

methods

for

classifying

queries

according

to

locality

and

empirically

established

that

this

is

desirable

and

feasible

for

many

queries

We

also

presented

some

first

thoughts

on

possible

query

mulation

and

result

reranking

strategies

that

utilize

locality

information

to

actually

improve

the

results

the

user

sees

Although

our

strategies

for

query

modification

and

result

reranking

are

preliminary

they

illustrate

a

promising

family

of

approaches

that

we

plan

to

investigate

in

the

future

so

that

we

can

exploit

the

classification

of

queries

based

on

their

geographical

locality

in

order

to

improve

search

result

quality

This

material

is

based

upon

work

supported

in

part

by

the

National

Science

Foundation

under

Grants

No

and

We

are

grateful

to

Claudia

Perlich

and

Foster

Provost

for

providing

us

with

their

adaptation

of

the

classifier

that

we

used

in

our

experiments

Also

we

would

like

to

thank

Thorsten

Joachims

for

answering

our

questions

on

SVM-Light

and

David

Parkes

for

his

helpful

comments

and

insight

-LSB-

-RSB-

D.

M.

Bates

and

D.

G.

Watts

Nonlinear

Regression

Analysis

and

its

Applications

Wiley

New

York

-LSB-

-RSB-

B.

E.

Boser

I.

M.

Guyon

and

V.

Vapnik

A

training

algorithm

for

optimal

margin

classifiers

In

Proceedings

of

the

Fifth

Annual

Workshop

on

Computational

Learning

Theory

Pittsburgh

-LSB-

-RSB-

A.

Bradley

The

use

of

the

area

under

the

ROC

curve

in

the

evaluation

of

machine

learning

algorithms

Pattern

Recognition

-LRB-

-RRB-

-LSB-

-RSB-

S.

Brin

and

L.

Page

The

anatomy

of

a

large-scale

hypertextual

web

search

engine

In

Proceedings

of

the

Seventh

International

World

Wide

Web

Conference

-LRB-

-RRB-

Apr.

-LSB-

-RSB-

C.

Buckley

J.

Allan

G.

Salton

and

A.

Singhal

Automatic

query

expansion

using

SMART

TREC

In

Proceedings

of

the

Third

Text

REtrieval

Conference

-LRB-

-RRB-

pages

April

NIST

Special

Publication

-LSB-

-RSB-

O.

Buyukkokten

J.

Cho

H.

Garc

ia-Molina

L.

Gravano

and

N.

Shivakumar

Exploiting

geographical

location

information

of

web

pages

In

Proceedings

of

the

ACM

SIGMOD

Workshop

on

the

Web

and

Databases

-LRB-

WebDB

-RRB-

June

-LSB-

-RSB-

S.

Chakrabarti

B.

Dom

P.

Raghavan

S.

Rajagopalan

D.

Gibson

and

J.

Kleinberg

Automatic

resource

compilation

by

analyzing

hyperlink

structure

and

associated

text

In

Proceedings

of

the

Seventh

International

World

Wide

Web

Conference

-LRB-

-RRB-

Apr.

-LSB-

-RSB-

W.

W.

Cohen

Learning

trees

and

rules

with

set-valued

functions

In

Proceedings

of

the

Thirteenth

International

Joint

Conference

on

Artificial

Intelligence

-LSB-

-RSB-

J.

Ding

L.

Gravano

and

N.

Shivakumar

Computing

geographical

scopes

of

web

resources

In

Proceedings

of

the

Twenty-sixth

International

Conference

on

Very

Large

Databases

-LRB-

VLDB

-RRB-

-LSB-

-RSB-

G.

W.

Flake

E.

J.

Glover

S.

Lawrence

and

C.

L.

Giles

Extracting

query

modifications

from

nonlinear

SVMs

In

Proceedings

of

the

Eleventh

International

World-Wide

Web

Conference

Dec.

-LSB-

-RSB-

M.

A.

Hearst

Trends

and

controversies

Support

vector

machines

IEEE

Intelligent

Systems

-LRB-

-RRB-

July

-LSB-

-RSB-

T.

Joachims

Estimating

the

generalization

of

performance

of

an

SVM

efficiently

In

Proceedings

of

the

Fourteenth

International

Conference

on

Machine

Learning

-LSB-

-RSB-

J.

Kleinberg

Authoritative

sources

in

a

hyperlinked

environment

In

Proceedings

of

the

Ninth

Annual

ACM-SIAM

Symposium

on

Discrete

Algorithms

pages

Jan.

-LSB-

-RSB-

K.

S.

McCurley

Geospatial

mapping

and

navigation

of

the

web

In

Proceedings

of

the

Tenth

International

World

Wide

Web

Conference

-LRB-

-RRB-

May

-LSB-

-RSB-

M.

Pazzani

C.

Merz

P.

Murphy

K.

Ali

T.

Hume

and

C.

Brunk

Reducing

misclassification

costs

In

Proceedings

of

the

Eleventh

International

Conference

on

Machine

Learning

Sept.

-LSB-

-RSB-

R.

Purves

A.

Ruas

M.

Sanderson

M.

Sester

M.

van

Kreveld

and

R.

Weibel

Spatial

information

retrieval

and

geographical

ontologies

An

overview

of

the

SPIRIT

project

In

Proceedings

of

the

ACM

International

Conference

on

Research

and

Development

in

Information

Retrieval

-LRB-

SIGIR

-RRB-

-LSB-

-RSB-

R.

J.

Quinlan

Programs

for

Machine

Learning

Morgan

Kaufman

-LSB-

-RSB-

G.

Salton

Automatic

Text

Processing

The

transformation

analysis

and

retrieval

of

information

by

computer

Addison-Wesley

-LSB-

-RSB-

T.

J.

Santner

and

D.

E.

Duffy

The

Statistical

Analysis

of

Discrete

Data

Springer-Verlag

New

York

-LSB-

-RSB-

C.

J.

van

Rijsbergen

Information

Retrieval

Butterworths

London

edition

-LSB-

-RSB-

G.

M.

Weiss

and

F.

Provost

The

effect

of

class

distribution

on

classifier

learning

An

empirical

study

Technical

Report

Computer

Science

Department

Rutgers

University

Aug.

O B I O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

information retrieval
query classification
search engine
search engines
query modification
web search
local page
web queries
global page
geographical locality
categorization scheme

query categorization / query classification