Query

Type

Classification

for

Web

Document

Retrieval

The

heterogeneous

Web

exacerbates

IR

problems

and

short

user

queries

make

them

worse

The

contents

of

web

documents

are

not

enough

to

find

good

answer

documents

Link

information

and

URL

information

compensates

for

the

insufficiencies

of

content

information

However

static

combination

of

multiple

evidences

may

lower

the

retrieval

performance

We

need

different

strategies

to

find

target

documents

according

to

a

query

type

We

can

classify

user

queries

as

three

categories

the

topic

relevance

task

the

homepage

finding

task

and

the

service

finding

task

In

this

paper

a

user

query

classification

scheme

is

proposed

This

scheme

uses

the

difference

of

distribution

mutual

information

the

usage

rate

as

anchor

texts

and

the

POS

information

for

the

classification

After

we

classified

a

user

query

we

apply

different

algorithms

and

information

for

the

better

results

For

the

topic

relevance

task

we

emphasize

the

content

information

on

the

other

hand

for

the

homepage

finding

task

we

emphasize

the

Link

information

and

the

URL

information

We

could

get

the

best

performance

when

our

proposed

classification

method

with

the

OKAPI

scoring

algorithm

was

used

H.

m

-LSB-

Information

Systems

Applications

-RSB-

Miscellaneous

Experimentation

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

SIGIR

July

August

Toronto

Canada

Copyright

ACM

The

Web

is

rich

with

various

sources

of

information

It

contains

the

contents

of

documents

web

directories

multi-media

data

user

profiles

and

so

on

The

massive

and

heterogeneous

web

document

collections

as

well

as

the

unpredictable

querying

behaviors

of

typical

web

searchers

exacerbate

Information

Retrieval

-LRB-

IR

-RRB-

problems

Retrieval

approaches

based

on

the

single

source

of

evidence

suffer

from

weakness

that

can

hurt

the

retrieval

performance

in

certain

situations

-LSB-

-RSB-

For

example

content-based

IR

approaches

have

a

difficulty

in

dealing

with

the

diversity

in

vocabulary

and

the

quality

of

web

documents

while

link-based

approaches

can

suffer

from

an

incomplete

or

noisy

link

structure

Combining

multiple

evidences

compensates

for

the

weakness

of

a

single

evidence

-LSB-

-RSB-

Fusion

IR

studies

have

repeatedly

shown

that

combining

multiple

sources

of

evidence

can

improve

retrieval

performance

-LSB-

-RSB-

-LSB-

-RSB-

However

previous

studies

did

not

consider

a

user

query

in

combining

evidences

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Not

only

documents

in

the

Web

but

also

users

queries

are

diverse

For

example

for

user

query

Mutual

Information

if

we

count

on

link

information

too

highly

well-known

site

that

has

mutual

funds

and

information

as

index

terms

gets

the

higher

rank

For

user

query

Britney

's

Fan

Club

if

we

use

content

information

too

highly

yahoo

or

lycos

's

web

directory

pages

get

the

higher

rank

instead

of

the

Britney

's

fan

club

site

Like

these

examples

combining

content

information

and

link

information

is

not

always

good

We

have

to

use

different

strategies

to

meet

the

need

of

a

user

User

queries

can

be

classified

as

three

categories

according

to

their

intent

-LSB-

-RSB-

topic

relevance

task

-LRB-

informational

-RRB-

homepage

finding

task

-LRB-

navigational

-RRB-

service

finding

task

-LRB-

transactional

-RRB-

The

topic

relevance

task

is

a

traditional

ad

hoc

retrieval

task

where

web

documents

are

ranked

by

decreasing

likelihood

of

meeting

the

information

need

provided

in

a

user

query

-LSB-

-RSB-

For

example

What

is

a

prime

factor

or

prime

factor

is

a

query

of

the

topic

relevance

task

The

goal

of

this

query

is

finding

the

meaning

of

prime

factor

The

homepage

finding

task

is

a

known-item

task

where

the

goal

is

to

find

the

homepage

-LRB-

or

site

entry

page

-RRB-

of

the

site

described

in

a

user

query

Users

are

interested

in

finding

a

certain

site

For

example

Where

is

the

site

of

John

Hopkins

Medical

Institutions

or

John

Hopkins

Medical

Institutions

is

a

query

of

the

homepage

finding

task

The

goal

of

this

query

is

finding

the

entry

page

of

John

Hopkins

Medical

Institutions

The

service

finding

task

is

a

task

where

the

goal

is

to

find

web

documents

that

provide

the

service

described

in

a

user

query

For

example

Where

can

I

buy

concert

tickets

or

buy

concert

tickets

is

a

query

of

the

service

finding

task

The

goal

of

this

query

is

finding

documents

where

they

can

buy

concert

tickets

Users

may

want

different

documents

with

the

same

query

We

can

not

always

tell

the

class

of

a

query

clearly

But

we

can

tell

most

people

want

a

certain

kind

of

documents

with

this

query

In

this

paper

we

calculate

the

probability

that

the

class

of

a

user

query

is

the

topic

relevance

task

or

the

homepage

finding

task

Based

on

this

probability

we

combine

multiple

evidences

dynamically

In

this

paper

we

consider

the

topic

relevance

task

and

the

homepage

finding

task

only

Because

the

proposed

method

is

based

on

the

difference

of

databases

we

can

apply

the

same

method

to

classify

the

service

finding

task

In

this

paper

we

present

a

user

query

classification

method

and

a

combining

method

for

each

query

type

In

section

we

describe

various

types

of

information

-LRB-

Content

Link

and

URL

information

-RRB-

Section

lists

the

differences

of

search

tasks

and

the

properties

of

Content

Link

and

URL

information

In

section

we

present

the

model

of

a

query

classification

In

section

we

experiment

with

our

proposed

model

Conclusion

is

described

in

section

In

this

section

we

explain

various

sources

of

information

for

the

web

document

retrieval

There

are

three

types

of

information

Content

information

Link

information

and

URL

information

Content

Information

There

are

multiple

types

of

representations

for

a

document

These

representations

typically

contain

titles

anchor

texts

and

main

body

texts

-LSB-

-RSB-

A

title

provides

the

main

idea

and

the

brief

explanation

of

a

web

document

An

anchor

text

provides

the

description

of

linked

web

documents

and

files

An

anchor

text

often

provides

more

accurate

description

of

a

web

document

than

the

document

itself

We

usually

use

tf

and

df

to

calculate

the

relevance

of

a

given

web

documents

-LSB-

-RSB-

tf

is

the

raw

frequency

of

a

given

term

inside

a

document

It

provides

one

measure

of

how

well

that

term

describes

the

document

contents

df

is

the

number

of

documents

in

which

the

index

term

appears

The

motivation

for

using

an

inverse

document

frequency

is

that

terms

that

appear

in

many

documents

are

not

very

useful

for

distinguishing

a

relevant

document

from

a

non-relevant

one

There

are

various

scoring

algorithms

that

use

tf

and

df

These

scoring

algorithms

include

the

normalization

and

the

combination

of

each

factor

tf

and

df

Link

Information

A

hyperlink

in

a

web

document

is

a

kind

of

citation

The

essential

idea

is

that

if

page

u

has

a

link

to

page

v

then

the

author

of

u

is

implicitly

assigning

some

importance

to

page

v

Since

we

can

represent

the

Web

as

a

graph

we

can

use

graph

theories

to

help

us

make

a

search

engine

that

returns

the

most

important

pages

first

The

PageRank

or

P

R

-LRB-

A

-RRB-

of

a

page

A

is

given

as

follows

-LSB-

-RSB-

P

R

-LRB-

A

-RRB-

-LRB-

d

-RRB-

-LRB-

-RRB-

d

-LRB-

P

R

-LRB-

T

-RRB-

C

-LRB-

T

-RRB-

P

R

-LRB-

T

n

-RRB-

C

-LRB-

T

n

-RRB-

-RRB-

We

assume

page

A

has

pages

T

T

n

that

point

to

it

The

parameter

d

is

a

damping

factor

that

can

be

set

between

and

Also

C

-LRB-

A

-RRB-

is

defined

as

the

number

of

links

going

out

of

a

page

A.

P

R

-LRB-

A

-RRB-

can

be

calculated

using

a

simple

iterative

algorithm

and

corresponds

to

the

principal

eigenvector

of

the

normalized

link

matrix

of

the

Web

-LSB-

-RSB-

URL

Information

The

URL

string

of

a

site

entry

page

often

contains

the

name

or

acronym

of

the

corresponding

organization

Therefore

an

obvious

way

of

exploiting

URL

information

is

trying

to

match

query

terms

and

URL

terms

Additionally

URLs

of

site

entry

pages

tend

to

be

higher

in

a

server

's

directory

tree

than

other

web

documents

i.e.

the

number

of

slashes

-LRB-

-RRB-

in

an

entry

page

URL

tends

to

be

relatively

small

Kraaij

et

al.

suggested

types

of

URLs

-LSB-

-RSB-

root

a

domain

name

-LRB-

e.g.

-RRB-

subroot

a

domain

name

followed

by

a

single

directory

-LRB-

e.g.

-RRB-

path

a

domain

name

followed

by

an

arbitrarily

deep

path

-LRB-

e.g.

-RRB-

file

anything

ending

in

a

filename

other

than

in-dex

html

-LRB-

e.g.

-RRB-

Kraaij

et

al.

estimated

a

prior

probability

-LRB-

URLprior

-RRB-

of

being

an

entry

page

on

the

basis

of

the

URL

type

for

all

URL

types

t

-LRB-

root

subroot

path

and

file

-RRB-

Combination

of

Information

We

can

combine

results

of

each

search

engine

or

scores

of

each

measure

to

get

better

results

Croft

proposed

the

IN-QUERY

retrieval

system

based

on

the

inference

network

to

combine

multiple

evidences

-LSB-

-RSB-

The

inference

network

model

is

a

general

model

for

combining

information

It

is

data-level

fusion

The

model

is

based

on

probabilistic

updating

of

the

values

of

nodes

in

the

network

and

many

retrieval

techniques

and

information

can

be

implemented

by

config-uring

the

network

properly

Several

researchers

have

experimented

with

linearly

combining

the

normalized

relevance

scores

-LRB-

s

i

-RRB-

given

to

each

document

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

score

-LRB-

d

-RRB-

i

i

s

i

-LRB-

d

-RRB-

-LRB-

-RRB-

It

requires

training

for

the

weight

i

given

to

each

input

system

For

example

we

can

get

a

better

result

by

combining

content

information

and

URL

type

information

with

the

following

weight

-LSB-

-RSB-

score

-LRB-

d

-RRB-

content

URLprior

-LRB-

-RRB-

In

this

section

we

show

properties

of

Content

information

Link

information

and

URL

information

in

each

search

task

Besides

we

will

propose

the

method

for

linearly

combining

information

for

each

task

We

use

TREC

data

collection

to

show

the

differences

of

each

search

task

We

made

a

simple

search

engine

that

use

the

variation

of

the

OKAPI

scoring

function

-LSB-

-RSB-

Given

a

query

Q

the

scoring

formula

is

score

t

-LRB-

QD

d

-RRB-

T

F

d

t

IDF

t

-LRB-

-RRB-

T

F

d

t

tf

d

t

tf

d

t

doclen

d

avg

doclen

-LRB-

-RRB-

IDF

t

log

-LRB-

N

df

t

-RRB-

log

-LRB-

N

-RRB-

-LRB-

-RRB-

N

is

the

number

of

documents

in

the

collection

tf

d

t

is

the

number

of

occurrences

of

an

index

term

t

in

a

document

d

and

df

t

is

the

number

of

documents

in

which

t

occurs

We

use

the

data

for

the

web

track

the

collection

-LSB-

-RSB-

distributed

by

CSIRO

-LSB-

-RSB-

We

use

topic

relevance

task

queries

-LRB-

topics

-RRB-

for

the

topic

relevance

task

and

queries

for

the

homepage

finding

task

-LSB-

-RSB-

For

the

homepage

finding

task

NIST

found

a

homepage

within

and

then

composed

a

query

designed

to

locate

it

We

used

the

anchor

text

representation

-LRB-

Anchor

-RRB-

and

the

common

content

text

representation

-LRB-

Common

-RRB-

for

indexing

Every

document

in

the

anchor

text

representation

has

anchor

texts

and

the

title

as

content

and

excludes

a

body

text

Consequently

the

anchor

text

representation

has

brief

or

main

explanations

of

a

document

We

used

two

other

evidences

for

a

scoring

function

besides

the

OKAPI

score

One

is

URLprior

for

URL

information

and

the

other

is

PageRank

for

Link

information

We

linearly

interpolated

Content

information

-LRB-

OKAPI

score

-RRB-

URLprior

and

PageRank

We

call

this

interpolation

as

CMB

rel

-LRB-

d

-RRB-

Content

Information

-LRB-

-RRB-

URL

Information

Link

Information

We

used

and

and

sum

operators

for

matching

query

terms

-LSB-

-RSB-

and

operator

means

that

the

result

document

has

all

query

terms

in

it

sum

operator

means

that

a

result

document

has

at

least

one

query

term

in

it

Table

shows

the

average

precision

of

the

topic

relevance

task

and

the

MRR

of

the

homepage

finding

task

-LSB-

-RSB-

The

first

column

in

the

table

means

the

method

that

we

used

for

indexing

and

scoring

For

example

Anchor

and

CMB

means

that

we

used

the

anchor

text

representation

for

indexing

and

operator

for

query

matching

and

the

OKAPI

score

PageRank

and

URLprior

for

scoring

The

average

precision

is

defined

as

the

average

of

the

precision

obtained

at

the

rank

of

each

relevant

document

P

avg

R

d

R

R

r

-LRB-

d

-RRB-

r

-LRB-

d

-RRB-

-LRB-

-RRB-

R

is

the

set

of

all

relevant

documents

and

R

r

-LRB-

d

-RRB-

is

the

set

of

relevant

documents

with

rank

r

-LRB-

d

-RRB-

or

better

MRR

-LRB-

Mean

Reciprocal

Rank

-RRB-

is

the

main

evaluation

measure

for

the

homepage

finding

task

MRR

is

based

on

the

rank

of

the

first

correct

document

-LRB-

answer

i

rank

-RRB-

according

to

the

Table

Topic

Relevance

Task

vs.

Homepage

Finding

Task

Topic

Homepage

model

P

avg

MRR

Anchor

and

Anchor

and

CMB

Anchor

sum

Anchor

sum

CMB

Common

and

Common

and

CMB

Common

sum

Common

sum

CMB

MAX

AVG

following

formula

M

RR

#queries

#queries

i

answer

i

rank

-LRB-

-RRB-

M

AX

represents

the

best

score

of

a

search

engine

that

submitted

in

AV

G

represents

the

average

score

of

all

search

engines

that

submitted

in

We

got

the

better

result

with

the

common

content

text

representation

than

the

anchor

text

representation

in

the

topic

relevance

task

A

title

and

anchor

texts

do

not

have

enough

information

for

the

topic

relevance

task

On

the

other

hand

we

could

get

the

similar

performance

with

the

anchor

text

representation

in

the

homepage

finding

task

URL

information

and

Link

information

are

good

for

the

homepage

finding

task

but

bad

for

the

topic

relevance

task

In

the

topic

relevance

task

we

lost

our

performance

by

combining

URL

and

Link

information

The

query

of

the

topic

relevance

task

usually

consists

of

main

keywords

that

are

relevant

to

some

concept

or

the

explanation

of

what

they

want

to

know

However

we

can

not

assume

that

other

people

use

same

expressions

and

keywords

to

explain

what

a

user

wants

to

know

Therefore

we

could

not

get

a

good

result

with

and

operator

in

the

topic

relevance

task

But

on

the

other

hand

the

query

of

the

homepage

finding

task

consists

of

entity

names

or

proper

nouns

Therefore

we

could

have

good

results

with

and

operator

when

we

can

have

a

result

document

However

the

MRR

of

Anchor

and

CMB

is

lower

than

that

of

Common

sum

CMB

in

the

homepage

finding

task

Anchor

and

CMB

method

did

not

retrieve

a

document

for

queries

To

compensate

for

this

sparseness

problem

we

combined

the

results

of

Anchor

and

CMB

and

Common

sum

CMB

This

combined

result

showed

in

the

homepage

finding

task

When

we

combined

the

results

of

Anchor

and

and

Common

sum

it

showed

in

the

topic

relevance

task

This

implies

that

the

result

documents

with

and

operator

are

good

and

useful

in

the

homepage

finding

task

We

can

conclude

that

we

need

different

retrieval

strategies

according

to

the

category

of

a

query

We

have

to

use

the

field

information

-LRB-

title

body

and

anchor

text

-RRB-

of

each

term

and

combine

evidences

dynamically

to

get

good

results

In

the

topic

relevance

task

the

body

text

of

a

document

is

good

for

indexing

sum

operator

is

good

for

query

term

matching

and

combining

URL

and

Link

information

are

useless

On

the

other

hand

in

the

homepage

finding

task

anchor

texts

and

titles

are

useful

for

indexing

and

operator

is

also

good

for

query

term

matching

and

URL

and

Link

information

is

useful

By

combining

results

from

main

body

text

and

anchor

texts

and

titles

we

can

have

the

better

performance

In

this

section

we

present

the

method

for

making

a

language

model

for

a

user

query

classification

Preparation

for

Language

Model

We

may

use

the

question

type

of

a

query

to

classify

the

category

of

a

user

query

For

example

What

is

a

two

electrode

vacuum

tube

is

a

query

of

the

topic

relevance

task

Where

is

the

site

of

SONY

is

a

query

of

the

homepage

finding

task

We

can

assume

the

category

of

a

query

with

an

interrogative

pronoun

and

cue

expressions

-LRB-

e.g.

the

site

of

-RRB-

However

people

do

not

provide

natural

language

queries

to

a

search

engine

They

usually

use

keywords

for

their

queries

It

is

not

easy

to

anticipate

natural

language

queries

In

this

paper

we

assume

that

users

provide

only

main

keywords

for

their

queries

We

define

a

query

Q

as

the

set

of

words

Q

-LCB-

w

w

w

n

-RCB-

-LRB-

-RRB-

To

see

the

characteristics

of

each

query

class

we

use

two

query

sets

For

the

topic

relevance

task

topic

relevance

task

queries

-LRB-

topics

-RRB-

are

used

For

the

homepage

finding

task

queries

for

randomly

selected

homepages

are

used

We

call

them

QU

ERY

T

T

RAIN

and

QU

ERY

H

T

RAIN

We

divided

into

two

sets

DB

T

OP

IC

and

DB

HOM

E

If

the

URL

type

of

a

document

is

root

type

we

put

this

document

to

DB

HOM

E

Others

are

added

to

DB

T

OP

IC

According

to

the

report

of

-LSB-

-RSB-

our

division

method

can

get

site

entry

pages

with

precision

Additionally

we

put

virtual

documents

into

DB

HOM

E

with

anchor

texts

If

a

linked

document

is

in

DB

T

OP

IC

then

we

make

a

virtual

document

that

consists

of

anchor

texts

and

put

it

into

DB

HOM

E

If

a

linked

document

is

in

DB

HOM

E

then

we

add

anchor

texts

to

the

original

document

Usually

a

site

entry

page

does

not

have

many

words

It

is

not

an

explanatory

document

for

some

topic

or

concept

but

the

brief

explanation

of

a

site

We

can

assume

that

site

entry

pages

have

the

different

usage

of

words

If

we

find

distinctive

features

for

site

entry

pages

then

we

can

discriminate

the

category

of

a

given

query

#DB

T

OP

IC

and

#DB

HOM

E

mean

the

number

of

documents

in

the

DB

T

OP

IC

and

DB

HOM

E

respectively

However

most

documents

in

the

DB

HOM

E

have

a

short

length

we

normalized

the

number

of

documents

with

the

following

equation

#DB

T

OP

IC

of

documents

in

DB

T

OP

IC

-LRB-

-RRB-

#DB

HOM

E

of

documents

in

DB

HOM

E

-LRB-

-RRB-

avg

doclength

HOM

E

avg

doclength

T

OP

IC

available

at

Distribution

of

Query

Terms

Earthquake

occurs

more

frequently

in

DB

T

OP

IC

But

Hunt

Memorial

Library

shows

the

high

relative

frequency

in

DB

HOM

E

General

terms

tend

to

have

same

distribution

regardless

of

the

database

If

the

difference

of

distribution

is

larger

than

expected

this

tells

whether

a

given

query

is

in

the

topic

relevance

task

class

or

the

homepage

finding

task

class

We

can

calculate

the

occurrence

ratio

of

a

query

with

the

following

equation

-LSB-

-RSB-

Dist

-LRB-

w

w

n

-RRB-

n

C

-LRB-

w

w

n

-RRB-

n

i

C

-LRB-

w

i

-RRB-

-LRB-

-RRB-

C

-LRB-

w

-RRB-

is

the

number

of

documents

that

have

w

as

an

index

term

df

of

w

is

used

for

C

-LRB-

w

-RRB-

C

-LRB-

w

w

n

-RRB-

is

the

number

of

documents

that

have

all

w

w

n

as

index

terms

To

see

the

distribution

difference

of

a

query

we

use

the

following

ratio

equation

dif

f

Dist

-LRB-

Q

-RRB-

Dist

HOM

E

-LRB-

Q

-RRB-

Dist

T

OP

IC

-LRB-

Q

-RRB-

-LRB-

-RRB-

If

a

query

has

only

one

term

we

use

the

chi-square

-LSB-

-RSB-

We

make

a

table

for

the

given

word

w

word

w

word

w

DB

T

OP

IC

a

b

DB

HOM

E

c

d

a

b

#DB

T

OP

IC

and

c

d

#DB

HOM

E

a

is

the

frequency

of

the

word

w

in

the

DB

T

OP

IC

and

c

is

the

frequency

of

the

word

w

in

the

DB

HOM

E

The

chi-square

value

shows

the

dependence

of

the

word

w

and

DB

If

the

chi-square

value

of

the

word

w

is

high

then

w

is

a

special

term

of

DB

T

OP

IC

or

DB

HOM

E

We

classify

these

words

that

have

a

high

chi-square

value

according

to

the

df

If

w

has

a

high

df

then

the

word

w

is

the

topic

relevance

task

query

Otherwise

w

is

the

homepage

finding

task

query

For

example

f

ast

shows

the

high

chi-square

value

since

it

is

used

a

lot

to

modify

proper

names

However

one

word

f

ast

is

not

the

proper

name

We

classify

a

word

that

has

a

high

chi-square

and

a

high

df

into

the

topic

relevance

task

If

the

chi-square

value

of

the

word

w

is

low

then

w

is

a

general

term

Fig.

shows

the

results

of

dif

f

Dist

of

queries

that

have

at

least

two

query

terms

The

mean

values

of

QU

ERY

T

T

RAIN

's

dif

f

Dist

and

QU

ERY

H

T

RAIN

's

dif

f

Dist

are

and

respectively

As

the

value

of

dif

f

Dist

of

a

given

query

is

higher

we

can

have

confidence

that

the

query

has

special

terms

On

the

other

hand

if

the

score

of

dif

f

Dist

is

near

the

mean

value

of

QU

ERY

T

T

RAIN

it

means

the

query

has

general

terms

not

a

special

expression

We

calculate

the

possibility

that

a

given

query

is

in

each

class

with

the

mean

value

and

the

standard

deviation

However

there

are

queries

that

show

high

dif

f

DIST

in

QU

ERY

T

T

RAIN

For

example

Jenniffer

Aniston

and

Chevrolet

Trucks

showed

and

respectively

Usually

proper

names

showed

high

dif

f

DIST

values

If

a

proper

name

is

frequently

used

in

the

DB

HOM

E

then

we

can

think

of

it

as

the

name

of

the

site

Mutual

Information

There

are

two

or

more

words

that

co-occur

frequently

These

words

may

have

syntactic

or

semantic

relations

to

if

length

-LRB-

Q

-RRB-

then

calculate

the

of

Q

if

then

if

df

of

a

query

the

topic

relevance

task

else

the

homepage

finding

task

else

the

topic

relevance

task

else

calculate

distributions

of

a

query

in

each

database

calculate

dif

f

Dist

-LRB-

Q

-RRB-

if

dif

f

Dist

-LRB-

Q

-RRB-

the

homepage

finding

task

else

unknown

Figure

The

Algorithm

of

Distribution

Difference

Method

Ratio

of

Distribution

Difference

P

e

r

c

e

n

t

a

g

e

o

f

O

b

se

r

v

at

i

o

n

s

QUERY-TOPIC-TRAIN

QUERY-HOMEPAGE-TRAIN

Figure

Distribution

of

Queries

each

other

We

say

these

words

have

some

dependency

For

example

tornadoes

formed

shows

similar

dependency

regardless

of

the

database

But

Fan

Club

has

a

high

dependency

in

DB

HOM

E

set

This

means

that

tornadoes

formed

is

a

general

usage

of

words

but

Fan

Club

is

a

special

usage

in

DB

HOM

E

Therefore

the

dependency

of

Fan

Club

can

be

the

key

clue

of

guessing

the

category

of

a

user

query

If

the

difference

of

dependency

of

each

term

is

larger

than

expected

this

tells

whether

a

given

query

is

the

topic

relevance

task

or

the

homepage

finding

task

For

two

variables

A

and

B

we

can

calculate

the

dependency

with

mutual

information

I

-LRB-

A

B

-RRB-

-LSB-

-RSB-

We

use

the

pointwise

mutual

information

I

-LRB-

x

y

-RRB-

to

calculate

the

dependency

of

terms

in

a

query

-LSB-

-RSB-

I

-LRB-

A

B

-RRB-

H

-LRB-

A

-RRB-

H

-LRB-

B

-RRB-

H

-LRB-

A

B

-RRB-

a

b

p

-LRB-

a

b

-RRB-

log

p

-LRB-

a

b

-RRB-

p

-LRB-

a

-RRB-

p

-LRB-

b

-RRB-

-LRB-

-RRB-

I

-LRB-

x

y

-RRB-

log

p

-LRB-

x

y

-RRB-

p

-LRB-

x

-RRB-

p

-LRB-

y

-RRB-

-LRB-

-RRB-

We

extend

pointwise

mutual

information

for

three

variables

We

use

the

set

theory

to

calculate

the

value

of

an

Ratio

of

MI

Difference

P

e

r

c

e

n

t

a

g

e

o

f

O

b

s

e

r

v

a

t

i

o

n

QUERY-TOPIC-TRAIN

QUERY-HOMEPAGE-TRAIN

Figure

Mutual

Information

of

Queries

intersection

part

like

two

variables

problem

I

-LRB-

A

B

C

-RRB-

H

-LRB-

A

B

C

-RRB-

H

-LRB-

A

-RRB-

H

-LRB-

B

-RRB-

H

-LRB-

C

-RRB-

I

-LRB-

A

B

-RRB-

I

-LRB-

B

C

-RRB-

I

-LRB-

C

A

-RRB-

a

b

c

p

-LRB-

a

b

c

-RRB-

log

p

-LRB-

a

b

-RRB-

p

-LRB-

b

c

-RRB-

p

-LRB-

c

a

-RRB-

p

-LRB-

a

b

c

-RRB-

p

-LRB-

a

-RRB-

p

-LRB-

b

-RRB-

p

-LRB-

c

-RRB-

-LRB-

-RRB-

I

-LRB-

x

y

z

-RRB-

log

p

-LRB-

x

y

-RRB-

p

-LRB-

y

z

-RRB-

p

-LRB-

z

x

-RRB-

p

-LRB-

x

y

z

-RRB-

p

-LRB-

x

-RRB-

p

-LRB-

y

-RRB-

p

-LRB-

z

-RRB-

-LRB-

-RRB-

In

principle

p

-LRB-

x

y

-RRB-

means

the

probability

that

x

and

y

are

co-occurred

in

a

specific

distance

-LSB-

-RSB-

Usually

x

and

y

are

consecutive

words

Since

the

number

of

words

and

documents

are

so

huge

in

IR

domain

it

is

not

easy

to

keep

statistics

Our

measure

assume

that

x

and

y

are

co-occurred

in

a

document

We

use

df

of

a

given

term

to

calculate

the

number

of

documents

that

contain

a

term

Like

the

distribution

difference

measure

we

use

the

ratio

difference

equation

to

see

the

difference

of

MI

If

pointwise

mutual

information

is

below

zero

then

we

use

zero

dif

f

M

I

-LRB-

Q

-RRB-

M

I

HOM

E

-LRB-

Q

-RRB-

M

I

T

OP

IC

-LRB-

Q

-RRB-

-LRB-

-RRB-

Fig.

shows

the

results

of

dif

f

M

I

The

mean

values

of

QU

ERY

T

T

RAIN

's

dif

f

M

I

and

QU

ERY

H

T

RAIN

's

dif

f

M

I

are

and

respectively

For

example

the

topic

relevance

task

query

mexican

food

culture

showed

but

the

homepage

finding

task

query

Newave

IFMO

showed

QU

ERY

H

T

RAIN

gets

a

slightly

high

standard

deviation

It

means

that

the

query

of

QU

ERY

H

T

RAIN

has

different

MI

in

DB

HOM

E

As

the

value

of

dif

f

M

I

is

higher

we

can

have

confidence

that

the

query

has

a

special

dependency

We

calculate

the

possibility

that

a

given

query

is

in

each

class

with

the

mean

value

and

the

standard

deviation

Usage

Rate

as

an

Anchor

Text

If

query

terms

appear

in

titles

and

anchor

texts

frequently

this

tells

the

category

of

a

given

query

is

the

homepage

finding

task

Titles

and

anchor

texts

are

usually

entity

names

or

proper

nouns

the

usage

rate

shows

the

probability

that

given

terms

are

special

terms

use

Anchor

-LRB-

w

w

n

-RRB-

-LRB-

-RRB-

C

SIT

E

AN

CHOR

-LRB-

w

w

n

-RRB-

C

SIT

E

-LRB-

w

w

n

-RRB-

C

SIT

E

-LRB-

w

w

w

n

-RRB-

C

SIT

E

-LRB-

w

-RRB-

means

the

number

of

site

entry

documents

that

have

w

as

an

index

term

C

SIT

E

AN

CHOR

-LRB-

w

-RRB-

means

the

number

of

site

entry

documents

and

anchor

texts

that

have

w

as

an

index

term

POS

information

Since

the

homepage

finding

task

queries

are

proper

names

they

do

not

usually

contain

a

verb

However

some

topic

relevance

task

queries

include

a

verb

to

explain

what

he

or

she

wants

to

know

For

example

How

are

tornadoes

formed

or

briefly

tornadoes

formed

contain

a

verb

formed

If

a

query

has

a

verb

except

the

be

verb

then

we

classified

it

into

the

topic

relevance

task

Combination

of

Measures

The

difference

of

distribution

method

can

apply

more

queries

than

the

difference

of

MI

The

usage

rate

as

anchor

texts

and

the

POS

information

show

small

coverage

However

four

measures

cover

different

queries

Therefore

we

can

have

more

confidence

and

more

coverage

by

combining

these

measures

We

use

a

different

combination

equation

as

the

number

of

query

terms

If

the

query

has

and

terms

in

it

we

use

pointwise

mutual

information

also

S

-LRB-

Q

-RRB-

diff

Dist

-LRB-

Q

-RRB-

diff

M

I

-LRB-

Q

-RRB-

-LRB-

-RRB-

use

Anchor

-LRB-

Q

-RRB-

P

OS

inf

o

-LRB-

Q

-RRB-

We

choose

and

with

train

data

-LRB-

QU

ERY

T

T

RAIN

and

QU

ERY

H

T

RAIN

-RRB-

If

S

-LRB-

Q

-RRB-

score

is

not

high

or

low

enough

then

we

make

no

decision

In

this

section

we

show

the

efficiency

of

a

user

query

classification

Query

Classification

We

used

four

query

sets

for

experimenting

our

query

classification

method

QU

ERY

T

T

RAIN

and

QU

ERY

H

T

RAIN

are

used

for

training

-LRB-

TRAIN

-RRB-

topic

relevance

task

queries

-LRB-

Topic

-RRB-

and

homepage

finding

task

queries

-LRB-

-RRB-

are

used

for

testing

-LRB-

TEST

-RRB-

We

call

two

test

sets

as

QU

ERY

T

T

EST

and

QU

ERY

H

T

EST

We

used

for

making

a

classification

model

We

classified

queries

with

our

proposed

method

If

the

score

S

-LRB-

Q

-RRB-

is

high

enough

to

tell

that

a

given

query

is

in

the

topic

relevance

task

or

the

homepage

finding

task

query

then

we

assigned

the

query

type

to

it

For

other

cases

we

did

not

classify

a

query

category

Table

shows

the

classification

result

of

our

proposed

language

model

Table

Query

Classification

Result

QUERY

TRAIN

TEST

Measure

Precision

Recall

Precision

Recall

Dist

MI

Anchor

POS

All

By

combining

each

measure

we

could

apply

our

method

to

more

queries

and

increase

precision

and

recall

Our

pro-Table

Average

Precision

of

the

Topic

Relevance

Task

model

OKAPI

TF-IDF

KL

DIR

MIXFB

KL

D

Lemur

MLemur

Table

MRR

of

the

Homepage

Finding

Task

model

OKAPI

TF-IDF

KL

DIR

MIXFB

KL

D

Lemur

MLemur

posed

method

shows

the

better

result

in

the

test

set

This

is

due

to

the

characteristics

of

the

query

set

There

are

queries

that

have

a

verb

in

QU

ERY

T

T

RAIN

and

queries

in

QU

ERY

T

T

EST

We

can

assume

that

the

POS

information

is

good

information

The

main

reason

of

misclassification

is

wrong

division

of

Since

our

method

usually

gives

the

high

score

to

the

proper

name

we

need

correct

information

to

distinguish

a

proper

name

from

a

site

name

We

tried

to

make

DB

HOM

E

automatically

However

some

root

pages

are

not

site

entry

pages

We

need

a

more

sophisticated

division

method

There

is

a

case

that

a

verb

is

in

the

homepage

finding

task

query

Protect

Preserve

is

the

homepage

finding

task

query

but

protect

and

preserve

are

verbs

However

Protect

and

Preserve

start

with

a

capital

letter

We

can

correct

wrong

POS

tags

There

are

queries

in

QU

ERY

T

T

EST

that

look

like

queries

of

QU

ERY

H

T

EST

For

example

Dodge

Recalls

is

used

to

find

documents

that

report

on

the

recall

of

any

dodge

automobile

products

But

user

may

want

to

find

the

entry

page

of

Dodge

recall

This

is

due

to

the

use

of

main

keywords

instead

of

a

natural

language

query

There

are

queries

in

QU

ERY

T

T

EST

and

queries

in

QU

ERY

H

T

EST

that

do

not

have

a

result

document

that

has

all

query

terms

in

it

We

could

not

use

our

method

to

them

is

not

enough

to

extract

probability

information

for

these

two

query

sets

To

make

up

this

sparseness

problem

we

need

a

different

indexing

terms

extraction

module

We

have

to

consider

special

parsing

technique

for

URL

strings

and

acronyms

in

a

document

Also

we

need

a

query

expansion

technique

to

get

a

better

result

The

Improvement

of

IR

Performance

We

used

the

Lemur

Toolkit

-LSB-

-RSB-

to

make

a

general

search

engine

for

the

topic

relevance

task

The

Lemur

Toolkit

is

an

information

retrieval

toolkit

designed

with

language

modeling

in

mind

The

Lemur

Toolkit

supports

several

retrieval

algorithms

These

algorithms

include

a

dot-product

function

using

TF-IDF

weighting

algorithm

the

Kullback-Leibler

-LRB-

KL

-RRB-

divergence

algorithm

the

OKAPI

retrieval

algorithm

the

feedback

retrieval

algorithm

and

the

mixture

model

of

Dirichlet

smoothing

MIXFB

KL

D

-LSB-

-RSB-

For

the

homepage

finding

task

we

add

the

URLprior

probability

of

a

URL

string

to

the

Lemur

Toolkit

Besides

Link

information

we

add

the

PageRank

of

a

document

We

normalized

PageRank

values

so

the

max

value

is

and

the

min

value

is

First

we

extracted

top

results

with

the

Lemur

Toolkit

Table

The

Retrieval

Performance

with

Classification

Method

OKAPI

TF-IDF

MIXFB

KL

D

Measure

DEFAULT

TOPIC

HOME

TOPIC

HOME

TOPIC

HOME

Dist

TOPIC

Dist

HOME

MI

TOPIC

MI

HOME

Anchor

TOPIC

Anchor

HOME

POS

TOPIC

POS

HOME

All

TOPIC

All

HOME

Then

we

combined

URL

information

and

Link

information

to

reorder

results

with

the

equation

Eq

We

presented

top

documents

as

the

answer

in

the

topic

relevance

task

and

documents

in

the

homepage

finding

task

We

call

this

modified

Toolkit

as

MLemur

Toolkit

Table

and

show

results

of

the

topic

relevance

task

and

the

homepage

finding

task

that

use

the

Lemur

Toolkit

and

the

MLemur

Toolkit

MIXFB

KL

D

showed

the

good

result

in

the

topic

relevance

task

but

showed

the

poor

result

in

the

homepage

finding

task

We

can

say

that

a

good

information

retrieval

algorithm

for

the

topic

relevance

task

is

not

always

good

for

the

homepage

finding

task

We

chose

three

algorithms

the

OKAPI

the

TF-IDF

and

the

MIXFB

KL

D

that

got

the

best

and

worst

score

in

each

task

for

the

test

of

performance

improvement

by

query

type

classification

Table

shows

the

change

of

performance

DEFAULT

means

the

default

category

for

an

unclassified

query

Digits

in

the

TOPIC

column

and

the

HOME

column

are

average

precision

and

MRR

respectively

From

the

result

the

OKAPI

algorithm

and

the

homepage

finding

task

as

a

default

class

method

shows

the

good

performance

Discussion

To

classify

a

query

type

we

need

the

document

frequency

of

a

query

term

in

each

database

This

lowers

the

system

efficiency

However

we

may

create

two

databases

as

proposed

in

this

paper

for

indexing

We

retrieve

two

result

document

sets

from

each

database

and

classify

a

query

type

at

the

same

time

And

then

according

to

the

category

of

a

query

merge

two

results

From

table

merging

the

results

of

the

anchor

text

representation

and

the

common

content

representation

shows

good

performance

We

need

more

work

to

unify

the

query

classification

work

and

the

document

retrieval

In

this

paper

we

proposed

a

user

query

classification

method

for

the

topic

relevance

task

and

the

homepage

finding

task

The

queries

of

the

homepage

finding

task

usually

consist

of

entity

names

or

proper

nouns

However

queries

of

the

service

finding

task

have

verbs

for

the

service

definition

For

example

Where

can

I

buy

concert

tickets

has

buy

as

the

service

definition

To

find

these

cue

expressions

we

need

more

sophisticated

analysis

of

anchor

texts

Since

the

service

in

the

Web

is

provided

as

a

program

there

is

a

trigger

button

Mostly

these

trigger

buttons

are

explained

by

anchor

texts

We

have

to

distinguish

an

entity

name

and

an

action

verb

from

anchor

texts

We

have

to

change

measures

for

the

query

classification

from

a

word

unit

to

entity

and

action

units

User

query

classification

can

be

applied

to

various

areas

MetaSearch

is

the

search

algorithm

that

combines

results

of

each

search

engine

to

get

the

better

result

-LSB-

-RSB-

-LSB-

-RSB-

proposed

CombMNZ

Multiply

by

NonZeros

is

better

than

other

scoring

algorithm

CombSUM

Summed

similarity

over

systems

But

if

we

consider

the

homepage

finding

task

we

are

in

a

different

situation

Table

and

show

the

improvement

of

performance

of

MetaSearch

algorithms

We

had

an

experiment

with

random

samplings

of

and

engine

results

The

score

is

the

average

improvement

of

tests

CombMNZ

was

good

for

the

topic

relevance

task

but

CombSUM

was

good

for

the

homepage

finding

task

It

also

tells

we

need

different

strategies

for

MetaSearch

as

the

class

of

a

query

Table

Performance

of

MetaSearch

in

the

Topic

Relevance

Task

engine

CombSUM

CombMNZ

Table

Performance

of

Metasearch

in

the

Homepage

Finding

Task

engine

CombSUM

CombMNZ

We

have

various

forms

of

resources

in

the

Web

and

consequently

purposes

of

user

queries

are

diverse

We

can

classify

user

queries

as

three

categories

the

topic

relevance

task

the

homepage

finding

task

and

the

service

finding

task

Search

engines

need

different

strategies

to

meet

the

purpose

of

a

user

query

For

example

URL

information

and

Link

information

are

bad

for

the

topic

relevance

task

but

on

the

other

hand

they

are

good

for

the

homepage

finding

task

We

made

two

representative

databases

DB

HOM

E

and

DB

T

OP

IC

for

each

task

To

make

databases

we

divided

text

collection

by

the

URL

type

of

a

web

document

If

the

URL

of

a

document

contains

a

host

name

only

then

we

put

it

into

DB

HOM

E

Also

we

make

a

virtual

document

with

an

anchor

text

and

put

it

into

DB

HOM

E

Other

documents

are

put

into

DB

T

OP

IC

If

given

query

's

distributions

in

DB

HOM

E

and

DB

T

OP

IC

are

different

then

this

tells

a

given

query

is

not

a

general

word

Therefore

we

can

assume

the

category

of

a

given

query

is

in

the

homepage

finding

task

Likewise

the

difference

of

dependency

Mutual

Information

and

the

usage

rate

as

anchor

texts

tell

whether

a

given

query

is

in

the

homepage

finding

task

or

not

We

tested

the

proposed

classification

method

with

two

query

sets

QU

ERY

T

T

EST

and

QU

ERY

H

T

EST

The

usage

rate

as

anchor

texts

and

the

POS

information

show

small

coverage

On

the

other

hand

distribution

difference

and

dependency

showed

good

precision

and

coverage

Also

each

classifier

applied

to

different

queries

We

could

get

the

better

precision

and

recall

by

combining

each

classifier

We

got

precision

and

recall

After

we

classified

the

category

of

a

query

we

used

different

information

for

a

search

engine

For

the

topic

relevance

task

Content

information

such

as

TFIDF

is

used

For

the

homepage

finding

task

Link

information

and

URL

information

besides

content

information

are

used

We

tested

our

dynamic

combining

method

From

the

result

our

classification

method

showed

the

best

result

with

the

OKAPI

scoring

algorithm

We

would

like

to

thank

Jamie

Callan

for

providing

useful

experiment

data

and

the

Lemur

toolkit

-LSB-

-RSB-

R.

Baeza-Yates

and

B.

Ribeiro-Neto

Modern

Information

Retrieval

ACM

PRESS

BOOKS

-LSB-

-RSB-

P.

Bailey

N.

Craswell

and

D.

Hawking

Engineering

a

multi-purpose

test

collection

for

web

retrieval

experiments

Information

Processing

and

Management

to

appear

-LSB-

-RSB-

S.

Brin

and

L.

Page

The

anatomy

of

a

large-scale

hypertextual

Web

search

engine

Computer

Networks

and

ISDN

Systems

-LRB-

-RRB-

-LSB-

-RSB-

A.

Broder

A

taxonomy

of

web

search

SIGIR

Forum

-LRB-

-RRB-

-LSB-

-RSB-

W.

B.

Croft

Combining

approaches

to

information

retrieval

In

Advances

in

Information

Retrieval

Recent

Research

from

the

Center

for

Intelligent

Information

Retrieval

pages

Kluwer

Academic

Publishers

-LSB-

-RSB-

CSIRO

Web

research

collections

trec

web

track

TRECWeb

-LSB-

-RSB-

E.

Fox

and

J.

Shaw

Combination

of

multiple

searches

In

Text

REtrieval

Conference

-LRB-

-RRB-

pages

-LSB-

-RSB-

D.

Hawking

and

N.

Craswell

Overview

of

the

web

track

In

Text

REtrieval

Conference

-LRB-

-RRB-

pages

-LSB-

-RSB-

E.

Jaynes

Information

theory

and

statistical

mechanics

Physics

Review

-LRB-

-RRB-

-LSB-

-RSB-

J.

H.

Lee

Analyses

of

multiple

evidence

combination

In

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

Retrieval

pages

-LSB-

-RSB-

C.

D.

Manning

and

H.

Schutze

Foundations

of

Statistical

Natural

Language

Processing

The

MIT

Press

-LSB-

-RSB-

P.

Ogilvie

and

J.

Callan

Experiments

using

the

lemur

toolkit

In

Text

REtrieval

Conference

-LRB-

-RRB-

lemur

pages

-LSB-

-RSB-

L.

Page

S.

Brin

R.

Motwani

and

T.

Winograd

The

pagerank

citation

ranking

Bringing

order

to

the

web

Technical

report

Stanford

Digital

Library

Technologies

Project

-LSB-

-RSB-

J.

M.

Ponte

Language

models

for

relevance

feedback

In

W.

B.

Croft

editor

Advances

in

Information

Retrieval

Recent

Research

from

the

Center

for

Intelligent

Information

Retrieval

pages

Kluwer

Academic

Publishers

-LSB-

-RSB-

S.

E.

Robertson

S.

Walker

S.

Jones

M.

Hancock-Beaulieu

and

M.

Gatford

Okapi

at

In

Text

REtrieval

Conference

-LRB-

-RRB-

pages

-LSB-

-RSB-

T.

Westerveld

W.

Kraaij

and

D.

Hiemstra

Retrieving

web

pages

using

content

links

urls

and

anchors

In

Text

REtrieval

Conference

-LRB-

-RRB-

pages

-LSB-

-RSB-

K.

Yang

Combining

text

and

link-based

retrieval

methods

for

web

ir

In

Text

REtrieval

Conference

-LRB-

-RRB-

pages

B O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O B O O O O B I I I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O B I O O O O O O O O O O O O O B O O O B O O O O O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O B O O O O O B O O O O O O O O O O B O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O B O O O O O O B O O O O O O B O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O B I O O O O O O O B O O B O B I O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O B I O O O B O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O B O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O B O B I O B I O O O O O O B O O O O O O B O O O O O B O O O O O O O B O B I O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O B O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O B O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O B O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O B O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O B O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O B O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O B B O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O B O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O B O O O O B O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O O O O O O O B O O O O B I O B I O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O B O O O O B O O O O O O O B O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O B O O B O O O O O O O O O O B O B I O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O

rate
model
query classification
combination of multiple evidences
link information
web document
task
url information
query
ir
url
improvement
frequency
information